diff --git a/CHANGELOG.md b/CHANGELOG.md index c0aeb8e..021c90b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,50 @@ This file exists because the version number was the only signal a consumer got (register C-111). Releases before 1.2.0 are summarised from their tags rather than reconstructed in detail. +## 1.3.0 — 2026-09-19 + +**No new failure modes.** A launcher that ran 1.2.0 sees nothing new stop. This release +moves two dependencies across a major version and picks up a statistics fix; the delivered +bytes are unchanged. + +### Dependencies a launcher will resolve differently + +- **views-frames `>=2.0.0,<3`** (was `<2`). views-frames 2.0.0 is that package's first + MAJOR: its frame constructors now type-check `index` and raise `TypeError` on a non-index + where they previously constructed silently. This repository passes real indexes and is + unaffected. **Delivery bytes are identical** — shards emitted through `views_frames.io.arrow` + hash the same under 1.10.2 and 2.0.0 at the pinned toolchain, and the ADR-013 §10 + byte-parity fixtures pass under 2.0.0 in CI. Not a contract change. +- **views-pipeline-core resolves to 3.3.0** on a locked install (was 3.0.1). The declared + range `>=3.0.0,<4.0.0` did not change; 3.3.0 is what lifted its own `views-frames <2` cap + and made the move above possible. A `pip install` from the tag already resolved the newest + 3.x, so a launcher installing from git sees no difference here. +- **pyarrow stays at 16.1.0.** The `<17` ceiling is deliberate and owned by #174. + +### Fix picked up from upstream + +- views-frames 1.11.0 corrected the published MAP-containment law, which was wrong on + **tied draws** — failing roughly 6% of rows on zero-inflated integer count posteriors, + this platform's primary data shape. Moving past 1.10.2 brings it in. This repository + volunteered to take that fix first. + +### Documentation a reader of the code will notice + +- Both products' `UPLOAD_ENABLED` docstrings previously named a precondition that had + already been met — faoapi's C-161 closure notice (delivered 2026-07-20) and the + views-crafdapi selection guard (deployed 2026-08-12). They now name the gate that actually + holds: the non-production Appwrite decision, views-appwrite#171. **The interlock itself is + unchanged and still closed.** Nothing uploads. +- `CRAFDPostProcessorManager` has an intent contract (`docs/CICs/`), stated as a delta + against the UN-FAO one. The README and `docs/architecture/role_and_seams.md` list every + `delivery/` module again, and a test now keeps them complete. + +### Upgrading + +Nothing to change in a launcher. Environments that hold `views-frames <2` for another +reason — views-evaluation's `[frames]` extra caps it — will refuse to resolve; this +repository requests neither that package nor that extra. + ## 1.2.0 — 2026-08-26 **A previously-passing delivery can now fail in three new ways. All three are diff --git a/README.md b/README.md index 361fbd4..2413100 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # views-postprocessing -[![Python 3.11+](https://img.shields.io/badge/python-3.11+-blue.svg)](https://www.python.org/downloads/) +[![Python 3.11](https://img.shields.io/badge/python-3.11-blue.svg)](https://www.python.org/downloads/) [![Poetry](https://img.shields.io/badge/dependency%20management-poetry-blueviolet)](https://python-poetry.org/) [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT) @@ -43,7 +43,18 @@ poetry install pip install views-postprocessing ``` -Requires **Python 3.11–3.14**. +Requires **Python 3.11**, and only 3.11. + +`pyproject.toml` still declares `>=3.11,<3.15`. **That declaration is wrong** and is +tracked as **#295**: the lockfile resolves on 3.11 alone, because `ingester3` caps +`levenshtein >=0.20,<0.21` and no release in that range publishes a 3.12+ wheel. +Measured 2026-08-25 against the index. CI and the delivery's own environment both run +3.11, so nothing in production is affected — the cost is that a contributor arriving on +3.12 or 3.13 is told the project supports them and then cannot install it. + +Narrowing the declaration is a one-line edit that invalidates `poetry.lock` and forces a +full re-resolve, which would move `pyarrow` off the 16.1.0 the ADR-013 §10 byte-parity +fixtures are pinned to (C-72). So it is #295's own change, not a documentation fix. ### Dependencies @@ -89,6 +100,21 @@ in #149; their rules survive as called invariants under `delivery/`. See the affected, confirm the fault offline, and supersede on the wire. The contract has no retraction primitive; a correction is a new complete run, manifest last. +### If a delivery fails loudly + +Since **1.2.0** a run can stop in ways it previously would not, and each one replaces a +silent failure with a refusal. A launcher may see: + +| exception | what it means | +|---|---| +| `DeliveryNotFindableError` | the upload succeeded but the consumer's own query does not find **this** run — the failure where every call reports success and the partner sees nothing | +| `FindabilityUnverifiedError` | the check above could not run; *"could not ask"* is deliberately not *"asked and got nothing"* | +| `ProducerClientUnavailable` | the producer client would not load, so the observed-data boundary is unknown. Refuses rather than shipping the unobserved tail as observed history | +| `TornRunError` | a run failed partway through uploading. Names every object confirmed uploaded and the one that failed; deletes nothing | + +If one fires after an upgrade it is reporting a condition that was already wrong and +already invisible. [`CHANGELOG.md`](CHANGELOG.md) carries the detail. + ### Output schema (geographic metadata columns) These 9 columns are the delivered geography contract, declared in @@ -132,7 +158,8 @@ views-postprocessing/ │ ├── draws.py # the §6 no-collapse gate │ ├── parity.py # sidecar covers exactly the forecast's cells │ ├── observed_range.py # fabricated-month decision - │ └── provenance.py # structured upload provenance + │ ├── provenance.py # structured upload provenance + │ └── findability.py # does the consumer's query find THIS run? ├── contract/ # HOW A DELIVERY IS BUILT — partner-neutral │ ├── wire/ # the ADR-013 contract (header, shard, sidecar, │ │ # run_manifest, sink, source_selection, naming) @@ -219,7 +246,8 @@ only `.py`; it now scans markdown too.)* |-----|----------------| | [`docs/architecture/role_and_seams.md`](docs/architecture/role_and_seams.md) | **Start here** — role vs the sibling repos + internal seams | | [`docs/ADRs/`](docs/ADRs/) | Architecture decisions (esp. ADR-011 mapper→lookup; ADR-012 ontology) | -| [`docs/CICs/`](docs/CICs/) | Class intent contracts (`UNFAOPostProcessorManager`) | +| [`docs/CICs/`](docs/CICs/) | Class intent contracts — one per partner manager; the CRAF'd one is stated as a delta against the UN-FAO one | +| [`CHANGELOG.md`](CHANGELOG.md) | **What changed for a consumer**, per release — behaviour a launcher can observe, failure modes first | | `reports/technical_risk_register.md` | Tracked risks — C-40 (the remaining pipeline-core inheritance), C-30/C-15 (delivery guards), C-43 (enrichment value verification) | --- diff --git a/docs/ADRs/004_rules_for_evolution_and_stability.md b/docs/ADRs/004_rules_for_evolution_and_stability.md index 942f247..f13f820 100644 --- a/docs/ADRs/004_rules_for_evolution_and_stability.md +++ b/docs/ADRs/004_rules_for_evolution_and_stability.md @@ -66,6 +66,26 @@ This ADR should be revisited when one or more of the following become true: At that point, a new ADR should supersede this one. +> **Audit note, 2026-08-26: at least three of those conditions are now true, and nothing +> noticed.** Recorded here rather than acted on, because writing the successor is an +> architectural decision and this was a documentation audit. +> +> - *"External users or downstream systems depend on this repository"* — views-models pins +> it in two launchers, views-faoapi and views-crafdapi consume its artifacts, and it has +> been published on PyPI since 1.0.0. +> - *"Breaking changes begin to incur real coordination or migration costs"* — the 1.1.0 → +> 1.1.1 pin move took eight days and a direct request (views-models#403); 1.2.0's is open +> as views-models#439. Register **C-111** and **C-112** exist for the two halves of that +> gap: consumers are told nothing but a version number, and nothing here can see what +> production actually runs. +> - *"Reproducibility across time becomes a contractual requirement"* — ADR-013 §10 pins +> delivery bytes against a golden fixture, and C-72 records what a toolchain bump costs. +> +> The deferral itself was sound and remains honest; what has changed is the world it was +> deferring against. **Whoever picks this up should write the successor rather than amend +> this document** — that is what its own last line asks for. The owner field above is still +> `—`, which is the part ADR-014 §4 would actually object to. + --- ## Non-Decisions (Explicitly Out of Scope for Now) diff --git a/docs/CICs/CRAFDPostProcessorManager.md b/docs/CICs/CRAFDPostProcessorManager.md new file mode 100644 index 0000000..1cba825 --- /dev/null +++ b/docs/CICs/CRAFDPostProcessorManager.md @@ -0,0 +1,103 @@ +# Class Intent Contract: CRAFDPostProcessorManager + +**Status:** Active +**Owner:** PRIO MD&D Team +**Last reviewed:** 2026-08-26 +**Related ADRs:** ADR-002, ADR-006, ADR-008, ADR-009, ADR-013 +**Sibling contract:** [UNFAOPostProcessorManager.md](UNFAOPostProcessorManager.md) + +--- + +> **This contract is stated as a delta, deliberately.** `CRAFDPostProcessorManager` is a +> clone of the UN-FAO manager (`docs/CLONING.md`, register **C-33**), and the two +> managers differ by fourteen lines on each side, none of which changes behaviour. Verify +> that rather than trusting this sentence: +> +> ``` +> diff views_postprocessing/unfao/managers/unfao.py \ +> views_postprocessing/crafd/managers/crafd.py +> ``` +> +> A second full contract was considered and rejected. Two 189-line documents describing +> one behaviour is not redundancy, it is **two things that can disagree** — and this +> repository has already paid for exactly that: register **C-75**, where the sibling CIC +> and this directory's other file asserted opposite things about the same call for weeks. +> One contract, one delta. If the managers ever diverge behaviourally, C-33's extraction +> trigger has fired and this document's form should be revisited along with the code. + +## 1. Purpose + +`CRAFDPostProcessorManager` orchestrates the end-to-end postprocessing pipeline that +delivers VIEWS conflict predictions to the **Complex Risk Analytics Fund (CRAF'd)**, +served by views-crafdapi. + +It is the single entrypoint for producing and delivering CRAF'd-formatted prediction data. + +## 2. What this contract inherits + +**Sections 2–11 of [UNFAOPostProcessorManager.md](UNFAOPostProcessorManager.md) apply +to this class unchanged**, substituting the partner identity below. That includes the +non-goals, the four-stage pipeline guarantee, the inputs and assumptions, the failure +modes and their loudness, the boundaries (ADR-002 topology: `crafd/` → `contract/` → +`delivery/`, one way only), and the test-alignment position. + +Two of those are worth naming here because they are the ones a reader most often assumes +differ, and they do not: + +- **The wire is partner-neutral** (#153). CRAF'd receives the same ADR-013 artifacts + built by the same `contract/` code. Only the *product* differs. +- **This class is one of the repository's only two importers of `views_pipeline_core`**, + mechanically pinned to an allowlist by `tests/test_doc_accuracy.py`. That is what keeps + C-40's blast radius at one file per partner. + +## 3. What differs — the whole of it + +| | `unfao` | `crafd` | +|---|---|---| +| consumer document `name` | `un_fao` | `un_crafd` | +| partner store | `unfao_bucket` | `crafd_bucket` | +| env tuple validated | `appwrite_env.UNFAO_ENV` | `appwrite_env.CRAFD_ENV` | +| consumer repository | views-faoapi | views-crafdapi (its ADR-034) | +| §11.4 interlock history | precondition met 2026-07-20 (faoapi C-161) | precondition met 2026-08-12 (views-crafdapi#53) | + +`TARGETS` and `S_MIN` are **the same values today** (`lr_ged_sb`, `lr_ged_ns`, +`lr_ged_os`; `S_MIN = 2`) but are independently declared per partner and may diverge +without either being wrong — CRAF'd naming an additional target is an Amendment A1 edit +to `crafd/product.py` alone. + +**Both partners are gated closed.** `product.UPLOAD_ENABLED` is `False` in each, and the +gate that holds is the same one for both: **views-appwrite#171**, the non-production +Appwrite project decision, without which the upload path cannot be rehearsed against a +real store (#18). The partner-specific preconditions in the table above are satisfied and +no longer gate anything. + +## 4. The uncertainty surface is the consumer's, not this class's + +CRAF'd is FAO *extended*: same forecasts, same PRIO-GRID geography, same cadence. The +additional surface CRAF'd wants — exceedance probabilities alongside HDI/MAP — is +computed in **views-crafdapi** (their ADR-034), not here. This producer ships the same +posterior-sample wire the FAO producer ships. + +This is a **non-goal** and belongs in a contract because the alternative is attractive and +wrong: a manager that starts summarising draws for one partner has taken a consumer +concern into the delivery, and the §6 no-collapse gate exists to prevent exactly that. + +## 5. Test alignment + +Covered by the same source-scan and seam tests as its sibling, parametrised over both +partners via `PARTNER_PACKAGES` in `tests/conftest.py` — including +`tests/test_clone_readiness.py`, which asserts the two partner packages are **independent** +(neither imports the other), and the line budgets in `tests/test_doc_accuracy.py`. + +**A note on the budget, current at this review:** `crafd/` sits at **699 of 700** lines. +The next line added anywhere in that package fails the guard, which is what the guard is +for. It is not a defect and is not scheduled; it is recorded so the next contributor meets +it here rather than in a red build. + +## End of Contract + +This document defines the **intended meaning** of `CRAFDPostProcessorManager`. + +Changes to behavior that violate this intent are bugs. +Changes to intent must update this contract — and, where the intent is inherited, its +sibling. diff --git a/docs/CICs/UNFAOPostProcessorManager.md b/docs/CICs/UNFAOPostProcessorManager.md index 1e1e242..84e715a 100644 --- a/docs/CICs/UNFAOPostProcessorManager.md +++ b/docs/CICs/UNFAOPostProcessorManager.md @@ -3,7 +3,7 @@ **Status:** Active **Owner:** PRIO MD&D Team -**Last reviewed:** 2026-08-19 +**Last reviewed:** 2026-08-26 **Related ADRs:** ADR-001, ADR-002, ADR-008, ADR-009 --- @@ -95,7 +95,7 @@ Assumptions that are not met **must cause failure**, not fallback behavior. The - **Launch config incomplete:** raises `LaunchConfigError` naming the missing key. A launcher that omits `wire_contract` or declares a `data_format` other than `feature_frame` is **refused**, never quietly routed into a fallback (ADR-003, register C-63) - **Region coverage mismatch:** Raises `CoverageError` in `_check_coverage()` (called from `_validate()`) if a pinned region's delivered cell count is wrong (S1/C-34) or a GAUL-uncovered excluded cell leaks into the delivery (S4/C-30) - **Fabricated historical tail:** `_read_historical_frame()` drops months beyond the producer's `last_valid_month_id` at the read (`_clip_observed_history` was the pandas equivalent, retired with that path in #149) so unobserved zero-padding is not shipped as observed history (S2/C-26). Two outcomes when the boundary is unavailable, and they are different on purpose (C-103, 2026-08-17): if the producer simply publishes no boundary — or the read fails — it **degrades open**, skipping the clip with a WARNING that states the unobserved tail will ship; if the producer client cannot be imported at all it **refuses** (`source_metadata.ProducerClientUnavailable`), because a broken environment is not a producer fact -- **Upload provenance:** the historical artifact's `description` carries structured provenance (lookup version, region, expected/actual cell counts, unmapped count) built by `delivery/provenance.py` (`build_provenance` → `compact_description`) via the manager's `_historical_frame_description()` (S5/C-15). The **forecast** side carries no such description: its guarantee is the wire's verified chain — per-shard content hashes recorded in the §4.2 run manifest, header asserts on load, and manifest-last commit ordering. That is identity and integrity, not the C-15 provenance field set; the §4.2 manifest's keys are exactly `contract_version`, `run_id`, `targets`, `shards`, `expected_months`, `expected_cell_count`, `sidecar` — and it carries **no** `lookup_version`, `region` or `unmapped_count`. `_delivery_description()` was the pandas-path equivalent and was deleted with it in #149 +- **Upload provenance:** the historical artifact's `description` carries structured provenance (lookup version, **the observed-range boundary this run clipped against**, region, expected/actual cell counts, unmapped count) built by `delivery/provenance.py` (`build_provenance` → `compact_description`) via the manager's `_historical_frame_description()` (S5/C-15). The **forecast** side carries no such description: its guarantee is the wire's verified chain — per-shard content hashes recorded in the §4.2 run manifest, header asserts on load, and manifest-last commit ordering. That is identity and integrity, not the C-15 provenance field set; the §4.2 manifest's keys are exactly `contract_version`, `run_id`, `targets`, `shards`, `expected_months`, `expected_cell_count`, `sidecar` — and it carries **no** `lookup_version`, `region` or `unmapped_count`. The `observed_through` field is **required and always emitted** (#297, 2026-08-25): an integer when the clip ran, and an explicit `null` when the producer boundary could not be read and the clip was therefore *skipped*, which is the case that most needs recording. A third state — never read — refuses at build time rather than collapsing to `null`, because reporting "clip skipped" for a run whose clip ran is the C-103 conflation one layer down. Added because #297 took a day to answer a question the artifact should have carried: *observed through when, and decided against what?* `_delivery_description()` was the pandas-path equivalent and was deleted with it in #149 The following **must never** fail silently: - Missing or None environment variables for Appwrite diff --git a/docs/CLONING.md b/docs/CLONING.md index 79844a7..11fe1e7 100644 --- a/docs/CLONING.md +++ b/docs/CLONING.md @@ -74,7 +74,9 @@ diff views_postprocessing/unfao/managers/unfao.py \ views_postprocessing/crafd/managers/crafd.py ``` -Sixteen lines differ on each side and **none of them changes behaviour**: the import, +The differing lines **change no behaviour**, and the count is deliberately not +repeated here — it was sixteen when this paragraph was written and is not now. What +differs is: the import, the class name, the two partner-named methods and their two call sites, the four env-var literals, one line that both selects which env tuple is validated and labels the store, one runtime refusal message, and four lines of prose. diff --git a/docs/architecture/role_and_seams.md b/docs/architecture/role_and_seams.md index 9af8da0..20624e3 100644 --- a/docs/architecture/role_and_seams.md +++ b/docs/architecture/role_and_seams.md @@ -106,7 +106,7 @@ The input-integrity guards are split into **two homes** on purpose: - `views_postprocessing/delivery/` — **representation-free invariants**. Primitives only (sets of ints, numpy arrays, scalars, dicts). **No pandas, no views_frames.** Each is a pure rule that raises or passes: `coverage.py`, `draws.py`, `parity.py`, - `observed_range.py`, `provenance.py`. + `observed_range.py`, `provenance.py`, `findability.py`. - `views_postprocessing/contract/frame_extraction.py` — **the representation seam**. It turns a `views_frames` frame into the primitives the invariants consume. @@ -171,7 +171,8 @@ views_postprocessing/ │ ├── draws.py the §6 no-collapse gate │ ├── parity.py sidecar covers exactly the forecast's cells │ ├── observed_range.py fabricated-month decision -│ └── provenance.py structured upload provenance +│ ├── provenance.py structured upload provenance +│ └── findability.py does the consumer's own query find THIS run? (C-94) ├── contract/ HOW A DELIVERY IS BUILT — partner-neutral, reusable by a clone │ ├── wire/ the ADR-013 contract (header, shard, sidecar, run_manifest, │ │ sink, source_selection, naming) diff --git a/poetry.lock b/poetry.lock index 6110028..d9690e3 100644 --- a/poetry.lock +++ b/poetry.lock @@ -3342,23 +3342,6 @@ pure-eval = "*" [package.extras] tests = ["cython", "littleutils", "pygments", "pytest", "typeguard"] -[[package]] -name = "stepshift" -version = "2.2.6" -description = "Implementation of the Views stepshifting modelling framework" -optional = false -python-versions = ">=3.8" -groups = ["main"] -files = [ - {file = "stepshift-2.2.6.tar.gz", hash = "sha256:398681a585c44eb056612b009ad0e3f6d48eb6a9be36f7fc911c8c0dd91af048"}, -] - -[package.dependencies] -pandas = ">=1.3.2" -PyMonad = ">=2.4.0" -toolz = ">=0.11.1" -xarray = ">=0.19.0" - [[package]] name = "strconv" version = "0.4.2" @@ -3505,35 +3488,33 @@ zstd = ["backports-zstd (>=1.0.0) ; python_version < \"3.14\""] [[package]] name = "views-evaluation" -version = "1.0.0" +version = "2.0.0" description = "" optional = false python-versions = "<3.15,>=3.11" groups = ["main"] files = [ - {file = "views_evaluation-1.0.0-py3-none-any.whl", hash = "sha256:b5b0b2e49e4dad25eebf0919ad587328ada52b8a9aff001775b7123c36f99c7e"}, - {file = "views_evaluation-1.0.0.tar.gz", hash = "sha256:8bff41285f3c6455e75e232e95e633140c72d38c15954b6998103634db57d093"}, + {file = "views_evaluation-2.0.0-py3-none-any.whl", hash = "sha256:f18c3427c6c0d57cb37badbc7c2eb21df685798d0f0d8a477f91e3ebb370f487"}, + {file = "views_evaluation-2.0.0.tar.gz", hash = "sha256:bde3feac02bf998948412c7807f514bcc03c6ec5e53ba0351a2db3d85d083869"}, ] [package.dependencies] numpy = ">=1.26.4,<2.0.0" -scikit-learn = ">=1.6.0,<2.0.0" scipy = ">=1.11,<2.0" [package.extras] -dataframe = ["pandas (>=1.5.3,<2.0.0)"] frames = ["views-frames (>=1.10.2,<2)"] [[package]] name = "views-frames" -version = "1.10.2" +version = "2.0.0" description = "The VIEWS platform data-contract layer: immutable array+identifier frames (numpy only, root of the dependency DAG)." optional = false python-versions = ">=3.10" groups = ["main"] files = [ - {file = "views_frames-1.10.2-py3-none-any.whl", hash = "sha256:ace7f6a09bdb743edbb39e7e95139a0c647ef7c6100ef4c612a3ce5975078ef0"}, - {file = "views_frames-1.10.2.tar.gz", hash = "sha256:96b4916afe9b64389045043ec1cf05fa2ea9b872ac0d12cb57f0d629ffd88106"}, + {file = "views_frames-2.0.0-py3-none-any.whl", hash = "sha256:389feb16d6bda995d9f2914de6f5619a38ff37025a0022216eb65d44e03f0b48"}, + {file = "views_frames-2.0.0.tar.gz", hash = "sha256:de13c9713fdecbeb705d0698d285941ea1c614c37e80e8bc2ed7b68f83733ac5"}, ] [package.dependencies] @@ -3545,27 +3526,29 @@ docs = ["jupyterlab (>=4,<5)", "matplotlib (>=3.8,<4)", "nbmake (>=1.5,<2)", "py [[package]] name = "views-pipeline-core" -version = "3.0.1" +version = "3.3.0" description = "Core orchestration, data and model-management library for the VIEWS conflict forecasting platform." optional = false python-versions = "<3.15,>=3.11" groups = ["main"] files = [ - {file = "views_pipeline_core-3.0.1-py3-none-any.whl", hash = "sha256:816e0093fda788d801bda38aa82c5de606867caa3cd561aa9652c98c740995fe"}, - {file = "views_pipeline_core-3.0.1.tar.gz", hash = "sha256:c05d88dfe8aa28feff5ae466bb1e5fd96c4a799aa54d665b274469d4a81403d9"}, + {file = "views_pipeline_core-3.3.0-py3-none-any.whl", hash = "sha256:7c0c7535ee9de7a74528bb64393621e379d57556ee125785d4d149a745dab515"}, + {file = "views_pipeline_core-3.3.0.tar.gz", hash = "sha256:8edf80123d35a626ce282aae8bac94c9b6a56cdf3b0163c0d98c9ddbfb5aa7d1"}, ] [package.dependencies] appwrite = {version = ">=13.4.1,<14.0.0", optional = true, markers = "extra == \"appwrite\""} art = ">=6.4,<7.0" ingester3 = "2.1.1" +pandas = ">=1.5.3,<3.0" polars = ">=1.34.0,<2.0.0" +pyarrow = ">=14.0.0,<17.0.0" pyprojroot = ">=0.3.0,<0.4.0" -views-evaluation = ">=1.0.0,<2.0.0" -views-frames = ">=1.10.2,<2.0.0" -views-transformation-library = ">=2.7.2,<3.0.0" +tqdm = ">=4.66,<5.0" +views-evaluation = ">=2.0.0,<3.0.0" +views-frames = ">=1.10.2,<3.0.0" viewser = ">=6.6.4,<7.0.0" -wandb = ">=0.18.7,<0.19.0" +wandb = ">=0.18.7,<1.0" [package.extras] appwrite = ["appwrite (>=13.4.1,<14.0.0)"] @@ -3624,26 +3607,6 @@ files = [ numpy = ">=1.20.0" pandas = ">=1.5.0,<2.0.0" -[[package]] -name = "views-transformation-library" -version = "2.7.2" -description = "A package containing data transformation functions used by the ViEWS team" -optional = false -python-versions = ">=3.8" -groups = ["main"] -files = [ - {file = "views_transformation_library-2.7.2-py3-none-any.whl", hash = "sha256:8869ebb508ae8b397fa0c9daff8a2c448bf233baa88cd296b599b2978fd87b5f"}, - {file = "views_transformation_library-2.7.2.tar.gz", hash = "sha256:1ce9d7c405d74bd6e609ad0c789d27be97f52007809b40f35fc09006b3ce4173"}, -] - -[package.dependencies] -ingester3 = ">=0.6.0" -pandas = ">=1.2.3,<2.0.0" -scikit_learn = ">=1.0.2,<2.0.0" -scipy = ">=1.6.2,<2.0.0" -stepshift = ">=1.2.0" -xarray = ">=0.19.0" - [[package]] name = "viewser" version = "6.6.4" @@ -3757,32 +3720,7 @@ docs = ["Sphinx (>=6.0)", "myst-parser (>=2.0.0)", "sphinx_rtd_theme (>=1.1.0)"] optional = ["python-socks", "wsaccel"] test = ["pytest", "websockets"] -[[package]] -name = "xarray" -version = "2024.3.0" -description = "N-D labeled arrays and datasets in Python" -optional = false -python-versions = ">=3.9" -groups = ["main"] -files = [ - {file = "xarray-2024.3.0-py3-none-any.whl", hash = "sha256:ca2bc4da2bf2e7879e15862a7a7c3fc76ad19f6a08931d030220cef39a29118d"}, - {file = "xarray-2024.3.0.tar.gz", hash = "sha256:5c1db19efdde61db7faedad8fc944f4e29698fb6fbd578d352668b63598bd1d8"}, -] - -[package.dependencies] -numpy = ">=1.23" -packaging = ">=22" -pandas = ">=1.5" - -[package.extras] -accel = ["bottleneck", "flox", "numbagg", "opt-einsum", "scipy"] -complete = ["xarray[accel,dev,io,parallel,viz]"] -dev = ["hypothesis", "pre-commit", "pytest", "pytest-cov", "pytest-env", "pytest-timeout", "pytest-xdist", "ruff", "xarray[complete]"] -io = ["cftime", "fsspec", "h5netcdf", "netCDF4", "pooch", "pydap ; python_version < \"3.10\"", "scipy", "zarr"] -parallel = ["dask[complete]"] -viz = ["matplotlib", "nc-time-axis", "seaborn"] - [metadata] lock-version = "2.1" python-versions = ">=3.11,<3.15" -content-hash = "302cbc3aa750f7dacffe848c6141e6ba542ce30f2f646d99a859409acfd23b49" +content-hash = "e305efbfd4f9ed430ac7051fca50378ec49692af59354a03c2072ebf5cfaade6" diff --git a/pyproject.toml b/pyproject.toml index d31e13d..04cc622 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,6 +1,6 @@ [tool.poetry] name = "views-postprocessing" -version = "1.2.0" +version = "1.3.0" description = "" authors = [ "Dylan Pinheiro ", @@ -17,7 +17,7 @@ python = ">=3.11,<3.15" # import the SDK unguarded. Without the extra the FAO delivery fails **at import**, not # at runtime, and inside a dependency rather than in our own code. Register C-07. views-pipeline-core = {version = ">=3.0.0,<4.0.0", extras = ["appwrite"]} -views-frames = ">=1.10.2,<2" +views-frames = ">=2.0.0,<3" pyarrow = ">=16.1.0,<17.0.0" # pytest was never declared here; it arrived transitively through pipeline-core, which diff --git a/reports/technical_risk_register.md b/reports/technical_risk_register.md index b1d07e5..afcddbb 100644 --- a/reports/technical_risk_register.md +++ b/reports/technical_risk_register.md @@ -4,9 +4,9 @@ |-------------------|--------------------------------------| | Project | views-postprocessing | | Owner | Dylan Pinheiro / PRIO MD&D Team | -| Last Updated | 2026-08-26 | -| Total Concerns | 112 | -| Open Concerns | 31 | +| Last Updated | 2026-09-14 | +| Total Concerns | 113 | +| Open Concerns | 32 | | Resolved Concerns | 81 | --- @@ -1341,8 +1341,45 @@ At the same time, `main` carries **29 commits since 1.1.1** — every guard from **The wider observation, recorded once so it is not rediscovered.** Eighty-four open issues across the organisation mention `views-postprocessing`; this repository tracks none of them and had never been swept. Most are informational, several were filed *by* this seat, and a few carry live asks (views-faoapi#390, views-crafdapi#55, views-models#362). No mechanism is proposed for that either — but a sweep belongs in the next repo-assimilation rather than being found by accident at the end of a sprint. +**2026-08-26 — the sweep was done, and the pin gap recurred within 48 hours of closing.** + +The version half first. views-models moved both launchers **1.1.0 → 1.1.1** on 2026-08-24 (their `d0c6969`), closing views-models#403 and the C-99 exposure this entry was written about. **1.2.0 was published two days later, and both launchers are behind again** — views-models#439 now asks for the same move a second time. That sharpens the finding: the eight-day lag was not an incident, it is the **steady state** of a system whose only signal to a consumer is an integer they must notice unaided. Nothing about the first fix made the second lag less likely. + +The issue half was executed rather than deferred. **All 66 open issues across the organisation naming this repository were read and triaged on 2026-08-26** (down from 84 on 2026-08-21, without deliberate effort here). Prompted by a `/falsify` audit that had sampled eight and declared the limit rather than claiming the sweep was clean. + +**Result: nothing is blocked by this repository that was not already known.** The live asks named above are unchanged in shape — views-faoapi#390 and views-crafdapi#55 are both *ours to them* (C-92; they owe a test driving their real query-construction path), and views-models#362 is trigger-gated on an ADR-013 amendment that is not adopted. views-pipeline-core#488 tracks **their** blocking pin, not ours to move. The commitment this repository made on views-crafdapi#55 — *"we are deleting our check"* — was honoured: `tests/test_product.py::test_the_declared_consumer_name_matches_the_registry` reads the public registry instead of scraping a sibling's source. + +**One thing the sweep found that no mechanism would have.** views-postprocessing#123 and views-datafactory#341 are the **same decision**, filed from both sides on 2026-07-20, and **neither names an owner** — 37 days. That is not a block, which is exactly why nothing surfaced it: a block has an owner and a queue position, and this has neither. Recorded on #123. It argues the sweep's value is not the asks it confirms but the *unowned* items it is the only way to see — which in turn argues for repeating the sweep rather than mechanising it. + +Still no mechanism proposed, and the trade named above is unchanged: a pin check would add a fifth repository whose `main` can redden this build (**C-86**, no bypass actors). What has changed is the evidence — one recurrence, and one unowned cross-repo decision found only by hand. + Cross-refs: **C-111** (the outbound half), **C-99** (the defect production is still running), **C-86** (the cost of adding another sibling to CI), **C-81** (what actually gates `main`), views-models#403. +--- + +### C-242: The consumer rename reached the forecast and not the historical file — FAO's historical delivery carries wire vocabulary + +| Field | Value | +|-------|-------| +| ID | C-242 | +| Tier | 3 — nothing is silent and nothing corrupts. The delivered numbers are right and the join keys are right. The cost lands on an FAO engineer who joins the two files and finds the forecast's value columns named one way and the historical file's another, and on the release note that committed to the first naming for both. | +| Source | Review of FAO Release Note 06 against the delivered artefacts, 2026-09-14 (found while checking a Pre-Release Note 07 claim, not predicted) | +| Trigger | The next `un_fao` delivery is cut, or the historical artefact's schema is touched — decide whether the consumer rename applies at this boundary, and if not, say so where the release note makes the commitment. | +| Owner | This repository, at the write boundary. The name itself is upstream and deliberate — see below. Tracked as **views-postprocessing#305**. | +| Location | `views_postprocessing/unfao/managers/unfao.py:440` writes `historical_dataset_{timestamp}.parquet` with the frame's columns unchanged. The name originates in `views-models/postprocessors/un_fao/configs/config_queryset.py:57-59` (`ged_sb_best` → `lr_ged_sb`, and the same for `ns`/`os`). | + +The delivered historical artefact carries **`lr_ged_sb`, `lr_ged_ns`, `lr_ged_os`** as its three value columns. Verified against the newest delivery — `historical_dataset_20260813_080043.parquet`, the 13 August re-delivery after the empty-bucket incident, 28,421,738 rows, 14 columns — and against the copy downloaded 2026-09-09. Both carry the wire names. + +**The name is deliberate, not legacy drift.** ADR-013's adoption record in this register states: *target vocabulary **decided**: `lr_ged_sb/ns/os`, producers rename at publish (models#146)*. So the wire vocabulary is correct and should not be changed at the producer; `views-datafactory` supplies `ged_sb_best` and the `un_fao` queryset config maps it deliberately. + +**What is missing is the consumer-side rename at this boundary.** `D-06` (RESOLVED) already settled the principle: the rename from internal names to consumer-facing names *"belongs in views-faoapi as a response-formatting step, coordinated with FAO"*. For the **forecast** that is now built and shipped — `json_contract.to_consumer_columns` is applied in `forecast/serialize/bulk_parquet.py`, `forecast/serialize/grid_parquet.py`, and the `hdi-map` route, producing `sb_map`, `ns_hdi90_upper` and the rest. It is applied on **no historical path**; `wire_reader.py` records the design as *"the served target keeps the wire vocabulary; mapping to the consumer name happens at the boundary"*, and the historical boundary has no such mapping. + +**Why it matters beyond tidiness.** FAO Release Note 06 — drafted, never transmitted — commits in Topic B that consumer-facing columns *"omit the internal VIEWS pipeline prefixes — source (`ged_`), scale/transform (`lr_`, and the deprecated `ln_`), and model-output (`pred_`)"*, and states that the deprecated `lr_`/`ln_` prefix *"must not be read as log space"*. The forecast honours that; the historical file does not. Pre-Release Note 07 (2026-09-14) now names the three columns explicitly so an FAO engineer is not surprised, which is the honest short-term answer but leaves the release note's commitment wider than the implementation. + +**Two ways to close it**, and the choice is not this register's to make: rename at this write boundary, mirroring what faoapi does for the forecast (small, and makes RN 06's commitment true); or scope the release note's naming commitment to the forecast columns and leave the historical file on wire vocabulary. The first costs a rename at one boundary; the second costs a paragraph of explanation in every note that touches the schema. + +Cross-refs: **D-06** (RESOLVED — the general principle, and the finding that no renaming layer existed at all; this is the residual for one artefact), **C-24** (a consumer-facing contract divergence nobody surfaced), **ADR-013** adoption record above (the wire vocabulary decision), **views-faoapi** `forecast/serialize/json_contract.py` (the implemented half), FAO project `reports/post_mortems/2026-09-14_prn07_writing_session.md` (the review that found it). + ## Disagreements ### D-12: Post-Run-0 infrastructure & naming intents — repo rename, internal-store transport, compute co-location diff --git a/tests/test_doc_accuracy.py b/tests/test_doc_accuracy.py index 7f9eeea..fea905c 100644 --- a/tests/test_doc_accuracy.py +++ b/tests/test_doc_accuracy.py @@ -140,6 +140,7 @@ def _governance_docs() -> list[Path]: #: `unfao/managers/unfao.py`, and no rule turns one into the other. _CIC_SUBJECT = { "UNFAOPostProcessorManager.md": ("unfao", "managers", "unfao.py"), + "CRAFDPostProcessorManager.md": ("crafd", "managers", "crafd.py"), } @@ -950,3 +951,33 @@ def kinds(src: str) -> list[str]: assert kinds( "from typing import TYPE_CHECKING\nif TYPE_CHECKING:\n import pandas\n" ) == ["type-only"], "a legitimate type-only import was flagged — the guard cries wolf" + + +@pytest.mark.parametrize( + "doc", ["README.md", "docs/architecture/role_and_seams.md"] +) +def test_the_orientation_docs_list_every_delivery_invariant(doc): + """Both orientation documents present `delivery/` as a complete list. Nothing checked + that it was. + + `delivery/findability.py` shipped on 2026-08-18 and was absent from both files until + 2026-08-26 — the README frozen since 2026-08-04, `role_and_seams.md` the same day. + These are the two files a new contributor opens first, and a *complete* list missing + an entry is worse than no list: it is read as exhaustive. + + Scoped to `delivery/` on purpose. It is the one directory both documents enumerate + exhaustively — the README collapses `contract/wire/` to a parenthetical, and a guard + that fired on that would be crying wolf about a deliberate abbreviation (ADR-014 §3). + """ + modules = { + f.stem for f in (_PKG / "delivery").glob("*.py") if f.stem != "__init__" + } + assert modules, "no delivery modules found — this guard is scanning the wrong place" + + text = (_REPO / doc).read_text() + missing = sorted(m for m in modules if f"{m}.py" not in text) + assert not missing, ( + f"{doc} presents delivery/ as a complete list but omits {missing}. Either add " + "them or stop presenting the list as exhaustive — findability.py was absent from " + "both orientation documents for eight days after it shipped." + ) diff --git a/tests/test_falsification_release_readiness.py b/tests/test_falsification_release_readiness.py index a076698..ac4a2fc 100644 --- a/tests/test_falsification_release_readiness.py +++ b/tests/test_falsification_release_readiness.py @@ -118,3 +118,31 @@ def test_the_publish_job_cannot_ship_untested_code(): "publish validates only version-greater-than-PyPI; nothing establishes that " "the code being shipped passes its own suite" ) + + +# ───────────────────────────────────────────────────────────────────────────── +# Third audit, 2026-08-26. Claim: *"nothing more to do here now; no pressing GH +# issues or registered risk; no repos upstream or downstream blocked by this repo."* +# +# Verdict: FALSIFIED. Both of its stubs are disposed of here rather than carried: +# +# **S5 DISCHARGED 2026-08-26.** It asserted that neither product names a precondition +# that has already been satisfied. Both did — `unfao` cited faoapi's C-161 closure +# notice (delivered 2026-07-20), `crafd` cited the views-crafdapi selection guard +# (views-crafdapi#53, closed 2026-08-12). Both docstrings now name the gate that +# actually holds, views-appwrite#171. Removed by hand rather than left to XPASS. +# +# **S6 WITHDRAWN 2026-08-26 — it was not a guard that could fire.** It asserted the +# "step 4 is incomplete" banner disappears from `docs/operations/correction_procedure.md`. +# That banner cannot come down until FAO answers Pre-Release Note 07 Decision B.1, which +# is not an action available in this repository at any effort. A test that can only go +# green on an external party's reply is decoration (C-102), and **#292 already tracks the +# item with more precision than an assertion can carry** — including the part the audit +# got wrong: the nine mislabelled cells (views-datafactory#387) **were** disclosed to FAO +# as Note 07 Topic G on 2026-08-21. The audit reported that disclosure as missing. It +# was not. +# +# Two further findings from that audit are facts about the issue tracker, not the tree, +# and are recorded in the sprint rather than as assertions: views-faoapi is blocked +# downstream (#294), and #272's second question is unanswered. +# ───────────────────────────────────────────────────────────────────────────── diff --git a/views_postprocessing/crafd/product.py b/views_postprocessing/crafd/product.py index b378a9a..077c435 100644 --- a/views_postprocessing/crafd/product.py +++ b/views_postprocessing/crafd/product.py @@ -27,8 +27,12 @@ it is a contract amendment. - ``UPLOAD_ENABLED`` — the §11.4 upload interlock: ``False`` means the sink writes artifacts locally and never calls the store. Overriding requires an explicit - launch-config declaration, and the first live enablement is gated on the - views-crafdapi consumer's selection guard being deployed in production. + launch-config declaration. The consumer-side precondition this once named — the + views-crafdapi selection guard in production — was **met 2026-08-12** + (views-crafdapi#53). + What gates it now is **ours**: the upload path has never run against a real store — + þing-02 D2 forbids testing against production and the org holds no other project + (#18). That decision is **views-appwrite#171**. """ from __future__ import annotations diff --git a/views_postprocessing/unfao/product.py b/views_postprocessing/unfao/product.py index c9bbcac..dd6af5c 100644 --- a/views_postprocessing/unfao/product.py +++ b/views_postprocessing/unfao/product.py @@ -21,8 +21,12 @@ contract amendment. - ``UPLOAD_ENABLED`` — the §11.4 upload interlock: ``False`` means the sink writes artifacts locally and never calls the store. Overriding requires an explicit - launch-config declaration (wired in the sink story), and the first live - enablement is gated on faoapi's C-161 closure notice. + launch-config declaration (wired in the sink story). The consumer-side precondition + this once named — faoapi's C-161 closure notice — was **met 2026-07-20**, its + residual shipping in faoapi v1.5.4 on 2026-08-21. + What gates it now is **ours**: the upload path has never run against a real store — + þing-02 D2 forbids testing against production and the org holds no other project + (#18). That decision is **views-appwrite#171**. """ from __future__ import annotations