Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 44 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,50 @@ This file exists because the version number was the only signal a consumer got
(register C-111). Releases before 1.2.0 are summarised from their tags rather than
reconstructed in detail.

## 1.3.0 β€” 2026-09-19

**No new failure modes.** A launcher that ran 1.2.0 sees nothing new stop. This release
moves two dependencies across a major version and picks up a statistics fix; the delivered
bytes are unchanged.

### Dependencies a launcher will resolve differently

- **views-frames `>=2.0.0,<3`** (was `<2`). views-frames 2.0.0 is that package's first
MAJOR: its frame constructors now type-check `index` and raise `TypeError` on a non-index
where they previously constructed silently. This repository passes real indexes and is
unaffected. **Delivery bytes are identical** β€” shards emitted through `views_frames.io.arrow`
hash the same under 1.10.2 and 2.0.0 at the pinned toolchain, and the ADR-013 Β§10
byte-parity fixtures pass under 2.0.0 in CI. Not a contract change.
- **views-pipeline-core resolves to 3.3.0** on a locked install (was 3.0.1). The declared
range `>=3.0.0,<4.0.0` did not change; 3.3.0 is what lifted its own `views-frames <2` cap
and made the move above possible. A `pip install` from the tag already resolved the newest
3.x, so a launcher installing from git sees no difference here.
- **pyarrow stays at 16.1.0.** The `<17` ceiling is deliberate and owned by #174.

### Fix picked up from upstream

- views-frames 1.11.0 corrected the published MAP-containment law, which was wrong on
**tied draws** β€” failing roughly 6% of rows on zero-inflated integer count posteriors,
this platform's primary data shape. Moving past 1.10.2 brings it in. This repository
volunteered to take that fix first.

### Documentation a reader of the code will notice

- Both products' `UPLOAD_ENABLED` docstrings previously named a precondition that had
already been met β€” faoapi's C-161 closure notice (delivered 2026-07-20) and the
views-crafdapi selection guard (deployed 2026-08-12). They now name the gate that actually
holds: the non-production Appwrite decision, views-appwrite#171. **The interlock itself is
unchanged and still closed.** Nothing uploads.
- `CRAFDPostProcessorManager` has an intent contract (`docs/CICs/`), stated as a delta
against the UN-FAO one. The README and `docs/architecture/role_and_seams.md` list every
`delivery/` module again, and a test now keeps them complete.

### Upgrading

Nothing to change in a launcher. Environments that hold `views-frames <2` for another
reason β€” views-evaluation's `[frames]` extra caps it β€” will refuse to resolve; this
repository requests neither that package nor that extra.

## 1.2.0 β€” 2026-08-26

**A previously-passing delivery can now fail in three new ways. All three are
Expand Down
36 changes: 32 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# views-postprocessing

[![Python 3.11+](https://img.shields.io/badge/python-3.11+-blue.svg)](https://www.python.org/downloads/)
[![Python 3.11](https://img.shields.io/badge/python-3.11-blue.svg)](https://www.python.org/downloads/)
[![Poetry](https://img.shields.io/badge/dependency%20management-poetry-blueviolet)](https://python-poetry.org/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)

Expand Down Expand Up @@ -43,7 +43,18 @@ poetry install
pip install views-postprocessing
```

Requires **Python 3.11–3.14**.
Requires **Python 3.11**, and only 3.11.

`pyproject.toml` still declares `>=3.11,<3.15`. **That declaration is wrong** and is
tracked as **#295**: the lockfile resolves on 3.11 alone, because `ingester3` caps
`levenshtein >=0.20,<0.21` and no release in that range publishes a 3.12+ wheel.
Measured 2026-08-25 against the index. CI and the delivery's own environment both run
3.11, so nothing in production is affected β€” the cost is that a contributor arriving on
3.12 or 3.13 is told the project supports them and then cannot install it.

Narrowing the declaration is a one-line edit that invalidates `poetry.lock` and forces a
full re-resolve, which would move `pyarrow` off the 16.1.0 the ADR-013 Β§10 byte-parity
fixtures are pinned to (C-72). So it is #295's own change, not a documentation fix.

### Dependencies

Expand Down Expand Up @@ -89,6 +100,21 @@ in #149; their rules survive as called invariants under `delivery/`. See the
affected, confirm the fault offline, and supersede on the wire. The contract has no
retraction primitive; a correction is a new complete run, manifest last.

### If a delivery fails loudly

Since **1.2.0** a run can stop in ways it previously would not, and each one replaces a
silent failure with a refusal. A launcher may see:

| exception | what it means |
|---|---|
| `DeliveryNotFindableError` | the upload succeeded but the consumer's own query does not find **this** run β€” the failure where every call reports success and the partner sees nothing |
| `FindabilityUnverifiedError` | the check above could not run; *"could not ask"* is deliberately not *"asked and got nothing"* |
| `ProducerClientUnavailable` | the producer client would not load, so the observed-data boundary is unknown. Refuses rather than shipping the unobserved tail as observed history |
| `TornRunError` | a run failed partway through uploading. Names every object confirmed uploaded and the one that failed; deletes nothing |

If one fires after an upgrade it is reporting a condition that was already wrong and
already invisible. [`CHANGELOG.md`](CHANGELOG.md) carries the detail.

### Output schema (geographic metadata columns)

These 9 columns are the delivered geography contract, declared in
Expand Down Expand Up @@ -132,7 +158,8 @@ views-postprocessing/
β”‚ β”œβ”€β”€ draws.py # the Β§6 no-collapse gate
β”‚ β”œβ”€β”€ parity.py # sidecar covers exactly the forecast's cells
β”‚ β”œβ”€β”€ observed_range.py # fabricated-month decision
β”‚ └── provenance.py # structured upload provenance
β”‚ β”œβ”€β”€ provenance.py # structured upload provenance
β”‚ └── findability.py # does the consumer's query find THIS run?
β”œβ”€β”€ contract/ # HOW A DELIVERY IS BUILT β€” partner-neutral
β”‚ β”œβ”€β”€ wire/ # the ADR-013 contract (header, shard, sidecar,
β”‚ β”‚ # run_manifest, sink, source_selection, naming)
Expand Down Expand Up @@ -219,7 +246,8 @@ only `.py`; it now scans markdown too.)*
|-----|----------------|
| [`docs/architecture/role_and_seams.md`](docs/architecture/role_and_seams.md) | **Start here** β€” role vs the sibling repos + internal seams |
| [`docs/ADRs/`](docs/ADRs/) | Architecture decisions (esp. ADR-011 mapper→lookup; ADR-012 ontology) |
| [`docs/CICs/`](docs/CICs/) | Class intent contracts (`UNFAOPostProcessorManager`) |
| [`docs/CICs/`](docs/CICs/) | Class intent contracts β€” one per partner manager; the CRAF'd one is stated as a delta against the UN-FAO one |
| [`CHANGELOG.md`](CHANGELOG.md) | **What changed for a consumer**, per release β€” behaviour a launcher can observe, failure modes first |
| `reports/technical_risk_register.md` | Tracked risks β€” C-40 (the remaining pipeline-core inheritance), C-30/C-15 (delivery guards), C-43 (enrichment value verification) |

---
Expand Down
20 changes: 20 additions & 0 deletions docs/ADRs/004_rules_for_evolution_and_stability.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,26 @@ This ADR should be revisited when one or more of the following become true:

At that point, a new ADR should supersede this one.

> **Audit note, 2026-08-26: at least three of those conditions are now true, and nothing
> noticed.** Recorded here rather than acted on, because writing the successor is an
> architectural decision and this was a documentation audit.
>
> - *"External users or downstream systems depend on this repository"* β€” views-models pins
> it in two launchers, views-faoapi and views-crafdapi consume its artifacts, and it has
> been published on PyPI since 1.0.0.
> - *"Breaking changes begin to incur real coordination or migration costs"* β€” the 1.1.0 β†’
> 1.1.1 pin move took eight days and a direct request (views-models#403); 1.2.0's is open
> as views-models#439. Register **C-111** and **C-112** exist for the two halves of that
> gap: consumers are told nothing but a version number, and nothing here can see what
> production actually runs.
> - *"Reproducibility across time becomes a contractual requirement"* β€” ADR-013 Β§10 pins
> delivery bytes against a golden fixture, and C-72 records what a toolchain bump costs.
>
> The deferral itself was sound and remains honest; what has changed is the world it was
> deferring against. **Whoever picks this up should write the successor rather than amend
> this document** β€” that is what its own last line asks for. The owner field above is still
> `β€”`, which is the part ADR-014 Β§4 would actually object to.

---

## Non-Decisions (Explicitly Out of Scope for Now)
Expand Down
103 changes: 103 additions & 0 deletions docs/CICs/CRAFDPostProcessorManager.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,103 @@
# Class Intent Contract: CRAFDPostProcessorManager

**Status:** Active
**Owner:** PRIO MD&D Team
**Last reviewed:** 2026-08-26
**Related ADRs:** ADR-002, ADR-006, ADR-008, ADR-009, ADR-013
**Sibling contract:** [UNFAOPostProcessorManager.md](UNFAOPostProcessorManager.md)

---

> **This contract is stated as a delta, deliberately.** `CRAFDPostProcessorManager` is a
> clone of the UN-FAO manager (`docs/CLONING.md`, register **C-33**), and the two
> managers differ by fourteen lines on each side, none of which changes behaviour. Verify
> that rather than trusting this sentence:
>
> ```
> diff views_postprocessing/unfao/managers/unfao.py \
> views_postprocessing/crafd/managers/crafd.py
> ```
>
> A second full contract was considered and rejected. Two 189-line documents describing
> one behaviour is not redundancy, it is **two things that can disagree** β€” and this
> repository has already paid for exactly that: register **C-75**, where the sibling CIC
> and this directory's other file asserted opposite things about the same call for weeks.
> One contract, one delta. If the managers ever diverge behaviourally, C-33's extraction
> trigger has fired and this document's form should be revisited along with the code.

## 1. Purpose

`CRAFDPostProcessorManager` orchestrates the end-to-end postprocessing pipeline that
delivers VIEWS conflict predictions to the **Complex Risk Analytics Fund (CRAF'd)**,
served by views-crafdapi.

It is the single entrypoint for producing and delivering CRAF'd-formatted prediction data.

## 2. What this contract inherits

**Sections 2–11 of [UNFAOPostProcessorManager.md](UNFAOPostProcessorManager.md) apply
to this class unchanged**, substituting the partner identity below. That includes the
non-goals, the four-stage pipeline guarantee, the inputs and assumptions, the failure
modes and their loudness, the boundaries (ADR-002 topology: `crafd/` β†’ `contract/` β†’
`delivery/`, one way only), and the test-alignment position.

Two of those are worth naming here because they are the ones a reader most often assumes
differ, and they do not:

- **The wire is partner-neutral** (#153). CRAF'd receives the same ADR-013 artifacts
built by the same `contract/` code. Only the *product* differs.
- **This class is one of the repository's only two importers of `views_pipeline_core`**,
mechanically pinned to an allowlist by `tests/test_doc_accuracy.py`. That is what keeps
C-40's blast radius at one file per partner.

## 3. What differs β€” the whole of it

| | `unfao` | `crafd` |
|---|---|---|
| consumer document `name` | `un_fao` | `un_crafd` |
| partner store | `unfao_bucket` | `crafd_bucket` |
| env tuple validated | `appwrite_env.UNFAO_ENV` | `appwrite_env.CRAFD_ENV` |
| consumer repository | views-faoapi | views-crafdapi (its ADR-034) |
| Β§11.4 interlock history | precondition met 2026-07-20 (faoapi C-161) | precondition met 2026-08-12 (views-crafdapi#53) |

`TARGETS` and `S_MIN` are **the same values today** (`lr_ged_sb`, `lr_ged_ns`,
`lr_ged_os`; `S_MIN = 2`) but are independently declared per partner and may diverge
without either being wrong β€” CRAF'd naming an additional target is an Amendment A1 edit
to `crafd/product.py` alone.

**Both partners are gated closed.** `product.UPLOAD_ENABLED` is `False` in each, and the
gate that holds is the same one for both: **views-appwrite#171**, the non-production
Appwrite project decision, without which the upload path cannot be rehearsed against a
real store (#18). The partner-specific preconditions in the table above are satisfied and
no longer gate anything.

## 4. The uncertainty surface is the consumer's, not this class's

CRAF'd is FAO *extended*: same forecasts, same PRIO-GRID geography, same cadence. The
additional surface CRAF'd wants β€” exceedance probabilities alongside HDI/MAP β€” is
computed in **views-crafdapi** (their ADR-034), not here. This producer ships the same
posterior-sample wire the FAO producer ships.

This is a **non-goal** and belongs in a contract because the alternative is attractive and
wrong: a manager that starts summarising draws for one partner has taken a consumer
concern into the delivery, and the Β§6 no-collapse gate exists to prevent exactly that.

## 5. Test alignment

Covered by the same source-scan and seam tests as its sibling, parametrised over both
partners via `PARTNER_PACKAGES` in `tests/conftest.py` β€” including
`tests/test_clone_readiness.py`, which asserts the two partner packages are **independent**
(neither imports the other), and the line budgets in `tests/test_doc_accuracy.py`.

**A note on the budget, current at this review:** `crafd/` sits at **699 of 700** lines.
The next line added anywhere in that package fails the guard, which is what the guard is
for. It is not a defect and is not scheduled; it is recorded so the next contributor meets
it here rather than in a red build.

## End of Contract

This document defines the **intended meaning** of `CRAFDPostProcessorManager`.

Changes to behavior that violate this intent are bugs.
Changes to intent must update this contract β€” and, where the intent is inherited, its
sibling.
4 changes: 2 additions & 2 deletions docs/CICs/UNFAOPostProcessorManager.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@

**Status:** Active
**Owner:** PRIO MD&D Team
**Last reviewed:** 2026-08-19
**Last reviewed:** 2026-08-26
**Related ADRs:** ADR-001, ADR-002, ADR-008, ADR-009

---
Expand Down Expand Up @@ -95,7 +95,7 @@ Assumptions that are not met **must cause failure**, not fallback behavior. The
- **Launch config incomplete:** raises `LaunchConfigError` naming the missing key. A launcher that omits `wire_contract` or declares a `data_format` other than `feature_frame` is **refused**, never quietly routed into a fallback (ADR-003, register C-63)
- **Region coverage mismatch:** Raises `CoverageError` in `_check_coverage()` (called from `_validate()`) if a pinned region's delivered cell count is wrong (S1/C-34) or a GAUL-uncovered excluded cell leaks into the delivery (S4/C-30)
- **Fabricated historical tail:** `_read_historical_frame()` drops months beyond the producer's `last_valid_month_id` at the read (`_clip_observed_history` was the pandas equivalent, retired with that path in #149) so unobserved zero-padding is not shipped as observed history (S2/C-26). Two outcomes when the boundary is unavailable, and they are different on purpose (C-103, 2026-08-17): if the producer simply publishes no boundary β€” or the read fails β€” it **degrades open**, skipping the clip with a WARNING that states the unobserved tail will ship; if the producer client cannot be imported at all it **refuses** (`source_metadata.ProducerClientUnavailable`), because a broken environment is not a producer fact <!-- legacy-ok: retirement record -->
- **Upload provenance:** the historical artifact's `description` carries structured provenance (lookup version, region, expected/actual cell counts, unmapped count) built by `delivery/provenance.py` (`build_provenance` β†’ `compact_description`) via the manager's `_historical_frame_description()` (S5/C-15). The **forecast** side carries no such description: its guarantee is the wire's verified chain β€” per-shard content hashes recorded in the Β§4.2 run manifest, header asserts on load, and manifest-last commit ordering. That is identity and integrity, not the C-15 provenance field set; the Β§4.2 manifest's keys are exactly `contract_version`, `run_id`, `targets`, `shards`, `expected_months`, `expected_cell_count`, `sidecar` β€” and it carries **no** `lookup_version`, `region` or `unmapped_count`. `_delivery_description()` was the pandas-path equivalent and was deleted with it in #149 <!-- legacy-ok: retirement record -->
- **Upload provenance:** the historical artifact's `description` carries structured provenance (lookup version, **the observed-range boundary this run clipped against**, region, expected/actual cell counts, unmapped count) built by `delivery/provenance.py` (`build_provenance` β†’ `compact_description`) via the manager's `_historical_frame_description()` (S5/C-15). The **forecast** side carries no such description: its guarantee is the wire's verified chain β€” per-shard content hashes recorded in the Β§4.2 run manifest, header asserts on load, and manifest-last commit ordering. That is identity and integrity, not the C-15 provenance field set; the Β§4.2 manifest's keys are exactly `contract_version`, `run_id`, `targets`, `shards`, `expected_months`, `expected_cell_count`, `sidecar` β€” and it carries **no** `lookup_version`, `region` or `unmapped_count`. The `observed_through` field is **required and always emitted** (#297, 2026-08-25): an integer when the clip ran, and an explicit `null` when the producer boundary could not be read and the clip was therefore *skipped*, which is the case that most needs recording. A third state β€” never read β€” refuses at build time rather than collapsing to `null`, because reporting "clip skipped" for a run whose clip ran is the C-103 conflation one layer down. Added because #297 took a day to answer a question the artifact should have carried: *observed through when, and decided against what?* `_delivery_description()` was the pandas-path equivalent and was deleted with it in #149 <!-- legacy-ok: retirement record -->

The following **must never** fail silently:
- Missing or None environment variables for Appwrite
Expand Down
4 changes: 3 additions & 1 deletion docs/CLONING.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,9 @@ diff views_postprocessing/unfao/managers/unfao.py \
views_postprocessing/crafd/managers/crafd.py
```

Sixteen lines differ on each side and **none of them changes behaviour**: the import,
The differing lines **change no behaviour**, and the count is deliberately not
repeated here β€” it was sixteen when this paragraph was written and is not now. What
differs is: the import,
the class name, the two partner-named methods and their two call sites, the four
env-var literals, one line that both selects which env tuple is validated and labels
the store, one runtime refusal message, and four lines of prose.
Expand Down
5 changes: 3 additions & 2 deletions docs/architecture/role_and_seams.md
Original file line number Diff line number Diff line change
Expand Up @@ -106,7 +106,7 @@ The input-integrity guards are split into **two homes** on purpose:
- `views_postprocessing/delivery/` β€” **representation-free invariants**. Primitives only
(sets of ints, numpy arrays, scalars, dicts). **No pandas, no views_frames.** Each is a
pure rule that raises or passes: `coverage.py`, `draws.py`, `parity.py`,
`observed_range.py`, `provenance.py`.
`observed_range.py`, `provenance.py`, `findability.py`.
- `views_postprocessing/contract/frame_extraction.py` β€” **the representation seam**. It
turns a `views_frames` frame into the primitives the invariants consume.

Expand Down Expand Up @@ -171,7 +171,8 @@ views_postprocessing/
β”‚ β”œβ”€β”€ draws.py the Β§6 no-collapse gate
β”‚ β”œβ”€β”€ parity.py sidecar covers exactly the forecast's cells
β”‚ β”œβ”€β”€ observed_range.py fabricated-month decision
β”‚ └── provenance.py structured upload provenance
β”‚ β”œβ”€β”€ provenance.py structured upload provenance
β”‚ └── findability.py does the consumer's own query find THIS run? (C-94)
β”œβ”€β”€ contract/ HOW A DELIVERY IS BUILT β€” partner-neutral, reusable by a clone
β”‚ β”œβ”€β”€ wire/ the ADR-013 contract (header, shard, sidecar, run_manifest,
β”‚ β”‚ sink, source_selection, naming)
Expand Down
Loading
Loading