Make both plot helpers backend-agnostic via a public pr_band

[hamed]

box_plot took pandas and plot_pr took Spark. The two plot helpers demanded opposite backends, neither matched the dispatching core, and a pandas user could not call plot_pr at all.

The Spark code moved rather than went away

The Spark calls inside plot_pr were not incidental. They reduced the curve table — 100 replicas by every distinct recall value — to plot size before anything reached the driver. Deleting them would send a large input to toPandas() whole.

So the reduction became a dispatched metric function. pr_band now sits beside confusion_table and calculate_pr, with an implementation for each backend:

from replicas import pr_band

band = pr_band(kpi, by="name", ci=0.90)  # name, recall, precision, low, high

It is public rather than private because the README says most users feed the metric output into their own plotting code, and those users want the band numbers, not the picture.

plot_pr calls it and draws. box_plot converts at the door, since at() has already reduced the data to one row per group and replica. Both now accept pandas, Polars, or Spark.

PySpark floor 3.3 → 3.5

pr_band uses F.percentile, added in PySpark 3.5. This is not a cosmetic preference — percentile_approx returns a different order statistic:

	q=0.05	q=0.95
pandas quantile	-1.2899645000000002	1.6916200499999998
Polars quantile(interpolation="linear")	-1.2899645000000002	1.6916200499999998
Spark F.percentile	-1.2899645	1.6916200499999998
Spark percentile_approx	-1.320431	1.689107

On 100 samples that is 2.4% off at the lower edge — a visible shift in a plotted band, not a rounding difference. Polars needed interpolation="linear" for the same reason; it defaults to "nearest", which gives a third answer. With both fixed, the three backends agree to within floating-point error, and test_pr_band_agrees_across_backends pins the interpolated values.

Spark 3.3 and 3.4 are both past end of life. The pandas UDF fallback stays: it is gated on applyInArrow at Spark 4.1, so it still serves 3.5-4.0.

Two CI jobs were already red on main — both fixed here

I should flag this plainly: the notebook job I added in #5 has been failing since it merged. I verified it locally and did not check the run. Two separate causes:

1. The address normalizer mangled figure dimensions. 0x[0-9a-fA-F]+ matches the 0x600 inside <Figure size 730x600 ...>, rewriting a dimension into the address placeholder. The pattern now requires a word boundary.
2. Figure dimensions are not stable across matplotlib releases. CI produced 730x600 where this machine produced 718x600, from legend spacing alone. Comparing them reports a dependency bump as a defect. Dimensions are now normalized away; the axes count in the same repr is still compared, so a facet that stops rendering still fails.

minimum-dependencies was also red, from before this series: tests/test_plotting.py imported matplotlib at module scope and that job installs no plotting extra. It now skips.

Notebook

examples/quickstart.ipynb drops two .toPandas() calls that are no longer needed and is re-executed. Three of its seven figures changed — the plot_pr bands moved from approximate to exact percentiles. Every text output is unchanged.

Verification

• ruff check . / ruff format --check . — clean.
• pytest -q — 127 passed (108 before; 19 new).
• scripts/check_example_notebook.py — clean, matches committed outputs.
• At the exact minimum floor (pyspark==3.5.0 pandas==1.3.0 polars==1.0.0 pyarrow==4.0.0 numpy==1.21.0, Python 3.9), the pr_band and metrics tests pass. That validates the three riskiest choices: F.percentile on 3.5.0, pandas 1.3 named aggregation on a SeriesGroupBy, and the polars 1.0 how="full", coalesce=True join.

Caveat on that last point: I could not run the Spark sampling tests at the floor locally. This machine has only Java 21 and Spark 3.5 requires Java 8/11/17 — they fail with UnsupportedOperationException: sun.misc.Unsafe ... not available. A control run confirms the limit is environmental rather than a regression: the old 3.3.0 floor fails worse on the same Java, erroring at session creation. The minimum-dependencies job uses Java 11, so CI is the real check on that job.

🤖 Generated with Claude Code

[hamed]

The Polars blocker was real and is fixed in 9698371. CI green on that commit. Replies to each point below.

1. Polars plotting dependencies — fixed

You were right, and the problem was slightly worse than described: it hit both helpers, not just box_plot, since plot_pr collects the reduced pr_band result the same way.

I built the exact environment you named to pin down which dependency was missing:

$ uv pip install -e '.[polars,plot]'
matplotlib 3.11.1   numpy 2.5.2   pandas 3.0.5   polars 1.43.2   seaborn 0.13.2
$ python -c "...; box_plot(polars_df)"
box_plot(polars): FAIL -> ModuleNotFoundError: No module named 'pyarrow'
plot_pr(polars):  FAIL -> ModuleNotFoundError: No module named 'pyarrow'

So pandas is present — seaborn pulls it — and PyArrow is the actual gap, because polars.DataFrame.to_pandas goes through Arrow.

Of your two suggested resolutions I took neither exactly, because both charge someone who does not benefit. Adding PyArrow to the plot extra bills every pandas plotting user for a conversion only Polars needs; requiring pandas explicitly in the docs would leave the README's "the plot extra is all you need" claim false.

Instead _to_pandas falls back to a column-wise copy when PyArrow is absent:

collect = getattr(df, "to_pandas", None)
if callable(collect):
    if find_spec("pyarrow") is not None:
        return collect()
    import pandas as pd
    return pd.DataFrame(df.to_dict(as_series=False))

The data is already reduced to plot size by then — that is the whole point of the pr_band split — so the slower path costs nothing that matters, and the Arrow path is unchanged wherever PyArrow exists. Re-verified in the same clean venv: both helpers now work, and pr_band still returns a native Polars frame.

I did also declare pandas in the plot extra. It arrived through seaborn either way, but both helpers build a pandas frame, so depending on that transitively was luck rather than design.

Two regression tests pin the path by making find_spec report PyArrow missing.

3. recall_round — I want to push back on this one

This is the point where I disagree, and I think it is a factual disagreement rather than a taste one.

recall_round cannot move to plotting. Plotting only ever sees the reduced frame. Rounding has to happen before the quantiles are taken, because replica curves rarely land on identical recall values — once you have grouped by exact recall and computed quantiles, every group holds one replica, the band collapses, and no downstream rounding recovers it.

So it is not an implementation detail leaking upward. It is a modelling decision — how finely to discretize the recall axis before comparing replicas — that is only expressible at the point of reduction. The default is None, which does nothing and keeps full curve resolution; a user who never thinks about it gets the honest answer.

I have made the docstring say this explicitly rather than describing it as a convenience.

2 and the API surface — partly taken

Agreed that the documentation had outgrown the operation. I cut the pr_band docstring by about a third.

I kept the validation. pr_band uses the same _groups / _validate_columns / reserved-column checks as confusion_table and calculate_pr — those are shared helpers, not new machinery, and removing them would make pr_band the odd one out rather than the lean one. The genuinely new surface is one function, one return schema, and one backend method each.

4. Spark floor — the requirement is hard

Cross-backend numerical equivalence is the README's headline claim, not an optional goal: "equivalent inputs produce the same source-row multiplicities across backends", with an explicitly documented carve-out list. A confidence band that differs by 2.4% depending on which engine you ran it on belongs on that carve-out list or gets fixed. I would rather fix it.

Verification

• pytest -q — 135 passed.
• ruff check . / ruff format --check . — clean.
• Notebook checker — clean, matches committed outputs.
• CI on 9698371 — all nine jobs green, including minimum-dependencies and notebook.
• Clean [polars,plot] venv with no PyArrow — both helpers work.