Skip to content

Skip NaN in quantile() and median() - #310

Open
vahid-ahmadi wants to merge 1 commit into
mainfrom
fix/quantile-skipna
Open

Skip NaN in quantile() and median()#310
vahid-ahmadi wants to merge 1 commit into
mainfrom
fix/quantile-skipna

Conversation

@vahid-ahmadi

Copy link
Copy Markdown

Closes #302

Problem

quantile had no NaN handling. NaN sorts to the end of np.argsort, and its weight still counted toward the cumulative distribution, so the inverse-CDF cutoff was pushed upward:

MicroSeries([1.0, np.nan, 3.0], weights=[1, 1, 1]).median()   # -> 3.0
MicroSeries([1.0, 3.0],         weights=[1, 1]).median()      # -> 1.0

Dropping a NaN row should give the same answer as never having had it. Instead the NaN row's weight made up a third of the distribution, moving the 0.5 cutoff past the second value onto the largest one.

To be precise about the expected value: this method is documented as inverse CDF (survey::svyquantile), so the correct answer is 1.0 — an observed value — not pandas' interpolated 2.0. This PR does not change the interpolation semantics, only the NaN handling.

Change

Verification

  • Three new tests. On main, test_quantile_skips_nan and test_quantile_skipna_false_propagates_nan fail; all three pass with the change.
  • The existing Skip zero-weight rows in MicroSeries.quantile #287 zero-weight regression still holds: MicroSeries([10, 20, 30], weights=[0, 1, 1]).quantile(0) -> 20.
  • Integer dtypes unaffected (pd.isna handles them, no np.isnan cast error).
  • make test — 65 passed on pandas 3.0.5 and pandas 2.3.3.

Marked as a minor bump in the changelog fragment: default median/quantile results change for data containing NaN.

Note: the lint job will be red until #307 merges (unrelated ruff 0.16 issue on main).

NaN sorts to the end of np.argsort and its weight still counted toward
the cumulative distribution, so the inverse-CDF cutoff was pushed
upward:

    MicroSeries([1.0, nan, 3.0], weights=[1, 1, 1]).median()  # 3.0
    MicroSeries([1.0, 3.0],      weights=[1, 1]).median()     # 1.0

Dropping a NaN row should give the same answer as never having had it.
Add skipna (default True), dropping NaN rows alongside the existing
zero-weight filter; skipna=False returns NaN when any value is NaN,
matching mean/var/std.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

quantile()/median() don't skip NaN, returning a too-high quantile

1 participant