Skip to content

BUG: fix weighted-nanquantile broadcasting issue. - #2

Open
cakedev0 wants to merge 1 commit into
mainfrom
fix/weighted_nanquantile-refresh
Open

cakedev0 wants to merge 1 commit into
mainfrom
fix/weighted_nanquantile-refresh

Conversation

@cakedev0

Copy link
Copy Markdown
Owner

Refresh of numpy#30066 on the current main (squashed, conflicts with the weak_q and NaT changes resolved). Staged here until there is room under the open PR limit upstream.

Fixes numpy#29709 and, for the weighted case, part of numpy#16575.

What it does

Weighted nanquantile / nanpercentile no longer has its own code path: the NaNs get a zero weight and the weighted quantile does the rest, as in scikit-learn and scipy.stats. This fixes the IndexError with 1-D weights.

A slice left with only NaNs or zero weights returns NaN with a RuntimeWarning, for the three cases discussed with seberg in numpy#30066:

x w main this PR
[1, 2] [0, 0] ValueError (invalid weights) NaN + warning
[1, nan] [0, 1] ValueError (invalid weights) NaN + warning
[nan, nan] [1, 1] NaN + "All-NaN slice encountered" NaN + warning

The warning is "Slice encountered with all NaN or 0-weight samples" in the three cases.

Changes compared to numpy#30066

  • object arrays keep working (np.isnan does not support them), and datetime/timedelta arrays get NaT for an empty slice
  • tests for 1-D weights with NaNs (the issue), the three cases above, an n-d case and NaT. The old branch had no test for the last commit.

To decide before going upstream

  • An integer array with an all-zero-weight slice now gives a float64 result (it has to hold the NaN). On main it raises.
  • np.quantile still raises for all-zero weights, so nanquantile and quantile now differ on input without NaNs.
  • The third case changes the message of an existing warning.
  • _remove_nan_1d keeps its now unused second_arr1d argument.
  • No release note yet.

Checks

numpy/lib and numpy/ma tests pass (9501 passed). On 400 random weighted calls (float, int and timedelta, 1 to 3 dims), results are the same as main except for the slices of the table above.

🤖 Generated with Claude Code

Give a zero weight to the NaNs and use the weighted quantile, instead of
a separate code path that failed to broadcast 1-D weights (numpygh-29709).

A slice left with only NaNs or zero weights gives NaN and a
RuntimeWarning.

Refresh of numpygh-30066 on the current main.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

BUG: nanpercentile/nanquantile does not broadcast weights as the doc suggested

1 participant