Skip to content

Check VCS index bounds in one parallel pass - #69

Merged
aarmey merged 1 commit into
mainfrom
parallel-bounds-check
Oct 14, 2026
Merged

aarmey merged 1 commit into
mainfrom
parallel-bounds-check

Conversation

@aarmey

@aarmey aarmey commented Oct 14, 2026

Copy link
Copy Markdown
Member

Summary

_VCSBase.__init__ checks that every index is in [0, n_minor) with indices.min() and indices.max(): two single-threaded passes over the whole array. On large files that is a large share of the load time, spent after the indices were already decoded in parallel.

  • New _ops.indices_in_bounds() checks arrays of at least 2²⁰ signed-integer indices with one parallel numba pass. Smaller or unsigned arrays keep the numpy min/max path.
  • The constructor uses it. Behavior is unchanged: an out-of-range index still raises ValueError before the dtype narrowing, so it can't wrap silently.

Results

full_IBD_cohort (1.19M cells, 3.6 billion indices = 14.6 GB of int32), warm cache:

Before After
Bounds check in the constructor ~1.2 s (min 0.6 s + max 0.6 s) one parallel pass
load_packed 5.30 s 3.50 s

What's left is mostly decoding the packed indices (~2.9 s).

Notes

  • This doesn't change index dtypes. Decoding straight into uint16 would halve the memory, but it would break the int32-minimum index policy from Fix/dtype narrowing #29 that the kernels rely on, so I left it out.

Test plan

  • test_parallel_bounds_check: the parallel path agrees with min/max for int32/int64 arrays with no bad index, a negative index, one too large, and 2**31 - 1.
  • test_large_out_of_bounds_index_raises: the constructor raises when the parallel check finds a bad index (VCSC and VCSR).
  • Full suite: 1588 passed, 71 skipped. ruff check, ruff format --check, codespell and ty check are clean; vulture, radon and xenon report nothing new.

https://claude.ai/code/session_01QBeTZy6BKKGKGJ7fwvkWgM

The constructor validated indices with indices.min() and indices.max():
two single-threaded passes over the whole array. On the IBD cohort file
(3.6 billion int32 indices, 14.6 GB) that took ~1.2 s of every load,
after the indices had already been decoded in parallel. Arrays of at
least 2**20 signed-integer indices are now checked by one parallel numba
pass; smaller or non-signed arrays keep the numpy path. Behavior is
unchanged: an out-of-range index still raises before any narrowing.
load_packed on the IBD file: 5.30 s -> 3.50 s.

Claude-Session: https://claude.ai/code/session_01QBeTZy6BKKGKGJ7fwvkWgM
@aarmey
aarmey merged commit 385ceec into main Oct 14, 2026
7 of 8 checks passed
@aarmey
aarmey deleted the parallel-bounds-check branch October 14, 2026 15:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant