Draft
Conversation
…e pair that measures it
Adds docs/plans/ndexpr-evaluate.md — a measured audit of the fused-expression engine
(np.evaluate / NDExpr: binding, NEP50 per-node typing, Tier-3B compilation, NDIter drive,
root reductions) and a seven-phase plan to a high-quality, high-performance, broad-coverage
DSL — plus benchmark/fusion/probes/{expr_probe.cs,numpy_twins.py}, which reproduce every
number in it (sections A semantics, B 15-dtype x 18-shape support sweep, C fused vs unfused vs
NumPy at 1K/100K/4M, D fixed cost at n=8, E kernel cache, F 2-D layouts; the twin gives the
NumPy 2.4.2 side on the same core with the same data).
Findings the plan is built on (all pinned by the probe):
* Two real defects and one shared engine gap:
- Where/LogicalNot/any zero-test over an int8 operand throws
"Zero-push unsupported for SByte" (EmitPushZeroPublic lacks SByte) — today EXCUSED in
MisalignedRegistry as W1-E; the plan deletes the excuse.
- Abs(complex128) types as Complex and returns (|z|, 0); NumPy's absolute is D->d
(float64).
- int64 vs uint64 comparisons promote to float64 and lose exactness past 2^53; NumPy 2.x
has dedicated qQ/Qq comparison loops (generate_umath.py:537-548). Fused AND unfused
NumSharp answer False where NumPy answers True.
* Fused float reductions are not NumPy-exact: the flat kernel folds four scalar accumulators,
f16/f32 sum/prod/mean accumulate in f64, Prod is folded where NumPy multiplies sequentially,
and a broadcast input lands 429 ULP from the unfused value. Raw-bit sweep (Appendix B) also
shows the CORE flat np.sum drifting from NumPy at large N (1 ULP @100k, 16 ULP @4m — the
multi-accumulator SIMD fold, not pairwise) while matching bit-for-bit up to N=1000, so the
fused target is set to NumPy's pairwise_sum (loops_utils.h.src:81), not to np.sum.
* Performance is bimodal. Homogeneous float chains: (a-b)/(a+b) 4.3x NumPy, sqrt(a*a+b*b)
4.45x, leaky relu 4.7x, mean((a-b)^2) 7.9x, sum(a*b) 3.3x at 4M. But any comparison /
Where / Min-Max / predicate node types Boolean and breaks the "one dtype" SIMD gate, so the
whole tree runs scalar: a>0.5 is 0.16x NumPy at 100K (0.71x at 4M), maximum(a,b) 0.46x,
where(a>b,a,b) 0.82x the unfused chain; Half trees are 0.15-0.18x the unfused chain
(CanUseSimd(Half) false); bare unary trees 0.65-0.87x; fused max(a*b) 0.44x NumPy and
sum(f32) 0.60x unfused (scalar 4-acc fold vs the SIMD pairwise np.sum).
* Fixed cost ~1.0 us + 3 KB managed garbage per np.evaluate at n=8 (bind clone, Dictionary
typing table, string cache key, broadcast, iterator, result) vs 0.36 us / 0.9 KB for the
unfused chain and 0.52 us for NumPy — every 1K-element case loses to the chain it replaces.
Kernel compile is ~1.2 ms per never-seen tree; each distinct literal value is a new kernel.
* The F/T layout cliff the committed fusion_results.md shows (11.8 ms) does NOT reproduce on a
pinned core: F 4.75 / T 3.91 / C 4.61 ms fused; F-order preservation works and matches
NumPy's K-order result layout.
* Coverage vs the np.* surface NumSharp ships: no Cast/astype node, no
positive/conjugate/real/imag/isposinf/isneginf, no fmax/fmin/copysign/nextafter/
logaddexp*/shifts, no NumPy-semantics logical and/or/xor, no any/all/argmax/argmin/std/var/
nan*/count_nonzero/ptp/average reductions, no tuple axis or flat keepdims, no where=/casting=/
order=/dtype=, no comparison operators, literals only for int/long/float/double (a ulong
literal binds to double; NumPy gives uint64).
Plan (ordered by measured loss; each phase carries acceptance numbers the probe re-checks):
P0 correctness (SByte zero-push, Abs(complex)->f64, exponent-array sign check, complex
Min/Max reductions, typed literals for all 15 dtypes, exact q/Q comparisons, operand-engine
dispatch) + an evaluate.jsonl differential tier whose oracle is NumPy's node-by-node
unfused chain, + a no-Python metamorphic fused-vs-unfused sweep;
P1 typed vector emission — masks as lane masks, Where -> ConditionalSelect, Min/Max via the
fuzz-validated EmitVectorMinOrMax, mask&One for mask->arithmetic (never a multiply), bool
I/O via the existing bool->lane expansion and mask->byte pack; targets a>0.5 >= 1.5x NumPy,
maximum >= np.maximum, leaky relu >= 5x;
P2 reductions — streaming pairwise schedule (bit-exact with NumPy's pairwise_sum regardless of
chunking; the 8 accumulators are 8 SIMD lanes), sequential Prod, result-dtype accumulation
(f16 mean via f32 per _methods.py), plus Any/All/ArgMax/ArgMin/Std/Var/nan*/CountNonzero/
Ptp/Average, tuple axis, flat keepdims, direct out=;
P3 fixed cost — CompiledExpression + implicit per-tree cache keyed by the operand dtype
signature, array-indexed node slots, identical-dims fast path, Param leaves; target
<= 450 ns / <= 600 B at n=8;
P4 coverage — every missing node (incl. Cast, Select, Clip, logical family, shifts,
copysign/nextafter/logaddexp), < > <= >= operators, where=/casting=/order=/dtype=, routing
np.evaluate through the packed-key ExecuteElementWise so the 2-D block kernels apply;
P5 Half widen-compute-narrow per node (npy_half model, Giesen converters) and mixed-width
lane groups (i4*2+f8 target >= 4x NumPy);
P6 CSE, constant folding, unary-only delegation to the direct kernels, opt-in threading with a
fixed merge order, a numexpr-style string front-end, NDExpr.Explain diagnostics.
Run the probe:
NS_PROBE_AFFINITY=4 DOTNET_TC_CallCountingDelayMs=0 OPENBLAS_NUM_THREADS=1 \
dotnet run -c Release benchmark/fusion/probes/expr_probe.cs [ABCDEF]
NS_PROBE_AFFINITY=4 OPENBLAS_NUM_THREADS=1 python benchmark/fusion/probes/numpy_twins.py [ACD]
…e.jsonl differential oracle tier
Phase 0 of docs/plans/ndexpr-evaluate.md: every correctness gap the plan ranked (G1-G8, G10)
is closed, and np.evaluate gets its own NumPy differential tier so every later phase is
measured against NumPy's unfused node-by-node chain instead of hand-picked probes.
Correctness (all pinned against NumPy 2.4.2 probes; gate NDEvaluateParityTests, 14 tests):
- G1 WhereNode.EmitPushZeroPublic covers SByte: Where / LogicalNot / any zero test over an
int8 operand no longer throws "Zero-push unsupported for SByte". The MisalignedRegistry
W1-E excuse is deleted so the where tier verifies it.
- G2 Abs(complex128) types to Double and emits NDComplexMath.Abs (npy_cabs): np.absolute's
complex loop is D->d, a float64 magnitude, not Complex(|z|, 0).
- G4 complex Min/Max reductions and elementwise Min/Max fold through the engine's
ComplexMinNaN/ComplexMaxNaN clamp (lexicographic (real, imag), first NaN sticks —
np.min([3-4j, 0, -1+1j, nan]) is (nan+0j)); identity seeds (+-inf, +-inf).
MinMaxNode and the reduce fold now use EmitScalarOperation(Maximum/Minimum) for EVERY
dtype — the ufunc kernels' own body — which also fixes the +-0 tie: Math.Max resolved a
(-0, +0) tie to +0 where np.maximum returns the SECOND operand.
- G5 a negative integer exponent inside an exponent ARRAY raises NumPy's verbatim
"Integers to negative integer powers are not allowed." per element (IL guard on the
signed-integer Power path; unsigned exponents skip it).
- G6 typed literals: ConstNode carries the CLR value + NEP50 kind (NDExprLiteralKind).
bool / uint / ulong / Complex join int/long/float/double as WEAK Python literals
(bool+True->bool, f4+True->f4, i1+True->i1; f4+1j->Complex; u8 + 2^64-1 -> uint64 wraps
to 0 while i8 + 2^63 raises "Python integer 9223372036854775808 out of bounds for
int64"); Half / decimal / char are STRONG (np.float16-scalar-like: f8+f16->f8,
i1+f16->f16). Decimal literals emit exactly (decimal.GetBits ctor, no double detour).
Exact operator overloads for ulong / Complex / Half / decimal / bool keep the literals
weak/strong as intended — System.Numerics.Complex converts implicitly from Half and
decimal, so without them `expr + (Half)2` typed complex.
- G7 int64 vs uint64 compares EXACTLY (NumPy's qQ/Qq loops): the comparison type for that
pair is Decimal (holds both exactly; the pair never vectorizes). Shared with the engine —
ComparisonKernelKey.GetComparisonType / ComparisonScalarKernelKey / the NDIter compare
route — so np.greater(u8[2^63+1], i8[2^63-1]) is True on every path. An out-of-range
Python-int literal in a COMPARISON compares instead of raising (uint64 > -1 is True,
int8 > 300 is False; arithmetic still raises OverflowError), also via Decimal adoption.
- G8 np.evaluate dispatches on the first array's TensorEngine (NDExpr.FirstArray), the
np.dot rule.
- G10 CallNode accepts all 15 dtypes (SByte/Half/Complex signatures); the audit-v2 T1.34
Call-Half test leaves OpenBugs.
- arctan2(int, int) picks the float loop PER INPUT like NumPy's default type resolver:
arctan2(int8, uint8) is float16 (int8+uint8's result_type int16 -> float32 was wrong).
- round(bool) is float16 (np.round is a function; bool has no rint loop) while floor/ceil/
trunc keep their '?->?' identity loops.
- integer reciprocal is NumPy's C 1/x with the probed per-dtype 1/0 sentinel (0x80..0 for
int32/int64/uint64, 0 for narrower types and uint32) — NDExprIntegerReciprocal mirrors
DefaultEngine.ReciprocalInteger; the generic emitter threw DivideByZeroException.
- complex mean divides by the count through NumPy's complex true_divide formula
((re + im*0)/n, (im - re*0)/n): a NaN in either part poisons both, as np.mean does.
Fused-tree bugs the new oracle exposed and fixed:
- sub-32-bit integer INTERMEDIATES were never wrapped: byte 200+100 sat on the IL stack as
300 (a root node is truncated for free by its stind.i1 store, a parent converting it to a
wider dtype is not), so sqrt(add(u8,u8)) / div(sub(i8,i8),...) / neg(u8) inside a tree
read unwrapped values. Binary and unary nodes now normalize narrow-int results
(EmitNormalizeNarrowInt: one conv.* opcode).
- bool add/multiply in the SIMD block emitted Vector256<bool> arithmetic (threw
NotSupportedException once 32+ elements ran the vector body; a broadcast bool constant
made the JIT fail outright): they are the normalized logical or/and byte-lane ops now,
and the vector locals take the SIMD lane type (bool -> byte lanes).
The oracle tier (test/oracle/gen_oracle.py evaluate -> Fuzz/corpus/evaluate.jsonl, 14,702
cases, floor 11,800; C# side OpRegistry.Evaluate.cs; gate FuzzCorpusTests.Evaluate):
trees are a prefix grammar over the node catalog (in<k>, weak li/lu/lf/lb/lc literals,
strong lh float16 literal, fn(...)), evaluated node by node with NumPy; 9 pairwise layouts x
16 dtype pairs x 40 templates (arithmetic, comparisons, where, min/max, logical, fused
compositions, every literal kind), the 37-op unary catalog x 6 layouts x 13 dtypes, 12
composite shapes over the single layouts, root reductions (sum/prod/min/max/mean; flat, every
axis, keepdims) over 8 layouts, and out= cases recording the returned view AND the whole out
base buffer. NumPy's verbatim errors (bool subtract/negative, negative int power, no-loop
bitwise) are recorded as error cells. Float/complex Sum/Prod/Mean reductions use a benign
[0.75, 1.25] pool until Phase 2 lands NumPy's pairwise schedule — the 4-accumulator fold is
excused at <=16 ULP (E1, tagged PENDING, to be deleted with Phase 2). The other excuses
mirror the ufunc tiers' documented scopes (float16 mod/floordiv W1-A, complex multiply/power/
abs FMA and hypot ULP, transcendental ~ULP, expm1/log1p, complex min/max NaN identity on a
negative-stride view: NumPy's reduce walks it in logical order, NDIter's KEEPORDER in memory
order — P2 canonical-order work). OracleSurfaceCoverageTests: evaluate is now a direct
corpus op.
Gates: NumSharp.Tests 14,365/0 (CI filter), NumSharp.Tests.Oracle 153/0 (24 host-pinned
inconclusive), NDExpr/evaluate/comparison/clip suites 665/0.
… the fused inner-loop shell
Phase 1 of docs/plans/ndexpr-evaluate.md. The first vector contract vectorized a fused tree only
when every operand AND every node shared one dtype, so any comparison, where, min/max, predicate
or logical node — and any bool operand — made the WHOLE tree scalar: a>0.5 ran at 0.16x NumPy
(100K), maximum(a,b) at 0.46x, (a>0.2)&(b<0.8) at 0.39x. "Vector v2" keeps one compute lane
dtype W per kernel and lets every Boolean-typed node ride as a LANE MASK of W.
The contract (NDExpr.Vector.cs — NDExprVectorPlan + NDExprVec + per-node CanEmitVectorV2):
- W = the unique non-bool operand dtype (SIMD-capable), or Boolean when every operand is bool
("byte mode": nodes carry canonical 0/1 bytes through the engine's normalized logical ops).
- a node typed W emits Vector<W>; a node typed Boolean emits a Vector<W> mask (all-ones / zero
lanes). Edges convert: mask -> number is `mask & One` (exact 1/0 lanes — never a multiply,
which turns inf*0 into NaN), number -> Boolean slot is `~Equals(x, 0)` (NumPy truthiness).
- ComparisonNode: the comparison kernel's own vector compare at the common dtype (NaN lanes
false, NotEqual as ~Equals so NaN != NaN is true, unsigned-aware generics); two bool masks
compare by truth table (eq ~xor, ne xor, lt b&~a, le ~a|b, gt a&~b, ge a|~b).
- WhereNode: ConditionalSelect on masks; a numeric condition is nonzero-tested at its dtype.
- MinMaxNode: EmitVectorMinOrMax(propagateNaN) — the fuzz-validated np.maximum body; bool -> or/and.
- UnaryNode: LogicalNot on a value is Equals(x,0), on a mask ~mask; IsNaN ~Equals(x,x), IsInf
|x|==inf, IsFinite |x|<inf (constant masks on integer lanes); floor/ceil/round/trunc are the
identity on integer lanes (they used to force the scalar path); bool add/multiply are the
logical or/and.
- ConstNode: a Boolean-typed literal is AllBitsSet / Zero.
The shell (DirectILKernelGenerator.InnerLoop.Fused.cs, CompileFusedInnerLoop — the production
Tier-3B shell is untouched): every operand arrives as ONE CLR vector type, Vector<lane(W)>; a
bool operand is expanded to a lane mask with the np.where kernels' EmitInlineMaskCreation; a bool
output is packed to 0/1 bytes (1-byte lanes: `mask & 1` whole; 2/4/8-byte lanes: the four masks
of an unrolled block narrow (ulong->uint->ushort->byte) into ONE 16/32-byte store, NumPy's
npyv_pack_b8 shape; the one-vector remainder uses MSB-extract + BMI2 PDEP, per lane without
BMI2). Runtime dispatch: all operands contiguous -> 4x-unrolled SIMD; every input contiguous OR
broadcast (stride 0 — hoisted ONCE, selected by a loop-invariant branch: an N-ary generalization
of the old binary-only scalar-lhs/rhs paths) with a contiguous output -> the broadcast-aware SIMD
loop; strided 32/64-bit lanes with no bool operand -> the existing AVX2 gather loop; else the
scalar strided fallback. The tail is stride-aware, so a stride-0 operand reads its one element.
Bool lanes/pack are 128/256-bit only (a 512-bit host keeps bool-free trees vectorized).
NDExpr.ForceScalar (thread-static test hook) compiles the scalar-only kernel under its own cache
key. NDEvaluateVectorTests (11 tests) replays 47 tree shapes x 8 lane dtypes x contiguous 1-D
(two sizes, so every lane width hits the unrolled block, the remainder vector and the tail) /
2-D / F-contiguous / strided / negative-stride / broadcast-column / 0-d layouts and compares the
vector kernel's bytes with the forced-scalar kernel's — the scalar body is what the
evaluate.jsonl oracle tier holds to NumPy (14,702 cases, still green).
Measured (Release, one pinned P-core, NPY/NS = NumPy ms / fused ms; unfused = NumSharp's own chain):
100K 4M
a>0.5 (bool out) 0.65x NumPy (was 0.16) 1.26-1.45x NumPy (was 0.71), 1.9x unfused
where(a>b,a,b) ~1.0x NumPy 2.6x NumPy, 1.64x unfused (was 1.33x)
maximum(a,b) 0.9x NumPy 1.8x NumPy, parity with np.maximum
(a>0.2)&(b<0.8) 1.26x NumPy (was 0.39) 1.85x NumPy (was 0.93)
leaky relu 17.6x NumPy 5.2x NumPy
f32 af>0.5 8x the unfused engine compare at both sizes
The 100K a>0.5 cell is a third fixed cost (~1 us per call: bind + typing + string key +
broadcast + iterator + result) — Phase 3's target; the unfused engine's own scalar-broadcast
comparison is 4x slower than the fused kernel there (a separate engine finding).
Trap re-confirmed while measuring: `dotnet run` file-based apps can keep a STALE #:project
build — the first probe run reported the pre-Phase-1 numbers verbatim; a copy of the script
under a fresh name (or a fresh script) measures the current tree.
Gates: NumSharp.Tests 14,375/0 (CI filter), NumSharp.Tests.Oracle 153/0 (24 host-pinned
inconclusive), NDExpr/evaluate suites 320/0.
…xpression handle, N-ary input broadcast Phase 3 of docs/plans/ndexpr-evaluate.md (fixed cost). np.evaluate redid, on EVERY call, everything that depends only on the tree and the operand dtypes: rebind the array leaves (a cloned tree), re-run the NEP50 typing pass (a reference-keyed Dictionary), rebuild the kernel's string cache key (a StringBuilder), re-plan the vector emission — ~1.0 us and ~3 KB of garbage per call at n = 8, three times the unfused chain's fixed cost, so every small-array call lost to the chain it replaces. NDExprProgram (Backends/Iterators/NDExpr.Program.cs) is that dtype-dependent work done ONCE per root NDExpr instance and cached on it: the bound tree, the operand dtype signature, the resolved result dtype and the compiled kernel (elementwise), or the reduction node with its lazily compiled flat / axis kernels. The embedded-array form owns its operand list (the same instances every call, so their dtypes cannot change); the positional form re-validates the dtype signature per call and recompiles on a change; a program built under the NDExpr.ForceScalar test hook never serves the vector mode. Programs are immutable snapshots, so the slot is a benign race (two threads racing on a fresh tree compile the same kernel, which the kernel cache dedups). The host (DefaultEngine.Evaluate.cs) now does only what depends on the SHAPES: iteration shape, result allocation, iterator. The inputs' broadcast is one N-ary pass (BroadcastInputDims): identical dims — the common case, and the only one for a single operand — is one clone; otherwise ONE fresh long[] computed NumPy's way. The pairwise Shape.Broadcast fold it replaces built two Shapes per operand (dims + strides each) and the host cloned the result twice more through Shape.Clean; and it could only name its running shape in the error, where NumPy lists EVERY operand — the mismatch now reads `operands could not be broadcast together with shapes (2,3) (2,3) (4,) ` (trailing space included, IncorrectShapeException, the engine's house type for this text). out= still joins through ResolveUfuncIterationShape (its non-broadcastable-output texts are unchanged). NDExpr.Compile() / Compile(params NPTypeCode[] inputTypes) -> CompiledExpression is the explicit handle (numexpr's NumExpr(expr) object): eager JIT, Evaluate(out) for embedded arrays / Evaluate(operands, out) for positional trees, IsPositional / IsReduction / OperandCount / InputTypes / ResultType. A positional handle PINS its dtype signature — a call with other dtypes is a TypeError naming both signatures, never a silent recompile. It is deliberately not the Tier-3C Compile(inputTypes, outputType, cacheKey), which emits a raw kernel computed at ONE output dtype. Plumbing: TensorEngine gained an internal virtual Evaluate(NDExprProgram, operands, out) (NotSupported default; DefaultEngine routes it to EvaluateCore). Literal-as-parameter (plan 3.3) ships as the 0-d operand form: NDExpr.Input(k) fed NDArray.Scalar(v) runs ONE kernel for every v where a literal bakes one kernel per distinct value (~1.1 ms JIT each). It is a STRONG scalar (np.float64(v) semantics — int32 * 0-d int64 -> int64) where a literal is weak; a named weak Param leaf is not done (Phase 6 optional). Measured (n = 8, ns / managed B per call; Release, one pinned P-core, DOTNET_TC_CallCountingDelayMs=0; probe section D, two runs): np.evaluate(expr) prebuilt 1006 / 2896 -> 404-419 / 584 (acceptance <= 450 / <= 600; unfused a*b+c 353 / 928; NumPy 524) np.evaluate(expr, out=) 831 / 2432 -> 266-339 / 184-248 np.evaluate(Sum(a*b)) prebuilt 775 / 2648 -> 295 / 464 (unfused np.sum 348 / 1048; NumPy 1717) np.evaluate(Where(a>b,a,b)) prebuilt 935 / 2968 -> 369-382 / 576 (unfused np.where 432 / 1040; NumPy 934) positional prebuilt - -> 377 / 584 a*b+k, k a 0-d operand 823 / 1560 -> 668 / 1368 (unfused a*b+2.5 453 / 1488) np.evaluate((NDExpr)a*b+c) rebuilt per call 901-1276 / 3128 (unchanged — misses the per-instance cache by construction) Section C at 1K with a PREBUILT tree: fusion beats the unfused chain on 16 of 22 rows (a*b+c 1.32x, (a-b)/(a+b) 1.82x, sqrt 2.05x, where 1.28x, a>0.5 1.53x, (a>0.2)&(b<0.8) 3.55x, leaky relu 3.23x, mean((a-b)^2) 2.75x, i8*i8+1 2.44x, ...); the six it loses are other phases' cells (Half 0.18x P5; sum f32 / max / sum ax0 0.83-0.96x P2) or single-op trees with nothing to fuse (maximum 0.65x, abs 0.67x — the engine's direct kernel route has a lower fixed cost than an NDIter pass, P6.3). Two residuals, both measured, both outside the per-instance cache and recorded in the plan (section 0.1): the inline-rebuilt spelling in a loop still pays bind + typing (~0.5 us + 2.5 KB over prebuilt) — hoist the tree or Compile() it; the lever is Phase 6.1's structural hash (a global program cache keyed by structure + dedup slots + dtype signature, VERIFIED by structural equality — a hash-only match would run the wrong kernel). And a broadcast (0-d) operand costs ~250 ns + ~800 B over identical dims INSIDE NDIterRef.MultiNew (per-operand broadcast Shapes) — the same cost every scalar-operand ufunc pays (unfused a*b+2.5 453 ns vs a*b+c 353), an iterator-level lever for the NDIter perf line. Tests: NDEvaluateProgramTests (14) — cache hit/miss semantics (per root, current operand contents, positional signature change, ForceScalar isolation), the handle (properties, both Evaluate forms, pinned signature TypeError, binding-form rejections), the every-operand broadcast text on elementwise / flat / axis reductions, column x row broadcast, zero-length axis, out never stretched, and the 0-d parameter form reusing one kernel (GeneratedDelegates.InnerLoopCount pinned) with its strong-scalar promotion. Docs: plan section 0.1 (landed-status table for P0/P1/P3, the Phase 3 numbers, the residuals), CLAUDE.md "Fused Expressions". Gates: NumSharp.Tests 14,389 passed / 0 failed / 179 skipped (CI filter, net10.0); NumSharp.Tests.Oracle 153 / 0 (24 host-pinned inconclusive) incl. the evaluate.jsonl tier; NDExpr/evaluate suites 334 / 0.
…erage for OnnxRuntime (117 -> 141) 24 more tests covering the ndarray.flags contract across every conversion path, bit-exact value fidelity through a real session, and realistic runtime scenarios. Verified on net8.0 / net10.0 x ORT 1.16.0 (floor) AND 1.29.0. New files (test/NumSharp.Tests.Interop.OnnxRuntime/): - FlagsTests.cs (12): the memory-layout flag contract with the exact NumPy num integer pinned per path. ToNDArray / ToNDArray<T> = OWNING C-contiguous (owndata=true, num 1285/1287); AsNDArray / AsNDArray<T> = non-owning VIEW (owndata=false, num 1281/1283); 1-D and 0-d are both C- and F-contiguous (1287); an EMPTY AsNDArray is owning, not a view (nothing to lease -> owndata=true); a column-major DenseTensor returns an F-contiguous view (num 1282) while ToNDArray transposes to a C-contiguous copy. Export / output flag rules: a read-only C-contiguous array CAN be exported as an input but is REFUSED as a pre-allocated output (isolating the WRITEABLE check from contiguity); an F-contiguous input is refused by AsOrtValue; a session round-trip preserves the copy-vs-view contract; setflags(write:false) on a view keeps owndata false; flags.ToString() is the six-line NumPy repr. - ValueFidelityTests.cs (6): NaN (payloads included) / +-inf / signed zero / subnormals / min-normal round-trip bit-exact through a real identity session for float64, float32 and float16; integer extremes (int64 / uint64 incl. 2^64-1, int8 / uint8) and bool are byte-for-byte. The interop never canonicalizes a value. - AdvancedScenarioTests.cs (6): concurrent inference on one session (ORT's Run is thread-safe; the interop uses per-call handles) is correct and leak-free; a contiguous slice is fed zero-copy in logical order; a broadcast input is refused zero-copy by AsOrtValue but copied by Tier-2 Run; a 5-D tensor round-trips; a reused session over 40 runs does not leak; a two-input broadcast add reads back exact. Docs: the test count updated 117 -> 141 in .claude/CLAUDE.md, both plan docs, the website interop page and the package README. Every behaviour was probed against live ORT before asserting (dotnet run). This branch is fast-forwarded to the journey3 merge tip (112dba9), so the commit is a clean fast-forward for journey3.
…fixtures + tests (141 -> 148) Adds the capability to read ORT outputs that are NOT dense tensors — the scikit-learn ZipMap sequence-of-maps, sequences of tensors, and string tensors — plus the fixtures and tests for them and for the two model-INPUT rejections. Verified on net8.0 / net10.0 x ORT 1.16.0 (floor) AND 1.29.0. New capability (src/NumSharp.Interop.OnnxRuntime/NDArrayOnnxInterop.NonTensor.cs): - ToNDArrays(OrtValue seq) -> NDArray[] : a sequence of tensors (copies). - ToMap(OrtValue map) -> (NDArray keys, values) : a map (ORT stores it as a keys tensor + a values tensor); a string-keyed map is refused (no NumSharp string dtype). - ToMaps(OrtValue seqOfMaps) -> (keys, values)[] : the ZipMap sequence(map(int64, float)) classifier output, one pair per input row. - GetMapKeys(OrtValue map) -> OrtValue : the keys tensor standalone. - ReadStringTensor(OrtValue) -> string[] : string tensor OUTPUTS (NumSharp has no string dtype). All readers COPY, so no ORT lease is held. Also fixed the EnsureTensor message (it named non-existent "ProcessSequence / ProcessMap visitors"; now names ToNDArrays / ToMap / ToMaps) and pointed the string message at ReadStringTensor. New fixtures (test/oracle/gen_onnx_models.py, each self-checked through onnxruntime): - zipmap_int64.onnx ZipMap (ai.onnx.ml) -> sequence(map(int64, float)). - string_io.onnx string Identity (string in AND out). - sequence_input.onnx SequenceLength(sequence<tensor(float)>) -> int64 (a non-tensor input). 20 -> 23 committed models (the existing 20 re-serialize byte-identical). Tests (NonTensorTests.cs, 7): sequence-of-tensors via ToNDArrays; the ZipMap sequence-of-maps via ToMaps; a single map element via ToMap plus the type guards; a string output via ReadStringTensor (and the numeric/string cross-refusals); null-argument guards; and the two Tier-2 Run input rejections (a string-typed input, a sequence input) — both now reachable with the new fixtures. ImportTests' sequence-output message assertion updated to the improved wording. Docs: test count 141 -> 148 and model count 20 -> 23 in .claude/CLAUDE.md, both plan docs, the website interop page and the README (which gains the non-tensor verbs row + a string-dtype nuance). Every ORT API used was probed against live ORT before asserting (OnnxType / GetValueCount / GetValue map layout, GetStringTensorAsArray). This branch is one commit ahead of journey3.
…DType
Continue the DType-as-single-spelling unification (Stage B) into the two info
objects and the interop dtype-map surface.
np.finfo.dtype / np.iinfo.dtype now return a DType descriptor (NumPy's
finfo.dtype / iinfo.dtype) instead of an NPTypeCode. For a complex finfo this is
the underlying REAL float dtype (float64 for complex128), matching NumPy. Being a
DType it exposes .kind / .name / .itemsize and compares structurally
(info.dtype == np.float64); an NPTypeCode still converts back implicitly, so
existing call sites are unaffected. ToString() renders dtype=<name>.
The public dtype-map helpers across every interop bridge take ONE DType parameter
instead of NPTypeCode, mirroring the rest of the library's dtype surface (a Type,
NPTypeCode, NPTypeCode?, or dtype string converts implicitly and binds the single
overload):
- MLNet: ToDataViewType / TryToDataViewType / ToDataViewClrType,
FromDataViewType / TryFromDataViewType (out DType)
- OnnxRuntime: ToTensorElementType / TryToTensorElementType /
ToTensorElementClrType, FromTensorElementType /
TryFromTensorElementType (out DType); InferenceSessionExtensions
updated to the DType out-param
- System.Numerics.Tensors: ToTensorElementClrType
- pythonnet: ToNumpyDtypeStr / ToBufferFormat
Each throws ArgumentNullException on a null DType. DtypeMapTests updated for the
new out DType shape.
DType.cs gains a load-bearing note on why the implicit Type/NPTypeCode/string ->
DType operators must not be removed: every dtype-taking overload and every interop
map relies on them for source-compatibility, and the reverse (DType -> Type) stays
EXPLICIT to avoid the Type.== ambiguity.
…path Add np.hypot(x1, x2) = sqrt(x1**2 + x2**2) without spurious overflow/underflow, the last float-tier binary ufunc in the arctan2 family (ee/ff/dd/gg loop signatures, same NEP50 promotion, path classification and kernel dispatch as logaddexp / logaddexp2 / nextafter / copysign — it reuses ExecuteFloatTierBinary, the scalar-scalar fold, and the EmitLogAddNextOperation IL routing). Numerics: the scalar kernel is the CORRECTLY-rounded result via Borges' FMA algorithm (NDHypotMath), verified bit-identical to CPython's math.hypot over 1.1M adversarial pairs incl. subnormals / overflow. NumPy calls the platform (win-amd64 UCRT) hypot, which is only faithfully rounded and disagrees with the exact result on 8.7% of float64 inputs, so NumSharp is bit-exact with NumPy on ~91% of float64 and within 1 ULP (more accurate) on the rest. float32/float16 round the correctly-rounded double down to the narrower type, which reproduces UCRT's hypotf EXACTLY -> bit-exact. Half via the double bridge, Decimal via DecimalMath. Perf: np.hypot has no SIMD in NumPy (a scalar BINARY_LOOP), so NDHypotMath.Simd adds a vectorized Borges kernel taken for the contiguous / scalar-broadcast, same-dtype (no per-element cast) float32/float64 cases (TryExecuteHypotSimd in Default.LogAddExp) — bit-identical to the IL scalar kernel and several-fold faster than NumPy; every other layout/dtype falls through to the scalar IL kernel. Wired through TensorEngine.Hypot (abstract) -> DefaultEngine, KernelOp.Hypot, and the DirectILKernelGenerator LogAddNext helper table. Gate: Math/HypotTests.cs.
…outs
np.piecewise(x, condlist, funclist) — evaluate a different function on each region
of x selected by its condition. The LAST true condition wins (forward overwrite,
the opposite of np.select), and one extra func in funclist is the default
("otherwise" = ~any(condlist)). Implemented as a composition over the existing
machinery plus a fused single-pass scalar IL kernel
(DirectILKernelGenerator.Piecewise): seed the output with the default
(funcVals[n]), then overlay each region forward so a later true condition wins.
Output dtype = x's dtype (zeros_like); scalar/constant funcs are weak-scalar cast
into x's dtype, callable funcs run through the composition path. Gate:
Indexing/np.piecewise.Test.cs.
Fix: the NDArray `!` operator (op_LogicalNot) read its operand with a linear
buffer walk, so it returned wrong results on a non-contiguous / F-layout /
strided / negative-stride input. np.logical_not was always correct (it goes
through the engine ufunc) — only the operator overload had the bug; it now reads
through the shape's strides. Gate: Operations/NDArray.NOT.Test.cs.
np.putmask(a, mask, values) writes a.flat[i] = values.flat[i % values.size] wherever mask is True, walking both in C-order (NumPy's PyArray_PutMask / npy_fastputmask). It shares place's whole structure — the writeable-first check, the same-size (not shape) mask contract, the non-bool-mask -> !=0 cast (NaN/inf -> True), the ascontiguousarray+copyto writeback for non-contiguous targets, and the arrays_overlap -> ENSURECOPY guard (NDMemOverlap.SolveMayShareMemory(maxWork:0), NumPy's NPY_MAY_SHARE_BOUNDS) so putmask(a, m, a[2:8]) reads the ORIGINAL a for every cyclic value — and differs in exactly two probed ways (2.4.2): the values cursor advances by POSITION (every element, j in lockstep with i, wrapping at nv) rather than per-True, and an empty values is a silent no-op rather than place's ValueError. Kernel (DirectILKernelGenerator.PutMask): Place's typed-MOV scatter with the cursor advance OUTSIDE the mask-True branch, plus a cursor-free nv==1 scalar-broadcast fast path. All 15 dtypes; scalars convert implicitly. Perf (NPY/NS, best-of-9, Release): geomean 1.80x, faster than NumPy on every cell. Gate: Indexing/SelectionTests.cs (PutMask_*, 30 — the position-vs-place contrast, empty-values no-op, transposed/negstride writeback, operand-overlap self-alias (values/mask aliasing a, full putmask(a,m,a)), complex/float16 mask->bool (both parts / half-truthiness), mask-with-more-dims (size not shape), NaN mask, char/decimal/complex). CLAUDE.md putmask section updated (overlap guard + test count 24 -> 30).
The five NumPy window generators (numpy/lib/_function_base_impl.py). Each takes a scalar point count M (kaiser also a shape parameter beta) and returns a 1-D float64 taper. M is double (NumPy's _FloatLike_co stub type): an int caller binds via the implicit int->double conversion, and a non-integer M reproduces NumPy's float-M behavior (length = len(arange(1-M, M, 2)); kaiser can carry a trailing NaN once |(n-alpha)/alpha| > 1). Cosine windows share M<1 -> empty and M==1 -> ones(1); kaiser has NO M<1 guard (0<M<1 -> a computed one-sample window). kaiser's i0 is the cephes Chebyshev modified-Bessel routine NumPy's np.i0 uses (internal np.BesselI0 helper — NOT a public np.i0; kaiser is its only consumer). BIT-IDENTICAL to NumPy 2.4.2: the elementwise transform is built as an np.evaluate fused expression in NumPy's exact operation order — float64 add/sub/mul/div are IEEE-exact, and cos/exp/sqrt are the same host ucrtbase libm NumPy calls on win-amd64. Fusion is also the fast path: NumPy materializes ~6 intermediate arrays per window, NumSharp fuses the transform into one pass (NPY/NS ~1.8x-9x at 100K/10M). Gate: Math/np.windows.Test.cs.
…windows
Wire the four new ops into the NumPy 2.4.2 differential-fuzz pipeline
(gen_oracle.py generators + OpRegistry dispatch + FuzzCorpusTests tiers +
committed corpus):
- hypot -> logic.jsonl (gen_binary over the arctan2 pair layouts). float32/
float16 bit-exact; float64 is <=1 ULP vs NumPy's faithfully-rounded UCRT hypot,
excused Double-only in MisalignedRegistry (correctly-rounded Borges FMA,
prefer-precise; a gross error still fails).
- piecewise -> groupa.jsonl (scalar funclists over int32/float64/uint8/
complex128 x nc / nc+1 / overlapping conditions x 2-D / transposed / all-false
layouts). Callable and weak-scalar overflow edges stay unit-tested.
- putmask -> putmask.jsonl (new corpus: 11 dtypes x 3 value modes {scalar nv==1 /
cycle nv==3 / long nv==size} x 8 layouts incl. the non-contiguous writeback).
- windows -> windows.jsonl (new corpus: bartlett/blackman/hamming/hanning/kaiser
over the empty/single/even/odd/multi-SIMD-chunk M corners + kaiser's beta sweep
crossing the i0 Chebyshev split + the float-M path). Host-libm gated
(RunHostLibmCorpus) like fft — hard on win-amd64, Inconclusive elsewhere.
New conversion package bridging NDArray and Apache Parquet through the fully-managed Parquet.Net (no Pandas.NET, no native code, no P/Invoke of its own). Parquet is columnar, so a column maps almost 1:1 onto an NDArray, and Parquet.Net's read API is buffer-based (ReadAsync<T>(field, Memory<T>)) — this bridge hands it a Memory<T> window over the NDArray's own unmanaged block, so a required (non-null) column decodes STRAIGHT into NDArray memory with no intermediate managed array and no extra copy. Surface: ParquetFile (NpzFile-style — pf["col"], pf.f.col, pf.Columns, pf.RowCount, column projection, bounded-memory row-group streaming), ParquetConvert, ParquetLoadOptions, NumSharpParquetExtensions, and a Memory<T> manager over the unmanaged buffer. Like the other interop bridges it is a pure conversion library: no TensorEngine seam, no ModuleInitializer, no native asset. Sole external dependency Parquet.Net [6.1.0, 7.0.0). Adds the parquet-dotnet reference clone as the refs/parquet-dotnet submodule, registers both projects in the solution, and grants the package + its test assembly friend access (Assembly/Properties.cs) for parity with the other interop bridges. Gate: test/NumSharp.Tests.Interop.ParquetNet.
- getting-and-setting-values.md: new guide mapping every way to get and set NDArray values (indexing, slices, boolean masks, fancy indices, fill / put / place / copyto / fill_diagonal, the flat iterator) with the view-vs-copy write-through rules; docs/toc.yml links it and the existing Iterating & Enumerating guide. - api/toc.yml: regenerate the DocFX API TOC to include the DType-system classes (DType, DTypeCasting, DTypeFlags, ArrayMethod, ArrayMethodContext, BuiltinCastingImpl, CastingImpl, ...). - docs/plans/scipy.md: planning note for a sibling ScipySharp repo (SciPy 1.16.3 layered on NumSharp's numpy 2.4.2 oracle, following the OptunaSharp consumer pattern).
Wire the iteration/enumeration MemoryDiagnoser benchmark (foreach / flat / flatiter / nditer / nested_iters / broadcast x kind x size) into the interactive benchmark menu.
…y suite A runnable examples project (examples/gist/NumSharp.GistExamples.csproj, net8.0;net10.0) that ports the NumPy-dependent numerical routines from the top-ten entries of a saved stars x NumPy-call-site gist ranking, plus Karpathy's RNN / Pong policy gradient / batched LSTM / microGPT / natural evolution strategies / stable-diffusion walk. Every port is real C# computation (forward passes, gradients, optimizer state) — none delegates back to Python; NumPy is only the independent test oracle. File-based runners (run.cs) drive the same compiled classes the tests exercise; SOURCES.md / sources.json pin each gist's link, revision and hash, and the VERIFICATION / PARITY / coverage records document the parity claims. References NumSharp.Core + the optional OpenBLAS package (one thread — needed for polyfit -> LAPACK and the matrix products the live parity suite uses). No model, dataset, checkpoint or Python interpreter is needed to RUN the demos.
…live) Gate the NumSharp.GistExamples ports two ways, wired into both test projects (each now references examples/gist/NumSharp.GistExamples.csproj): - NumSharp.Tests/Examples (managed, no Python): deterministic regression tests over the gist + Karpathy ports and their captured demo output. - NumSharp.Tests.Interop (pythonnet-live): Gist*/Karpathy* live parity suites run the hash-pinned ORIGINAL gist / Karpathy Python definitions through pythonnet and byte-check every NumSharp result against NumPy 2.4.2 (GistParity asserts EXACT or a stated ULP budget, never an auto-fallback). Karpathy's original min-char-rnn / pg-pong / batched-lstm / microgpt / nes / stablediffusionwalk sources ride as embedded resources (Fixtures/Karpathy). - NumSharp.Tests/Documentation/ClosedIssueExamplesTests + Interop DocExamples.ClosedIssues: mirror the closing-comment snippets of the GitHub issues resolved by 0.70.0 (#628 and the bundled feature PRs) so a closed issue's claim cannot silently regress. Both test projects build on net8.0 and net10.0. The live suites are gated on a present Python/torch environment (they skip otherwise), matching the existing pythonnet interop tests.
outputs/ holds dated gist / Karpathy demonstration + parity run artifacts (~95 MB) produced locally by the example runners; it is scratch, not repo content. Ignore it so it stays out of the tree and cannot be swept in by git add -A.
…umPy 2.4.2 parity)
Adds the three remaining members of NumPy's divmod family, all probed byte-for-byte
against NumPy 2.4.2 and wired into the differential-fuzz gate.
np.remainder
Exact alias of the existing np.mod (in NumPy `remainder` IS the `mod` ufunc — floored
remainder, result takes the DIVISOR's sign). Array + scalar-divisor overloads; classified
as an EquivalentAlias(mod) in OracleSurfaceCoverageTests.
np.fmod (new BinaryOp.Fmod)
C-library remainder: truncated division, result takes the DIVIDEND's sign — fmod(-7,3) == -1
where mod(-7,3) == 2. Slots into ExecuteBinaryOp exactly like Mod/FloorDivide (scalar-only,
auto via CanUseSimdForOp), so it inherits ALL layouts, NEP50 promotion (integer stays integer,
bool->int8), out=/where=/dtype=, and every dtype for free:
- core EmitScalarOperation -> new EmitFmodOperation/GetFmodMethod -> NDDivision.Fmod* helpers
- Half via EmitHalfOperation (float32), Decimal via EmitDecimalOperation
- integer fmod = C# `%` with the div0->0 and int/long MIN%-1 guards; float fmod = C# `a % b`
(bit-identical to C fmod incl. -6%3=-0, inf%3=NaN, 3%inf=3); Complex refused (NumPy TypeError)
np.divmod (two-output tuple ufunc)
Returns (floor_divide(x1,x2), remainder(x1,x2)) — verified byte-for-byte across every edge
(int div0 -> (0,0), signed MIN/-1 -> (MIN,0), float div0 -> (+/-inf, nan), +/-inf, nan, -0.0).
- out=(q,r)/where=/dtype= path composes the already-validated FloorDivide + Mod (full parity)
- common path runs a FUSED single-pass IL kernel (DirectILKernelGenerator.DivMod.cs): a
4x-unrolled scalar two-in/two-out loop calling NDDivision.Divmod*(a,b, out mod)->floordiv
over contiguous promoted operands (the np.modf materialize pattern)
- integer Divmod* helpers use the ONE-IDIV form q=n/d; r=n-q*d (not a second n%d), which
halves the integer kernel cost (verified bit-identical over 2M random incl. MIN/-1, div0)
Perf (NPY/NS, Release, best-of-11): divmod 100K 2.1-2.3x / 10M f64,i32 1.7-1.8x (10M i64 1.26x,
memory-bandwidth-bound); fmod 100K 1.6-1.9x / 10M 1.34-1.45x (bandwidth-bound). Faster than NumPy
at every non-negligible cell.
Fixes W1-A (rode along): EmitHalfOperation computed float16 mod/floor_divide via a double
`a - floor(a/b)*b` path, so float16 finite/0 gave NaN instead of NumPy's +/-inf (a known bug,
excused in MisalignedRegistry). It now computes float16 mod/floor_divide/fmod in FLOAT32 via the
NDDivision *Single helpers, exactly matching NumPy's HALF loops (astype 'e'->'f'). float16
division is now bit-exact with NumPy and the W1-A excuse is removed (so any regression fails the
gate). Also adds the b==0 guard to NDDivision.DivmodDouble/DivmodSingle (npy_divmod's own guard —
it lived only in the FloorDiv/Rem wrappers, so the raw fused kernel gave (NaN,NaN) for div0).
Oracle: fmod added to the divmod_power tier (369 cases); divmod added to the multioutput tuple
tier (24 cases: int/float dtype x contig/negstride/broadcast/mixed, all edges). OpRegistry +
OpRegistry.Kinds (tuple) cases; remainder alias + Shrinker entry.
Gates green: differential replay 246/246 vs NumPy (11 dtypes x 7 layouts + 5 mixed pairs),
edge 59/59 (real view paths, unroll/tail sizes, out=/where=, complex-raises, Char/Decimal, 0-d),
FuzzCorpusTests 66/66, targeted main-suite math/binary/dtype 634/634.
Corrects one test that asserted the old W1-A bug (B33_Half_FloorDivide_FiniteOverZero) to expect
NumPy's +/-inf.
…ion/differentiation
Implement NumPy 2.4.2's `np.trapezoid` (composite trapezoidal integration) and
`np.gradient` (2nd-order central differences), both BIT-IDENTICAL to NumPy across
every dtype, spacing mode, axis form, edge_order and memory layout. (`np.trapz` was
removed in NumPy 2.x, so only `trapezoid` is provided.)
## np.trapezoid(y, x=None, dx=1.0, axis=-1)
Pure composition mirroring NumPy's own expression `sum(d*(y[1:]+y[:-1])/2.0, axis)`,
so NEP50 promotion, broadcasting and every edge case fall out of the existing
operators:
* dx / the literal 2.0 are WEAK NEP50 scalars → float32 stays float32, float16
stays float16, integer/bool → float64, complex128 stays complex128.
* 1-D y reduces to a 0-D scalar; x is NOT sorted (a decreasing x integrates in
reverse). A 1-D x broadcasts along axis; an N-D x is diff'd along axis.
* Float-family inputs take a FUSED single-pass np.evaluate for d*(y1+y2)/2.0 (no
intermediate temporaries) then a pairwise np.sum — 2–9x faster than NumPy at
100K+ while byte-identical. INTEGER-family inputs take the plain-operator
composition, because the fused kernel promotes the intermediate y1+y2 to float
too early and would NOT wrap a uint8 overflow (200+180 must be 124, not 380);
the operators each materialise at their own dtype and wrap correctly.
## np.gradient(f, *varargs, axis=None, edge_order=1)
Faithful port of NumPy's slicing+arithmetic implementation. Same shape as the input;
returns a bare array for one axis, a tuple otherwise (GradientResult: implicit
NDArray / NDArray[] conversions, Deconstruct, indexer, IsSingle).
* otype: float-family kept; integer/char → float64; BOOLEAN raises NumPy's verbatim
"numpy boolean subtract" message.
* Uniform spacing feeds the stencil WEAK C# double coefficients (f32/f16 preserved);
non-uniform (a coordinate array) feeds STRONG float64 coefficients that cast down
to otype on store — exactly NumPy's pre-alloc-and-assign precision.
* Perf: the interior stencil computes straight into the out view via ufunc out=
(uniform) or a fused np.evaluate (non-uniform), avoiding the intermediate temps
and slice-copies → 100K 2.3–5.2x, 10M unit 1.89x / coord 1.66x / edge2 2.48x,
2-D 2.7–4.5x. Small 1-D arrays sit at NumSharp's documented per-op setup floor.
* C# API: NumPy's keyword-only axis/edge_order cannot coexist with positional
*varargs, so spacing is spread over arity overloads and axis/edge_order are named.
Two documented resolution quirks: `gradient(f, 2, 3)` binds spacing=2/axis=3 (use
doubles or an object[] for two scalar spacings), and a tuple axis with no spacing
needs `gradient(f, Array.Empty<object>(), axis: new[]{...})`.
* Verbatim error taxonomy: ValueError (too small / edge_order>2 / repeated axis /
coord length or rank mismatch), TypeError (invalid number of arguments), AxisError
(out of bounds). gradient's axis normalization is done inline because the shared
normalize_axis_tuple helper silently de-duplicates instead of raising "repeated
axis".
## Verification
* Differential vs NumPy 2.4.2 through the real API: trapezoid 57 cases + gradient 56
cases, all dtype+bytes bit-identical (C/F/transposed/strided/negative-stride,
x=None/dx/coord/reversed, axis variations, 1/2/3-D, degenerate empties).
* Oracle: trapezoid (array, 203 cases) + gradient (tuple, 154 cases) added to
gen_multioutput → multioutput.jsonl; FuzzCorpusTests.MultiOutput bit-exact
(uint8-overflow bug caught here and fixed). Surface + coverage-strength guards
pass; zero-leak gate clean for both ops.
* Unit tests: test/NumSharp.Tests/Math/np.{trapezoid,gradient}.Test.cs (37 tests).
…ed-precision inputs np.gradient with non-uniform (coordinate) spacing on a float16/float32 input computed the one-sided boundary (edge) differences in the input's REDUCED precision instead of float64, diverging from NumPy 2.4.2 by 1-14 ULP at the two edge elements (the interior was already correct). Caught by a fresh validation differential; the committed oracle's gradient cases used only uniform spacing and missed it. Root cause: the edge coefficients are 0-d NDArray.Scalar(double) values, and NumSharp demotes a 0-d float64 scalar to WEAK against a float16/float32 array (Scalar(f64) * f32array -> f32), so the edge product/quotient ran in reduced precision. NumPy keeps the float64 coordinate coefficients STRONG (np.array(1.6) * f32 -> f64), computes the edge in float64, and rounds ONCE to the output dtype on store. The interior was correct because it multiplies by 1-D float64 coefficient ARRAYS, which NumSharp correctly keeps strong. Fix: GradientImpl computes edgeWiden = (otype is float32/float16) ? float64 : null; Edge1/Edge2 widen their edge slices to it (exact widening) in the non-uniform branch, so the edge arithmetic runs in float64 and the store rounds once to otype -- bit-identical to NumPy. float64/complex128/Decimal inputs pass null and are numerically unchanged (the float64 scalar already promotes up to them); the previously-leaked Scalar temporaries are now disposed. Validated vs NumPy 2.4.2: fresh 307-case C-contiguous differential (305 bit-exact incl. all f16/f32 non-uniform edges now fixed) and a 240-case trapezoid layout differential (232 bit-exact). Perf unchanged (non-uniform f32 gradient 1M = 2.1x NumPy; only two edge-hyperplane astypes added). Two ACCEPTED divergences found and documented in source (not fixed here, both pre-existing/library-level): (1) complex128 divided by a non-power-of-2 real scalar is 1 ULP off -- the .NET System.Numerics.Complex division vs NumPy's complex-divide formula; reproducible in isolation as np.divide(complex, 3.0); complex byte-exactness is contractual for unary ufuncs only. (2) trapezoid on a non-C-contiguous (Fortran/transposed/strided) float16/complex128 input is 1 ULP off -- np.sum is layout-order-dependent in NumPy itself, and NumPy preserves the intermediate's F layout while NumSharp's fused half is C-contiguous. Pinned by Gradient_NonUniform_Float32_EdgesMatchFloat64Cast and Gradient_NonUniform_Float16_EdgesMatchFloat64Cast (invariant: gradient(fReduced, x) == gradient(f64, x).astype(fReduced), bit-for-bit).
…cal-multiply)
np.divide / the `/` operator / any binary true-division on complex128 was 1 ULP off
NumPy on ~33% of finite operands — most visibly `np.divide(z, 3.0)`, a complex array
divided by a non-power-of-2 real scalar (a power of two like /2.0 is exact both ways,
which is why np.trapezoid dodged this).
Root cause: ComplexDivideNumPy's finite branch deferred to the BCL
System.Numerics.Complex.op_Division, whose comment falsely claimed "ULP-identical to
NumPy for finite inputs". Both use Smith's algorithm, but they diverge two ways:
- NumPy (CDOUBLE_divide, numpy/_core/src/umath/loops.c.src) forms scl = 1/denom ONCE
and MULTIPLIES; the BCL DIVIDES each component by denom. So z/3.0 is z*(1.0/3.0) in
NumPy but z/3.0 in the BCL — 1 ULP wherever a*(1/c) != a/c.
- NumPy pivots on |c| >= |d|; the BCL on |d| < |c| (they pick different formulas
exactly at |c| == |d|).
Fix: ComplexDivideNumPy is now a byte-for-byte transcription of NumPy 2.4.2's
CDOUBLE_divide (Smith's algorithm, reciprocal-multiply, un-fused mul+add to match
NumPy's MSVC /fp:precise build; RyuJIT never introduces an FMA on its own). The
divide-by-zero sub-branch (divide by |c|/|d|, both +0.0 via Math.Abs) subsumes the old
explicit (0+0j) special-case and yields NumPy's component-wise IEEE inf/NaN result.
Verified byte-exact vs NumPy 2.4.2 over a 3M random complex/complex + 2.4M
complex/real-scalar (incl. 3.0, 7.0, 0.1) + a full 0/-0/inf/nan/subnormal/huge edge
grid: 0 finite/inf mismatches (both Smith branches). The only residual is
non-contractual NaN sign/payload in pathological inf/nan/+-0-divisor edges — the oracle
tokenizes complex NaN for binary ops (NaN-sign is contractual for unary ops only), and
the value is NaN either way.
Perf (NPY/NS, Release, best-of-21, warm): complex/real-scalar 2.05x (100K) / 1.93x (1M),
complex/complex 1.22x / 1.15x — the inlined reciprocal-multiply (fewer divisions, no BCL
call) now BEATS NumPy's scalar CDOUBLE_divide loop.
Oracle: the "complex division ~1 ULP (npy_cdivide vs System.Numerics.Complex)"
MisalignedRegistry excuse is now DEAD and REMOVED — divide/true_divide are gated
bit-exact (Binary_Arith/Specials pass without it; the 29 committed complex+real/mixed
divide cases now enforce the fix). corrcoef's excuse STAYS but its residual is
re-attributed to cov's managed complex GEMM (np.dot), not the division (now exact).
Doc header, B2 cross-reference, Fuzz/README ledger, and np.corrcoef code comment updated
to match.
…es / histogramdd / histogram2d (NumPy 2.4.2 parity)
Implements the full histogram family bit-exact with NumPy 2.4.2, ports of
numpy/lib/_histograms_impl.py and _twodim_base_impl.py.
API (result structs follow the meshgrid INDArrayCarrier pattern — implicit→NDArray,
Deconstruct, named fields):
- np.histogram(a, bins=10, range=None, density=False, weights=None) -> (hist, bin_edges)
- np.histogram_bin_edges(a, bins=10, range=None, weights=None) -> bin_edges
- np.histogramdd(sample, bins=10, range=None, density=False, weights=None) -> (H, edges)
- np.histogram2d(x, y, bins=10, range=None, density=False, weights=None) -> (H, xedges, yedges)
bins overloads: int / estimator string / NDArray edges / double[] / int[] / object[] (mixed per-dim).
All 8 estimators: sqrt, sturges, rice, scott, fd, doane, auto, stone.
Bit-exactness (validated: 849 live-NumPy .npy round-trip fuzz cases + 42 unit tests +
verbatim error-message parity; histogram counts 100% bit-exact, dd/2d H 100% bit-exact,
weighted-density/weighted-non-uniform within a few ULP = NumPy's own assert_almost_equal
tolerance from np.sum/argsort order):
- Bin dtype B = FloatPromoteBinType(input): Half->Half, Single->Single, else->Double.
Shared by histogram AND histogramdd (dd edges are float32 for a float32 sample, not
float64 — NEP50 weak-float `num` never widens a strong float array).
- HistogramLinspace computes float32/float16 edges IN-dtype (NumSharp's np.linspace is
double-internal-then-cast, which diverges ~1 ULP from NumPy whose endpoints are B-typed).
- Weighted path blocks at 65536 and accumulates float64-per-block-then-casts to the weight
dtype (both load-bearing for last-ULP parity); complex weights split real/imag.
- bool->uint8, NaN dropped as outlier, inclusive last bin, signed-int-overflow ranges, and
the full verbatim error taxonomy (ValueError/TypeError).
Performance (NPY/NS, Release, best-of, higher = NumSharp faster): 26/28 measured cells
>=1.5x; everything at 100K/10M is 1.76-5.71x. Two sub-1.5x cells are both at N=1000 (the
per-op allocation floor). Fused kernels in DirectILKernelGenerator.Histogram.cs
(generic-closure-cached-per-dtype, the WeightedSum idiom):
- HistogramCount (unweighted uniform, privatized accumulators like bincount)
- HistogramWeighted (fused per-block float64 scatter, no indices temp)
- HistogramSearchCount / HistogramSearchIndex (non-uniform + dd per-dim binning) via a
branchless SIMD "count edges <= x" (Vector256.LessThanOrEqual + popcount, NaN-padded tail)
below 256 edges, binary search above. This replaces np.searchsorted in the hot loops
(its per-key overhead made non-uniform 172ms->14ms and dd3_20 87ms->23ms at scale).
Oracle/gate wiring:
- The 4 new np.* names classified SiblingOwned in OracleSurfaceCoverageTests (tuple returns,
polymorphic bins, multi-array samples — no single-operand corpus representation, like bmat;
the dedicated np.histogram.Test.cs suite + live differential is the gate).
- The fused-count privatization's NativeMemory site allowlisted in
NativeAllocationChokepointTests (audit debt, same pattern as np.bincount).
…— KEEPORDER relayout of the reduction input
np.trapezoid ends in np.sum(d*(y1+y2)/2, axis), and floating-point summation
is not associative, so np.sum is LAYOUT-ORDER-DEPENDENT: when the reduced axis
is the contiguous (inner) one NumPy sums it pairwise — and for float16
accumulates in float32, narrowing once — whereas a strided/outer reduced axis
sums sequentially (float16: narrowing at every step). Which fires is decided by
the layout of the reduction input, and NumPy allocates that intermediate in
KEEPORDER (its ufunc output layout, matching the operand's stride permutation),
so an F-contiguous / transposed / permuted `y` produces a non-C intermediate
that sums in a different order than a plain C one.
NumSharp's fused `half` was always C-contiguous, so for a non-C-contiguous `y`
it summed in C-order where NumPy summed in KEEPORDER — a 1-ULP divergence,
previously an ACCEPTED [Misaligned] gap for float16/complex128 (and, on cleanly-
F/transposed inputs, occasionally float32/float64; integer sums are exact and
never diverged). This was broader than the docs claimed and is now fixed
outright.
Root cause and fix (probed against NumPy 2.4.2):
- NumSharp's np.sum ALREADY reproduces NumPy's per-layout reduction bit-for-bit
across C/F/transposed/permuted (verified 108/108 in isolation), so the sole
gap was `half`'s layout. RelayoutHalfToNumpyKeepOrder re-lays the float-family
intermediate into NumPy's exact KEEPORDER before the sum. The element VALUES
are layout-independent (an elementwise expression), so this is a pure strided
copy into a fresh owned array carrying the KEEPORDER element-strides
(Shape(dims, strides) over a contiguous buffer) — no view/ARC entanglement.
- KEEPORDER is computed by NumpyKeepOrderPerm, a faithful port of NumPy's
PyArray_CreateMultiSortedStridePerm (stable insertion sort, largest |stride|
outermost, C-order wins ties/conflicts). The final d*(y1+y2) multiply is
KEEPORDER over BOTH operands, so a full N-D coordinate `x` adds a second
voting operand: its d = diff(x, axis) inherits x's stride permutation (diff is
itself a KEEPORDER subtract ufunc), so the ORIGINAL x's strides are the exact
comparator proxy — NumSharp's own np.diff output layout is deliberately not
used (it need not match NumPy's).
- Guarded to `halfFloat && !y.Shape.IsContiguous`: the C-contiguous common fast
path and integer/bool inputs both short-circuit with zero copy; cleanly-F
operands whose evaluate output already equals KEEPORDER also skip the copy.
`half.typecode` (not y's) gates it, so an integer `y` with a float coordinate
`x` — whose float `half` IS order-sensitive — is handled too.
Verified bit-for-bit against NumPy 2.4.2 across a 31,056-case differential
(3 seeds x 9 shapes x {f16,c128,f32,f64,i32,i64,u8} x {C,F,transposed,strided,
negative-stride,perm102,perm021} x every axis x {dx=1, dx=2.5, 1-D f32/f64 x,
N-D f64 x in C/F/T} ) — 31056/31056 exact — plus a special-values sweep
(NaN/inf/-0.0/subnormals) and empty/degenerate F-contiguous shapes. New unit
tests (Trapezoid_Float16_FortranLayout/Transposed/3D_Permuted and
Trapezoid_Complex128_FortranLayout_MatchesNumpyKeepOrder) pin the exact NumPy
bit patterns and assert the C-order result differs (non-vacuous). All 21
trapezoid tests green on net8.0/net10.0; the oracle multioutput tier passes.
…ity (not a bug)
A downstream report (DR-4 zero-copy-crossing gate / NDEvaluator.Materialize)
claimed np.copy(np.broadcast_to(a, shape)) returns flags.c_contiguous == false
because "NumSharp doesn't recompute the flag after copying a broadcast view."
This is a NumPy-semantics misunderstanding, NOT a bug — and deliberately no core
change was made, because changing np.copy here would BREAK NumPy 2.4.2 parity.
Root cause of the confusion: np.copy defaults to order='K' (KEEPORDER), which
mirrors the SOURCE's memory order. A row-broadcast (leading stride-0 axis, e.g.
(3,)->(4,3)) copied with 'K' lays out F-contiguous (PyArray_NewLikeArray KEEPORDER
/ PyArray_CreateSortedStridePerm), so the fresh dense owned buffer legitimately
reports c_contiguous=false, f_contiguous=true — EXACTLY what NumPy returns.
flags.c_contiguous simply reads Shape.IsContiguous (NDArray.flags.cs), recomputed
from the copy's real strides; there is no stale/un-recomputed flag. Verified
byte-for-byte against NumPy 2.4.2 across 68 broadcast x order cases (strides + C/F
flags) plus 3-D neither-contiguous cases — all identical. This exact scenario was
already gated as parity.copy_bcast_K = {C:0,F:1} in FlagsOracleTests.
The correct tool for a GUARANTEED C-contiguous dense copy (e.g. gating a zero-copy
crossing on c_contiguous) is np.copy(x, order='C') or np.ascontiguousarray(x), both
of which correctly yield c_contiguous=true. The fuzz oracle compares only
dtype+shape+bytes (not strides/flags), so contiguity parity is pinned by unit tests.
Changes (test-side only, no np.* or engine change):
- gen_flags_oracle.py + FlagsOracleTests.BuildLayout (twins): 5 new parity.* recipes
— copy_bcast_{C,F,A}, ascontig_bcast, asfortran_bcast — and the flags_oracle.jsonl
corpus regenerated from real NumPy 2.4.2 (+5 cases, purely additive, deterministic).
Pins the CORRECT C-contiguous tools: 'C'/'A'/ascontiguousarray -> {C:1,F:0};
'F'/asfortranarray -> {C:0,F:1}. Layout-matrix floor comment updated 32 -> 37.
- CopyCastCoreParityTests.CopyOfBroadcast_Contiguity_AndIndependence_MatchNumpy:
end-to-end regression over all 15 dtypes x 6 producers asserting the full contract
the report doubted — correct flags per order, owned + writeable + !broadcasted +
independent backing buffer + correct logical values (vs an independent stride-walk).
Gates: FlagsOracle + CopyCastCoreParityTests 30/30 green on net8.0 and net10.0.
… (npy_cpow port)
Same class as the complex-divide 1-ULP fix: NumSharp deferred complex power to BCL
System.Numerics.Complex.Pow, which ALWAYS evaluates the polar form exp(b*log a). For an
integer exponent that is catastrophically imprecise — measured up to ~1e14 raw ULP on
np.power(z, 3)/z**5/z**-1/z**-2 (effectively the wrong value) vs NumPy 2.4.2's
np.power. NumPy's npy_cpow (numpy/_core/src/npymath/npy_math_complex.c.src) instead
special-cases integer exponents in (-100, 100) with EXACT repeated multiplication.
Two fixes, both routing complex power through the faithful port:
- DirectILKernelGenerator.ComplexPowNumPy: a byte-for-byte transcription of npy_cpow —
b==0 -> 1; complex-zero base -> 0 / complex-NaN by sign of Re(b); integer real
exponent in (-100,100) -> z*z / z*(z*z) / binary exponentiation via the naive cmul
(ComplexMulNumPy, == np.square), with cdiv(1,r) for negatives through the now-byte-
exact ComplexDivideNumPy; every other exponent -> Complex.Pow (NumPy's host cpow
branch, the only residual). Wired into EmitComplexOperation, so the whole complex
power surface (function + operator, every layout) uses it.
- Default.Power.cs: the scalar-exponent FAST PATHS (exp==2 -> lhs*lhs, exp==-1 ->
reciprocal, exp==0.5 -> sqrt) are generic and WRONG for complex — z**2 must be
cmul(a,a) (== np.square), not the complex Multiply kernel, and z**-1 must be
cdiv(1,z), not CDOUBLE_reciprocal. TryScalarExponentFastPath now bails for a complex
base so complex power always reaches ComplexPowNumPy, exactly as NumPy routes every
complex power through npy_cpow. Real/integer bases keep the fast paths unchanged.
Verified byte-exact vs NumPy 2.4.2 np.power over 500K wide-magnitude complex operands:
z**2/z**3/z**5/z**-1/z**-2 all 0 finite mismatches (was up to ~1e14 ULP). np.square /
reciprocal / conjugate stay 0. Non-integer/complex exponents (z**0.5, z**(2+1j)) remain
on Complex.Pow — NumPy computes those via the host cpow, an inherent ~ULP/libm branch.
Perf (NPY/NS, Release, best-of-15, warm): np.power(complex,2) 4.7x, power(complex,3)
5.9x, power(complex,-1) 3.8-4.0x at 100K/1M — FASTER than NumPy's scalar npy_cpow.
Oracle: the "complex power ~ULP" excuse is SCOPED OFF integer exponents — a value-level
PowerExponentAllIntegerBranch decode of the exponent operand (mirrors npy_cpow's
integer-branch predicate: real, integral, |n|<100) keeps integer complex power GATED
bit-exact (a regression fails), while the ≤512 element-magnitude-ULP envelope still
covers the genuine non-integer/out-of-range host-cpow branch. Scoping this exposed and
correctly re-excused the corpus's int32-exponent=127 cases (>=100 -> NumPy's cpow branch
in BOTH libraries). Binary_DivModPower green; MisalignedRegistry doc header + Fuzz/README
ledger updated to the integer/non-integer split.
…offset-safe complex weights, weak-endpoint edge dtype, float-repr range messages Validating the histogram family (np.histogram / histogram_bin_edges / histogramdd / histogram2d) against NumPy 2.4.2 with an 843-case bit-exact differential (real np.save oracle .npy round-trip) surfaced four genuine parity gaps in the committed implementation. All four are fixed here: the family is now 802/809 value cases BIT-EXACT with every error case (34/34) verbatim-identical, the sole residual being density normalization's ≤few-ULP np.sum reduction-order class (documented, library-wide, not histogram-specific). 1. Mixed-dtype column promotion (histogram2d + histogramdd sequence-of-columns). NumPy builds its internal sample via atleast_2d(sample).T, which STACKS a sequence of coordinate columns into one array and thereby promotes every column to result_type. The port kept each column at its own dtype, so histogram2d(float32_x, float64_y) returned float32 x-edges where NumPy returns float64 (edge dtype AND values diverged); likewise histogramdd([f32,f64]) / [i32,f32] (→f64) and [f16,f32] (→f32). PromoteColumnsToCommon folds the columns' dtypes through NEP50 promote_types and casts each to the common dtype; a same-dtype sequence (the common case) and the (N,D)-array form are untouched (no copy). 2. Offset-unsafe read of a trivially-contiguous view (complex weights, uniform path). EnsureContiguousDtype judged "already contiguous, no copy" from Shape.IsContiguous alone, but a size-1 array is trivially contiguous whatever its offset, and np.imag(complexWeights) is a stride-2 OFFSET-1 view that keeps the buffer's base address (only a simple contiguous SLICE re-seats it). The kernels then read (void*)a.Address — the buffer base, i.e. the REAL part — so np.histogram([x], bins, weights=[re+im*j]) via the uniform (int-bins) path returned <re,re> instead of <re,im>. The guard now also requires Shape.offset == 0, forcing the copy that folds the offset into a fresh buffer. 3. Weak-endpoint edge dtype / computation (histogram + histogramdd). NumPy's linspace computes internally in result_type(first, last, num) and only then casts to the requested dtype. When the endpoints are "weak" — a supplied range (Python floats) or an empty input (the 0/1 int defaults) — they cannot widen past float64, so: histogramdd edges collapse to float64 for a ranged/empty float32/float16 column (e.g. histogram2d(f32,f32,range=)); and np.histogram, though its RESULT dtype stays the array's float width, computes the edge VALUES in float64-then-cast (a ~1 ULP fix for ranged/empty float32/float16). HistogramLinspace now takes separate compute/result dtypes; GetBinEdges and histogramdd pass weak = range || empty. 4. Non-finite range message renders bounds like Python str(float). The "supplied/autodetected range of [.,.] is not finite" messages used .NET's default double formatting (1.0 -> "1"), where NumPy's f-string uses str(np.float64) (1.0 -> "1.0", 1e20 -> "1e+20"). FormatEdge now routes finite bounds through NumSharp's byte-exact 0-d scalar repr (verified equal to Python str across whole/decimal/scientific magnitudes), so the message is verbatim (e.g. "[1.0, inf]", "[100.0, inf]", "[1e+20, inf]"). Density normalization (n / db / n.sum()) stays a faithful composition of NumPy's exact formula: raw histogram counts, weighted sums, edges and dtypes are bit-exact; only np.sum(n)'s reduction ORDER can differ by <= a few ULP (amplified for complex density by complex-division cancellation), the documented library-wide np.sum characteristic and out of scope to reproduce inside histogram. Tests: 10 new regression tests (mixed-dtype promotion incl. the same-dtype guard, weak-endpoint edge dtype for empty/range dd/2d, float32-range bit-exact edges, complex-weight imaginary survival, verbatim non-finite range messages). Full histogram suite 52/52 green on net8.0 + net10.0.
…Py 2.4.2 parity
Port NumPy's exact interpolation into the QuantileEngine IL kernel, closing the
long-standing [known bug] divergences (W6-A/B/C) that the fuzz gate had to excuse.
Verified bit-exact vs NumPy 2.4.2 over tens of thousands of differential cases
across every dtype x method x axis x layout x scalar/array-q, plus the committed
Stat / NanReduce / DecimalStat oracle tiers (green on net8.0 and net10.0).
Root cause and fixes (all in WriteCell / ComputeIndex / QuantileEngine):
* Two-branch _lerp (the headline fix). NumPy's _lerp is NOT the naive a+(b-a)t:
it computes a+(b-a)t for t<0.5 and b-(b-a)(1-t) for t>=0.5
(numpy/lib/_function_base_impl.py::_lerp — the where=t>=0.5 overwrite). The old
single-branch form diverged by up to ~200 ULP on ~7% of float64 inputs. WriteCell
now emits the two-branch form at the exact per-dtype precision.
* Per-dtype arithmetic mirrors NumPy's ufunc promotion of _lerp(previous,next,gamma),
which subtracts diff_b_a = b-a in the OPERAND's own dtype before promoting:
- float32/float16 + scalar(weak) q -> stays float32/float16 throughout (f16 rounds
every op and the weak weight to f16, matching NumPy's f16 loops);
- float32/float16 + array(strong) q -> float64, but diff rounds to f32/f16 first;
- every integer width -> the subtraction WRAPS at that width, so e.g. int16
1-(-32768) overflows to -32767 and its percentile is a large POSITIVE value
(the historical W6-B 'gross error'/sign-flip — NumPy raises a RuntimeWarning but
returns the wrapped result, which is the contract).
* np.median / np.nanmedian now use NumPy's mean-of-middle reduction ((a+b)/2 at the
output precision), NOT the q=0.5 lerp. NumPy computes median via mean(part[middle]),
a different code path that disagrees with quantile(0.5) by up to 1 ULP on ~10% of
inputs. Threaded through as QuantileEngine.Compute(medianMean:) -> the baked kernel.
* midpoint uses next = prev+1 (never the Ceiling(vi) collapse at integer indices), so
a +/-inf neighbour yields a + inf*0 = NaN exactly as NumPy does on non-finite slices.
* Clamped-index gamma for the identity-fix_gamma methods (linear/hazen/weibull/
median_unbiased/normal_unbiased/interpolated_inverted_cdf): NumPy derives gamma from
its out-of-bounds clamped index (prev=-1 above -> gamma=vi+1 -> b-branch; prev=0
below -> gamma=vi -> a-branch), which fixes the SIGN of a +/-0 endpoint. midpoint and
averaged_inverted_cdf are excluded (their fix_gamma transforms gamma differently).
* QuantileEngine passed staged.Address WITHOUT folding in Shape.offset, so a
contiguous view with offset!=0 (e.g. a simple_slice_offset) read from the buffer
start and ignored the slice. Now uses (byte*)staged.Address + offset*itemsize, the
same logical-element-0 base np.average/np.cov use.
Residual (documented non-contractual, both sides identical VALUE): the SIGN of a +/-0
result (NumPy's -0<+0 introselect partition order + clamped fix_gamma branch on a +/-0
endpoint), and complex np.median NaN-propagation (NumPy _median_nancheck; complex is
not a NumPy percentile/quantile dtype at all).
Third and last of the complex BCL-deferral family (after divide ab69cf8 and integer power 58cd55d). NumSharp's complex `*` / np.multiply routed through the BCL's naive Complex.op_Multiply (a_re*b_re - a_im*b_im); NumPy's ARRAY multiply is the fused vfmaddsub kernel simd_cmul (loops_arithm_fp.dispatch.c.src), taken for EVERY normal layout (contiguous/strided/broadcast/in-place/transposed — verified: all differ from naive; the naive loop_scalar tail only runs on genuine src/dst overlap). The naive path diverged from NumPy on ~14% of operands, up to ~2840 ULP in the catastrophic- cancellation regime. Fix: NDComplexMath.Multiply ports simd_cmul's exact arrangement — real = fused(a_re*b_re) - (a_im*b_im) [Fma.MultiplySubtractScalar] imag = fused(a_re*b_im) + (a_im*b_re) [Fma.MultiplyAddScalar] the a_im*· products being the pre-rounded addend (NumPy's `ab_iiir`), so the last bit matches vfmaddsub. It is the two-operand generalisation of the already-byte-exact NDComplexMath.Square (simd_csquare). The fused/addend split is load-bearing and must NOT be swapped the way Square can (there a_re*b_im == a_im*b_re, so either slot is finite- identical; for a general multiply they differ and the fused product is full-precision while the addend is pre-rounded). EmitComplexOperation(Multiply) now emits this instead of op_Multiply; off x86 the portable Math.FusedMultiplyAdd keeps the finite/overflow bits identical (a NaN component's sign is host-dependent there, tokenized for this non- contractual binary op). Verified byte-exact vs NumPy np.multiply over 500K wide-magnitude complex operands (0 finite mismatches, incl. the cancellation regime). Perf (NPY/NS, best-of-15, warm): 2.5x at 100K, 1.96x at 1M — FASTER than NumPy's simd_cmul. polydiv fix (rode along): NumPy has TWO complex multiplies — its np.complex128 SCALAR multiply is the un-fused scalarmath (a_re*b_re - a_im*b_im), while its ARRAY multiply is FMA simd_cmul (proven: scalar 0/200000 vs naive, array 52387/200000). polydiv's `d = scale * r[k]` is a SCALAR*SCALAR product NumPy computes naively; NumSharp has one (now FMA) array multiply and did it as a 0-d-array multiply, drifting polydiv complex by 1-2 ULP (the poly tier is held bit-exact). NaiveComplexScalarMultiply computes that one scalar coefficient un-fused to match NumPy, while `d * v` (scalar*array) stays the FMA array multiply — restoring polydiv complex to byte-exact. Gate: FuzzMatrix green on all complex-multiply tiers (Binary_Arith/Specials/Products) and Poly; the complex-multiply MisalignedRegistry excuse (≤16 element-magnitude ULP) is now DEAD and its removal + the Fuzz/README ledger row are staged in the working tree but held OUT of this commit because MisalignedRegistry.cs is being co-edited by a parallel session (quantile/median T12 work) — committing it would clobber their WIP. Main-suite complex tests 934/934.
…mPy 2.4.2 parity) np.float_power is the last unimplemented arithmetic binary ufunc in the divmod/power neighborhood. It is np.power restricted to NumPy's two floating loops (dd->d / DD->D): every real input (bool/int/float16/float32/decimal/char) promotes to float64 and a complex operand to complex128, so the result is always an inexact float — and, lacking an integer loop, a negative integer exponent is LEGAL (float_power(2,-1)=0.5 where power raises "Integers to negative integer powers are not allowed"). Design — zero new kernel: - Default.FloatPower resolves the float loop (complex128 if either operand is complex, else float64) and DELEGATES computation to the bit-exact Power engine, so the arithmetic is byte-for-byte power on those loops (same Math.Pow / ComplexPowNumPy). No BinaryOp.FloatPower, no DirectILKernelGenerator changes. - power's scalar-exponent fast paths are preserved: when both operands already carry the loop dtype, it delegates to plain Power (no dtype override), so float_power(x, 2.0) is x*x and float_power(x, 0.5) is sqrt(x). - Error taxonomy AND order are NumPy's (read-only out -> non-bool where -> dtype-no-loop -> complex-input-can't-cast-to-float64 -> out-cast -> shape), raised in the wrapper with the 'float_power' ufunc name, reusing the shared ufunc validators; the delegated Power re-validates identically (naming 'power') but never raises, so the name never leaks. Surface: np.float_power(x1, x2, out=, where=, dtype=) mirroring NumPy's ufunc signature, plus the (NDArray, object) scalar/array-like convenience overload ([NDScoped], like np.power). New abstract TensorEngine.FloatPower (DefaultEngine is the only subclass). Parity (probed against NumPy 2.4.2): - 30/30 direct bit-exact differential (all 13 NumPy dtypes, promotion, specials, out/where/dtype, layouts C/F/strided/negstride/broadcast/0-d/empty) + char/decimal->float64. - Error messages verbatim incl. co-occurring order. - Complex float_power inherits the documented complex-power divergence (F5: Complex.Pow vs npy_cpow host cpow ~ULP for non-integer/large-integer exponents + inf/NaN edges) — bit-exact on the integer-exponent branch, F5-excused otherwise (same MisalignedRegistry scope as complex power). Perf (NPY/NS, Release): scalar-exponent fast path 4.5-51x; int->float64 1.05-1.22x; general fractional-exponent path ~parity (0.86-0.99x) — the Math.Pow physical ceiling (NumPy equally scalar-pow-bound, no SIMD pow; the same ceiling as arcsinh/arccosh). Oracle: float_power added to the divmod_power (main dtype x layout matrix, 342 cases — more than power since int**neg is legal), out_where and specials tiers; OpRegistry + OpRegistry.Kinds + Shrinker + MisalignedRegistry F5 + surface classification. Carved out of the Char proxy (like power — a large char exponent would give a host-libm-dependent non-exact float64; the main tier covers every NumPy dtype). MinCases floors bumped. Gates green in an isolated worktree at HEAD: divmod_power/out_where/specials + OracleSurfaceCoverage + 13 unit tests (Math/np.float_power.Test.cs); leak-clean (delegates to Power, mints no temps). Files: Math/np.float_power.cs, Backends/Default/Math/Default.FloatPower.cs, Backends/TensorEngine.cs (abstract FloatPower), + oracle generator/harness wiring and the three regenerated corpus tiers.
Convert every em-dash (U+2014) to an ASCII hyphen-minus (-) in the hand-authored docfx source under docs/website-src/. 1574 occurrences across 42 tracked markdown/YAML files. Scope: only git-tracked *.md / *.yml content in docs/website-src/. The generated api/*.yml (gitignored) and the refs/data report content pulled in at build time are untouched. Method: byte-level replacement of the UTF-8 encoding E2 80 94 -> 2D, so line endings (CRLF/LF), any BOM, and every other byte are preserved exactly. The diff is a balanced 1438/1438 (each touched line is one delete + one add), so there is no whitespace or EOL churn. UTF-8 is self-synchronizing, so the 3-byte sequence cannot collide with part of another character. Safety checks made before converting: - No em-dash starts any line (after leading whitespace), so none could turn into a stray "- " markdown list item. - The only YAML occurrence was mid-value (docs/toc.yml, "Under the hood - internals"); the list-item dash sits at line start and is unaffected. - Em-dashes inside fenced code blocks were all prose comments (// ... - ...), never executable syntax. En-dashes (U+2013, 27 occurrences) are intentionally left as-is; only em-dashes were requested.
Nucs
changed the title
…p.ma.masked stays as NumPy alias
NumSharp's port of numpy.ma.MaskedArray now follows the ND* type convention of NDArray.
Renames (symbols, done with the IDE rename)
- class MaskedArray -> NDMaskedArray (file Ma/MaskedArray.cs -> Ma/NDMaskedArray.cs)
- class MaskedConstant -> NDMaskedConstant
- tests MaskedArrayTests -> NDMaskedArrayTests (file Ma/MaskedArrayTests.cs -> Ma/NDMaskedArrayTests.cs)
- prop np.ma.masked -> np.ma.NDMasked
- prop np.ma.masked_singleton -> np.ma.NDMaskedSingleton
- the ma oracle harness (FuzzCorpusTests.Ma / OpRegistry.Ma) and three gate comments follow the type rename.
MaskedArrayModule (the np.ma facade) and MaskedPrintOption keep their names.
NumPy API parity kept: np.ma.masked / np.ma.masked_singleton
- NumPy 2.4.2 (probed): np.ma.masked is the ONE numpy.ma.core.MaskedConstant instance (a MaskedArray
subclass), a 0-D float64 0.0 under a True mask, `masked is masked_singleton` is True, repr 'masked',
and a fully-masked reduction returns it (`ma.array([1.,2.], mask=[1,1]).sum() is masked`).
- Both NumPy names are re-added on MaskedArrayModule as EXPRESSION-BODIED aliases of NDMasked, so all four
spellings (NDMasked, NDMaskedSingleton, masked, masked_singleton) are the same object. A second stored
`new NDMaskedConstant()` would have broken the reference-identity contract: the ufuncs/reductions return
NDMasked, and `ReferenceEquals(r, np.ma.masked)` (NumPy's `r is np.ma.masked`) must hold. Ported NumPy
code (`x[i] = np.ma.masked`) compiles unchanged; NumSharp code can use the ND* name.
- New test NDMaskedArrayTests.MaskedConstant_NumPyNames_AliasTheNDMaskedSingleton pins identity across all
four names, the NDMaskedConstant type, the repr token, the 0-D float64 0.0 / True-mask payload, the
fully-masked sum returning np.ma.masked, and `y[1] = np.ma.masked` masking without touching data.
Repaired: the IDE's "text occurrences" pass damaged prose, NumPy-verbatim strings, and the oracle gate
The rename ran with text-occurrence replacement, which rewrote EVERY whole word `masked` -> `NDMasked`
(1,104 comment/string hits) and `MaskedArray` -> `NDMaskedArray` even where the text names NumPy's own
class. Left as-is it would have shipped:
1. A red oracle gate. FuzzCorpusTests.Ma's `case "masked":` became `case "NDMasked":`, but the committed
ma_* corpus spells the kind "masked" (~62K cases), so every masked case fell to `default` ->
"unknown ma expected.kind 'masked'". Verified: MaReduce / MaScan / MaExtras all failed that way.
2. Broken NumPy-verbatim messages: "A masked array does not own its data and therefore cannot be
resized.", "Cannot process masked data." (x2), "Unavailable for masked array.", NumPy's own
NotImplementedError "MaskedArray.tofile() not implemented yet.", the view() itemsize message, and the
NDMaskedConstant.ToString() token "masked" (NumPy's repr(np.ma.masked)). Their tests had been rewritten
in lockstep, so the unit suite stayed green while parity broke.
3. English turned into nonsense, including in files the rename never needed to touch: "masked-off slots
keep prior contents" -> "NDMasked-off slots" in Default.Fabs/Frexp/Ldexp/Positive/Reciprocal and
DefaultEngine.CompareOp, "NumPy's MaskedArray._insert_masked_print" -> "NDMaskedArray._insert_…" in
ArrayFormatter, the masked-corpus schema docs in FuzzCorpus.cs, "masked positives" in
UndisposedIntermediateTests, and the pythonnet decode tests that describe the PYTHON-side numpy
MaskedArray. Those 9 files outside the ma rename are byte-identical to HEAD again (not in this commit);
UndisposedIntermediateTests keeps only its correct NDMaskedArray type mention.
How the repair was done (mechanical, then reviewed)
- HEAD and the renamed tree were tokenized (words + single non-word chars). The rename was strictly 1:1
word-for-word, so both token streams have equal length and zip into exactly 1,889 substitutions.
- Each substitution was classified with Roslyn at its position:
code identifier / doc-comment cref ....... 701 kept (real references to the renamed symbols)
string / interpolated-string literal ....... 19 reverted (NumPy-verbatim text, corpus kind, repr)
comment / XML doc text: `masked` ......... 1,087 reverted (the English word / NumPy's constant)
comment / XML doc text: masked_singleton ..... 3 reverted (NumPy's name)
comment / XML doc text: MaskedArray ......... 79 split by context: 29 name NumSharp's own class
(kept as NDMaskedArray / NDMaskedArrayTests);
50 name NumPy's class (reverted: numpy.ma.MaskedArray,
"NumPy's <c>MaskedArray.x</c>", MaskedArray._mask, …).
10 of those 50 were caught in manual review (the "NumPy's" sat on the previous doc line, which the
context heuristic did not see).
- The +2-char rename had pushed aligned trailing `//` comments right; the 7 affected lines are put back
on HEAD's comment column. Two articles fixed to house style ("an NDMaskedArray"; the codebase writes
"an NDArray" 196x vs "a NDArray" 2x).
Docs
- docs/MA_ORACLE_DESIGN.md, docs/MA_MODULE_AUDIT.md: current-state pointers now name
Ma/NDMaskedArray.cs / NDMaskedArrayTests, plus a dated rename note; the historical pass logs keep the
names that were in use at the time.
Verification (isolated detached worktree at b779d95 + exactly this changeset; the shared tree carries
an unrelated in-flight collections rename that is NOT part of this commit)
- Build: NumSharp.Tests + NumSharp.Tests.Oracle (Core, OpenBLAS interop) - 0 errors.
- NumSharp.Tests `NumSharp.Tests.Ma.*`: 91/91 on net10.0 and net8.0 (90 existing + 1 new).
- Oracle (net10.0), each in its own test host: MaUnary, MaBinary, MaReduce, MaScan, MaManip,
MaConstruct, MaSelect, MaSortSetops, MaExtras, MaSurface_IsCoveredOrExplicitlyClassified,
OracleCoverageStrengthTests (3), OracleApplicabilityTests (1) - all pass.
- No remaining code references to the old identifiers in any tracked .cs; no API snapshot, coverage
mapping (coverage/object_surfaces.py maps ma.MaskedArray with no CLR owner) or website page names them.
Pre-existing, NOT introduced here (reproduced 3/3 on a clean b779d95 worktree)
- Running several ma oracle tiers in ONE test host intermittently kills the host: it vanishes mid-test
(the crash point varies from after MaReduce to during OracleApplicabilityTests), with no managed
exception, no stderr, and no Application event-log entry. It is timing-sensitive: a --diag run of the
9 tiers completed with exit code 0. Every test passes in its own host.
Not included (other work present in the tree): the staged OrderedDict/ConcurrentOrderedDict ->
*Dictionary collections rename and ConcurrentPointingDict.cs deletion, benchmark BenchmarkBase.cs /
Program.cs / ConcurrentOrderedDictBenchmarks.cs edits, and the dirty refs/data submodule.
…entPointingDict removed, docs and crefs fixed
Lands the rename of NumSharp's ordered concurrent maps (an IDE rename, finished here) and deletes the experimental
ConcurrentPointingDict, with every doc, cref, test, benchmark and probe brought in line.
TYPE / FILE RENAMES (namespace NumSharp.Collections unless noted)
- ConcurrentOrderedDict<TKey,TValue> -> ConcurrentOrderedDictionary<TKey,TValue> (COD)
- ConcurrentOrderedCompactDict<TKey,TValue> -> ConcurrentOrderedCompactDictionary<TKey,TValue> (COCD)
- OrderedDict<TKey,TValue> -> OrderedDictionary<TKey,TValue> (lock-free value-once variant; the
proposal's reference implementation)
- ConcurrentOrderedDict.{TODO,COMPACT}.md -> ConcurrentOrderedDictionary.{TODO,COMPACT}.md (design ledgers)
- ConcurrentDictionary.NumSharp.cs -> ConcurrentDictionary.RefAccessors.cs (the vendored clone's one
NumSharp-only member). GetValueRefOrNullRef is now PUBLIC (was internal) - deliberate part of the rename; its
remarks now say the three caller obligations (lifetime inside one serialized section, single-field <=word writes
only, caller-side write serialization) bind EVERY caller, and that it bypasses the tear-free discipline the REST of
the public surface keeps.
- All 11 test classes/files now read ...Dictionary... (the IDE rename had covered only the two primary suites;
the Advanced/Concurrency/Memory/Randomized mirrors of both types and COCD's Specific suite still said "Dict").
- ConcurrentOrderedDictBenchmarks -> ConcurrentOrderedDictionaryBenchmarks. LOAD-BEARING, not cosmetic: the rename
changed menu option 15 to `--filter "*ConcurrentOrderedDictionary*"`, and BenchmarkDotNet's --filter globs
`Namespace.Class.Method` (categories are not matched), so with the class still named ...DictBenchmarks the menu
entry matched ZERO benchmarks. Verified: `--list flat` now lists all 43.
REMOVED
- ConcurrentPointingDict<TKey,TValue> (src/NumSharp.Core/Collections/Concurrent/ConcurrentPointingDict.cs,
1,030 lines). Experimental "hash -> int32 index -> value list" variant on the vendored ConcurrentDictionary; it had
no tests, no benchmark rows and no in-tree consumer, and its own /optimize run already recommended not shipping it.
It also carried two concurrency defects reproduced before removal (isolated worktree, x64): its validated read
checks keys[idx] BEFORE loading values[idx], so an in-place TryRemoveSwapBack let a reader of the removed key
return the moved entry's value (31,268 wrong-value hits / 6 s with the default comparer; COD/COCD 0) - the exact
chained-layout tear ConcurrentOrderedDictionary.COMPACT.md section 7 warns about - and its "lock-free" read spun
until an interior removal finished re-indexing (4.1-4.5 ms GC-free stall at N=1M vs ~0.35 ms for COD/COCD).
Nothing else referenced it; docs/MA_MODULE_AUDIT.md keeps its dated pass-11 mention (a historical log).
DOC / CREF FIXES
- The IDE rename double-applied inside crefs: 22 `<see cref="ConcurrentOrderedDictionaryionary{TKey,TValue}"/>` /
`ConcurrentOrderedCompactDictionaryionary` (src 4, benchmark 1, tests 17). All collapsed to the real names. The
csproj NoWarn suppresses the whole CS1574 family, so the compiler never flags a broken cref - docfx is the check.
- Crefs docfx cannot bind on net10.0, where the BCL signatures are `ref readonly`:
* `<see cref="Volatile.Read(ref int)"/>` x3 (OrderedDictionary class doc - rendered as unlinked plain text on the
public API page - plus COD/COCD private count docs) -> `<c>Volatile.Read</c>` (the paragraphs already link the
Volatile class; a `ref readonly` cref would break the net8.0 doc build instead).
* `<see cref="Unsafe.IsNullRef{T}(ref T)"/>` -> `(ref readonly T)` on GetValueRefOrNullRef - newly visible because
the method is now public, so docfx renders it.
- Ledgers (ConcurrentOrderedDictionary.TODO.md / .COMPACT.md): type, file, test-suite, benchmark-class and
menu-filter names; the seam's new file name and public signature.
- docs/proposals/ConcurrentOrderedDictionary.md: reference implementation is now
`NumSharp.Collections.OrderedDictionary<TKey, TValue>`; added a naming note because the rename makes three names
collide in one document - the reference impl shares its simple name with the BCL
`System.Collections.Generic.OrderedDictionary<TKey, TValue>` the proposal compares against, and NumSharp's own
node-based `NumSharp.Collections.ConcurrentOrderedDictionary<TKey, TValue>` shares the PROPOSED type's name while
NOT being the reference implementation. Section 8's comparator is now fully qualified.
- benchmark/collections/probes/*.cs (4 `dotnet run` reproduction scripts the ledgers cite; outside every build, so
nothing else would have caught them): type names updated - they no longer compiled against the renamed types.
NAMING CAVEAT (not changed, flagged): NumSharp.Collections.OrderedDictionary<TKey,TValue> now shares its simple name
with System.Collections.Generic.OrderedDictionary<TKey,TValue> (.NET 9+). A net10.0 consumer importing both
namespaces gets CS0104 and must alias/qualify (net8.0 has no generic BCL OrderedDictionary). Same situation as the
pre-existing public NumSharp.Collections.Concurrent.ConcurrentDictionary. Nothing in-tree is affected.
VERIFICATION (isolated `git worktree` = HEAD + exactly the paths in this commit; the shared tree holds a parallel
session's unrelated pending edits)
- Build: NumSharp.Core + NumSharp.Tests (net8.0 + net10.0) + NumSharp.Benchmark.CSharp - 0 errors. The benchmark
project needed that parallel session's still-uncommitted BenchmarkBase.cs fix to build at all: at HEAD it has
CS0160 (`catch (IncorrectTypeException)` after `catch (TypeError)`, and IncorrectTypeException : TypeError). That
fix is NOT part of this commit.
- Tests: `FullyQualifiedName~ConcurrentOrdered` -> 167/167 passed on net10.0 AND net8.0 (Release).
- BenchmarkDotNet: `--filter "*ConcurrentOrderedDictionary*" --list flat` -> 43 benchmarks (= every method).
- docfx metadata: 0 warnings for any collection file (repo total 423, all pre-existing elsewhere); the API
reference regenerates 11 collection pages under the new names.
- Probes: `dotnet build -c Release <probe>.cs` -> all 4 compile.
- Leftover sweep: no ConcurrentOrderedDict / ConcurrentOrderedCompactDict / OrderedDict / ConcurrentPointingDict /
ConcurrentDictionary.NumSharp / "Dictionaryionary" in tracked files outside the dated MA audit log and the
gitignored optimize-runs/.
NOT IN THIS COMMIT: the parallel session's pending work in the shared tree (Ma MaskedArray -> NDMaskedArray rename,
Default.* math / ArrayFormatter / Oracle fuzz / interop test edits, BenchmarkBase.cs, refs/data).
…n list, probe self-name, blame-ignore the rename A second, independent verification pass over 5635182 (the *Dict -> *Dictionary rename), running checks the first pass did not, plus the three small fixes it surfaced. VERIFIED, NO CHANGE NEEDED - Mirror-suite integrity. For each of the 5 COD/COCD test-suite pairs, substituting ConcurrentOrderedDictionary -> ConcurrentOrderedCompactDictionary in the COD file and diffing against its COCD mirror leaves exactly the documented residual deltas (the MIRROR header, cod-/cocd- thread names, the memory-doc pointer TODO -> COMPACT, the `== 0` vs `<= 72` presized-append bound, the AddRange comment variant) - and those residuals are IDENTICAL (name-normalized) before and after the rename, so the IDE rename kept both suites in lockstep. - Compiler-level cref validation, stronger than docfx (every member incl. private, BOTH TFMs): rebuilt NumSharp.Tests (pulls Core) in a clean HEAD worktree with `-p:NoWarn=NONE -p:GenerateDocumentationFile=true`. The override surfaces 782 CS1574 repo-wide, and ZERO XML-doc diagnostics (CS1574/1580/1581/1584/0419/1570-1573/1711/ 1712/1734) in any src or test collection file - their only warnings are 668 CS1591 (undocumented test members). - Cross-reference integrity: every file path / file name cited by the 26 files of 5635182 resolves to a tracked file (the only non-resolving tokens are upstream dotnet/runtime paths) - except one pre-existing stale self-reference, fixed below. - Broader leftover sweep (`Dict(?!ionary)` in scope; case-insensitive `ordered[ _-]?dict` repo-wide): only BCL Dictionary shorthand ("COD vs Dict", the DictC adapter), Python-dict semantics (AddRange_UpsertsWithPythonDictSemantics) and generic "ordered dict" prose remain - none name these types. - Rename-collateral audit (the Rider text-occurrence hazard): every changed string literal is the type naming itself (reentrancy messages, nameof, BDN category, menu text/filter); the rename touched nothing outside the collection scope (no Python OrderedDict exists in any tracked .py). FIXED - ConcurrentDictionary.cs header: its "deviations from the upstream source" list said the public key-API is otherwise unchanged, but the clone now carries a PUBLIC NumSharp-only member (GetValueRefOrNullRef in ConcurrentDictionary.RefAccessors.cs, made public in 5635182). Added the `partial` + public-accessor bullet and scoped the closing "unchanged" line to the upstream members. - benchmark/collections/probes/compact_ordered_dict_complexity.cs: its title comment named the file "complexity_probe.cs" (stale since it was committed under its real name). - .git-blame-ignore-revs: records 5635182, per the file's convention for mechanical renames (test-project rename, Npy* -> ND*, src -> refs relocation). Verified: a renamed test line (ConcurrentOrderedDictionaryConcurrencyTests.cs:110) blames to 5635182 without the file and back to its origin 1fcaea0 with it. The entry states the caveat measured while verifying: git also re-attributes the commit's few genuinely NEW doc lines to neighbouring older lines (the proposal's naming note blames to e3d5b77 with the file on); `git blame --no-ignore-revs-file` shows their true origin. Every earlier entry's SHA is an ancestor of origin/master (branches merge, not squash), and an unknown SHA in the file is harmless (probed: blame still exits 0). BUILD: NumSharp.Core (net8.0 + net10.0) and the edited probe (`dotnet build -c Release <probe>.cs`) compile in a clean HEAD worktree. LEFT AS IS, DELIBERATELY: ledger line widths (already 19/43 lines over 120 chars - tables and repo paths; the rename added one each), generic "ordered dict" prose, and the dated history entries in docs/MA_MODULE_AUDIT.md.
… bool-mask crash, leaks, host-pinned tests PR #631 (journey4) had never run on Linux/macOS CI. Run 35818966511 failed in five jobs, and CI stops at the first red step, so later steps hid more. Every job was replicated locally to find all of it: Windows directly, Linux in WSL Ubuntu 24.04 (every step of the `test` job plus the docs job's benchmark smoke), and ARM64 via DOTNET_EnableSSE41=0 (Vector128 stays accelerated, Sse41.IsSupported turns false, the same shape the kernel emitter sees on Apple silicon). Six root causes, two of them real Core bugs. 1. CORE BUG: np.argsort wrote past a zero-byte buffer (native heap corruption) AxisSort.ArgSortInto guarded only an empty AXIS (N == 0), not an empty ARRAY. For (0,3) along the last axis (or (2,0,3)/-1) there are zero lines, but the all-but-axis NDIter still makes one kernel call, and the radix line kernels ignore the per-call count: each call writes a full line of N int64 indices. Those 24 bytes landed in dst's ZERO-byte buffer. glibc never noticed. Windows' heap manager killed the test host with 0xC0000374 (STATUS_HEAP_CORRUPTION) in whatever test ran next, which aborted the whole Windows Oracle run inside IndexOracleTests.Index_Curated/Index_Dtype. Bisected 88 FuzzCorpusTests tiers -> MaSortSetops -> the single case ma.sort/empty_2d/bool.some/748 -> np.argsort (np.ma.sort = argsort of the filled keys + take_along_axis), then isolated in a standalone repro: every dtype, not just bool, at any zero-lines shape with a length >= 2 sort axis. This is the "OPEN pre-existing Ma-tier testhost crash" noted with 2ea7ba6. SortInPlace and AxisPartition (partition/argpartition) already carried the size guard; argsort was the one sibling without it. Fix: `if (N == 0 || src.size == 0) return;` in ArgSortInto, plus a defensive `ops[0].size == 0` early-out in DriveAllButAxis so a future caller cannot repeat it. Gate: Sorting/ArgsortZeroLinesTests (all 15 dtypes, covering every argsort kernel family, x 6 empty shapes/axes, axis=None, the ma.sort corpus case) with native-heap churn after each call. Verified to have teeth: the unfixed build crashes the test host; the fixed build passes 3/3 in 0.6 s. 2. CORE BUG: np.evaluate threw PlatformNotSupportedException on ARM64 (macOS) The fused NDExpr shell (vector v2) widens a STREAMED bool operand into a Vector<W> lane mask with DirectILKernelGenerator.EmitInlineMaskCreation, whose 2/4/8-byte path calls the x86-only Sse41/Avx2 ConvertToVector*Int*. The plan/shell gated bool operands on vector WIDTH only (FusedBoolLanesAvailable: 128 or 256 bits), and ARM64 has 128-bit AdvSimd Vector128 with every x86 ISA unsupported. The DynamicMethod compiles fine and throws only when it runs, so every tree mixing a bool array with a 2/4/8-byte lane (where(m1,a,b), a*m1, a+m1, where(m1&(a>b),a,-b)) threw on the macos-latest runner (30 cases across 7 NDEvaluateVectorTests). np.where / np.select / np.piecewise gate the same helper with their own needsX86 checks; only the NDExpr shell lacked one. Fix: new DirectILKernelGenerator.InlineMaskCreationSupported(simdBits, elementSize) (the helper's real ISA contract) + FusedBoolInputMasksAvailable, applied in both NDExprVectorPlan.TryPlan and FusedSimdViable. Such trees take the scalar shell on ARM64, which is the byte-for-byte oracle the vector path must reproduce, so results are unchanged. The gate is precise: a bool OUTPUT is packed portably and stays vectorized, and so does a bool 0-d PARAMETER (a constant Zero/AllBitsSet mask, no x86). TryPlan now receives isParam so it can tell the two apart. x86 behaviour is unchanged (SSE4.1/AVX2 present). Tests: VectorPlan_DecidesLaneAndScalarFallback now states the plan against the host capability, plus param/output cases and an InlineMaskCreationSupported truth table. Verified: with DOTNET_EnableSSE41=0 the 30 failures reproduce before the fix and all NDEvaluate tests pass after it. 3. Undisposed intermediates (Oracle Corpus_AllOps_LeaveNoUndisposedIntermediates; 38 families / 54 cases, all on this branch) - np.logspace / np.geomspace: CastSpacingResult dropped the float64 compute array after astype -> exactly 1 escaped buffer per non-float64 call. It now disposes the replaced array (ReferenceEquals-guarded). Ownership transfer is documented on the parameter. - np.piecewise (1-3 escaped per call): the bool-condition aliases, ToBoolCondition's astype temp, ComputeElseCondition's partial ORs and its logical_not result, the coerced 0-d scalar constants (composition path), and the fused path's scalar + astype pair were all dropped live. PiecewiseCore now owns and releases them in a finally, and also releases the half-written result on a throw. The bare-array and scalar-bool overloads release the condition arrays they build. Deliberately NOT [NDScoped]: an ambient scope would reclaim arrays a user callback allocates or keeps (the reason apply_along_axis stays unscoped too), so disposal is explicit and never touches caller-owned or user-callback arrays. asanyarray can hand back an EXISTING array for NDArray input, so the scalar-constant dispose is safe only because that case returns earlier. That is documented at the site. 4. Stale tests / benchmark gates behind the earlier failures - benchmark CS0160 (Deploy Docs build): IncorrectTypeException now derives from TypeError (3ad4360), so `catch (IncorrectTypeException)` after `catch (TypeError)` was unreachable. Reordered derived-before-base and kept it explicit so the benchmark survives a future re-parenting. - ModfBenchmarks (next failure in the same job once it compiled; benchmark smoke: "expected to be rejected, but the operation succeeded"): np.modf accepts every REAL dtype since 59f9932 (bool/int -> narrowest float per width, float16 kernel); measured 14/15 accept, only Complex is rejected, the same as NumPy 2.4.2 (per-dtype probe on both sides, result dtypes identical). The C# gate now rejects only Complex, and the NumPy twin (numpy_benchmark.py) times every non-complex dtype so the new C# rows have peers. The old comment explaining the float32/float64-only restriction was describing a C# rejection that no longer exists. - MisalignedRegistryTightnessTests B2/B3 asserted that two excuses STILL existed, but 074d4d0 (complex multiply now bit-exact) and 7fd70a3 (cumprod size<=1 widening) removed them on purpose. Inverted to *_NoLongerExcused so they pin the tight contract. 5. Tests that need an optional dependency CI does not provide - 25 Examples tests (Gist/Karpathy demos + polyfit companions) called OpenBlasEngine.Enable unconditionally, and CI's test job stages no native binary, so they threw DllNotFoundException on all 3 OSes. New Examples/ExampleBlasBackend follows the house LapackEigTests.RequireLapack pattern (Inconclusive where no CBLAS/LAPACK loads). KarpathyLstmTests binds in [TestInitialize] without failing: its 4 semantic/tolerance tests now run everywhere (verified passing with no backend), and only the 2 exact-value tests require the backend. With the binary staged, all 104 example tests still pass (0 skipped). - 9 interop tests import SciPy, which the interop job does not install ("No module named 'scipy'"). They now call the existing SkipUnless("scipy") as their FIRST statement: their mid-test failures had leaked export pins, which cascaded into NumpyNet_BootsOnTheSharedEngine / Lifetime_TheirWrapperIsTheOnlyHolder (LiveExports 9 != 0). 6. Host-dependent exact values (the tests, not the product) - np.emath arctanh(0.5) (glibc) and arccos(0.5)/arccos(0.3) (Apple libm), plus np.sinc(0.25) (both), differ from the win-amd64 ucrtbase bits by 1 ULP. The matching corpus tiers are already host-pinned (RunHostLibmCorpus: Inconclusive off-Windows). New Utilities/HostLibm gives unit tests the same pin but keeps checking: bit-exact on Windows, <= 4 ULP elsewhere, with NaN-ness, infinities and the sign bit (incl. signed zero) always strict. - 4 interop live-parity tests (NES final weights/trace, forecast mpe, ranking dcg) measured 1 ULP off live NumPy on macos-latest arm64 only. Their authors deliberately refuse to widen per-host budgets ("Other hosts deliberately expose a disagreement instead of silently widening the gate"), so they use the existing SkipByteExactOnArm64 x64 pin (the NEON 128-bit reduction-width class) with the measured drift in their docs. Verification (all local; macOS arm64 itself is only reachable through CI). Other sessions landed 2ea7ba6, 5635182 and 4499d3e while this was in progress (none touch these files), so each figure names the base it ran on: - Windows (CI filter): NumSharp.Tests net8.0 16022/0 (b779d95) and net10.0 16024/0 (2ea7ba6); Oracle net8.0/net10.0 182/0 without OpenBLAS (2ea7ba6; was: test-host crash) and 206/206 with it staged (4499d3e); Interop net8.0 733/733 with SciPy and 724 passed / 9 skipped / 0 failed with SciPy hidden, the CI path (2ea7ba6). At 4499d3e all four affected projects build and the 220 targeted tests pass. - Linux (WSL Ubuntu 24.04, CI test-job replica, every step run even after a failure, b779d95): NumSharp.Tests net8.0 15958/0, net10.0 15959/0; Oracle net8/net10 167/0; Analyzer net8/net10 321/0 (baseline before the fix: 28 + 28 + 3 + 3 failures). The docs job's benchmark body smoke runs all 499 bodies OK on net8.0, and the unary twin join check is OK. - ARM64 simulation (DOTNET_EnableSSE41=0): NDEvaluate 30 failures -> 0. Not changed here (noted for follow-up): benchmark/scenarios.html is a generated artifact that already disagrees with its own (non-CI) test_scenarios_audit.py totals (python 1296 vs 1699); regenerate it with benchmark/scripts/generate_scenarios_html.py. NumpyNet lifetime tests assert ABSOLUTE LiveExports counts, so any interop test that leaks on failure cascades into them; comparing deltas would localise such failures.
…ntOrderedDictionary ([OpenBugs]), pin the compact sibling's comparer exception safety Two new test classes. Every [OpenBugs] test asserts the CORRECT outcome promised by the type's own docs and by docs/proposals/ConcurrentOrderedDictionary.md, and fails today (CI excludes the category until each fix lands; remove the attribute together with the fix). Five passing controls pin the behaviour the compact sibling and OrderedDictionary already get right. Tests only - no product code changed. == test/NumSharp.Tests/Collections/OrderedDictionaryOpenBugsTests.cs (6 tests, all [OpenBugs]) == 1. Enumerate_WhileAWriterRegrowsTheTable_NeverThrowsAndNeverYieldsAnUnwrittenSlot GetEnumerator() = new Enumerator(_t._values, Volatile.Read(ref _t._count)) reads the volatile generation field TWICE: a growth published between the reads pairs the old (shorter) values array with the new count -> IndexOutOfRangeException or phantom default values. Violates "snapshot enumeration never throws / consistent prefix" (proposal 7.1). Fix: read `_t` once into a local. Storm (writer Clear+refill 64 keys, readers foreach); fails in ~3 ms; ~160K IOOR/s measured on the shipped code. 2. WideValueReplaceThenAppend_ReaderHoldingTheOlderGeneration_NeverReadsAnUnwrittenValue (DETERMINISTIC) A replace of a non-atomically-writable TValue (decimal/Guid/wide struct/long on 32-bit) clones ONLY the values array; the new generation SHARES _index/_keys with the old one, so the next in-place append writes a word + key into those shared arrays and a reader still holding the old generation resolves the new key and returns its own never-written values[slot] = default for a present key. Exactly the aliasing rule ConcurrentOrderedDictionary.COMPACT.md 4.7 forbids (the compact sibling copies the whole generation). Fix: full generation copy-on-write. Made deterministic with a comparer hook: TryGetValue captures its generation and only THEN calls GetHashCode, so the hook runs the writer (wide replace + append, on another thread) inside that window. Result today: found=True, value=0 for a key whose value is 150.25 (~47K/s with a trivial comparer). 3. LockFreeReplace_RacingCopyingResizes_NeverLosesACompletedWrite The lock-free SetByKey does a plain value store then volatile-reads _resizing/_t; a resize volatile-writes _resizing = true then copies the values array. That is the store-buffering (Dekker) litmus: release/acquire cannot forbid it on x86-TSO or ARM64, so the resize can copy the OLD value while the replacer sees the flag clear -> the completed write is lost when the resize publishes. Violates "no update is ever lost" (proposal 7.1). Fix: Interlocked.MemoryBarrier() after the store AND after every `_resizing = true` - verified in a scratch A/B: 3.26% lost -> 0 (0.84% -> 0 with one slot per cache line), +3.3 ns per replace (4.9 -> 8.2 ns), still ~5x a locked replace. The test uses one WRITER PER KEY with read-your-own-write re-reads: a storm with several writers per key (the original ~280M-op gun) cannot see this - last-writer-wins hides the lost intermediate write. 4. NullKey_IsRejectedOnEveryKeyEntryPoint_LikeTheSiblingsAndTheBcl No null check anywhere; EqualityComparer<string>.Default.GetHashCode(null) is 0, so a null key is stored, found and enumerated. All 11 key entry points accept it (COD/COCD/Dictionary/ConcurrentDictionary throw ArgumentNullException). Fix: the siblings' NullCheck (with the typeof(TKey).IsValueType Debug-boxing guard). 5. HugeCapacity_IsRefusedWithArgumentOutOfRange_InsteadOfSpinningForever `while (len < cap * 100 / 70 + 1) len <<= 1;` on an int: from capacity ~752M the shift wraps to int.MinValue then 0 and `0 << 1 == 0` spins forever (AppendGrow's `newLen <<= 1` has the same overflow). The compact sibling throws ArgumentOutOfRangeException above ~715M. Runs the ctor on a lowest-priority background thread with a 10 s join (a spinning managed thread cannot be stopped; no memory is allocated before the spin). 6. ComparerThrowDuringAResize_DoesNotLeaveTheReplacePermanentlyLocked Every generation-replacing resize sets _resizing = true, then allocates and rebuilds the index through the comparer, then clears the flag - with no try/finally. A comparer/OOM throw in that window leaves the collection consistent (nothing published) but _resizing stuck true forever, so every later "lock-free" replace fails its re-check and silently takes the write lock. Observable: SetByKey on an existing key no longer completes while another thread holds the lock (an AddRange whose source blocks inside the locked foreach). A healthy-instance baseline check in the same test proves the harness. Fix: clear the flag in a finally. == test/NumSharp.Tests/Collections/OrderedCollectionsComparerExceptionSafetyTests.cs == ConcurrentOrderedDictionary mutates its key map FIRST and then re-indexes the shifted/moved keys THROUGH the map (one comparer call per key) BEFORE publishing the new list-path store, in: the order-preserving interior removal (RemoveCoreUnderLock), both swap-back branches (in place for word-sized types, copying for wide ones) and RemoveWhere (whose finally re-runs the throwing re-index, so its "publish the consistent prefix" never happens). A comparer that throws there - including the collection's OWN LockRecursionException raised by the documented "evil comparer write-back" defence, which the docs say is "refused ... instead of being allowed to corrupt" - aborts the operation half-applied. 4 x [OpenBugs], deterministic, single-threaded: - interior TryRemove(3), poison key 7: key 3 still enumerates but ContainsKey(3) is false; IndexOf(4..6) wrong - in-place TryRemoveSwapBack(3), poison 9 (the moved key): key 9 enumerates TWICE - copying TryRemoveSwapBack(3) on <int,decimal>: key 3 enumerates but no longer resolves - RemoveWhere(even), poison 7: keys 0/2/4/6 enumerate but no longer resolve (15 disagreements) Fix: resolve every node ref to be re-indexed BEFORE the first map mutation, then mutate throw-free (the same "allocate before touching the map" discipline the type already applies to allocations). The existing Reentrancy_EvilComparerWritingDuringALockedLookup test only covers the refusal on the FIRST lookup (before any mutation), which is why this was not caught. 5 passing controls (run in CI, regression guards): ConcurrentOrderedCompactDictionary never consults the comparer after its probe (tags travel with the index words) and probes BOTH swap-back keys before mutating (in-place and copying branches), so the same attacks either never fire or change nothing; OrderedDictionary rebuilds a private generation and publishes last, so its published state survives the throw (its stuck-flag defect is test 6 above). The audit is generic over the value type: count/keys/values agree, no key enumerates twice, IndexOf/TryGetValue/ ContainsKey agree with every enumerated position, the value law holds, and no key resolves without enumerating. == Notes == - Both files put `using NumSharp.Collections;` INSIDE the namespace block: on net10.0 a file-level using next to `using System.Collections.Generic;` makes `OrderedDictionary<,>` ambiguous with the BCL's .NET 9 generic OrderedDictionary (CS0104 - verified with a compile check). The using attached to the namespace declaration is consulted first. Any net9+ consumer with ImplicitUsings that imports NumSharp.Collections and names OrderedDictionary<,> hits the same error. - Storm tests stop at the first anomaly (a broken build fails in milliseconds; a fixed one runs its 2-3 s budget). They are statistical by nature: on a 1-2 core box the races may not reproduce within the budget. Verification (isolated worktree, Release): - HEAD b7d9093 + these two files: net10.0 -> 10 failed / 5 passed (exactly the [OpenBugs] set fails). - At 5635182 + these files: net10.0 and net8.0 each 10 failed / 5 passed, every failure message checked for the right reason; CI filter (FullyQualifiedName~NumSharp.Tests.Collections & TestCategory!=OpenBugs) -> 172/172 on net10.0 and net8.0 (167 existing + 5 new controls). - A single-threaded differential fuzz of all three types against a List-of-pairs oracle (765K ops over int/string keys, constant-hash / mod-8 / case-insensitive comparers, int/long/decimal/Guid/string values, growth, tail pops, interior removals, AddRange, Clear, every read surface) found NO functional divergence: the defects are concurrency, exception-safety and edge-contract ones. Related findings NOT covered by these tests (for reference): - ConcurrentOrderedDictionary.ReplaceExistingUnderLock, non-atomic TValue: the key-map node is swapped BEFORE the values-array Clone() allocation, so an OutOfMemoryException there leaves the key path holding a value the list path never publishes - contrary to the type's own strand-proofing rule (AddRange's wide-value branch does it in the right order). Inspection only (no deterministic in-process OOM). - docs/proposals/ConcurrentOrderedDictionary.md still says the reference implementation is "concurrency-verified" and "complete" and that 7.1 holds (no lost updates / no garbage reads / never-throwing enumeration); tests 1-3 contradict it, and the proposed surface (KeyValuePair indexer, IDictionary family, TryUpdate/AddOrUpdate) is a superset of what OrderedDictionary implements. - AddRange semantics differ inside the family: OrderedDictionary = add-if-absent (first value wins), ConcurrentOrderedDictionary / ConcurrentOrderedCompactDictionary = upsert (last value wins). Both documented. - OrderedDictionary accepts a negative capacity (documented: <= 0 -> default); both siblings and the BCL throw ArgumentOutOfRangeException. - OrderedDictionary.TryRemove is O(n) even for the LAST entry (no O(1) tail path, so a pop-back drain is quadratic) and both TryRemove and growth re-hash every key through the comparer (no stored hashes/tags) - the compact sibling pops the tail in O(1) and rebuilds from stored tags. - GetValueRefOrNullRef on the public vendored ConcurrentDictionary is now public (documented as intentional in 4499d3e); it bypasses the class's "all public members are thread-safe" guarantee by design.
…ed-collection bugs (+1 by inspection) docs/ORDERED_COLLECTIONS_FIX_HANDOVER.md - a self-contained brief for a fresh session: revalidate each problem first (the fe756f9 [OpenBugs] tests; expected 10 failed / 5 passed on net8+net10, CI filter 172/172), and only if confirmed apply the fix and flip the test to a CI gate in the same commit. Covers, with exact file:line sites at fe756f9, root cause, fix design and acceptance gates: - OrderedDictionary B1 enumerator double-read of the generation field; B2 wide-value replace sharing index/keys across generations (fix = whole-generation copy, COMPACT.md 4.7); B3 lock-free replace lost updates (store-buffering; fix = full fence on BOTH sides, measured 3.26% -> 0 at +3.3 ns/replace, with a proof sketch); B4 null keys; B5 int overflow in the capacity/growth loops (mirror COCD IndexLengthFor); B6 _resizing stuck after a throw (try/finally). - ConcurrentOrderedDictionary B7-B10: comparer called during the post-map re-index of interior removal, both swap-back branches and RemoveWhere; recommended fix = node-handle seam in the vendored clone's NumSharp partial (FindNode / NodeValue / TryRemoveNode by reference) so every comparer call happens before the first mutation; weaker pre-pass fallback documented. B11 (inspection): wide-value replace swaps the map node before the Clone() allocation. - Ground rules for this multi-session repo (isolated worktree, commit --only, no attribution lines, full XML docs, docfx as the only cref check, CS0104-safe test usings), acceptance gates (both TFMs, 20x storm loop, 182 green, measured costs, proposal 7.1 update) and the user decisions NOT to take unilaterally (OrderedDictionary name vs the BCL type, AddRange semantics split, negative capacity, O(n) tail removal, public GetValueRefOrNullRef).
…y_gauss, host-libm sin(2^32) out_where cell After b7d9093, run 35824898562 was green on Windows and Ubuntu (test + interop + every other job) and Deploy Docs passed. macOS arm64 had two remaining reds, both test-harness classes, no product change. (The 4 Karpathy interop failures were ALREADY in the first run 35818966495; the previous pass missed them.) 1. interop-test (macos-latest): 4 Karpathy seeded-initialization tests PackedInitialization_ExactSeededNumpy, OriginalPinnedClassAndBothOriginalChecks_RunThroughPythonNet, Full200By6400_SeededInitializationForwardAndBackward_Exact, ShortPong_AllStepsAndElevenEpisodes_ByteExactAgainstPinnedOriginal. Root cause: NumPy's OWN seeded Gaussian stream differs on arm64. legacy_gauss (numpy/random/src/legacy/legacy-distributions.c) computes `r2 = x1*x1 + x2*x2`. NumPy's `min` CPU baseline is ASIMD on aarch64, where fused multiply-add is base ISA, and clang's default contraction turns that line into fma(x1, x1, x2*x2). The x86-64 baseline is X86_V2 (no FMA3), so every x86-64 wheel rounds both products, and so does NumSharp (RyuJIT never contracts). log(r2) amplifies that 1 ULP near the unit circle: ~14% of draws differ, by 1 to ~50,000 ULP. Evidence: a C# replica that fuses exactly as fma(x1,x1,x2*x2) reproduces the FIRST mismatching element macos-latest reported for each seeded test: seed 7 -> element 24 (LSTM, row 0 is overwritten), seed 23 -> 10 (short Pong), seed 3 -> 44 (full Pong). The other operand order gives 20/22/0. For all these seeds the accept/reject decisions and the number of MT19937 doubles consumed are identical (316, 624, 65276, 1629594), so the stream position and every later uniform() draw still agree. NumSharp keeps the literal (x86-64) stream on every architecture on purpose. It is what every x64 NumPy returns and keeps seeded NumSharp output portable. The win-amd64-authored random_parity_host corpus and the NumSharp.Tests random unit tests pin it; making NumSharp fuse on arm64 would break those, and would still leave gamma/beta/... differing, since clang contracts other lines there too. Fix (test harness, zero coverage loss): InteropTestBase.NumPyLegacyGaussianIsLiteral (true for an X64/X86 PROCESS, since the in-process CPython loads the wheel built for it) and DefineLegacyRandn(), which defines Python `legacy_randn(rs, *shape)`: - where NumPy is literal (x64): exactly rs.randn(*shape), so x64 keeps comparing against NumPy's own output; - elsewhere: re-evaluates legacy_gauss literally over rs's OWN live MT19937 doubles (random_sample = the same next_double), with separately rounded NumPy-ufunc products and sum, and math.log from the platform libm (the one .NET's Math.Log calls). It honours and leaves rs's cached Gaussian and advances rs by exactly the doubles the literal sampler consumes, so later draws on rs still come from NumPy's generator. It never patches NumPy, per KarpathyOriginalSource's stated rule. The pinned original LSTM's own init keeps its direct comparison on x64. On a fusing host that check is split: the original's deterministic bias row must match exactly, and the matrix is compared against the same formula over legacy_randn. The short Pong driver's pg_initialize draws through legacy_randn, so all 44 ticks / 11 episodes / RMSprop update stay byte-exact on arm64 too. The same root cause behind two tests pinned last pass: Nes_Complete300IterationOptimization and Nes_EveryStateAndRewardAcrossOriginal300Updates were marked SkipByteExactOnArm64 as a "1-ULP NEON reduction-width drift". That diagnosis was WRONG. NumSharp's NES trajectory is bit-identical with AVX2, with 128-bit vectors only (DOTNET_EnableAVX2=0) and with hardware intrinsics off; its final weights[1] is 4F2FA1136C6CBB3F, the exact C# value macos-latest printed, while macOS NumPy printed 502F... A fused r2 moves a typical draw ~1 ULP, which the 0.0002 step rounds away almost every iteration. That is why the trace first diverged only at iteration 135. Both tests now draw through legacy_randn and the arm64 skip is REMOVED, restoring their arm64 coverage. Nes_SeededNoiseAndOneStep and NesOriginalObjective_TwelveCompleteUpdates make the same seeded-stream claim and passed on macOS only by luck (8 unaffected draws; 12 steps end before the first visible effect), so they use legacy_randn too. The rule is now uniform: a test that claims NumSharp's seeded Gaussians equal NumPy's draws its expectation through legacy_randn. Tests that feed NumPy-drawn (or NumSharp-exported) inputs are untouched. 2. test (macos-latest) Oracle: OutWhere, 12/6631 comparisons (6 cases x 2 slots) out_unary/sin/2x3x4/float32 at input 2^32: expected 0xbeec8981 (win-amd64 ucrtbase), actual 0xbeec8982 (Apple libm), 1 ULP. (The report prints little-endian BYTES, 81 89 ec be, which read like a factor-4 error; they are not.) |x| = 2^32 is past NumPy's sin Cody-Waite limit (117435.992). There the simd_sincos_f32 port calls MathF.Sin, exactly as NumPy's own kernel calls npy_sinf, so on every host both libraries return that host's libm answer, and the corpus holds ucrtbase's. The whole-tier libm pin (RunHostLibmCorpus) would throw away 6,631 portable out=/where= cells for one element, and the generic "unary ~ULP" excuse cannot reach out_unary (two operands) and deliberately excludes the float32 sin/cos port. Fix: MisalignedRegistry branch (H1) + internal predicate IsHostLibmSinCosHandoff and the IsLibmReferenceHost flag (the same Windows check RunHostLibmCorpus uses). It is narrow by construction: - never on the reference host, so Windows stays strict; - only float32 sin/cos, the plain op or out_unary params.ufunc; - only a float32 input; - every differing element must BE this host's MathF.Sin/Cos answer for a past-limit input in the case's input buffer, and within 2 ULP of the reference. An in-range lane (the bit-exact port) therefore cannot qualify, and a port regression still fails on every host. The host is a parameter so both sides are pinned anywhere. New MisalignedRegistryTightnessTests: - H1_PastLimitHostLibmAnswer_OneUlp_ExcusedOnlyOffTheReferenceHost: sin+cos x plain+out tier; excused off-host, strict on-host, and Classify follows this host. - H1_InRangeLane_GrossMiss_OtherOpOrDtype_NotExcused: an in-range 1-ULP port miss, with and without a past-limit input in the case; a 64-ULP miss; a value within 1 ULP that is NOT the host libm answer; tan; float64. Verification: - Python reference: rs.randn == legacy_randn with the literal path forced on win-amd64 NumPy 2.4.2. 126 draws in 63 seeded sequences over 9 seeds (odd counts, pre-cached Gaussians, empty, 1.28M-element), values + full state tuple + continuation byte-identical. - Interop (Windows, net8.0, OpenBLAS staged, SciPy present): the 32 tests in the 5 touched classes pass on the x64 path, with only the Python flag forced literal-off, and with NumPyLegacyGaussianIsLiteral itself forced false (the arm64 branch, confirmed by reading the property back from the built DLL by reflection). Full suite 733/733 on the final build. - Oracle (Windows): net8.0 and net10.0 184 passed / 24 skipped / 0 failed (208 incl. the 2 new H1 tests). - Oracle (Linux, WSL Ubuntu x64, clone at 035f24f + this diff, where H1 is LIVE): net8.0 and net10.0 169 passed / 39 skipped / 0 failed; both H1 tests pass (Classify != null off Windows). - fe756f9 (another session's tests-only commit that rides along in this push): Collections tests with CI's filter, 175/175 on net8.0 and net10.0. macOS arm64 itself is only reachable through CI; this push is its check. Still pinned to x64, cause NOT the Gaussian stream (no randn involved), left as is: AllThirtyForecastScores (mpe) and RankingDiscounts (method-0 dcg), each 1 ULP on macos-latest.
…u leg (SVML runners broke byte-exact tests) Run 35828348090 (6ee94ae): interop-test macOS and Windows went GREEN (macOS 632 passed / 101 skipped / 0 failed on net8.0 and net10.0; before: 626 / 103 / 4 failed; the two NES tests un-skipped by 6ee94ae now run and pass on arm64). But interop-test (ubuntu-latest) failed 18 byte-exact tests that NO commit touched: GoogLeNet_LocalResponseNormalization_ByteExact, GoogLeNet_BothSourceLrnStageShapes x2, StableWalk_All200FullSizeLatentFrames, WalkOriginalSlerp (False), SourceSizedForwardAllIntermediatesAndFourGradients x3, OriginalPinnedClassAndBothOriginalChecks, SixUpdateRecurrentTrainingShortRun, CompleteTransformer (16,4,1), AdamFourUpdates, ShortTrainingLedger, ShortPong_AllStepsAndElevenEpisodes, FullForwardLossAndBptt x2, BoundedTraining_AllFortyLosses, FourSourceSizedUpdates. All of them passed on the ubuntu leg of the previous run (35824898562, b7d9093). Root cause: runner hardware changes the LIVE NumPy oracle, not NumSharp. The job prints numpy.show_config(). The failing runner's "SIMD Extensions" found X86_V3 + X86_V4 + AVX512_ICL; the passing runner found only X86_V3. NumPy's Linux x86-64 wheel links Intel SVML (numpy/_core/meson.build: use_svml = linux AND x86_64 AND X86_V4 in dispatch). On AVX512_SKX hardware its float64 exp/log dispatch to __svml_exp8_ha/__svml_log8_ha (loops_exponent_log.dispatch.c.src), and ~20 more f32/f64 ufuncs go to SVML (loops_umath_fp.dispatch.c.src: power, arccos, tan, cbrt, expm1, log1p, ...). The AVX512F float32 exp/log instantiations also replace the AVX2-FMA3 ones NumSharp ports bit-exactly. On an AVX2 runner, and in the win-amd64 reference wheel, the same ufuncs call the platform libm, which NumSharp's .NET Math also calls. That is why the failures are LSTM/RNN/transformer/Pong training (exp/log/tanh chains), LRN (power) and slerp (arccos): every one is a transcendental that SVML answers differently. The Windows wheel is immune by construction: meson_cpu disables X86_V4 ("Considered broken by Highway on MSVC") and AVX512_ICL under MSVC, so it dispatches nothing past X86_V3. The macOS arm64 wheel has none of these features. So only the ubuntu leg can draw an AVX-512 runner and change answers, and whether a run is red depended on which runner GitHub assigned. Fix: interop-test job env NPY_DISABLE_CPU_FEATURES: ${{ matrix.os == 'ubuntu-latest' && 'X86_V4 AVX512_ICL AVX512_SPR' || '' }} NumPy's runtime override (npy_cpu_features.c npy__cpu_check_env): a dispatched feature the CPU HAS is turned off, and one it LACKS is silently accepted for "disable", so AVX2 runners are unaffected. An empty value counts as unset. Both Windows and macOS get the empty value; on Windows the names would only earn an ImportWarning ("not part of the dispatched optimizations (X86_V3)", measured). The pin sits at job level, so "Install numpy" prints show_config() WITH it applied: on an AVX-512 runner X86_V4/AVX512_ICL now appear under "not found", making the pin visible in every log. Verified locally (no AVX-512 hardware here, so the AVX-512 branch itself rests on NumPy's source): - the workflow YAML parses; interop-test env carries both variables; - Linux x86-64 numpy 2.4.2 (WSL, AVX2) with the pin under `python -W error`: imports cleanly, __cpu_features__ X86_V3 True / X86_V4 False / AVX512_ICL False, and np.exp bytes identical with and without the pin (a no-op on AVX2, as intended); - Windows numpy 2.4.2: the names would only trigger the ImportWarning above, and the import succeeds. NumSharp's own AVX-512 code paths are deliberately NOT pinned here. The Oracle test job replays the committed win-amd64 corpus on whatever runner it draws, so it keeps covering them, and nothing in this run implicates them. The nightly fuzz-soak generates fresh expectations with live NumPy on ubuntu runners and can be affected the same way; it is not part of PR CI and is left for a follow-up.
…spatch Run 35828946169 (33a4d2c) is the first FULLY GREEN Build and Release on PR #631: all 16 jobs, the release-only jobs skipped on a PR as designed, and Deploy Docs is green too. - interop-test ubuntu: 638 passed / 95 skipped / 0 failed, net8.0 and net10.0, with numpy.show_config() reporting found [X86_V3] and not found [X86_V4, AVX512_ICL, AVX512_SPR]. - The 6ee94ae run before it: macOS test job green (NumSharp.Tests 15964/15965, Oracle 168 + 40 host-pinned skips, Analyzer 321, net8.0 and net10.0). Its Oracle TRX records exactly "12x float32 sin/cos past NumPy's Cody-Waite limit ... [host-libm pin]" in both frameworks. macOS interop: 632 passed / 101 skipped / 0 failed. What that green run could NOT show is whether its ubuntu runner even had AVX-512. show_config() lists NumPy's EFFECTIVE dispatch, which the NPY_DISABLE_CPU_FEATURES pin (33a4d2c) makes read "X86_V3 only" on every runner, and nothing else in the job names the hardware. So the pin had not been observed working on the hardware it exists for. Hosted ubuntu-latest runners are a mix: run 35824898562 drew an AVX2 box, 35828348090 an AVX-512 (ICL) one, and whether the 18 byte-exact live-NumPy tests went red depended on that. This adds one line to "Report Python host" (all three OSes): cpu AVX512F=<hw> AVX2=<hw> ASIMD=<hw> | numpy dispatch X86_V4=<eff> AVX512_ICL=<eff> read from numpy._core._multiarray_umath.__cpu_features__, the same private module numpy.show_config() reads. NumPy's runtime override clears only the named dispatch GROUPS (npy__cpu_check_env sets npy__cpu_have[feature_id] for each listed dispatch name), so the raw per-instruction flags survive it. "AVX512F=True ... X86_V4=False" is therefore the pin at work, and a future byte-exact red on this leg can be attributed to hardware (or ruled out) from the log alone. f.get() tolerates a missing key, and on the macOS arm64 wheel the x86 flags simply read False, with ASIMD=True. Verified: extracted the step body from the YAML (yaml.safe_load) and ran it under Git Bash on Windows and under WSL Ubuntu python3, with and without the pin. Every run printed the line; this AVX2 host reads AVX512F=False AVX2=True, dispatch X86_V4=False AVX512_ICL=False.
…G1-G13) + 4 upstream ORT issues
A full review of NumSharp.Interop.OnnxRuntime against the complete ONNX Runtime surface: the ORT C# API
and the native C API in refs/onnxruntime (a full checkout at v1.29.0, tags through v1.30.0), with NuGet
current = Microsoft.ML.OnnxRuntime.Managed 1.30.0. Written to docs/plans/onnxruntime-gap-review.md as the
companion of docs/plans/onnxruntime.md (design, decisions not re-litigated) and the business review.
Method: every finding was EXECUTED, not inferred. The existing suite is 159/159 on ORT 1.16.0 (floor) and
on 1.30.0. Three scratch dotnet-run probes (real InferenceSessions, 10 purpose-built .onnx models + the
committed ones) ran on 1.29.0 and 1.16.0 with identical outcomes except the version-dependent cells noted
in the doc. The NumSharp ownership analyzer was run over the package sources (0 diagnostics) and over a
consumer file.
High:
- G1 Fortran-order AsDenseTensor<T> builds a reverseStride DenseTensor that ORT's only DenseTensor->OrtValue
projection (PinAsTensor, OrtValue.shared.cs:1567) refuses ("Tensor of reverseStride is not supported"),
and the documented OrtTensor<T>.Memory -> CreateTensorValueFromMemory route feeds it TRANSPOSED with no
error (identity output row0 [0,4,8,1] for source [0,1,2,3]). The existing test never fed it to ORT.
- G2 ORT's managed GetTensorBufferRawData narrows the byte length with (int) (OrtValue.shared.cs:712-716):
AsNDArray/ToNDArray throw ArgumentOutOfRangeException for 2-4 GB tensors and silently wrap at >= 4 GB
(lease memory pressure 42x too small at 4.4 GB). Export (AsOrtValue) is fine; the README/website/
TooLargeForSpanMessage advice "share it zero-copy instead" is wrong for import.
- G3 OrtIoBinding.BindInput/Output (native IOBinding copies the OrtValue), SessionOptions.AddInitializer
(session reads NumSharp memory for its lifetime) and OrtValue.CreateSequence (TensorSeq::Add shares data)
keep dereferencing NumSharp memory after the OrtTensor handle and its ARC pin are gone (LiveExports == 0
while the binding still reads the buffer) - a use-after-free once the NDArray dies. Proposes one "keeper"
concept: NDArrayIoBinding, AddInitializer(name, NDArray), ToOrtSequence(NDArray[]).
Medium:
- G4 Run(dict) with default outputNames throws on any non-tensor output (skl2onnx classifiers: label +
ZipMap); a string label output is unreadable through the tier at all.
- G5 ToMap/ToMaps refuse string-keyed maps (every string-labelled ZipMap) though ORT reads them.
- G6 the Run tier accepts only NDArray inputs: no string/sequence/map/raw-OrtValue, so a model with one
string + one float input cannot go through it.
- G7 PrepareInput refuses optional(tensor) inputs (with a wrong CreateSequence/CreateMap hint) and
overridable initializers - both of which ORT accepts (inference_session.cc:3016, LookupInputMetadata).
Low / docs:
- G8 no RunAsync tier; ORT's own RunAsync pins its arrays until completion only from 1.30.0 (#32015).
- G9 ToNDArrays/ToMap(s) copy twice (GetValue already returns a fresh ORT copy) - 1.26x on 16 MB.
- G10 IsCpuAccessible refuses QnnHtpShared (CPU host-accessible, MemTypeDefault) and CpuAligned4K on newer
ORT; plain Run outputs are always CPU, device tensors only arise through IoBinding.
- G11 the multi-output Run result is a non-disposable IReadOnlyDictionary the analyzer cannot see.
- G12 BFloat16 bridgeable today by widen/RNE-narrow without a Core dtype.
- G13 CI "current" pin 1.29.0 (NuGet 1.30.0, suite green), "148 tests" (now 159), the 2 GB and F-order
claims, and the csproj FileLoadException rationale (ORT 1.16.0's DLL has AssemblyVersion 0.0.0.0).
Upstream ORT (documented, not fixable here): a None optional graph output AVs inside ORT's own C#
session.Run (onnxruntime_typeinfo.cc:169-176 null deref); the (int) truncation above; float8/int4 graph
I/O unreachable from C# (metadata lookup throws "Unregistered TensorElementType ... DataTypeMax"); no
sparse-tensor API in C#.
Also records what was verified NOT to be a gap and a recommended implementation order (G1 first).
Documentation only - no code, test or CI change in this commit.
…ce, all corpus families, catalogue + completeness gate (NOT merged: gate red until np.ma/histogram leaks are fixed) Side branch off journey4 (256224d). This is work in progress, parked here so the shared PR branch stays green: the extended gate intentionally fails until the leaks it finds are fixed. Full state, audit numbers and ordered next steps are in docs/plans/leak-audit-completion.md. WHY The leak oracle (UndisposedIntermediateTests + ScopeAudit, [FuzzMatrix]+[ScopeAudit]) was green, but its "every op gated at zero" held only for ops with ordinary corpus rows. Cross-referencing the ApiInventory surface (coverage/NumSharp.Tools.ApiInventory: 8 [ModuleName] modules, 847 methods + 91 properties + 42 fields) against the op keys the sweep actually MEASURED found 265 of 961 module members never leak-measured: ndarray 94, np 75, np.ma 62, np.linalg 21, np.random 13. Measured with the gate's own protocol: - np.ma (the ma_* tiers were excluded) leaks in 36,696 of 68,860 cases: 84 of 149 ops, up to 29 buffers per call (std 29, median 28, var 22, average 12, anom 11, mean 9). The replay takes 12m47s while leaking, over CI's 10-minute Oracle step. - The histogram family has no corpus rows and leaks 4-13 buffers per call; apply_along_axis 1. - 2,441 error paths (expects_throw), now measured: 0 leaks. - 276 LAPACK cases always threw (backend suppressed), so they were never measured. - NDW012 already flags 214 sites statically (176 in NDMaskedArray.cs, 34 in histogram). WHAT (test/NumSharp.Tests.Oracle/Fuzz/) - ScopeAudit: MeasureConfirmedTraffic / MeasureConfirmed (the screen, settle, confirm protocol, shared by every replay). - UndisposedIntermediateTests (now partial): - SharedSweep, a Lazy with one sweep per process, shared with the completeness gate. - SweepResult/SweepAccumulator gain per-family counts, MeasuredByOp (+ rnd:/grnd: coverage keys), ErrorMeasuredByOp and ThrewByOp. - Error paths are measured, not skipped. Per-family non-vacuity floors; the index and error-path floors are estimates to verify on the first run. - FIX: the bypass freshness range now starts at Storage.InternalArray.Address. It used `Address - Offset*isz`, but a corpus operand's Storage.Address IS the base, so the range sat shifted down by the offset. That was a latent false bypass for contiguous slices in the last `offset` elements (0 disagreements today). - AssertNoUnclassifiedEscapes extracted so every gate shares one verdict. - UndisposedIntermediateTests.Families: ma_* replay (NDMaskedArray operands; the keep-set is recomputed per execution because mutators swap operand masks) and index_* replay (indexer get/set, bases built under a harness NDScope). Also DisposeAny, and MaskedSingletons(): np.ma.nomask and the masked constant's arrays are returned BY mask_or/getmask/.mask, and disposing them would recycle a shared scalar slot process-wide. - UndisposedIntermediateTests.Backend: the whole ordinary corpus under OpenBLAS (threads=1), Inconclusive where none loads; BackendOnlyOpKeys must be measured when it does. - LeakCatalogue(.NDArray/.Masked): ~360 direct invocations for members no corpus row reaches, plus the LeakFixture. NOT YET COMPILED. - LeakSurfaceCoverageTests: LeakSurface.Enumerate() (the inventory tool's discovery rule, operators, the object_surfaces.py owners, NDMaskedArray, NDArray<T>) and EveryInventoryMember_IsLeakAudited with self-retiring mappings. NOT YET COMPILED. - IndexOracleTests helpers and OracleSurfaceCoverageTests alias maps widened to internal (with docs) for reuse. NEXT: see docs/plans/leak-audit-completion.md. In order: compile fixes, the catalogue and reflective-read runner, np.ma hand-scopes (the weaver refuses class carriers), the histogram [NDScoped], the README and chokepoint pin, then merge to journey4.
…whole-generation wide replace, single-read enumerator, null keys, capacity guard, unstickable resize flag Fixes the six OrderedDictionary<TKey,TValue> bugs pinned as [OpenBugs] in fe756f9 (handover docs/ORDERED_COLLECTIONS_FIX_HANDOVER.md, B1-B6) and flips their tests to CI tests. Every bug was first revalidated in an isolated worktree at 035f24f: the 15-test revalidation filter failed 10 / passed 5 on net10.0 AND net8.0 with exactly the failures the handover predicted (the other 4 failures are ConcurrentOrderedDictionary's, fixed in the next commit). B1 - enumerator read the volatile generation field twice GetEnumerator() was `new Enumerator(_t._values, Volatile.Read(ref _t._count))`: a growth published between the two reads paired the OLD value array with the NEW count -> IndexOutOfRangeException or phantom default values mid-foreach. Now reads `_t` once into a local and takes both fields from it (as ToArray/AsValuesSpan already did). Grepped the type: no other expression reads `_t` twice (SetByKey's `ReferenceEquals(_t, t)` re-read is the intentional generation check). B2 - a non-atomic (wide TValue) replace shared the index and keys across generations The wide SetByKey cloned ONLY the value array and published `new Tables(t._index, t._keys, values, ...)`. The next in-place append wrote its index word and key into those SHARED arrays, so a reader still holding the older generation resolved the new key to its OWN never-written value slot: `(true, default)` for a present key. Fix (ConcurrentOrderedDictionary.COMPACT.md section 4 rule 7): ReplaceWideUnderLock publishes a WHOLE new generation - index cloned word-for-word (same length/shift/slots, so nothing is re-hashed and no user code runs), fresh keys, fresh values. No array is ever shared by two generations now, which is what the count-gate-free read relies on. Cost (measured, below): 1.95x at N=1K, 2.59x at N=100K for the O(n) wide replace - three arrays instead of one; the correctness price, same O(n) class. B3 - the lock-free replace lost completed writes (store-buffering / Dekker race) Replacer: plain value store, then volatile loads of _resizing/_t. Resizer: volatile store of `_resizing = true`, then copy the value array. Release/acquire cannot forbid "each side's load misses the other side's store" on x86-TSO or ARM64, so the copy could read the old value while the replacer saw the flag down -> the resize published a generation without the completed write. Fix, both sides fenced: * replacer: Interlocked.MemoryBarrier() between the value store and the re-check; the flag is read with Volatile.Read BEFORE the generation (a flag already lowered by a finished resize then guarantees its publish is visible, so the generation check cannot miss it); * _resizing is now an int raised by BeginResize() = Interlocked.Exchange(ref _resizing, 1) (full fence) before any read of the live value array, lowered by EndResize() = Volatile.Write(ref _resizing, 0); * TryRemove reads the value it RETURNS inside the fenced window too - read before the flag it could be the value a lock-free replace had just overwritten, a replace that then committed: a lost-update variant on the removal's result that the handover did not list. Evidence (in-process A/B, pre-fix vs fixed source embedded in one script, B3's single-writer-per-key read-your-own-write storm, 4 replacers + 1 churn thread, 3 s x 2 reps each, i9-13900K): pre-fix: 3.43 % / 3.79 % / 1.40 % / 3.83 % / 4.23 % / 1.61 % of completed replaces lost (6 runs, 68M replaces) fixed: 0 lost - 18.3M replaces under 9.0M copying resizes with this commit's code, and 53M more under 11.7M resizes with the build-first TryRemove variant described below. B4 - null keys were accepted EqualityComparer<string>.Default.GetHashCode(null) is 0, so a null key was silently stored, found and enumerated. Added the siblings' NullCheck (`!typeof(TKey).IsValueType && key is null` - the typeof guard keeps Debug codegen from boxing value-type keys; Release folds it) to TryGetValue, ContainsKey, IndexOf, TryAdd, SetByKey, TryRemove and per pair in AddRange; GetByKey / this[key] get / GetOrAdd get it through TryGetValue (their first call), this[key] set through SetByKey. All 11 entry points of the test now throw ArgumentNullException. B5 - a capacity of ~752M or more hung the constructor `int len = 8; while (len < cap*100/70 + 1) len <<= 1;` wrapped len to int.MinValue then 0 (`0 << 1 == 0`) and spun forever; AppendGrow's `newLen <<= 1` had the same overflow. Now IndexLengthFor(int) computes the length in long (RoundUpToPowerOf2 of max(8, cap*100/70+1)) and refuses > 2^30 words (the largest power-of-two int[]) with ArgumentOutOfRangeException BEFORE allocating; AppendGrow refuses to double a 2^30-word index the same way. The limit is MaxCount = 751,619,276 entries (2^30 x 70 %), identical for the constructor and growth; ThrowTooManyEntries mirrors the compact sibling's refusal (AOOR, paramName "capacity"). The test's `new OrderedDictionary<byte,byte>(800_000_000)` now throws in microseconds. B6 - a throw during a resize left _resizing stuck true forever Every resize set the flag, re-hashed survivors through the user comparer, then cleared it with no try/finally, so a comparer throw silently turned every later "lock-free" replace into a locked one. Now every generation replacement (TryRemove, Clear, AppendGrow, the wide replace) lowers the flag in a finally. Clearing after a failed resize is safe: nothing was published, so a replacer that stored into the live generation stored into the generation that stays live. Resize shape, per operation (all under the write lock): * AppendGrow and ReplaceWideUnderLock build their new keys + index BEFORE raising the flag (GrownWith / the clone), so their fenced window holds only the live-value copy and the publish; a lock-free replace that overlaps the (comparer-calling) rebuild commits without waiting. Measured: growth path 1.01x. * TryRemove keeps the handover's prescribed order - flag (full fence) -> removed-value read -> copy keys and values -> rebuild -> publish -> lower in finally. Building the index before the flag was tried and measured 1.16-1.21x slower single-threaded (the values must be copied right after the keys; allocating the value array inside the window, dropping the try/finally or the fence each changed nothing - diagnostic variants), while the prescribed order measured 1.00x (3 processes). The trade-off it gives up (a replace overlapping the rebuild falls back and waits, exactly as before the fix) was measured too: in the churn storm the build-first order completed ~60 % more replaces but ~38 % fewer removals. Measured cost (in-process A/B, pre-fix vs fixed, pinned P-core, 2.5 s spin-up, 150 ms warm-up per op, best-of-15 x 5 interleaved blocks, median of 3 processes; A/A harness noise on the same rows 0.99-1.04): SetByKey existing <int,int> lock-free 1.86 -> 4.88-5.08 ns (+3.0-3.2 ns: the fence; handover expected ~+3.3; +2.6-3.2 across every run of the session) SetByKey existing <int,long> lock-free 1.86 -> 4.88 ns SetByKey existing <string,int> 5.27 -> 6.74 ns (+1.5 ns) TryGetValue <int,int> / <string,int> 1.46 -> 1.46 / 5.18 -> 5.18 ns (null check folded / one compare) wide SetByKey <int,decimal> N=1K / 100K 1.95x / 2.59x (B2's whole-generation copy) TryRemove interior@0 + re-add 1.00x at 1K (prescribed order; see above) build 100K from empty via TryAdd 1.01x Context: the lock-free replace stays ~3x faster than the sibling maps' locked replace (14.6 ns). Documentation (every edited member fully XML-documented, WHY comments on the protocol): * type remarks rewritten for the fenced handshake, generation ownership (no shared arrays), the measured fence cost, null keys + the 751,619,276-entry limit, and TWO NAMED RELAXATIONS of the lock-free replace that the store-then-verify design cannot avoid while a replace is retrying under the lock: (1) another reader can see new -> old -> new for that key; (2) an overlapping TryRemove can return the retrying replace's value while the retry (an upsert) re-adds the key. Once SetByKey returns every later read sees its value or newer. Neither sibling map has them (their replace takes the lock). * docs/proposals/ConcurrentOrderedDictionary.md: section 7.1 describes the fenced protocol (and why release/acquire is insufficient), whole-generation replace, single-read snapshots; 7.2 replaces "per key, operations are linearizable" with the precise contract + the two relaxations, fixes the memory-model claim (release/acquire PLUS full fences), adds null keys and the capacity limit; 7.3 notes the retry flicker; section 8 stops calling the implementation "complete" (it lists what the reference implementation lacks of the proposed API), says its perf numbers predate these fixes, and replaces the "0 lost updates" gun claim with the five defects + fixes + the storm evidence, citing the contract tests. Tests: test/NumSharp.Tests/Collections/OrderedDictionaryOpenBugsTests.cs -> OrderedDictionaryContractTests.cs (git mv, history kept; the class name no longer says "OpenBugs" now that every test in it passes), the six [OpenBugs] attributes removed, each test's doc rewritten from "Bug:" to "Defect caught:" + the fix. Verification (isolated worktree; the final code): * the 15 revalidation tests: 15/15 on net10.0 and net8.0, Release AND Debug; * storm tests (enumerator regrow, lost-update, stuck-flag) looped 20x per TFM twice (once before the TryRemove order decision, once on the final binaries): 80/80 runs, zero flakes; * Collections CI filter: 172 -> 182 on both TFMs (the +10 are this commit's 6 and the next commit's 4); * full NumSharp.Tests CI filter green on both TFMs (net10.0 16,039 passed / 0 failed; net8.0 16,038 / 0). Not changed (handover section 4 - the user's decisions): the OrderedDictionary<,> name vs the .NET 9 BCL type (CS0104), AddRange's keep-first semantics vs the siblings' upsert, negative capacity = default, the O(n) tail removal / re-hash-on-resize perf opportunity.
…xception-safe (node handles, trusted one-pass path), wide replace allocates before it swaps Fixes the four ConcurrentOrderedDictionary (COD, the shipped node type) bugs pinned as [OpenBugs] in fe756f9 (handover docs/ORDERED_COLLECTIONS_FIX_HANDOVER.md, B7-B10) plus B11 (by inspection), flips the four tests to CI tests, and marks the handover executed. Revalidated first in an isolated worktree at 035f24f: all four failed on net10.0 and net8.0 with exactly the predicted audits (key 3 enumerates but does not resolve; key 9 enumerates twice; 0/2/4/6 enumerate but do not resolve); the five COCD/OrderedDictionary controls passed. Root cause (B7-B10) Interior removal (RemoveCoreUnderLock), both TryRemoveSwapBack branches and RemoveWhere removed the key from the map FIRST, then re-indexed the shifted/moved keys through the map - one call into the user's key comparer per key - and only then published the new Store. Any comparer throw in that window aborted the removal half-applied, permanently. That includes COD's OWN LockRecursionException: the documented "evil comparer" defence refuses a write-back from inside GetHashCode, and the refusal itself corrupted the collection. RemoveWhere was worse: its finally re-ran the comparer and skipped the publish altogether. Fix: resolve every node first, then mutate without the comparer New internal members in the NumSharp partial of the vendored clone (ConcurrentDictionary.RefAccessors.cs, never the vendored body): * NodeHandle - a readonly struct around one live Node (typed object: Node is private and may not appear in an internal signature); Value = ref to the node's value field via Unsafe.As (FindNode is the only producer). * FindNode(key) - GetValueRefOrNullRef's exact walk (same hash + comparer calls, same value-type/default- comparer split), returning the node; AggressiveInlining like the seam. * TryRemoveNode(handle) - unlink BY IDENTITY: bucket from the node's stored _hashcode, chain compared by reference, TryRemoveInternal's stripe lock + tables-changed retry + _countPerLock bookkeeping, no comparer. Handles share the ref seam's lifetime rule (valid inside one serialized critical section; a growth re-creates nodes, so a stale handle is reported absent by TryRemoveNode). COD then: * TryRemove / RemoveAt resolve the removed key's node once (FindNode) and unlink it by handle - one hash walk fewer than before (GetValueRefOrNullRef + TryRemove(key)); * TryRemoveSwapBack resolves the MOVED key's node before any mutation (both branches); * RemoveCoreUnderLock / RemoveWhere resolve every node they will unlink or re-index before the first mutation, then write indices through the handles (RewriteIndexUnderLock now takes a handle); * RemoveWhere pre-allocates the Store its finally publishes (an OOM there, after unlinking keys, would strand them), so the finally can no longer throw. A refused write-back now aborts the removal with nothing changed; the four tests' full two-path audits pass. Two paths, because resolve-first is NOT neutral (the handover predicted ~neutral; measured otherwise) Micro-decomposition on the vendored map, per shifted key: one-pass lookup+write 1.18-1.25 ns; resolve-first (lookup + write-barriered handle store, then a second pass over the nodes) 2.0 ns cache-resident, 3.1 ns at 100K (the second pass is memory-bound). Pooling the handle array (ArrayPool) made small N worse and changed nothing at 100K, so allocation was not the cost - the extra traffic is intrinsic. But the hazard only exists when a lookup can run USER code, so: * _lookupsRunNoUserCode = TKey is a framework type whose default equality/hashing no user code can override (primitive, enum, string, decimal) AND the map uses the default comparer: those lookups cannot throw or write back, so the re-index keeps the one-pass form (pre-fix order), which is exception-safe there; * everything else (any custom comparer - even a StringComparer - and every user key type) takes the resolve-first path. Placement trap found on the way: the trusted one-pass loop was byte-identical in the tier-1 JIT output (DOTNET_JitDisasm) to the pre-fix loop yet ran ~0.4 ns/key slower inlined in the now-larger RemoveCoreUnderLock; as its own method (ReindexOnePassUnderLock) it runs at or above the pre-fix speed. B11 - out-of-memory could strand a wide-value replace ReplaceExistingUnderLock (non-atomic TValue) swapped the map node BEFORE the O(n) values.Clone() and the new Store, so an OOM left the key path ahead of the list path. Now: clone + fill + new Store first, then the node swap (the vendored indexer hashes and allocates the replacement node before linking, so it is strand-proof), then the throw-free publish - the order AddRange's wide branch already used. Measured (in-process A/B: pre-fix and fixed COD sources embedded in one script, pinned P-core, 2.5 s spin-up, 150 ms warm-up, best-of-15 x 5 interleaved blocks; median of 3 processes; A/A harness noise 0.99-1.01): interior TryRemove@0 + re-add, default comparer N=1K 0.87x N=10K 0.89x N=100K 0.98x (faster) same, custom comparer (resolve-first) N=1K 1.53x N=10K 1.62x N=100K 2.37x (the safety price) RemoveWhere(even) N=100K default 0.95x custom 1.33x TryRemoveSwapBack drain N=100K default 0.95x custom 0.97x TryRemoveSwapBack drain <int,decimal> (copying) 1.00x tail-pop drain N=100K 0.99x SetByKey existing <int,decimal> (B11 reorder) 0.99x Recorded in ConcurrentOrderedDictionary.TODO.md ("Comparer exception safety"). New finding, reported not fixed (OOM-only, hot add path, wants its own measurement) The vendored map grows its table AFTER inserting a key (TryAddInternal -> GrowTable), and AddRange allocates its published Store in its finally, so an out-of-memory raised by either can leave the key just added resolving without enumerating. The class remarks and the TODO's thread-safety contract now say so instead of claiming full OOM-tightness; the fix sketch (pre-grow before the insert via a partial member; allocate AddRange's holder when it goes fresh) is in the TODO. Docs (every new/edited member fully XML-documented, WHY comments on each non-obvious body): * COD class remarks: exception-tightness now covers comparer calls, names the two OOM-only gaps; RemoveWhere remarks: a throwing comparer aborts the pass with nothing removed. * ConcurrentOrderedDictionary.TODO.md: design paragraph, Gap B3 (re-index made exception-safe), thread-safety contract (comparer clause, corrected; the open OOM gap), a new "Comparer exception safety" section with the decomposition + A/B table + the placement trap, memory-churn rows (+8 B per resolved entry on the resolve-first path only), clone deviations (the partial's four members and the internals they read), gates. * ConcurrentDictionary.cs header (NumSharp-authored) + RefAccessors header list the new partial members. * docs/ORDERED_COLLECTIONS_FIX_HANDOVER.md: an "EXECUTED" status block with the gates, the deviations from the plan and the new findings. Tests: OrderedCollectionsComparerExceptionSafetyTests - the four [OpenBugs] attributes removed, class remarks and each test's doc rewritten as regression guards; the five COCD/OrderedDictionary controls unchanged. Verification (isolated worktree; final code): the 15 target tests 15/15 on net10.0 + net8.0, Release and Debug; Collections CI filter 182/182 on both TFMs; full NumSharp.Tests CI filter green on both TFMs (net10.0 16,039 / net8.0 16,038 passed, 0 failed); docfx metadata: no new cref warnings in Collections.
…ates run; 9 Core leak/bug fixes (shape setter, ReplaceData ARC, ToJaggedArray OOB, poly1d, mvn, tofile, ogrid, apply_*, vectorize) Continues 8df2598 on the side branch (NOT merged to journey4: the catalogue gate is still red on the np.ma and histogram families by design until those leaks are fixed - see docs/plans/leak-audit-completion.md "Session 2 progress" + "Next steps"). Harness (test/NumSharp.Tests.Oracle/Fuzz): - Catalogue/gate files now COMPILE (masked-indexer casts, GetData(int[]), FromMultiDimArray<double>). - UndisposedIntermediateTests.Catalogue.cs (new): SharedCatalogue + Catalogue_EveryEntry_Leaves- NoUndisposedIntermediates - warm + ScopeAudit.MeasureConfirmedTraffic per entry, pool-bypass check via FreshBytesAny against the fixture's base-buffer ranges (LeakFixture.Ranges), backend-only entries under OpenBLAS threads=1 (skipped + reported where none loads), unrunnable entries fail as harness errors. LeakCase.Throws + T(...): an always-throwing member is measured as an ERROR path and a Throws entry that starts returning fails as stale. - UndisposedIntermediateTests.Properties.cs (new): SharedPropertyReads + EveryPropertyAndField_Read_ LeavesNoUndisposedIntermediates. Every surface property/field (288 members) is read - settable ones round-trip written - on targets built INSIDE the measured region (so a getter caching an allocation on its owner is caught at the owner's disposal); two reads per region tell owner-held parts (same reference) from fresh ones (disposed, also when the setter throws); MissingBackendException reads re-run with OpenBLAS. GREEN: 539 reads + 15 error paths. - CollectDisposables: the one result walk DisposeAny, the freshness check and the stable-part detection share (a lazy IEnumerable<NDArray> is enumerated exactly once). SweepAccumulator.ToResult + Direct count. Core fixes (each found by the new gates, documented in place): - ndarray.shape / Shape setters -> SetShapeInPlace = NumPy's array_shape_set over the existing ReshapeCore port: adopt the no-copy view's dims/strides (WRITEABLE/ALIGNED kept), else AttributeError("Incompatible shape for in-place modification. Use `.reshape()` to make a copy with the desired shape.") verbatim. The old UnmanagedStorage.Reshape route COPIED a non-contiguous array in place and swapped its buffer under the NDArray's ARC reference: 2 pooled buffers stranded per assignment + a NumPy divergence (probed 2.4.2: T.shape=(4,3) ok strides (8,32); (12,)/(2,6) AttributeError; [:, ::2] (2,6) ok). - NDArray.ReplaceData (6 overloads): MoveArcReference re-points the array's counted reference to the new buffer. Before: the old buffer stayed referenced forever AND Dispose released a reference on the adopted buffer it never took - x.ReplaceData(nd) could free nd's buffer under nd. - ToJaggedArray: no densified Storage.GetData<T>() temp (leaked for views); rank-2 now reads by coordinate (GetValue<T>(i, j)). It returned WRONG values for a transposed view (0,5,10 vs 0,4,8) and read OUT OF BOUNDS on a column slice (-3.0e-241). - ndarray.tofile (binary): the C-order copy of a non-contiguous view is released in a finally. - np.ogrid: [NDScoped] NdGridLines (each line's wrapper stranded one buffer per slice). - poly1d: the constructor DETACHES its field (Returns re-tracked it into the caller's scope, which released a live polynomial's coefficients); FromFresh/WrapQuotient release every intermediate behind the operators, deriv, integ; Call(double), ==, indexer get/set, np.polyval(poly1d, poly1d) seed fixed; np.polymul disposes its temporary polynomials. - np.random.multivariate_normal (both overloads) + ComputeSvdTransform: 7 raw UnmanagedMemoryBlock scratch buffers per call freed in finally (FreeScratch, the dirichlet DangerousFree idiom), throw paths included. - np.apply_along_axis releases buff/inarr; np.apply_over_axes releases superseded expand_dims views; np.vectorize signature mode releases its broadcast views. Ownership contract (documented in each XML doc): arrays a user CALLBACK returns, and slices handed to it, stay the caller's - never disposed - so the catalogue drives these APIs with non-allocating callbacks. Found, not fixed (measured as error paths, reason in the entry label): NDArray.AsStringArray (legacy ToString layout), GetStringAt/SetStringAt (ndim vs ndim-1 coordinates), NDArray.Normalize (NotImplemented). State: catalogue 443/451 measured (8 by-design throwers on error paths); the only remaining catalogue escapes are np.ma (48 families) and histogram (4 ops). Corpus sweep / backend pass / completeness gate not yet rerun.
…e-on-host oracle; pin its CPU dispatch below AVX-512 The nightly Fuzz Soak (.github/workflows/fuzz-soak.yml, schedule -> master) has been RED EVERY NIGHT since 2026-09-06: 18 consecutive runs (35840931915 latest), each seed of the 10 failing ~620-690 of its 200000 cases. It was reliably green on master 61506de before that, with one unrelated red night (below). The first red night is the day PR #628 (journey3, merge 9fdcaf5) brought 760bb5b to master: the complex128 NaN-sign byte contract (ComplexNanContractOps + raw-byte NaN compare on x64). Root cause (same family as 33a4d2c: the live Linux NumPy oracle is not the win-amd64 reference) Every committed corpus is win-amd64 NumPy bytes, replayed. The soak instead RECOMPUTES `expected` with the Linux NumPy of an ubuntu-latest runner. For complex128 sqrt/log/exp/sin/cos/tan, NumPy does not compute the answer itself: it calls the platform's C99 complex library. Linux wheels use glibc. The win-amd64 wheel uses UCRT's cexp/csin/ccos/ctan, plus NumPy's own msun ports for csqrt/clog, because npy_config.h blocklists those under _MSC_VER. C99 Annex G leaves the SIGN of many NaN and infinite results of a non-finite argument unspecified, so the hosts disagree. NumSharp reproduces the win-amd64 bits and the harness enforces that on x64, so every Linux-recomputed complex NaN became a "NaN-sign/ signed-zero contract violation". Evidence - The CI log prints the first 60 divergences per seed. All 600 printed in the latest run are that violation, in exactly those six ops (sin 130, sqrt 112, cos 95, exp 89, log 88, tan 86). - Reproduced with a REAL Linux NumPy 2.4.2 (WSL Ubuntu, glibc 2.39, in a venv; the WSL `python3` here turned out to be a shim to the Windows interpreter) and today's harness: seed 1, 673/200000. The report cap was lifted locally to print every divergence: all 673 are the six ops' complex NaN-sign class, nothing else. - The same seed generated on both hosts: identical operands, `expected` differing in 4306 cases: 2348 NaN-sign only, in ops the harness tokenizes (reductions, real log, complex add/divide); 1284 last-bit libm rounding in real ops; 674 in the six ops, in NaN signs, in infinity SIGNS (csin -> (+nan, -inf) on win-amd64 vs (+nan, +inf) on glibc) and in last-bit rounding. 673 of these fail the harness, the same 673 the replay reported. So merely tokenizing the NaN sign would NOT have made the soak green. - The ops NumPy computes itself (square, reciprocal, negative, conjugate, sign) agree across hosts. Fix 1 — test/oracle/fuzz_random.py, following its own precedent for undefined casts (_defuse_cast; soak run 29722530598): the recompute-on-host tier must not emit host-defined values. - COMPLEX_LIBM_UNARY = {sqrt, log, exp, sin, cos, tan}. - _defuse_complex_nonfinite(base) rewrites only the NaN/inf COMPONENTS of those ops' complex inputs with finite _FLOAT_POOL values. It works through base.real/base.imag views, so every strided, reversed or transposed view sees the repair, and finite components survive, -0.0 and the 1e20/3.5e38 overflow edges included. From a finite argument these functions are fully specified, so what remains is ULP-level rounding that the complex-unary excuse already bounds. - assert_portable audits the serialized buffers for it, alongside the cast audit. - Consumes no rng, so every seed keeps its case sequence; only those operand buffers change. - The committed unary (24 per op, Windows-pinned), specials (5 per op, replayed on EVERY OS) and nan tiers still carry non-finite complex inputs for all six ops as win-amd64 bytes, so the NaN-sign contract stays gated. Fix 2 — .github/workflows/fuzz-soak.yml - Job env NPY_DISABLE_CPU_FEATURES: 'X86_V4 AVX512_ICL AVX512_SPR' (the job is ubuntu-only), with the rationale in the job comment. On AVX512_SKX runners the Linux wheel links Intel SVML (float64 exp/log and ~20 more ufuncs) and dispatches its AVX512F float32 exp/log kernels. The win-amd64 reference never runs either, and NumSharp's float32 exp/log/sin/cos cells are held bit-exact (the ported-kernel carve-out), so they cannot ride a ULP excuse. The soak's green era (61506de) predates that carve-out on master, so it says nothing about AVX-512 runners. The pin is proven on real AVX-512 hardware by run 35830326891 (interop ubuntu: AVX512F=True, X86_V4=False, 638/0). - "Install NumPy" now logs `cpu AVX512F=<hw> AVX2=<hw> | numpy dispatch X86_V4=<eff> ...`. - The header says why the oracle must not vary with the host. Docs: Fuzz/README.md "Host-dependent values" gains the complex class and the runner-ISA pin, plus a note that a Linux regeneration needs a real Linux interpreter. Verification (WSL Ubuntu 24.04 x64, glibc 2.39, real Linux NumPy 2.4.2, harness at 5f2821d) - The 10 seeds of the latest failing night, the fixed 1..5 and its random 5958011133604509022, 178170414394467171, 1709832944129576645, 1799821189561194548, 9145333823791418102, at 200000 cases each: 0 divergences on every seed (2,000,000 cases). CI had 620-688 per seed. - Negative control through the same flow with the OLD generator: seed 9145333823791418102 632 divergences, seed 3 648. The replay has teeth. - Cross-host diff of the fixed seed 1: no NaN- or infinity-sign differences remain in the six ops. The remaining differences are in tokenized ops or are last-bit rounding, and the Linux replay is green. - Helper unit checks: every non-finite component removed (views included), finite components untouched (sign bits included), real arrays untouched, assert_portable catches a leaked complex NaN into sin and allows `square` and real `sin`. - fuzz-soak.yml parses (yaml.safe_load); its new install step body, run with the pin against the real Linux NumPy, prints `cpu AVX512F=False AVX2=True | numpy dispatch X86_V4=False AVX512_ICL=False`. Under `python -W error` the pin is a silent no-op on AVX2 and leaves np.exp bytes unchanged. - This push also carries another session's unpushed 256224d (docs), dc32aec and c3ebaed (collections fixes). Their Collections tests with CI's filter pass 185/185 on net8.0 and net10.0 at c3ebaed. CORRECTION to earlier commit messages (commits are never amended here). 33a4d2c's "Linux x86-64 numpy 2.4.2 (WSL, AVX2) with the pin under `python -W error`: imports cleanly..." and 5f2821d's "ran it ... under WSL Ubuntu python3" did NOT run Linux NumPy. That WSL `python3` is ~/.local/bin/python3, a shim to the Windows interpreter, and WSL does not forward NPY_DISABLE_CPU_FEATURES without WSLENV. Both properties hold nonetheless: the pin is proven on a real AVX-512 runner (35830326891), and the no-op and diagnostic behaviour were re-verified above with the real Linux NumPy. The one earlier red night (2026-08-28, run 33187446032, master 61506de) was unrelated: 1/200000, where/random/1205 returned garbage bytes (0x10/0x44/0x84 for 01/00/00). Regenerating that seed with the 61506de-era generator and replaying it on today's harness: all 914 divergences are the complex class above and the `where` case passes. Its signature matches the ~NDArray finalizer use-after-free that 09b47a4 fixed; that commit was not on master on 08-28 and is now. It no longer reproduces. Scheduled runs execute master, so the nightly turns green once this reaches master via PR #631. workflow_dispatch on journey4 exercises it before then.
…o | grep -q` under pipefail failed a FOUND pattern Run 35856837911 (b1a8211) failed package-consumer-smoke (macos-latest) in verify_build_override.sh step 10 ("a poisoned cache entry is discarded and re-downloaded"): verify_build_override.sh: line 218: echo: write error: Broken pipe FAIL: the discarded entry must be re-downloaded Line 218 was `echo "$POISON" | grep -q "NumSharp.Interop.OpenBLAS: downloading" || fail ...` under `set -euo pipefail`. $POISON is a full `dotnet build -t:Rebuild -v n` log, far bigger than a pipe buffer. grep -q exits on its FIRST match while echo is still writing; echo then dies of SIGPIPE, and pipefail makes the pipeline's status echo's failure. So the check failed precisely BECAUSE the line was present. The same job passed on this commit's Linux/Windows legs and on every earlier run today; whether it trips depends on where the match sits in the log and on scheduling. This is a latent flake that can red any PR #631 run, independent of what the commit changes. The script already guards the identical trap for `find | head` (its step-10 comment); the echo|grep form was not covered. Reproduced in isolation (Git Bash 5.2): with pipefail, `echo "$big" | grep -q "^5$"` over 300000 lines reported "not found" in 20/20 runs. `grep -q "^5$" <<< "$big"` reported it in 20/20, and still returns 1 for an absent pattern. A here-string has no writer process that grep's early exit can kill. Fix (both scripts the job runs, both `set -euo pipefail`): - All 24 `echo "$VAR" | grep FLAGS "PAT"` checks become `grep FLAGS "PAT" <<< "$VAR"`: 10 in verify_build_override.sh, 14 in verify_package_consumer.sh. It is a mechanical regex rewrite and no echo|grep remains. The small captures ($RUN, $PUBRUN, ...) were safe in practice, but $POISON and the nupkg $LISTING are not, and one convention is simpler than judging each capture's size. - Step 10's failure-path excerpt (`grep -i ... <<< "$POISON" | tail -30`) gains `|| true`. If the excerpt matched nothing, pipefail + set -e would end the script before `fail` explained the failed rebuild. - A note under each `set -euo pipefail` states the convention and cites this run. Verified: - `bash -n` passes on both scripts, and they stay LF (git ls-files --eol: i/lf w/lf, eol=lf). - The package-consumer-smoke job replayed end-to-end on Windows/Git Bash against this exact tree (fetch_openblas.py for all 8 RIDs, then both scripts in CI order), with NUMSHARP_OPENBLAS_CACHE_DIR pointed at a scratch directory so step 10's deliberate poisoning never touched the real per-user cache: verify_package_consumer.sh: "ALL CHECKS PASSED — ... loads from a PackageReference restore on ... (win-x64)" verify_build_override.sh: "ALL CHECKS PASSED — the override delivery chain is intact" Step 10 ran in full: it poisoned the entry's .entry.json, the entry was discarded and re-downloaded, and staging completed. - Linux and macOS legs of both scripts run in CI on the push (the job's matrix).
…coped], measured-only completeness gate, FlatIterator/SetData/polyfit leaks; ScopeAudit green on net10 + net8 Completes docs/plans/leak-audit-completion.md (steps 1-6). The extended leak oracle built on this branch in sessions 1-2 (8df2598, 4539456) now runs whole and GREEN: every family of the committed corpus, a backend pass, a direct-invocation catalogue, a property/field read gate and a completeness gate that proves every public surface member was actually MEASURED. GATE STATE (identical counts on net10.0 and net8.0) - ScopeAudit category 14/14 green; full Oracle suite 212/212 green (48 s). - Corpus sweep: 213,264 success cases (ordinary 138,660 / masked ma_* 68,860 / index_* 5,744) + 9,123 error paths, 0 GC-inconclusive. The ma_* replay took 12m47s before the np.ma fixes (every leaking case forced a GC settle, over CI's 10-minute Oracle step); the whole ScopeAudit category now runs in ~25 s. - Backend pass (OpenBLAS, threads=1): 138,936 success + 2,441 error paths over 77 files. - Catalogue: 482 entries = 474 measured + 8 always-raise members measured as error paths, 0 harness errors. - Property/field reads: 288 members, 539 reads + 15 error paths. - Completeness: 1,348 / 1,348 surface members credited by a MEASUREMENT (not by a declaration). - Main suite NumSharp.Tests (CI filter TestCategory!=OpenBugs&TestCategory!=HighMemory): net10.0 16,250 passed / 12 skipped / 0 failed; net8.0 16,249 passed / 12 skipped / 0 failed. CORE LEAK FIXES (each found by the new gates; each documented in place) 1. np.ma (src/NumSharp.Core/Ma/NDMaskedArray.cs) - leaked in 36,696 of 68,860 corpus cases, 84 of 149 ops, up to 29 pooled buffers per call (std 29, median 28, var 22, average 12, anom 11, mean 9, domained unary 4-7, domained binary 8, every reduction >= 1). The [NDScoped] weaver refuses class carriers (ScopeWeaver.ImplementsCarrierInterface requires IsValueType), so the masked methods use HAND-WRITTEN scopes: `using var scope = NDScope.Open(); ... return Yield(scope, result);` where Yield() Returns() the result's _data and _mask (NDMaskedConstant skipped). Field stores go through Own(...) = NDScope.Detach (the poly1d lesson: a field egress is Detach, never Returns). Shared helpers fixed first (Unary/Binary/DomainedBinary/ ReduceIdentity/Scan/Map1/MapSeq/MaskPropagate/AverageCore/MedianAxisMasked), then every still-leaking public method. A superseded mask is NEVER disposed by the library (it may be shared with another masked array) - SupersedeMasked/ReleaseParts/ReleaseUnlessKept encode that ownership explicitly. 2. Histogram family (np.histogram*.cs) - histogram 4/call, density 8, weights 7, "auto" 11, histogram_bin_edges 4, histogramdd 13, histogram2d 11. Every public overload of histogram, histogram_bin_edges, histogram2d and histogramdd is now [NDScoped] (the result structs are INDArrayCarriers), with a <remarks> naming what is reclaimed and what leaves. 3. FlatIterator (APIs/np.flatiter.cs) - found by the new object-indexer catalogue entries: slice/ fancy GET stranded 2 buffers per call and SET 6. ResolveSlice's arange + view, NormalizeMany(NDArray)'s flatten, Scatter's astype and ConvertToBase's 1-element source + cast (the int-target scalar route) are now `using`-released; XML docs added on all four. 4. UnmanagedStorage.SetData sub-array path (UnmanagedStorage.Setters.cs) - CastIfNecessary returns a FRESH bare storage on a cross-dtype write (a float64 row into an int32 array); nothing ever took a counted reference on its pooled block, so every cross-dtype sub-array assignment stranded one buffer until a GC (the index.set escape). Released after the copy with the NDIter overlap-temp idiom (TryAddRef + Release: 0 -> 1 -> 0 frees straight back to the pool; a lone Release on a refcount-0 block would be a no-op). 5. np.polyfit (Polynomial/np.polyfit.cs) - the non-full paths carried the lstsq diagnostics (residuals / rank / singular_values) in the result regardless of `full`, so the ordinary `NDArray c = np.polyfit(x, y, deg)` stranded 2 pooled arrays per call. Those slots are now null unless full=true (NumPy returns them only then) and the scope reclaims them. NUMPY-PARITY BUGS FIXED ON THE WAY (all probed against NumPy 2.4.2) - Hard-mask __setitem__ restored data under the OLD mask only, so a slot the VALUE masks kept the new data. - Hard-mask put/putmask now OR into the LIVE mask in place (NumPy writes self._mask in place, so the mask object survives). - np.ma.fromfunction defaulted to np.indices' int64 grid dtype; NumPy's default is dtype=float (pinned by NDMaskedArrayTests.Fromfunction_DefaultDtype_IsFloat64LikeNumPy). - The np.ma callback APIs (apply_along_axis / fromfunction) release the arguments they hand a callback unless it returns one VERBATIM (reference identity) - the one place np.ma's contract differs from np.apply_*'s (where a callback's result is always the caller's). HARNESS (test/NumSharp.Tests.Oracle/Fuzz/) - Completeness gate (LeakSurfaceCoverageTests): Resolve() now credits ONLY measured evidence via LeakCoverageEvidence over the four shared runs (SharedSweep, SharedCatalogue, SharedPropertyReads, SharedBackendSweep). DirectRunResult.Attempted/Skipped and BackendSweepResult.Attempted answer "did a region actually run this id". GC-inconclusive measurements are recorded per id (SweepAccumulator.Inconclusive + SweepResult.InconclusiveIds / ErrorInconclusiveIds) and credited - the member ran, and inconclusive is never red - while backend-skipped ids are credited only when NO library loads. The missing-member report now says when an entry was DECLARED but never ran, which is a different fix from a missing entry. - The backend pass is a shared Lazy (SharedBackendSweep / RunBackendSweep, BackendSweepResult), with floors of its own (success > 131,900, error paths > 2,300). - OperatorOwners += DType, NDArrayFlags, poly1d, plus an owner-drift guard that fails when ANY object owner declares op_* members outside the list (their operators were silently unenumerated before). - New catalogue entries: every DType / NDArrayFlags / poly1d operator, and the object indexers FlatIterator.Item (scalar, int->int cast route, slice / int[] / long[] / NDArray get+set), NpzFile.Item, NDIterator.Item (0-d + external-loop views), NDArrayFlags.Item, poly1d.Item, NDArray<T>.Item (typed coordinate get/set; typed slice get/set on an owned copy). - NpzFile.Item's apparent escape was the ENTRY's fault: NpzFile's cache is a BORROWED memo - the first reader owns the array - so the entry now returns the array for disposal and boxes the second-spelling identity check instead of returning the cached field twice. - np.ma.apply_over_axes entries reduce with keepdims-shaped SLICES (views, no buffer): an allocating callback would strand every superseded per-axis result BY DESIGN under the apply_* contract. unshare_mask's entry releases the mask it superseded itself (it built the copy, so it is that mask's only holder - what the member's documentation tells a caller to do). - OpRegistry.Ma flatten_mask replay: getmaskarray MATERIALIZES an all-False mask for an unmasked operand - a temp of the replay expression, released after flatten_mask copies it (a masked operand's own mask is left alone; the fixture still needs it). - Floors tightened to ~5% under the measured counts: corpus ordinary > 131,700, masked > 65,400, index > 5,450, error paths > 8,650. The session-1 guesses (index > 5,400, errors > 8,600) were sound. - NativeAllocationChokepointTests: NDIter.cs allowlist pin 2 -> 1 (15154b0 folded the separate allocations into the single state block + inline arena, recycled through the per-thread cache). UNIT PINS (test/NumSharp.Tests, all green on net10 + net8) - Manipulation/ShapeSetterInPlaceTests.cs (10): the in-place `shape` setter, NumPy 2.4.2 probed - transposed t=arange(12.).reshape(3,4).T strides (8,32): (4,3) / (4,3,1) -> (8,32,32) / (1,4,3) -> (32,8,32) / (4,-1) adopt without a copy; (12,) / (2,6) / (6,2) / (-1,) raise AttributeError "Incompatible shape for in-place modification. Use `.reshape()` to make a copy with the desired shape."; strided [:, ::2] of arange(24.).reshape(3,8): (2,6) -> (96,16), (12,) -> (16,), (3,2,2) -> (64,32,16), (6,2) -> (32,16), (2,3,2) -> (96,32,16); broadcast (3,4): (12,) raises, (3,2,2) -> (0,16,8) stays read-only; 0-d -> (1,) / (); size mismatch "cannot reshape array of size 6 into shape (4,)"; two unknowns -> ValueError "can only specify one unknown dimension". The refcount pin builds its base IN PLACE (b.shape = ...) because np.arange(n).reshape(..) leaves an undisposed parent holding a 2nd ARC reference. - Lifetime/ReplaceDataArcTests.cs (4): ReplaceData MOVES the ARC reference (old block released, new block held once) - the session-2 UAF fix. - Casting/NDArray.ToJaggedArray.Views.Test.cs (4): transposed / sliced / strided views return C-order values without the old out-of-bounds read. - Ma/NDMaskedArrayScopeTests.cs (8): the np.ma scope/ownership contract (Yield, superseded-mask survival, callback-argument release). DOCS - test/NumSharp.Tests.Oracle/Fuzz/README.md: Scope-gate section rewritten (the retired KnownEscapeFamilies_AreFixed [OpenBugs] pin line removed; coverage-completion subsection with the five layers and the crediting rules; chokepoint text). - .claude/skills/oracle: SKILL.md, references/architecture.md, triage.md (two new ScopeAudit red shapes: "N surface members are not leak-audited" and "catalogue entries could not run"), add-op.md (a new op's corpus rows credit the completeness gate; an API without corpus route needs a LeakCatalogue entry). - docs/plans/leak-audit-completion.md: header COMPLETE, measured cost notes, Session 3 section. KNOWN, NOT FIXED (deliberately) - Size-0 arrays' byte strides: NumSharp (0,8) for (3,0) on every path; NumPy (8,8) via reshape but (0,0) via np.zeros - NumPy is itself inconsistent, and the repo's standing policy (LayoutParityOracleTests.Verify) treats size-0 strides as non-contractual. The shape-setter pin asserts shape + flags only there, documented in its remarks. TRAPS RECORDED (for the next session on this gate) - A catalogue entry must RETURN every array it owns (the harness disposes results) and must never return a fixture array or a member's OWN field - report its size instead. - NDScope Returns() is for yielded values; a FIELD store must Detach, or the next scope exit frees the object's own state. - The weaver cannot scope class carriers - NDMaskedArray methods need hand scopes.
Lands the leak-oracle coverage completion (docs/plans/leak-audit-completion.md, steps 1-6) on the active branch. Three commits: 8df2598 (inventory cross-reference, every corpus family in the sweep, catalogue + completeness gate), 4539456 (catalogue + property/field read gates running; 9 Core leak/bug fixes: shape setter in-place adopt + AttributeError, ReplaceData ARC move, ToJaggedArray view values + OOB read, poly1d ctor Detach, multivariate_normal raw scratch, tofile, ogrid, apply_*, vectorize) and 77b967d (np.ma hand scopes, histogram [NDScoped], FlatIterator/SetData/ polyfit leaks, the measured-only completeness gate, floors, unit pins, docs). What the gate proves after this merge: every public surface member of the 8 inventory modules (1,348) is leak-MEASURED - by the corpus sweep (ordinary + ma_* + index_* + error paths, 213,264 + 9,123 cases), the OpenBLAS backend pass (138,936 + 2,441), the direct-invocation catalogue (482 entries) or the property/field read gate (288 members) - and every measurement returns the buffer pool to zero. Merge: textually clean. The only file both sides touched is test/NumSharp.Tests.Oracle/Fuzz/README.md, in different sections (journey4's b1a8211 "Host-dependent values" complex/ISA paragraphs vs this branch's "Scope gate" rewrite). journey4's other changes since the fork point 256224d (the ordered collections fixes dc32aec/c3ebaed1, the fuzz-soak complex-defuse + AVX-512 pin b1a8211, the OpenBLAS verify-script here-strings 45c7913) touch no file on this branch. The merged tree was built and its full suites run on net10.0 and net8.0 before journey4 was fast-forwarded to this commit.
…e pack reuses Build's Release outputs instead of recompiling Core
The standalone `verify-signing` job (ubuntu-latest, every push and PR) took 69 s in run
35858283137, and about 45 s of it repeated work the `test` job already does on the same runner
image: its own checkout (9.3 s "Fetching the repository"), a cold .NET start (5.3 s before the
first "Determining projects to restore") and a from-scratch Release compile + NDScope weave +
re-sign of NumSharp.Core (~31 s inside the 54 s pack step). The six non-Core packs took 14.5 s
and the verifier 2.7 s.
Change (.github/workflows/build-and-release.yml):
- The `verify-signing` job is removed.
- The `test` job gains two ubuntu-only steps whose commands are verbatim from the old job:
* "Strong naming: key is present and intact" (id: snk), BEFORE Build, so a missing key or a
dropped `*.snk binary` rule in .gitattributes still reports as itself instead of as a
confusing signing failure.
* "Strong naming: pack (throwaway) and verify every shipped assembly is signed", LAST, with
`if: ${{ !cancelled() && matrix.os == 'ubuntu-latest' && steps.snk.outcome == 'success' }}`
and timeout-minutes: 10. Its verdict stays independent of the tests, as the separate job's
was: a red test cannot hide a signing regression, and a signing regression cannot skip the
tests. It builds whatever it needs itself, so it does not require the Build steps to pass;
it only requires the key check, whose failure already names the root cause. A cancelled run
still stops it (always() would not).
- ARCHITECTURE.md CAP-SIGN-01 evidence link now points at those steps.
Why the pack is cheap inside `test`: the Build steps already produce Release outputs of
NumSharp.Core (compiled, woven and re-signed), NumSharp.Bitmap, NumSharp.Interop.OpenBLAS and
NumSharp.Build with the identical -p:NoWarn, so each `dotnet pack` is an incremental build.
Core's CoreCompile and its NDScopeWeave target (Inputs=@(IntermediateAssembly),
Outputs=$(IntermediateOutputPath)NDScopeWeave.marker, run in lockstep with CoreCompile) are up to
date, so Core is neither recompiled nor re-woven, and only pythonnet, OnnxRuntime and MLNet
compile. If the Build steps' properties ever diverge from the pack's, the cost is a recompile,
never correctness: the pack still builds and verifies exactly what it packs.
Why `test` rather than `interop-test`: `test` already builds 4 of the 7 packable projects in
Release (Core, Bitmap and OpenBLAS through NumSharp.Tests/Oracle; NumSharp.Build through the
Analyzer tests). `interop-test` builds 3 (Core, pythonnet, OpenBLAS) and stages all 8 OpenBLAS
RIDs, which the OpenBLAS pack would then zip (~186 MB of runtimes) for nothing, since the verifier
skips runtimes/**/native.
Why ubuntu only: build-nuget packs every release on ubuntu-latest, so this leg checks the
platform whose output ships; Directory.Build.props enables signing with no OS or TFM condition;
and ubuntu is the shortest `test` leg (11.5 min against Windows' 15.7 in that run), so the added
~20-25 s does not lengthen the run.
Measured (Linux under WSL, fresh `--depth 1` clones of journey4 2180151, pinned to 4 CPUs with
taskset to mirror the 4-vCPU runner, build servers shut down between the builds and the pack to
mirror the ~10-minute gap the test steps leave on CI):
- merged: the `test` job's three Release builds (44.7 + 2.9 + 1.8 s, existing cost), then the
verbatim pack loop in 17.1 s + verifier 1.2 s. `grep -c woven pack.log` = 0: Core was neither
recompiled nor re-woven. verify_strong_name: 14 assemblies, 7 packages, 0 failures.
- standalone (the old job): pack loop 25.0 s (Core 15.7 s, 2 weave lines) + verifier 1.5 s;
same 14/7/0.
Local cores are faster than the runner's, so CI numbers run roughly 2x these; the old job's own
CI log (six non-Core packs = 14.5 s with Core already built) puts the merged step at ~20-25 s.
Effect per push/PR: one job fewer (runner time -69 s, +~20-25 s on the ubuntu `test` leg) and
wall time unchanged. The signing verdict now arrives ~11.5 min into a run instead of ~1 min; the
run's own verdict still waits for the 15.7-min Windows leg. The check now also gates releases:
validate-release already `needs: test`, whereas `verify-signing` was never in that list
(build-nuget's own packed-artifact check is unchanged).
Branch protection: master has no classic protection (GET .../branches/master/protection is 404)
and no rulesets, so no required status check named "Verify strong naming" is left waiting.
Landed index-only: another session's uncommitted edits to the same workflow file (moving
package-consumer-smoke into interop-test) stay in the working tree untouched. This commit's blob
is HEAD plus these hunks only, verified by reverse-applying their diff to the live file and
round-tripping it back byte-identically.
…is validated job-env knobs The nightly soak swept five fixed seeds and five fresh random seeds, 200K cases each. fuzz_random.py is deterministic, so every fixed seed regenerates a byte-identical corpus each night: half of each night's 2M cases were the same inputs as the night before. From 2026-09-11 to 09-23 master did not move at all (9396799 on all 13 scheduled runs), so that half re-tested identical inputs against identical code for 13 nights. Fresh seeds are where new ground comes from — measured on the win-amd64 reference host (NumPy 2.4.2): a 200K-case seed is ~67.5% distinct inputs, and 10 seeds reach 959K distinct inputs of 2M, the 10th still adding 78K. One fixed seed keeps both things a fixed seed is for: * a deterministic canary: it turning red while its draw stayed put points at the code, not at new cases; * a corpus fingerprint comparable night to night: its source_sha256 in the evidence file should repeat while the generator and the NumPy pin are unchanged. The old evidence already showed why that matters, and nothing compared it: on the same master SHA and generator, fixed seed 1 came out 8ee0d4b0... on five nights and 993a8df6... on 09-17 and 09-19 — NumPy answering differently on different runners (the ISA lottery the NPY_DISABLE_CPU_FEATURES pin closes). Seed 1 is the one kept, so its nightly history stays comparable. What changed (.github/workflows/fuzz-soak.yml) * SOAK_FIXED_SEED_COUNT=1 / SOAK_RANDOM_SEED_COUNT=9 in the job env — env rather than script variables, so the random-seed generator's Python heredoc reads its count from the environment it already has. The header and the env block document the 1 + 9 rationale. * The sweep step validates both knobs as positive integers before anything uses them: the `-ne` tests would read a malformed value as 0 and accept the wrong shape silently. * workflow_dispatch `seeds` keeps its name, so existing `-f seeds=N` dispatches still bind, but it now takes exactly one seed (default '1'). A five-seed dispatch is refused with "Expected exactly 1 fixed seed(s), got 5: 1 2 3 4 5". * The random-seed generator draws SOAK_RANDOM_SEED_COUNT distinct seeds, never the fixed one, and the step checks the count: mapfile over a process substitution drops the generator's exit status, so a crashed generator surfaces there, as a short count (comment added). * requested_total_cases is derived from the two seed lists instead of the literal `COUNT * 10`. * Evidence schema unchanged (fixed_seeds stays a list, now of one); runtime unchanged (still 10 seeds x 200K — ~7 min of the 45-min budget). Docs: Fuzz/README.md, the oracle skill's references/architecture.md and CLAUDE.md's CI section called the soak "~1M cases/night", stale since b6935f3 made it 5 + 5 (2M); they now say one fixed + nine fresh seeds x 200K (~1.8M fresh cases/night), and the README explains the fixed seed's canary/fingerprint role. Only the soak line of CLAUDE.md is in this commit: the working tree also holds a parallel session's uncommitted CLAUDE.md and build-and-release.yml edits, which stay unstaged (index-only landing via git apply --cached). Verification * actionlint 1.7.12 and shellcheck 0.11.0: clean on the edited workflow and its extracted sweep step (both were also clean on the pre-edit baseline). * Stub harness — heavy commands stubbed, the step extracted from the edited YAML with GitHub's `||` input semantics, run under a GitHub-like `bash --noprofile --norc -eo pipefail` with no BASH_ENV — 17 scenarios, all green: - scheduled defaults (seed 1 + 9 random, 200K), dispatch seed 7, an empty seed input falling back to '1': 11 evidence records, fixed first, 9 distinct random seeds in [1, 2^63-1] never equal to the fixed one, total = 10 x count, 10 builds, 10 tests, 10 TRX, no failing corpora; - refusals, each exiting 1 before any dotnet call with its exact ::error:: text: two seeds, the old five-seed dispatch, a non-integer seed, a negative seed, count 0, a malformed / zero / missing split knob, and a crashed random-seed generator; - per-seed failures — fixed-seed generation failure, short corpus, fixed-seed divergence, every random seed diverging, copied-corpus mismatch — each finish all 10 seeds, exit 1, and keep exactly the failing seeds' own corpora. The same suite against the pre-edit workflow fails all 17 scenarios (40 checks), so it discriminates. * Real end-to-end run of the edited step in a detached worktree at HEAD — real fuzz_random.py on NumPy 2.4.2, real Oracle Release build, real FuzzRandom replay — at count 1600 (the MinCases floor): all 10 seeds (fixed seed 1 + 9 fresh) generated, built and replayed green, step exit 0; the evidence has 11 records in order with 1600 source/copied cases and matching hashes per seed, 10 TRX files, no failing corpora — and fixed seed 1's corpus sha256 (ba801e3b...) came out identical in two separate runs, the night-to-night fingerprint property in miniature. Noted, not changed here: FuzzCorpusTests.MinCases floors random_smoke.jsonl at 1600, so a dispatch with count < 1600 fails every seed's replay, which the step reports as "found a divergence"; and on a Windows runner the random-seed heredoc's stdout would carry "\r" into every seed (Windows Python writes "\r\n" to a pipe; the job is ubuntu-only today).
… its last two steps
The nupkg-consumer gate for NumSharp.Interop.OpenBLAS (verify_package_consumer.sh +
verify_build_override.sh) ran as its own 3-OS job, `package-consumer-smoke`. It now runs as the
final two steps of `interop-test`: "Package consumer: pack, restore by PackageReference, load in
every layout" and "Package consumer: build-time version override, end to end". The scripts
themselves are unchanged.
Why interop-test and not test:
- interop-test already sets up what the scripts need: Python 3.12 (fetch_openblas.py, the nupkg
listing, the macOS realpath compares) and the staged OpenBLAS bundle (verify_build_override.sh
packs WITHOUT RequireOpenBlasAssets, so it relies on the "Stage OpenBLAS native assets" step).
`test` has neither.
- Wall time. In run 35858283137 interop-test took 3.3-3.9 min and `test` 11.5-15.7 min; the two
scripts cost 197/241/260 s (ubuntu/windows/macos). Inside interop-test that job lands at
~6.5-8.5 min, still under `test`, so the run gets no longer. On `test` the same minutes would add
straight onto the longest job.
- One macOS slot. The SciSharp org is on the GitHub Free plan: 5 concurrent macOS jobs org-wide.
Every push used all five (test, interop, onnx, mlnet, consumer-smoke); in run 35858283137
package-consumer-smoke (macos-latest) was created at 12:04:58 and started at 12:06:06, 9 s after
the previous run's test (macos-latest) released its slot at 12:05:57. Now 4 per push.
Ordering and conditions:
- LAST, after the interop tests: both scripts `-t:Rebuild` Core and NumSharp.Interop.OpenBLAS in
Release inside the repo tree and delete NumSharp / NumSharp.Interop.OpenBLAS <Version> from the
global NuGet packages folder so the scratch consumer restores the fresh pack. Running after the
tests keeps the interop suite on the untouched build, as when the gate had its own runner.
- `if: ${{ !cancelled() }}` on both: a red interop test must not hide a package-layout regression
(a separate job's verdict never depended on the interop tests), and the build-override script
still runs when the consumer script failed. A cancelled run still stops them (always() would
not).
- timeout-minutes: 15 each (a normal run is 1.5-2.5 min per script, PyPI included); they were
uncapped before, i.e. the 6-hour job default.
- Job env: NUMSHARP_PYTHONNET_REQUIRE_ENGINE and NPY_DISABLE_CPU_FEATURES are inert for the
scripts - their scratch consumers reference only NumSharp + NumSharp.Interop.OpenBLAS, so nothing
starts pythonnet or imports numpy. A comment now forbids adding NUMSHARP_OPENBLAS_CACHE_DIR to
that env block: verify_build_override.sh asserts the DEFAULT per-user cache root on Linux/macOS
and, unlike verify_package_consumer.sh, does not unset the variable, so a job-level value would
fail its step 1 on two of the three legs.
- validate-release drops package-consumer-smoke from `needs`; needing interop-test covers it.
Effect per push/PR, on top of 2adf21d (verify-signing folded into `test`): 15 -> 12 jobs,
5 -> 4 macOS jobs, three fewer checkouts and .NET setups (setup-dotnet alone took 58 s on the old
job's Windows leg).
Branch protection: master has no classic protection (GET .../branches/master/protection is 404)
and no rulesets, so no required check named "package-consumer-smoke (...)" is left waiting.
Verified:
- YAML parses, and a structural check asserts: the old job is gone from `jobs` and from every
`needs`; every remaining need resolves; interop-test keeps its 3-OS matrix; the step order is
Setup Python < Stage OpenBLAS < Test (net8.0) < Test (net10.0) < consumer < override < Upload;
both new steps carry !cancelled(), a timeout and bash; Upload keeps always(); no other job runs
either script; the job env never sets NUMSHARP_OPENBLAS_CACHE_DIR. actionlint 1.7.12 (release
zip sha256-verified against its checksums file): clean, before and after 2adf21d landed
underneath.
- Local replay of the merged job's Windows leg, in job order and with the job env, in a detached
worktree of 2180151: Stage OpenBLAS (8/8 verified) -> Build 140 s -> Ensure native ->
Test (net10.0) 733/733 passed, 119 s -> verify_package_consumer.sh ALL CHECKS PASSED, 31 s ->
verify_build_override.sh 15/15 ALL CHECKS PASSED, 59 s. Isolated from this machine: NUGET_PACKAGES
and (build-override only; its default-root assertion is Linux/macOS-only) NUMSHARP_OPENBLAS_CACHE_DIR
pointed at scratch dirs, because the scripts otherwise evict and refill the real global cache's
NumSharp 0.70.0 - a RELEASED version - with this branch's pack. The real caches' timestamps and
the scratch caches' contents confirm the isolation held.
- Not verifiable locally: the Linux and macOS legs (where the macOS dylib-layout branches run).
The next CI run on journey4 is their gate.
Landed alongside a concurrent session: 2adf21d (verify-signing -> test) was committed index-only
while these hunks sat uncommitted in the same file; this commit contains only these hunks
(git diff HEAD matched the reviewed patch byte-for-byte at commit time).
Docs: .claude/CLAUDE.md's OpenBLAS paragraph names the gate's new home and its two ordering
constraints (runs after the interop tests; the job env must not set NUMSHARP_OPENBLAS_CACHE_DIR).
…t; run the pythonnet suite in two isolated Python venvs (parity + ecosystem) Why the two jobs existed, and why they don't need to ---------------------------------------------------- `onnxruntime-interop-test` (f5077cd) and `mlnet-interop-test` (bec868c) were each "its OWN job for the same reasons as interop-test" - but interop-test's reasons (b5b4948: the embedded-CPython setup must not destabilise the main gate; a red must be attributable) do not apply to them. Both suites are pure .NET: no Python, no native asset beyond what their NuGet packages restore, no env var they read. They were separate by precedent. After 2adf21d (verify-signing -> test) and f12a03f (package-consumer-smoke -> interop-test) they were the last two standalone 3-OS jobs. Effect per push/PR: 12 -> 6 jobs (test x3 + interop-test x3), 4 -> 2 macOS jobs. The SciSharp org is on the GitHub Free plan - 5 concurrent macOS jobs ORG-WIDE - and this workflow used to take all five per push (run 35858283137: a macOS job waited 68 s for a slot). NumSharp.Core now compiles once in the interop job instead of three times. Wall time: the suites' measured costs (run 35858283137) add to ~10-13 min per leg, still under `test`'s 11.5-15.7 min. The "version collision with interop-test's Python" -------------------------------------------------- No record of one exists in git, plans or docs, and none is possible for the two .NET suites (they never start Python; separate `dotnet test` processes). Two REAL collisions exist on the Python side: - The ORT floor's Python twin: onnxruntime==1.16.0 publishes no cp312 wheel (cp38-cp311 only; `uv pip compile` -> "no wheels with a matching Python ABI tag") and was compiled against NumPy 1.x - in a py3.11 venv with numpy 2.4.2 its import prints "A module that was compiled using NumPy 1.x cannot be run in NumPy 2.4.2"; with numpy 1.26.4 it imports. Any Python work tied to the ORT floor needs a different interpreter than the pythonnet suite's 3.12 + numpy 2.4.2. - Inside the pythonnet suite: the byte-exact live-NumPy tests need numpy pinned to the release whose bundled OpenBLAS NumSharp ships, in a process nothing else perturbs, while its bridge tests need torch/pandas/scipy/pyarrow/pillow/polars/opencv, which pin numpy ranges of their own and load their own native runtimes. CI installed numpy alone, so those 93 tests were Inconclusive on every run. Today they all co-resolve with numpy 2.4.2 on win_amd64 / manylinux_2_28 x86_64 / macOS 14 arm64 (uv pip compile per platform), but nothing kept it that way; plain PyPI torch on Linux also drags ~3 GB of CUDA wheels. Python environments (test/NumSharp.Tests.Interop/python-envs/) -------------------------------------------------------------- make_env.py (stdlib venv + pip; runs on a bare interpreter) builds a named environment from the interpreter that RUNS it, at <repo>/.venvs/<name>-py<maj><min>/ (gitignored) - so one machine holds the same environment for several Pythons side by side. stdout is exactly one line, the environment's interpreter path (progress on stderr), so `PY=$(python make_env.py parity)` captures it. - parity: numpy ONLY, at the OpenBLAS manifest's numpy_version (single source of truth - the pin the bundled binary is byte-identical to). - ecosystem: the same numpy + torch==2.13.0 from PyTorch's CPU-only index (torch-cpu.txt; its own file because --index-url is per-file) + ecosystem.txt: pandas 3.0.5 (PandasTestGate's exact pin), scipy 1.16.3, pyarrow 23.0.1, pillow 11.3.0, polars 1.38.1, opencv-python-headless 5.0.0.93 - exact pins, the versions validated locally. numpy goes in first; every later file installs under a `-c numpy==<pin>` CONSTRAINT, so a library that demands another numpy fails the install instead of moving the oracle. macOS: numpy 2.4.2 ships Accelerate-linked macosx_14_0_* wheels for BOTH arm64 and x86_64 (the old step only handled arm64), which pip prefers on macOS 14+; the script downloads the scipy-openblas tag per arch and installs that file. Then it VERIFIES what numpy reports (show_config dicts: blas name scipy-openblas at the manifest's openblas_version) and fails otherwise - checking the result, not the request. PythonSession: venv-aware, binding interpreter selection -------------------------------------------------------- - NUMSHARP_PYTHONNET_PYTHON (new EnvVars.PythonnetPython): the interpreter to embed, checked before PYTHONNET_PYDLL. BINDING: an unusable named interpreter fails the session with that reason; no other Python is tried (a typo cannot silently test against PATH's numpy). - Venv adoption: a venv has no libpython; pythonnet loads the BASE install's, which alone starts an interpreter whose sys.prefix is the base - the venv's site-packages are not on sys.path and the base's packages answer. When the probe finds sys.prefix != sys.base_prefix, Start sets PythonEngine.ProgramName to the venv's python before Initialize: CPython 3.11+ getpath.py takes executable = abspath(program_name), reads the venv's pyvenv.cfg, and site.venv() makes the venv the prefix - exactly what running the venv's python does. PROVEN with a witness package: base holds torch 2.12.1+cu126, venv holds 2.13.0+cpu; embedding with the program name set yields prefix=venv, numpy and torch from the venv's site-packages; without it, prefix=base and torch 2.12.1+cu126. - The embedded sys.prefix is then verified against the probe's for a venv (mismatch -> session unavailable, reason names both). Base installs are deliberately NOT checked: their prefix comes from the loaded library, and demanding the probe's exact spelling (macOS framework vs toolcache alias) would add a failure mode to the configuration that already works on all three OSes. - The probe appends sys.prefix and sys.executable as lines 6-7 (lines 1-5 keep their meaning). - NUMSHARP_PYTHONNET_REQUIRE_PACKAGES (new EnvVars.PythonnetRequirePackages, default false): an absent or wrong-version optional package FAILS instead of Inconclusive - CI sets it for the ecosystem run, where every package is installed on purpose. Test routing: [PythonEcosystem] ------------------------------- New TestCategoryBaseAttribute (category "PythonEcosystem"): 8 whole classes (PyTorchInteropTests, PyTorchInteropEdgeCaseTests, PandasInteropTests/EdgeCase/RareScenario, Example09_PyTorch, Example10_Pandas, GistSignalLiveParityTests) + 12 methods of mixed classes (7 on the NpFrombuffer docs page, 3 in example 05, 2 in GistSignalSourceDemonstrationsLiveTests) = 94 tests, derived from the TRX of run 35858283137 (every test Inconclusive for a missing package). DRIFT GUARD: SkipUnless, PyTorchTestGate.Require and PandasTestGate.Require FAIL a caller without the tag, on every machine, before looking for the package - untagged, a new package test would run only in the numpy-only environment, skip there, and never run anywhere (green-by-skipping). The current test is recorded in InteropTestBase's [TestInitialize] from MSTest's TestContext, now declared ONCE on InteropTestBase (GistMetricsLiveParityTests and GistSignalLiveParityTests dropped their own copies, which would have hidden it). ReportMissingPackage is [DoesNotReturn]. Workflow (.github/workflows/build-and-release.yml) -------------------------------------------------- interop-test now runs, per OS: Setup Python (pip cache keyed on python-envs/*.txt + the OpenBLAS manifest) -> "Python env: parity" / "Python env: ecosystem" (make_env.py; outputs.python) -> Report Python host (per environment: python, numpy + its BLAS, prefix/base_prefix/executable, INSTSONAME/LIBDIR, cpu flags vs numpy dispatch) -> Stage OpenBLAS -> pythonnet: Build -> Test parity net8.0/net10.0 (TestCategory!=PythonEcosystem) -> Test ecosystem net8.0/net10.0 (TestCategory=PythonEcosystem, REQUIRE_PACKAGES=1) -> ONNX Runtime: Build floor, Test floor net8.0/net10.0, Test current net10.0 (1.29.0) -> ML.NET: same (2.0.1 / 4.0.2) -> the two "Package consumer:" steps (still LAST: they -t:Rebuild Core in-tree and evict the NuGet cache) -> Upload. Independence, which separate jobs had for free: a job stops at its first failing step, so every step past the first suite carries `!cancelled()` plus its own prerequisites' outcomes (step ids pyenv_parity, pyenv_ecosystem, stage_openblas, pythonnet_build, ort_build, mlnet_build) - a red ORT suite cannot hide ML.NET's verdict, a failed ecosystem install cannot stop the parity oracle. NUMSHARP_PYTHONNET_PYTHON is set per STEP, never job-wide. TRX files are named per suite/run (pythonnet-parity-net8.0.trx, onnxruntime-current-net10.0.trx, ...). validate-release needs [test, interop-test]. Removed jobs: onnxruntime-interop-test, mlnet-interop-test. Fixed along the way: the old comment "The second build uses its own obj/bin (-p: changes the restore)" was false - the test csprojs redirect nothing; the current-version run re-restores and rebuilds the SAME obj/bin, harmless only because it runs after both floor steps. The comment now says so. Verification ------------ - Windows (Anaconda 3.12.12 base): make_env.py parity 7 s, ecosystem 74 s (torch 2.13.0+cpu). pythonnet parity 639/639 (0 skipped - was 93 Inconclusive in CI), ecosystem 93/94 (1 hardware skip: no CUDA/MPS), on BOTH net10.0 and net8.0. ORT 159/159, ML.NET 165/165 (net10.0). - Negative controls: (1) removing one method's tag -> that test FAILS with the drift-guard message even in the ecosystem env where scipy is installed; (2) parity env + REQUIRE_PACKAGES=1 -> the scipy class FAILS naming the interpreter, without the flag -> 7 Skipped; (3) NUMSHARP_PYTHONNET_PYTHON pointing at a missing file -> binding failure, "no other Python was tried". - Linux (WSL Ubuntu 24.04) with setup-python's EXACT interpreter: the actions/python-versions 3.12.14 linux-24.04-x64 build. make_env.py builds both envs; parity 637 + 2 platform skips (4-byte wchar_t), ecosystem 93 + 1 hardware skip, 0 failed, both TFMs. Also proved the Linux venv adoption against ~/np242 (numpy only in the venv, NONE in base /usr: a failed adoption could not pass). Ubuntu's SYSTEM python3.12 is not usable for this: it lacks ensurepip (python3.12-venv) and lib2to3 (python3-lib2to3; 9 Karpathy tests + cascading leak checks fail) - packaging, not NumSharp; documented in the suite's CLAUDE.md. - actionlint 1.7.12 (release zip sha256-verified): clean, as is HEAD's version. A structural check (PyYAML) asserts the job set, needs, step order, every `steps.<id>` referencing an EARLIER id, `!cancelled()` on every post-first-suite step, and the env routing; the env and report steps' run scripts were extracted from the YAML and executed as written. - NOT verified locally: the macOS leg (framework-build venv adoption, the per-arch wheel download). The next CI run on journey4 is its gate. Docs: test/NumSharp.Tests.Interop/CLAUDE.md (discovery order, venv adoption, a "Python environments" section: table, commands, tagging rule, Debian ensurepip note, the ORT-floor interpreter case); .claude/CLAUDE.md (ORT/ML.NET gate rows name the interop-test steps; consumer steps run after every suite); docs/plans/onnxruntime.md; the examples' requirements.txt points the suite at make_env.py.
… .NET's unfused Complex.Abs, 35.5% of values off); MPS test uses float32; live spectrum parity tests Found by the first CI run (35874958875, 3ef6b0e) that installed the [PythonEcosystem] packages on macOS: 3 of 94 ecosystem tests failed on macos-latest (both TFMs); windows/ubuntu green. 1. Complex magnitude was NOT bit-exact with NumPy - on x64 too. GistSignal FrequencyPipelines / SyntheticDemoHarmonics: the harmonic-product-spectrum estimate (a product of 7 FFT magnitudes -> log -> parabolic vertex) was 20 ULP off NumPy on macOS. Root cause is stage 1, complex abs. NumPy 2.4.2 computes np.abs(complex) with the SIMD kernel simd_cabsolute (loops_unary_complex.dispatch.c.src): sqrt(npyv_muladd(ratio, ratio, 1.0)) * larger, ratio = smaller/larger. npyv_muladd_f64 is a REAL fused multiply-add on every current dispatch target: vfmaq_f64 on NEON (FMA is in the arm64 baseline) and _mm256_fmadd_pd on x86-64 X86_V3 (AVX2+FMA3). NDComplexMath.Abs deferred finite inputs to System.Numerics.Complex.Abs, the same formula UNFUSED (1.0 + ratio*ratio). Measured over 1M seeded random complex values on this AVX2+FMA3 host: numpy.abs == fused form 1,000,000/1,000,000; numpy.abs == NumSharp 644,728/1,000,000 (35.5% off). The x64 byte-exact tests had passed on lucky inputs (integer pools make ratio^2 exact, so both forms agree); macOS's input differs (Apple libm sin builds the signal) and exposed it. Fix: NDComplexMath.Abs = the kernel's formula with Math.FusedMultiplyAdd (hardware FMA where present, correctly-rounded software FMA elsewhere -> same answer on every host); inf/NaN handling unchanged (inf beats NaN -> +inf; NaN -> positive NPY_NAN), larger == 0 -> +0 as NumPy's masked division gives. After: NumSharp == numpy.abs on 1,000,000/1,000,000. Caveat documented: an x86-64 CPU WITHOUT FMA3 makes NumPy fall back to X86_V2's unfused muladd - no current runner/desktop. All kernels reach it (the IL ComplexAbs MethodInfo -> NDComplexMath.Abs); comment updated. Oracle FuzzMatrix 133/133 green after the change (net10.0). 2. PyTorchInteropEdgeCaseTests.AcceleratorTensor_ForceCopiesToCpu_WhenAnAcceleratorIsAvailable: the macOS runner HAS an MPS device, and the test created torch.arange(dtype=float64, device='mps') -> "Cannot convert a MPS Tensor to float64 dtype as the MPS framework doesn't support float64". It had only ever run where no accelerator exists (Inconclusive). Now float32 on MPS (the widest float both accelerators share), float64 kept on CUDA; the *CPU* message check still holds (torch's own "Use Tensor.cpu()" text, matched case-insensitively). 3. New SpectrumLiveParityTests (parity env, numpy only): complex abs over 100K seeded random values (13 decades per component) and over edge values (+-0, subnormals, ratio^2 underflow, overflow, inf beside NaN, NaN) through contiguous / strided / reversed / transposed exports; np.fft.rfft of the seven-harmonic signal at n = 1024, 1000 (mixed radix), 1021 (Bluestein). Byte-exact vs live NumPy on the same exported buffer, failure message = count of differing lanes + first lane's hex + input. The rfft test is also the open question's probe: whether NumPy's arm64 pocketfft (clang default -ffp-contract=on, NumPy sets no flag) contracts butterflies into FMA - the next macOS run answers it. Verified locally (Windows x64, AVX2+FMA3): parity 642/642 (639 + 3 new), ecosystem 93/94 (1 = no CUDA/MPS), net10.0; Oracle FuzzMatrix 133/133. macOS confirmation = the CI run on this commit.
…ansposed planes, astype-semantics conversion, IL-emitted rank>=4 allocation; float16/complex128/0-d/T!=dtype now work NDArray.ToMuliDimArray<T>() - and the explicit (Array)nd cast, which dispatches to it on the dtype - is rewritten to fill the pinned .NET result in ONE pass. The old body walked the array into a full-size managed T[] via Storage.ToArray<T>() and then Buffer.BlockCopy'd that into Array.CreateInstance's result (decimal: a boxed SetValue per element): two copies, 2x peak memory, a per-element coordinate walk for every view, and hard failures on the inputs below. Behaviour changes (each was an exception before): - float16 / complex128: Buffer.BlockCopy accepts CLR primitives only, so ToMuliDimArray<Half>/<Complex> and therefore (Array)nd threw ArgumentException for both dtypes. Now copied like every other dtype. - T != dtype: threw ArrayTypeMismatchException. Now converts with astype semantics - the same NDIter cast core astype uses - e.g. int32 -> double, complex -> double (imaginary part discarded, as astype). - 0-d arrays: threw "Must provide at least one rank." (.NET has no rank-0 array). Now a one-element T[]. - EMPTY decimal arrays: threw (the per-element decimal path read one element unconditionally). - A non-dtype T (Guid, nint, DateTime, ...) now fails up front with NotSupportedException naming it. Unchanged: a dimension above int.MaxValue -> InvalidOperationException (same text); rank > 32 -> the runtime's TypeLoadException; the result shares no memory with the source; 1-D -> SZ T[], rank N -> T[,..]. Mechanism (one method region of Casting/NDArrayToMultiDimArray.cs, signature unchanged; 14 private members, well over half of the ~700 lines are XML docs and WHY comments; ToArray<T> untouched): 1. Allocation: rank-specialized `new T[a,b(,c)]`; a 1-D result of >= 1024 elements comes from GC.AllocateUninitializedArray(pinned: true) - every element is written, and the pin is then free; rank >= 4 goes through RankNAllocator<T>, a DynamicMethod per (T, rank) emitting `newobj T[,...]::.ctor(int...)` over the NumSharp dims, each narrowed by the same ManagedLength check (so its InvalidOperationException surfaces unwrapped), cached in a per-T table. Measured ~37-52 ns vs Array.CreateInstance's ~215-240 ns (4.6-6x) for small rank-4..8 arrays; rank > 32 and hosts without dynamic code (NativeAOT) keep CreateInstance and its exceptions. 2. Lean hot path: the shape is read by reference; contiguous same-dtype = one Buffer.MemoryCopy from the logical start (Address + offset * itemsize - right both for re-seated contiguous slices and for offset views such as np.split children); one element = one assignment; a ONE-element conversion stays inline (NDIterCasting.ConvertValue); only the multi-element conversion setup is outlined ([NoInlining] ConvertInto), which keeps the frame of the tiny same-dtype calls small. 3. Strided same-dtype (CopyStrided -> CopyPlane per 2-D plane, odometer over the outer axes): unit column stride -> a memcpy per row; |row stride| < |column stride| (transposed / F-order) -> TransposeBands: 64-column bands walked down in 8-row strips, with 8x4 (8-byte) / 8x8 (4-byte) AVX register transposes when the source columns are element-contiguous; plain strided rows of 4/8-byte elements -> AVX2 stride-2 deinterleave (the [::2] case) or hardware gathers (any other non-zero stride, negative included); everything else (1/2/16-byte elements, stride 0) -> an indexed scalar loop. The SIMD paths only MOVE lanes, so NaN payloads, signalling NaNs and -0.0 survive bit-exactly, and no load touches memory outside the view (the deinterleave's final block is scalar for that). 4. Conversion: ConvertInto -> NDIter.Copy into a non-owning wrapper over the pinned result (astype's own cast core, any layout; the wrapper's block has Disposer.Null, nothing to free) - except a contiguous decimal -> double, which runs DecimalToDoubleKernel: an AVX2 replica of the runtime's (double)decimal, RN(RN(lo64) + hi32 * 2^64) / 10^scale then the sign (the 2^52 magic-number split makes each 32-bit half exact, so the recombination rounds once), enabled only where a start-up probe proves it bit-identical to the running runtime. Findings worth keeping (measured during the search): - Square tiles thrash L1 on power-of-two pitches: every destination row of a 1024-pitch f64 matrix maps to the same L1 set, so a 32x32 tile keeps 32+ lines of one set live and goes L2-bound. 64-column bands walked in 8-row strips keep only the 8 destination lines being filled live (f64 1024^2 transposed: 0.105 -> 0.85 of the copy-out floor). Full-width strip sweeps lose - the columns stop streaming. - .NET 8's ulong -> double rounds TWICE for values >= 2^63 (convert as signed, then add 2^64), so on .NET 8 the replica disagrees with (double)decimal on ~1.3% of random mantissas (51,409 / 4,000,000) while on .NET 10 it matches every one - hence the runtime probe (mantissas >= 2^63, 96-bit mantissas, scale 28, negative zero). On .NET 8 the probe fails and astype's scalar conversion runs. - Never outline the ONE-element conversion: doing so cost the 0-d conversion ~40% (0.55 -> 0.32 of the floor). Outlining the multi-element setup is what lifts the tiny same-dtype calls. - The byte-strided cell is bimodal for a pointer-increment row loop (0.117 <-> 0.213 across windows) and steady for the indexed loop kept here (0.20-0.21). Performance (search harness: paired ABBA timing pinned to one core; each cell is the ratio of a bare "allocate the result + memcpy its bytes" yardstick to the call, so 1.0 = the physical floor of a copy-out; old -> new): f64 1024^2 contiguous 0.52 -> 1.03 | u8 2048^2 0.51 -> 1.07 | bool 4M 1-D 0.51 -> 1.14 decimal 256^2 0.10 -> 1.01 | char 1024^2 0.41 -> 1.02 | f64 64^2 0.46 -> 0.98 | f64 3x4 0.66 -> 1.63 f64 1024^2 transposed 0.105 -> 0.85 | f32 1024^2 F-order 0.120 -> 0.67 | i32 [::2, ::2] 0.16 -> 0.63 f32 broadcast rows 0.17 -> 0.95 | i64 column slice 0.30 -> 0.96 | i32 5-element 1-D 0.18 -> 0.57 previously throwing: c128 512^2 1.03 | f16 1024^2 1.02 | 0-d f64 0.92 | i32 -> f64 1024^2 0.92 | f16 -> f32 512^2 0.95 | f64 -> f32 512^2 0.79 | decimal -> f64 256^2 0.78 Geomean speed 0.288 -> 0.926; peak memory = the result alone (was the result + a full-size flat temp). Remaining headroom: byte-wide strided views (~0.2), f32 transposes (~0.67), tiny calls (0.6-0.9). Tests - test/NumSharp.Tests/Casting/NDArray.ToMuliDimArray.Test.cs (62 cases): - all 15 dtypes x 18 plane layouts (transposed, F-order, [:, ::2], [:, ::3], [:, ::-1], [::-1, :], offset windows, stepped transposes incl. a negative column stride, row / column / scalar broadcasts, single row and column) on a (70, 150) base whose transpose has ragged strips, ragged block columns and a second 64-column band, plus 3-D permutations and 1-D stepped / reversed / offset views - each compared BYTE for BYTE (decimal by value, the oracle convention) with NumSharp's own C-order walk, and cross-checked through the (Array) cast; - 27 conversion pairs x every route (contiguous, transposed, strided, broadcast, one-element, 0-d, empty, 1-D >= 1024) against astype; - decimal -> double bit-identical to (double)decimal on 4,099 adversarial decimals through the kernel, its tail and the strided route; - 0-d; empty shapes up to rank 6 incl. decimal; ranks 4..32 through the emitted allocator for 6 element types; rank-5 permuted / stepped views; rank 33 -> TypeLoadException; a dimension > int.MaxValue at ranks 1/2/4 -> InvalidOperationException (broadcast views, nothing allocated); Guid / DateTime / nint -> NotSupportedException; the float16 / complex128 (Array) cast; result independence; NaN-payload / sNaN / -0 / subnormal / inf bits through every strided path for f64 / f32 / f16; 1-D lengths 1023 / 1024 / 1025 / 4099 around the uninitialized-allocation threshold. Teeth: against the pre-change implementation 43 of the 62 fail (ArrayTypeMismatchException, "Must provide at least one rank.", BlockCopy's ArgumentException); the 19 that pass are exactly what the old code handled (same-dtype layouts of the 13 primitive dtypes, the > int.MaxValue and rank-33 errors, f64/f32 special bits). docs/website-src/docs/NDArray.md gained the any-layout + conversion line and table note, pinned in NDArrayDocExamplesTests.Interop_DotNetArrays. Verification: full NumSharp.Tests (TestCategory!=OpenBugs&TestCategory!=HighMemory) net10.0 16323 passed / 0 failed / 11 skipped, net8.0 16322 / 0 / 11; the Oracle leak catalogue (Catalogue_EveryEntry_LeavesNoUndisposedIntermediates + LeakSurfaceCoverageTests) green on both TFMs. During the search every candidate passed a byte-exact correctness gate before it was timed: a 1,470-case optimize matrix and a 1,080-case held-out matrix (every dtype x layout x conversion x degenerate shape, ranks 0..40) - the winner passed both in full, and passed them again with DOTNET_EnableAVX=0 (the pure-scalar fallbacks). Provenance: /optimize-code run tomulidimarray-20260923-1544 (optimize-runs/, gitignored): 24 trials, 49 programs, 4 islands; winner P0048 (optimize 1470480.58 +/- 4.91 vs baseline 334183.50; held-out 1080454.59 +/- 2.7, best in the final same-window comparison). The rank >= 4 IL allocator follows the user's mid-run directive to replace reflection-driven activation with generated IL.
… Gist FFT/HPS cells strict on x64, Inconclusive-with-measurement on arm64; platform-independent rfft input with SHA-256 fingerprints What macos-latest still failed in run 35878504557 (fe30edf); windows/ubuntu were green: parity SpectrumLiveParityTests.Rfft_HarmonicSignal_* : "np.fft.rfft, n=1024: 751 of 1026 float64 lanes differ; first at lane 3: NumSharp=0xBD54E59CB4F4F299, NumPy=0xBD54E59CB4F4F298" (1 ULP). ecosystem GistSignalLiveParityTests.FrequencyPipelines_* (2/3 exact, max 20 ULP) and GistSignalSourceDemonstrationsLiveTests.SyntheticDemoHarmonics_* (element 3: NumPy 0x4077FFECF0F7C9E6 vs NumSharp ...C9FA, 20 ULP). Both are the SAME cell - the harmonic-product-spectrum estimate - over the same 1024-sample signal. Everything else passed on macOS, including the new ComplexAbs live tests (fe30edf's fused simd_cabsolute port is byte-exact on arm64 too), the MPS accelerator test, ORT 159x3, ML.NET 165x3 and both package-consumer scripts. Correction to fe30edf's narrative: it attributed the macOS HPS 20-ULP failure to complex abs. The failure persists with abs fixed, so it comes from rfft. The abs fix stands on its own evidence (35.5% of random complex values off on x64, 1M/1M after). Diagnosis: NumPy's arm64 wheels target a baseline ISA with fused multiply-add and are built with the compiler's default floating-point contraction (clang -ffp-contract=on on macOS; numpy/fft/meson.build passes no fp-contract flag). Every `a*b + c*d` written as ONE C expression becomes fmuladd(a, b, c*d) - clang's tryEmitFMulAdd fuses the LEFT product and rounds the right one first. pocketfft (numpy/fft/pocketfft/pocketfft_hdronly.h) spells exactly its hot arithmetic that way: the twiddle products in sincos_2pibyn::operator[], MULPM in every radf codelet, radf5's `x + c*y + d*z` rotations, and the complex codelets behind Bluestein. The x86-64 wheels target X86_V2 (no FMA), so the same source cannot fuse there (MSVC's /fp:contract has nothing to contract to), and RyuJIT never fuses - which is why NumSharp's scalar port equals x86-64 NumPy bit for bit. It is the mechanism already proven for NumPy's legacy Gaussian sampler (InteropTestBase.NumPyLegacyGaussianIsLiteral). Evidence (scratchpad replica, this session): a C# replica of rfftp (factorize, comp_twiddle, sincos_2pibyn, radf2/radf4/radf5, NumPy's r2c packing) in two modes. - literal mode == NumSharp's np.fft.rfft in every lane (n = 1024 and 1000); - fused per clang's rule changes 723 of 1026 lanes at n=1024 on the sin signal (macOS measured 751 - its signal itself differs, Apple libm sin vs ucrtbase, so its lane values cannot be matched from Windows); - on the NEW platform-independent input (below): 745/1026 lanes at n=1024, 887/1002 at n=1000. Predicted for the next macos-latest run (SHA-256 prefixes the test now prints): n=1024 input 8FEA7D41177CB101 NumSharp 5892C0B212DAB7AD arm64 NumPy 012CC43CDBA4187A n=1000 input B44EED2AE5B15E2C NumSharp 86696C2139124384 arm64 NumPy 22242EF71197D0DF A match proves the mechanism byte for byte (a NumSharp-hash mismatch would instead mean Apple libm's twiddle cos/sin differ from ucrtbase's, making the check inconclusive, not wrong). Changes 1. InteropTestBase (additive): - NumPyRoundsEachProduct: ProcessArchitecture is X64 or X86 (the in-process CPython loads the wheel built for the process; everything else counts as fusing - a byte-exact claim needs the guarantee). - PocketFftFusedArithmetic: the const naming the fused expressions, used in every message. - AssertExactUnlessNumPyFuses(cell, fusedArithmetic, assertExact): the comparison ALWAYS runs; on x86/x64 a failure propagates untouched (exception filter, stack trace intact); elsewhere an AssertFailedException becomes Inconclusive CARRYING the failure's message; any other exception propagates on every host. Deliberately not SkipByteExactOnArm64: an arm64 cell that happens to match still PASSES as a real byte check, and a mismatch leaves its measurement in the CI log. Verified: AwesomeAssertions 9.3.0 raises MSTest's AssertFailedException for both collection equality and numeric comparisons (probe), so the filter sees every assertion these cells make. 2. SpectrumLiveParityTests.Rfft_*: - Input = PortableHarmonicSignal(n): the Gist demo's 7 harmonics of 384 Hz @ 8192 Hz + seeded System.Random noise (amplitude 5e-4), quantized to 2^-20 -> the same bytes on every host (quantizing erases the libm last-bit differences; seeded Random is integer arithmetic + one scale). Without the noise the quantized harmonics repeat every 64 samples exactly, most bins become exact zeros, and contraction moved only 50 of 1026 lanes - the noise keeps every bin live. - All three sizes (1024 radix-4, 1000 mixed 2/4/5, 1021 Bluestein) are measured before the decision, so one arm64 log carries all three. - AssertSameDoubles split into DescribeMismatch (null when byte-identical, else lane count, first lane's two bit patterns, SHA-256 prefixes of input/NumSharp/NumPy; a byte-LENGTH difference still asserts on every host - no arithmetic difference explains it) + the AssertSameDoubles wrapper. The two ComplexAbs tests keep their strict assertion on every architecture. 3. GistSignalLiveParityTests: - FrequencyPipelines: autocorrelation now asserted separately and STRICT on every host (at this size SciPy's correlate picks its direct method, np.convolve, which reduces through OpenBLAS ddot; NumSharp's sliding-dot seam routes the same positions to the same bundled OpenBLAS - it matched on macOS); FFT + HPS go through AssertExactUnlessNumPyFuses. - LegacyHps: wrapped (every pass is rfft-derived; it matched on macOS this time and still passes there when it matches). 4. GistSignalSourceDemonstrationsLiveTests: SyntheticSine = crossings strict + FFT wrapped; SyntheticDemoHarmonics = crossings + autocorrelation strict, FFT + HPS wrapped (expected[[0,2]] / expected[[1,3]] against separately built NumSharp arrays). 5. Docs: test/NumSharp.Tests.Interop/CLAUDE.md section 4 "arm64 contraction trap" (when and how to wrap a cell); .claude/CLAUDE.md FFT section (arm64 NUMPY is the exception, not NumSharp; stale docs/FFT_PARITY.md path -> docs/stale-docs/FFT_PARITY.md); website compliance.md FFT section (the byte-parity reference is x86-64 NumPy; arm64 NumPy's own bins sit an ULP or so away). Verified locally (Windows x64, AVX2+FMA3, net10.0): SpectrumLiveParityTests 3/3 in the parity venv; GistSignal* 14/14 in the ecosystem venv with NUMSHARP_PYTHONNET_REQUIRE_PACKAGES=1; Release build of both TFMs clean. The arm64 behaviour, and the hash check above, is the CI run on this commit.
…replica reproduces both macOS hashes); correct "NumSharp returns the x86-64 bytes on every host" - twiddles come from the platform libm Run 35890253503 (cc8d56c) is green on all six jobs (test x3, interop-test x3). On macos-latest the wrapped cells reported Inconclusive with their measurements, as designed: np.fft.rfft n=1024 748/1026 lanes input 8FEA7D41177CB101 NumSharp 319422554ED4EAB6 NumPy 2FA2BA5D8FE0417A np.fft.rfft n=1000 882/1002 lanes input B44EED2AE5B15E2C NumSharp F494345710A79842 NumPy 483E089585F3D64E np.fft.rfft n=1021 999/1022 lanes input 7013E8BADA3F806B NumSharp 4040ABD5372D44D6 NumPy 68287B11EDCE0101 Gist FrequencyPipelines FFT/HPS: HPS 20 ULP; SyntheticDemoHarmonics: HPS 20 ULP, FFT estimate equal; LegacyHps and SyntheticSine matched and PASSED on arm64; autocorrelation and crossings strict and passed. The input hashes equal the Windows ones (the platform-independent input works). cc8d56c's predicted RESULT hashes did not match, and on NumSharp's side too: NumSharp on macOS produced different bytes from NumSharp on Windows for identical input. The only host-dependent input to NumSharp's rfft is its twiddle table (Math.Cos/Math.Sin -> the platform libm; NumPy uses std::cos/std::sin, the same libm). Proof, both sizes byte for byte: 1. Correctly rounded twiddles (mpmath at 200 bits over every angle sincos_2pibyn::calc can form; Windows ucrtbase misses correct rounding on 45 cos + 21 sin of the 1025 n=1024 angles and on 33 + 28 of the 1001 n=1000 ones). n=1000: the literal replica gives F494345710A79842 (= macOS NumSharp), the fused replica 483E089585F3D64E (= macOS NumPy), 882 lanes apart (= macOS). 2. n=1024: search over one-ULP perturbations of the table's 140 (angle, cos|sin, +-1) candidates, singles then pairs, matching ONLY NumSharp's hash -> exactly one hit: Apple libm's sin(72*pi/4096) and sin(216*pi/4096) are one ULP above correctly rounded. That table's FUSED mode then gives 2FA2BA5D8FE0417A = macOS NumPy's hash, with nothing fitted to it. So on arm64: NumSharp == pocketfft evaluated literally; NumPy == pocketfft with clang -ffp-contract=on (fmuladd, left product fused) at the twiddle products (sincos_2pibyn::operator[]), MULPM (every radf codelet) and radf5's rotations. n=1021 (Bluestein, complex codelets) has no replica yet; same source pattern, 999/1022 lanes apart. Doc corrections. cc8d56c said NumSharp returns the x86-64 bytes on every host, and that is false for large n. Both stacks take their twiddles from the platform libm, and the libms disagree in the last bit on some angles: - InteropTestBase.PocketFftFusedArithmetic remarks: the proof and the four hashes replace the "same share of lanes" argument. - SpectrumLiveParityTests (Rfft remarks): reproducing a host's hashes needs that host's libm twiddle values; result hashes are not host-independent, NumSharp's included. - test/NumSharp.Tests.Interop/CLAUDE.md, arm64 contraction trap: 73-98 % of lanes (all three sizes), and how the hashes were used and what they need. - .claude/CLAUDE.md FFT section: "proven"; new TWIDDLE TRAP. fft.jsonl (win-amd64 bytes) is portable only because ucrtbase, glibc and Apple agree on its sizes' twiddles (green on all three OSes). A larger corpus size can turn host-pinned. - website compliance.md FFT section: NumSharp evaluates pocketfft the x86-64 way, with platform-libm twiddles. It matches the NumPy beside it on x86-64 (live on Windows and Linux); arm64 NumPy fuses; large-transform bits can differ between hosts, NumSharp's too. Comments and docs only, no behavior change. Interop test project rebuilds clean (Release, both TFMs). Replica, table generator and search live in the session scratchpad: rfft_contract_probe{2,3,4}.cs and cr_twiddles.py.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
journey4carries everything committed afterjourney3(#628) was merged: 155 commits dated 2026-09-08 → 2026-09-18, touching 586 files (+288,157 / −42,366). Of those lines, +144,555 / −39,649 are the regenerated.jsonloracle corpora, so code, tests and docs account for +143,602 / −2,717. The work was accidentally committed straight onto a localmasterand never pushed; it has been moved here unchanged (same SHAs, tipb779d953) so it goes through CI and review like the previous journey branches.Draft: opened to run CI and start review. Nothing in this branch has been release-noted yet.
✨ New APIs
numpy.ma, the masked-array module (newMaskedArraytype): ufunc wrappers, masked reductions / operators / creation / extrema / manipulation, the extras (average,median,dot, set operations,sort/unique,cov/corrcoef,convolve/correlate,polyfit,apply_along_axis/apply_over_axes), theMaskedArrayindexer, hard-mask semantics and byte-exact maskedstr/repr(first commit5316a614; parity audit and gap registere425740c).np.emath: automatic-domainsqrt,log,log2,log10,logn,power,arccos,arcsin,arctanh(1755e1b9).np.*functions (none had a public declaration onmaster), all targeting NumPy 2.4.2 parity:hypot,divmod,fmod,remainder,float_power,heaviside,spacing,frexp,ldexp,gcd,lcm,sinc,fabs,signbit,fix,i0,real_if_close,unwrap,trapezoid,gradient,binary_repr,base_reprbartlett,blackman,hamming,hanning,kaiserhistogram,histogram_bin_edges,histogramdd,histogram2dpackbits,unpackbits,bitwise_countpiecewise,putmask,put_along_axisbroadcast_shapes,apply_along_axis,apply_over_axes,vectorize,frompyfuncarray_equiv,may_share_memory,shares_memorylogspace,geomspacetypenameout=/where=onmin/max(d41eb4ab);out=/where=/dtype=onleft_shift/right_shift,logical_and/or/xor/notandmodf(59f99320).np.array_equal(equal_nan=)(68081a74).np.round/aroundwithdecimals != 0, a port ofPyArray_Round(9071b539).np.indices/indices_sparseacceptint[]/long[]/Shape(abc02351).NPY_TYPESenum andDType.type_num(60df1061).np.evaluate/ NDExpr Phases 0, 1 and 3 (merged fromexprs,45c68405), plus a structural program cache with 0-d inputs passed as kernel parameters (670afdd1).ValueErrorinstead ofNumSharpException(60024b44).ValueErrorderives fromArgumentException, socatch (NumSharpException)no longer catches these writes.finfo.dtype/iinfo.dtypereturnDType(wasNPTypeCode); the interop dtype maps takeDType(5e3d6c31).🐛 NumPy-parity fixes
ab69cf8a,dbc0b3b3,58cd55dc).a43dc6fd).sum/prodpath (ac95ae85).cbrt,floor,ceil,trunc,deg2rad,rad2deg,floor_divideandmodraises NumPy's exactTypeError(ae43ba13);IncorrectTypeExceptionnow derives fromTypeError(3ad43600).owndata=True) (db4ab83f).arctan2(complex)crash (0c5d72c0),np.chooseindex leak (35dd5eb7),!operator on non-contiguous layouts (10a49f68), edge cases in the histogram family,gradient,trapezoidandgeomspace(7c3e14d8,23c45213,cb74b5ff,281fece0),PyLiteralrepr of non-printable chars (2ba16e76).np.mabugs surfaced by the new masked-array corpora: a strided-mask read bug (b14ff3a0) and a heap-corruption crash inmedianof an empty array (e7f73845).⚡ Performance
670afdd1).array_equal/array_equiv(b53f5e54).sumfor int32 → int64 and uint32 → uint64 (fcdacd29).np.hypot: correctly rounded Borges FMA with a SIMD fast path (831b664c).📦 Interop, collections & examples
NumSharp.Interop.ParquetNet(NDArray ↔ Apache Parquet) with its test project (1b82a7c9,42cffc42).OrtValues (sequence / map / string) plus more coverage (merged fromonnxruntime,179b2f6a).src/NumSharp.Core/Collections/Concurrent/:ConcurrentOrderedDict,ConcurrentOrderedCompactDict, the lock-freeOrderedDictand an experimentalConcurrentPointingDict, plus aSystem.Collections.ConcurrentAPI proposal (e3d5b772).NumSharp.GistExamples(10 NumPy gist ports + a Karpathy suite, parity-gated), a Unity N-body gravity sandbox, and a falling-sand game with a standalone terminal player (3710d260,77e419b0,2b67f296,6847cd7f).🧪 Oracle & tests
9a368db8,9af8fa36,08d5d13a, and the per-feature commits).846f53aa,ff7283f7,35dd5eb7).np.madifferential corpus, two passes (b14ff3a0,e7f73845).writeableschema field (3ceb6fa9).d35955ac).📚 Docs & build
60a0ea6a,d58207ed,d2f04679,6bc7d4a0,ff84f7f5,2b57d07d,36fc1aed).cd39ef5d,6a88d55c,4bfcde52,b779d953).bea2f602,13b7e0d3).refs/parquet-dotnetandrefs/TensorSharp2. Both pinned commits exist upstream, and no workflow checks out submodules, so they cannot affect CI.Review notes
45c68405(exprs) and179b2f6a(onnxruntime).0c47162eand moved back to theexprsbranch in89066194; they are not in the net diff.