feat: add lazy normalization for Lasso and Elastic Net

[jolars]

Closes #479

Current behavior

Lasso copies the design matrix to center and scale its columns. Elastic Net also stores an augmented design matrix with an identity block for the L2 penalty. These allocations increase memory use during fitting.

New expected behavior

The optional lazy-normalization feature lets Lasso and Elastic Net apply normalization to a borrowed design matrix through lazymatrix. Elastic Net also applies the identity block without storing the augmented matrix. Both DenseMatrix and ndarray-bindings are supported.

Enable the feature, then select the lazy path with LassoParameters::default().with_lazy_normalization(true) or ElasticNetParameters::default().with_lazy_normalization(true). The eager path remains the default, including when another dependency enables the feature. The existing normalize option still controls whether columns are centered and scaled. Parameter searches use a fixed lazy_normalization setting.

The lazy path reduces memory use but adds repeated arithmetic. Fit time can increase for both estimators, including Elastic Net on large inputs. The solver still stores a dense Gram matrix with one row and column per feature. Comparative time and memory benchmarks in smartcore-benches remain pending. The local benchmark example and results CSV have been removed.

This PR covers dense matrices only. Sparse matrix support remains separate from this change, pending #455.

Breaking change and release target

This PR targets 0.7 because it includes a breaking source compatibility change. Enabling lazy-normalization adds a public lazy_normalization field to LassoParameters, LassoSearchParameters, ElasticNetParameters, and ElasticNetSearchParameters. Downstream struct literals without ..Default::default() must add lazy_normalization: false to retain the eager path, or true to select the lazy path. This also applies when another dependency enables the feature through Cargo feature unification. Missing fields default to false during deserialization.

lazymatrix 0.6.0 declares Rust 1.85 support and an MIT OR Apache-2.0 license. It is an optional dependency with default features disabled.

The ndarray from_iterator fix is in #483. This branch still includes it because the ndarray tests need it. Once #483 merges, this branch must be updated from main to remove the fix from this PR's diff.

Change logs

• Add lazy matrix products, Gram construction, and runtime selection for Lasso and Elastic Net, using the existing numeric and matrix traits.
• Add tests that compare lazy and eager products and fits across matrix layouts, scalar types, normalization settings, and penalty options. Add tests for numerical edge cases, parameter searches, and serialization defaults.
• Add CI builds and tests for the feature alone and with ndarray-bindings. Existing all-feature jobs also cover MSRV and WebAssembly.
• Add README guidance and a changelog entry.

Validation

The following commands passed locally on NixOS with Rust 1.99.0:

cargo fmt --all -- --check
cargo build --all-features
cargo test --all-features
cargo clippy --all-features -- -Drust-2018-idioms -Drust-2024-compatibility -Dwarnings

The test run passed 612 unit tests, 34 integration tests, and 75 doctests. It reported five unfulfilled lint expectations in test builds outside the changed files. MSRV and WebAssembly checks were not run locally for this update.

[Mec-iS]

Thanks @jolars, this is a well-tested PR and the memory reduction is real. I reviewed the diff (head 80d6e12) and the CI state. I have not built or run it locally; CI was still running when I looked (coverage, wasm32, x86_64 Linux, macOS, Windows x86_64 and both lazy-normalization check_features jobs were in progress; lint, MSRV 1.85 and i686 were green). Findings are ordered by importance.

What looks good

• Public estimator APIs are unchanged, and the eager optimize path is preserved by refactoring into optimize_with_products with closures.
• The memory numbers are consistent with the arithmetic. At 65,536x32 f64 the design is ~16.8 MiB. Eager Lasso holds 2 copies (37.6 MiB) and eager Elastic Net holds 3 (53.7 MiB: original, scaled, augmented). Lazy sits at ~21 MiB, which is roughly one copy plus baseline.
• The tests are thorough: eager-vs-lazy products, both layouts, f32/f64, ndarray, panel boundaries (255/256/257), non-finite values, large offsets (1e12) and constant columns.
• Centering before accumulation, so small variation at large offsets is preserved, is the right call.

Blocking / should address

1. The feature is not additive. lazy-normalization replaces the eager path with #[cfg(feature)] / #[cfg(not(feature))]. Cargo unifies features across the dependency graph, so any transitive crate enabling it silently changes performance and numerics for every user of Lasso/ElasticNet. A user who wants the eager path (faster for Elastic Net, see 2) can no longer get it. Please consider making both paths always compiled when the feature is on, selected by a runtime option (e.g. a ElasticNetParameters/LassoParameters setting), or at least gating only the dependency and keeping the choice explicit. It would also remove the ~12 duplicated cfg blocks in fit, which are hard to maintain.
2. Elastic Net regresses in speed. From your CSV, lazy is slower at every size: 0.174s -> 0.289s at 65536x32 (+66%), 0.029 -> 0.049 at 8192x32 (+68%), 0.0033 -> 0.0056 at 1024x32 (+72%). Lasso is break-even only at the largest size. The likely cause is (raw - center) / scale computed per element in every matvec and transpose-matvec of every interior-point iteration, plus strided get_col access on row-major DenseMatrix. Options: precompute 1/scale (note this changes rounding vs eager, so loosen the exact-equality tests), cache a normalized column-panel, or document the trade-off per estimator. The README currently says "can be slower, especially on smaller inputs", but the data show Elastic Net is slower at large sizes too.
3. Eager path can get the same savings without a new dependency. Eager Elastic Net makes two full copies (rescale_x then augment_x_and_y). Building the augmented matrix directly from x plus (center, scale) removes one copy, and in-place or fused scaling would remove another. Worth quantifying how much of the 53.7 -> 21.8 MiB gain this alone captures, since it would benefit everyone with no feature flag. Lazy remains the right design for the sparse/out-of-core goal (Epic: Data layer — CSR sparse support, slices vs Vec in Array traits, parquet IO #455), but the motivation for this PR on dense data should be compared against that baseline.
4. New external dependency. lazymatrix 0.6.0 is a new, young crate. Please confirm its license is compatible with smartcore's (Apache-2.0), its MSRV (the MSRV job does not build with lazy-normalization, so a 1.85 break could slip through), and the maintenance plan. Notably, a large share of the logic (panel Gram, adapter traits, UnitWeights, MatrixOutput) lives in smartcore, so the crate provides mostly LazyMatrix::from_parts and trait definitions. Please explain what is gained over a ~100-line in-tree LazyDesign.
5. from_iterator fix should be its own PR and in the CHANGELOG. The old axis != 0 branch did reversed_axes() then a RowMajor reshape, which is wrong for rectangular input, so the fix is correct. But it is a user-visible behaviour change on a public trait impl (Array2::from_iterator for ndarray::Array2) unrelated to lazy normalization and is not listed in CHANGELOG.md (only in the PR text). Please split it out, add a ### Fixed entry, and double-check callers that pass axis = 1 don't rely on the old behaviour. The result is now an F-order array, so any code assuming a contiguous row-major slice could be affected.

Suggestions

• Panics in library code. check_vector_length uses panic_any, and centers()/scales() call .unwrap() (they panic on a design built without normalization). Given the recent direction (predict method on DecisionTreeRegressor panics when tree is not fit #469/Decision tree panic #470: return Err(Failed) instead of panicking), prefer Result or Option for these, even for pub(super) helpers.
• Unit weights in the Gram. gram() goes through the generic weighted kernel with UnitWeights, so every panel does an extra n*p multiply by 1.0 and a VectorView call. A dedicated unweighted path is simpler and cheaper. The bundle also over-generalizes (weights, ScaledSubSlice, SumEntries, etc. that the solver doesn't use).
• Exact float equality in tests. panel_gram_preserves_eager_accumulation_across_boundaries uses assert_eq! on floats, tied to the eager ab() summation order. A future matmul optimization or FMA differences across targets (wasm, aarch64) would break it without any real bug. A tight relative tolerance would keep the intent.
• Looser tolerances. f32 fit tests use 1e-2 / 5e-3; fine, but please note why (conditioning of the interior-point solver) in a comment.
• Pre-existing solver instability at zero L1. The test comment says the solver is unstable at a zero L1 penalty. This looks like a real, separate issue; consider opening an issue instead of leaving it only in a comment.
• solve_mut(&self.ata, ...). Previously x was passed and (I believe) ignored by the optimizer's mat_vec_mul. Passing &self.ata is harmless, but please confirm the BiconjugateGradientSolver implementation doesn't read the matrix argument elsewhere.
• README placement. The new paragraph is inserted directly under the "Data access and readers" heading, before the CSV bullets, so it reads as part of that section. It belongs under a features or performance section.
• Benchmark artefacts. I'd keep examples/lazy_normalization.rs (it's useful), but move lazy_normalization_results.csv out of examples/ (e.g. to the PR description or docs), since it's machine-specific and can't be reproduced without the environment. Please record the CPU, OS, rustc and --release flags, and note that results use 2-3 repeats per process so timings are noisy. A criterion bench would be more robust long-term.
• CI coverage. Add lazy-normalization to the wasm32 and MSRV jobs, since the tests carry wasm_bindgen_test attributes but are not exercised there.
• PR text. "The review reported that the all-features build, test suite, and required Clippy checks passed" is unclear about who ran what. Please replace it with the exact commands you ran (e.g. cargo test --all-features, cargo clippy ...) so reviewers can reproduce.

Scope

On your question about sparse support: I'd recommend keeping this PR dense-only and landing sparse in a follow-up after #455 is settled. On the example: keep it.

Happy to re-review once the feature-gating design (1) and the Elastic Net slowdown (2) are addressed. @Mec-iS, you were requested as a reviewer, so tagging you for the dependency and design decisions.

[Mec-iS]

check smartcore-benches repo for performance comparison.

[jolars]

Thanks for the review! I have fixed the feature-unificiation problem. I'm looking into the elastic net performance regression. But some performance regression is to be expected, given that this is essentially a trade-off between memory and speed (in the dense matrix case).

[jolars]

@Mec-iS - do you want me to break the from_iterator fix out into its own PR? It is already a separate commit.

[jolars]

check smartcore-benches repo for performance comparison.

Thanks, but can you please clarify what you mean? That I should use criterion here as well for the benchmarks? These were intended mostly to give you an idea of the performance tradeoff, but I can add Criterion as a depenency here if you'd like?

That repo doesn't have any lasso or elastic net benchmarks, but if you'd like me to add those to that repo I'd be happy to do it too. But I'm not sure if it would make a difference for this PR, or?

[Mec-iS]

yes, for me to accept these changes I need complete information about how/what they improve in quantified terms. So we need to run comparative benchmarks in smartcore-benches, we need a "dev" branch with both arms implemented and two benches that measure the same exercise in both arms.
We can accept/reject a trade-off only if it is thoroughly measured first.
Also we need to be careful to not disrupt usage downstream, so probably this will be something to go with the new minor 0.7. You have plenty of time to open different PRs here and run the benches.

[codecov]

Codecov Report

❌ Patch coverage is 52.39726% with 139 lines in your changes missing coverage. Please review.
✅ Project coverage is 63.47%. Comparing base (9eaae9e) to head (d122cc4).
⚠️ Report is 196 commits behind head on main.

Files with missing lines	Patch %	Lines
src/linear/lazy_normalization.rs	45.39%	77 Missing ⚠️
src/linear/lasso.rs	54.38%	26 Missing ⚠️
src/linear/elastic_net.rs	65.00%	21 Missing ⚠️
src/linear/lasso_optimizer.rs	58.62%	12 Missing ⚠️
src/linalg/ndarray/matrix.rs	40.00%	3 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##             main     #480       +/-   ##
===========================================
+ Coverage   43.97%   63.47%   +19.49%     
===========================================
  Files          85       97       +12     
  Lines        7281     8797     +1516     
===========================================
+ Hits         3202     5584     +2382     
+ Misses       4079     3213      -866     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[jolars]

Got it! I'll remove the example and run a benchmark through your benchmark suite instead. I'll also add benchmarks for elastic net/lasso there in a separate PR.

[jolars]

I've improved code coverage a bit too, but the missing coverage is mainly due to the currrent call to taupaulin missing some parts. I'm filing a separate PR to update that workflow.

[jolars]

I've now run the benchmarks in smartcorelib/smartcore-benches#9. I compared #483 with both the eager and lazy paths in this PR. Using #483 as the baseline gives both versions the ndarray iterator fix.

Lazy was faster in all 24 cases, by 14-51% for Lasso and 13-57% for Elastic Net. I didn't see a slowdown above 2% with the eager path, which is still the default.

Here are the results for 65,536 rows and 32 columns with a row-major DenseMatrix:

Estimator	Baseline fit	PR eager fit	PR lazy fit	Baseline peak RSS	PR eager peak RSS	PR lazy peak RSS
Lasso	486 ms	494 ms	389 ms	37.8 MiB	37.9 MiB	22.1 MiB
Elastic Net	406 ms	404 ms	338 ms	53.9 MiB	53.9 MiB	22.1 MiB

That's about 42% less peak memory for Lasso and 59% less for Elastic Net. These memory numbers cover the whole process, including input creation.

I wasn't expecting it to be faster, so I checked Gram construction separately. For the same size and layout, it takes 197 ms with eager Lasso versus 21 ms with lazy, and 155 ms versus 21 ms for Elastic Net. Across all cases, lazy Gram construction was 86-89% faster. The resulting Gram matrices were exactly equal.

This seems to explain the speedup. The lazy kernel computes one triangle and mirrors it, and it works on small panels. The eager kernel computes both triangles separately. Elastic Net also avoids storing and traversing the identity block. I haven't measured those changes individually. The Gram savings are larger than the total fit savings in the large row-major case, so lazy appears to spend more time elsewhere. Improving the eager Gram kernel could recover much of this benefit too.

I also compared the fitted models on every dataset. The baseline and PR eager path gave exactly equal coefficients, intercepts, training MSE, and objective values. Between eager and lazy, the largest coefficient difference was 1.2e-8, the largest prediction difference was 3.9e-8, and the largest relative objective difference was 2.9e-10.

For reference, I ran this on an AMD Ryzen 9 7900 with NixOS 26.11 and Rust 1.99.0, using Cargo release defaults and pinning the runs to logical CPU 2. Criterion used 30 samples, a one-second warmup, and three seconds of measurement time, extended where needed. Data creation was outside the timed loop, but model destruction was included. The memory figures are medians from five fresh processes per version and case, alternating between versions.

The benchmarks cover 1,024, 8,192, and 65,536 rows with 32 columns, plus 1,024 rows with 128 columns. Each uses row-major and column-major DenseMatrix inputs and row-major ndarray inputs. All fits use f64, normalization, alpha = 0.1, tol = 1e-4, and max_iter = 1000. Elastic Net uses l1_ratio = 0.5.

The commits were 04f4dfc for the baseline, 459f808 for this PR, and bf9b24b for the benchmark suite. Both PR builds enable lazy-normalization, but only the lazy run sets with_lazy_normalization(true). I discarded an initial lazy run after finding that Cargo had reused the eager binary from a shared build directory. The results here use a separate lazy build directory with an assertion that the option is enabled.

This is one full timing run per version on one machine, with fixed synthetic data. I'd expect the trade-off to depend on the data and hardware, but these results look promising. The full timing table is below.

Full fit-time results (means in milliseconds)

Estimator	Layout	Size	Baseline	PR eager	PR lazy	Lazy change vs. baseline
ElasticNet	DenseMatrix column-major	1024x128	78.843	78.282	36.895	-53.2%
ElasticNet	DenseMatrix column-major	1024x32	7.257	7.310	5.373	-26.0%
ElasticNet	DenseMatrix column-major	65536x32	391.561	389.736	243.802	-37.7%
ElasticNet	DenseMatrix column-major	8192x32	53.112	53.604	37.557	-29.3%
ElasticNet	DenseMatrix row-major	1024x128	78.107	77.998	37.525	-52.0%
ElasticNet	DenseMatrix row-major	1024x32	7.253	7.339	5.430	-25.1%
ElasticNet	DenseMatrix row-major	65536x32	406.007	404.056	337.514	-16.9%
ElasticNet	DenseMatrix row-major	8192x32	53.600	53.902	46.528	-13.2%
ElasticNet	ndarray row-major	1024x128	77.491	77.004	33.496	-56.8%
ElasticNet	ndarray row-major	1024x32	7.286	7.359	4.703	-35.5%
ElasticNet	ndarray row-major	65536x32	437.151	441.881	235.117	-46.2%
ElasticNet	ndarray row-major	8192x32	53.906	54.634	33.160	-38.5%
Lasso	DenseMatrix column-major	1024x128	72.581	73.635	38.284	-47.3%
Lasso	DenseMatrix column-major	1024x32	8.595	8.631	6.272	-27.0%
Lasso	DenseMatrix column-major	65536x32	471.201	473.871	291.757	-38.1%
Lasso	DenseMatrix column-major	8192x32	62.546	62.778	40.370	-35.5%
Lasso	DenseMatrix row-major	1024x128	76.163	74.489	39.871	-47.7%
Lasso	DenseMatrix row-major	1024x32	8.007	7.966	6.648	-17.0%
Lasso	DenseMatrix row-major	65536x32	485.787	493.686	388.632	-20.0%
Lasso	DenseMatrix row-major	8192x32	58.086	58.218	49.781	-14.3%
Lasso	ndarray row-major	1024x128	70.928	71.784	34.648	-51.2%
Lasso	ndarray row-major	1024x32	7.942	8.015	5.525	-30.4%
Lasso	ndarray row-major	65536x32	472.281	438.661	281.123	-40.5%
Lasso	ndarray row-major	8192x32	54.914	55.307	35.906	-34.6%

Negative changes mean faster fits.