Skip to content

Migrate RecursiveFeatureAddition to narwhals, add polars support - #1083

Open
solegalli wants to merge 4 commits into
narwhals-migrationfrom
narwhals-recursive-feature-addition
Open

solegalli wants to merge 4 commits into
narwhals-migrationfrom
narwhals-recursive-feature-addition

Conversation

@solegalli

Copy link
Copy Markdown
Collaborator

Stacked on #1070 (selection base classes). Until #1070 is merged, its commits also show in this diff. The commit for this PR is the last one.

Summary

Migrates RecursiveFeatureAddition to narwhals, so it works with pandas and polars:

  • fit uses BaseRecursiveSelector.fit, which returns (nw_X, y). Each round's model is trained on a native frame: X[features] for pandas, and nw_X.select(nw.col(*features)).to_native() for other backends. The cross-validation call is shared by the baseline model and the loop in a small _cross_validate method.
  • The importance ranking no longer uses Series.sort_values, because feature_importances_ is a dict for polars. It uses numpy with exactly the same ordering as sort_values(ascending=False), including for tied importances (see Behaviour). The attribute is then rebuilt with _importance_series, so it stays a pandas Series for pandas input and is a dict otherwise, as in the base class.
  • transform is inherited from BaseSelector.
  • Docstring: fit no longer says "pandas dataframe", and there is a polars example. The shared feature_importances_ / feature_importances_std_ attribute docstrings in _docstrings/fit_attributes.py now describe the Series-or-dict container, using the same wording as BaseRecursiveSelector. RFE uses these docstrings too.
  • User guide: added a "With polars" section, and updated outputs that no longer matched what the code returns: initial_model_performance_, and the drift values, which now print as np.float64(...) with numpy 2.

Benchmarks

Cross-validation takes almost all of the fit time. The selector's own work is building the frame for each round (one per feature, with a growing list of columns) and ranking the importances. Times are medians, in ms. Other jobs were running on the machine, so the whole-fit numbers are noisy.

Column subsetting, summed over all rounds of one fit:

backend rows × cols narwhals select(nw.col(...)).to_native() native (X[cols] / polars X.select(cols)) numpy to_numpy()
pandas 10k × 10 5.9 1.6 11.3
pandas 10k × 50 52.4 7.2 105.9
pandas 100k × 10 4.7 2.1 23.1
pandas 100k × 50 80.4 7.0 458.7
pandas 500k × 10 2.7 0.9 24.0
pandas 500k × 50 57.7 7.7 2210
polars 500k × 10 1.4 0.5 51.1
polars 500k × 50 35.6 16.6 887
polars 1M × 10 0.6 0.4 53.8
polars 1M × 50 36.2 12.1 1008
polars 5M × 50 39.3 19.1

Whole fit (LinearRegression, cv=3):

backend rows × cols this PR alternative
pandas 10k × 10 255 287 (narwhals subsetting)
pandas 100k × 10 526 583 (narwhals subsetting)
pandas 100k × 30 2141 2351 (narwhals subsetting)
pandas 500k × 10 3236 5941 (narwhals subsetting)
pandas 500k × 30 14577 11538 (narwhals subsetting)
polars 500k × 10 7294 5305 (polars X.select)
polars 500k × 30 15759 17281 (polars X.select)
polars 1M × 30 17163 21582 (polars X.select)
  • pandas: native X[cols] is 2-10x faster than narwhals for the subsetting, and the whole fit is about 10% faster at 10k-100k rows. At 500k rows the whole fit is dominated by noise, in both directions. pandas keeps its own path.
  • polars: native select is 10-25 ms faster per fit than narwhals, but a fit takes 5-20 s. That difference is below the whole-fit noise, which goes both ways in the table. So polars uses the narwhals path, and there is no third branch.
  • numpy is the slowest way to build the inputs, and it would break estimators that select columns by name, such as pipelines with a ColumnTransformer.
  • Ranking: numpy argsort is 2-20 µs for 10-100 features, against 18-36 µs for pandas sort_values, whether or not the Series is rebuilt afterwards. Both are negligible, so one numpy path is used for both backends.

Behaviour

  • Outputs were recorded on main for 18 pandas cases: RF / logistic regression / Lasso / decision tree / linear regression; int, KFold and generator CV; GroupKFold with groups; threshold keeping all and dropping all; a variables subset; confirm_variables; a categorical column; KNN (permutation importance); NaN with HistGradientBoosting; integer column names; and 25 features with tied importances (Lasso and a shallow tree). For each case I compared features_to_drop_, performance_drifts_(std_), feature_importances_(std_) including their order, initial_model_performance_, and transform on reordered columns. All 18 are bit-identical on this branch.
  • Polars gives bit-identical values to pandas in the 17 cases that apply (integer column names are pandas only). The only difference is the container of feature_importances_(std_): a dict for polars, as decided in Migrate the selection base classes and helpers to narwhals, add polars support #1070.
  • Tied importances: pandas sort_values uses quicksort, which is not stable for more than 16 values. So with many features, tied features are not ranked in column order. A stable Python sorted() would have changed the ranking, and so the selection, for any model that gives several features the same importance (for example Lasso zeros or unused tree features). The numpy code reproduces pandas' ranking exactly, and test_ranking_of_features_with_equal_importance covers it.
  • cross_validate is no longer called with return_estimator=True, because the fitted estimators were never used. Scores are unchanged.

Tests

  • test_recursive_feature_addition.py is rewritten to the conventions: init error tests with the full message, test_init_param_assignment, and one test per behaviour on pandas and polars through make_df. Targets are built with make_series, with one test for list and numpy targets. Expected values are explicit, floats are compared with pytest.approx, and outputs are checked with isinstance + frame_to_dict. The diabetes data now has its real column names, so it runs on polars. The pandas-only tests cover integer column names and the Series type of the importances.
  • test_recursive_feature_selectors.py: I removed only the RFA parts, which are now covered in the RFA file: RFA from _selectors, and the RFA blocks of the combined tests. The RFE PR is migrating RFE in parallel and edits the same file. The two PRs touch neighbouring lines, so whichever merges second may need a small rebase.
  • Failing tests in tests/test_selection, one file per pytest call: 137 on narwhals-selection-base, 123 on this branch. No new failures. The 8 old RFA tests, the 2 RFA test_fit_initial_model_performance cases and the 4 RFA check_estimator cases now pass. tests/parametrize_with_checks_selection_v16.py has the same result before and after: 289 failed, 204 passed.
  • flake8 feature_engine tests is clean. mypy feature_engine gives the same 2 errors as the base, none in this module.

Needs decision

solegalli and others added 4 commits September 19, 2026 11:44
…s support

BaseSelector.transform() returns the retained features in the train set
order, in the same library as the input (pandas X[features], narwhals
select otherwise). BaseRecursiveSelector.fit() trains the estimators on
native frames and returns (nw_X, y). The helpers in
base_selection_functions no longer import pandas: correlations are
computed with numpy (np.corrcoef, or matrix products for pairwise
complete observations when there are missing values), and feature
importances are pandas Series for pandas input and dicts otherwise.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant