Migrate the selection base classes and helpers to narwhals, add polars support

[solegalli]

Summary

Migrates the shared selection code to narwhals so the selectors can be migrated one by one in follow-up PRs, with pandas and polars both supported:

• BaseSelector.transform() returns the retained features in the column order of the train set, using the same library as the input. For pandas this is X[features], and for other backends it is a narwhals select. _get_feature_names_in() accepts native or narwhals frames.
• BaseRecursiveSelector.fit() hands native frames to sklearn, and polars frames go to sklearn directly. It now returns (nw_X, y), matching BaseNumericalTransformer._fit_setup. Variable selection reuses _select_numerical_variables.
• base_selection_functions no longer imports pandas:
  • find_correlated_features: the correlation matrix is computed with numpy (see benchmarks). When values are missing, each pair is still compared on the rows where both values are finite, as DataFrame.corr() does. This is vectorised with matrix products. Pairs where that sum is numerically unstable (for example a variable that is almost constant on the shared rows) are recomputed one by one. Kendall and callables loop over the pairs as pandas does, and for pandas input they still call .corr(). The grouping loop is vectorised with numpy and returns identical groups.
  • single_feature_performance: builds a one-column native frame per feature.
  • find_feature_importance and BaseRecursiveSelector.feature_importances_(_std_): these are a pandas Series indexed by the features for pandas input, as before, and a dict {feature: value} for other backends (see Needs decision).
• The threshold error message now follows the convention: threshold must be an integer or a float. Got {threshold} instead.

The individual selectors are untouched.

Benchmarks

Median of 3-5 alternating runs, times in ms. Other jobs were running on the machine, so treat small differences as noise.

find_correlated_features, whole function (main = X[variables].corr()):

method	missing	rows × cols	main (pandas)	this PR (pandas)	this PR (polars)
pearson	no	10k × 50	21.9	0.8
pearson	no	100k × 50	259	8.9
pearson	no	100k × 200	3920	42.9
pearson	no	500k × 50	1218	34.9	59.3
pearson	no	500k × 200	19994	282	667
pearson	no	1M × 50			108
pearson	5%	10k × 50	32.4	6.5
pearson	5%	100k × 50	213	25.7
pearson	5%	100k × 200	3229	140
pearson	5%	500k × 50	951	101	127
pearson	5%	500k × 200	15271	651	1342
pearson	5%	1M × 50			573
spearman	no	100k × 50	2339	814
spearman	no	500k × 10	476	460	83.7
spearman	no	500k × 50	10384	2940	494
spearman	5%	500k × 10	7621	7266	3188

Candidates tried for each piece, and which one won:

• pearson, no missing values: pandas .corr (1218 ms at 500k × 50) vs np.corrcoef (31 ms) vs polars DataFrame.corr (54 ms, which is corrcoef underneath) vs polars pl.corr expressions per pair (102 ms). numpy wins.
• pearson with missing values: pandas .corr (926 ms at 500k × 50) vs narwhals per-pair expressions (4614 ms on polars) vs a numpy per-pair loop (5210 ms) vs vectorised numpy (101 ms pandas / 109 ms polars). Vectorised numpy wins, on both backends.
• spearman, no missing values: pandas .corr (8956 ms at 500k × 50) vs pandas rank + corrcoef vs scipy rankdata + corrcoef (2905 ms), and on polars narwhals rank + corrcoef (79 ms at 500k × 10) vs rankdata (476 ms) vs pl.corr per pair (600 ms). Winners: scipy for pandas, narwhals rank for polars.
• spearman with missing values, kendall, callables: pandas keeps .corr(), which is 6x faster than narwhals expressions on pandas. Polars uses narwhals per-pair expressions for spearman (3452 ms vs 3821 ms for a numpy loop at 500k × 10) and a numpy per-pair loop for kendall and callables.
• Grouping loop: vectorised numpy is 0.16 ms vs 6.6 ms at 200 variables, and 0.5 ms vs 56 ms at 1000 variables.

BaseSelector.transform, whole transform (half the columns dropped, input columns in reverse order):

backend	rows × cols	main X[fni].drop()	X[keep]	narwhals select(keep)	narwhals select(fni).drop()	polars X.select(keep)
pandas	100k × 50	0.191	0.112	0.416	0.827
pandas	500k × 200	0.245	0.141	1.340	2.791
polars	500k × 50			0.117	0.223	0.097
polars	1M × 200			0.347	1.012	0.281

pandas keeps its own path. For polars, narwhals select is within 0.02-0.07 ms of native polars, so the non-pandas path uses narwhals.

Frames handed to cross_validate (10 columns, 3-fold CV; time for all features):

helper	estimator	backend	rows	pandas X[f].to_frame() / X[[f]]	narwhals get_column().to_frame()	polars select(f)	numpy
single_feature_performance	tree	pandas	100k	840 / 879	823		785
single_feature_performance	logreg	pandas	500k	1753 / 1664	1723		1621
single_feature_performance	logreg	polars	1M		4037	3855	2901
all variables	logreg	polars	1M		689		542

sklearn's fit dominates the timing, and how the frame is built makes no measurable difference. So one narwhals path is used for both backends. Passing numpy arrays is 10-30% faster with polars. It is not used because a user's estimator can be a pipeline that selects columns by name, which would break without the dataframe.

Behaviour

• Old and new outputs were recorded on main for 95 pandas cases. These cover transform with reordered columns, NaN, and every drop pattern; variable selection with and without confirm_variables; all 4 correlation methods at 3 thresholds, with and without NaN, including constant and integer columns; single_feature_performance and find_feature_importance with int, splitter, generator and groups CV and a numpy y; and BaseRecursiveSelector with RF, logistic regression, KNN (permutation importance), a variables subset, confirm_variables and a generator CV. All 95 are identical on this branch.
• Polars gives the same values as pandas in 93 of the 94 cases that apply (integer column names are pandas only). The exception is the error raised for an unknown variable (see below).
• find_correlated_features was stress-tested on 1773 random datasets: ties, 2-20% missing values, inf, nullable Int64, shuffled variable order, 4 methods including a random callable, and 3 thresholds. Groups are identical between main and the pandas path, and between pandas and polars.
• numpy computes the correlation values in a different order from pandas, so they can differ from pandas' in the last bits: about 1e-15 for well-scaled data, and up to 1e-10 in extreme cases (a column around 1e5 with a spread of 1e-6). In those extreme cases the vectorised result is closer to an exact two-pass computation than pandas' own. Only the groups are returned, so the result changes only if a correlation lies within that distance of the threshold.

Tests

• test_base_selector.py and test_base_selection_functions.py are rewritten to the conventions, and test_base_recursive_selector.py is new. Every test runs on pandas and polars through make_df, targets are built with make_series, and there is one test each with list and numpy targets. pandas-only tests cover integer column names, the pandas index, and the Series type of the importances. data_classification (a dict) is added to tests/test_selection/conftest.py; the existing pandas fixtures stay for the selector tests.
• Failing tests in tests/test_selection, run one file at a time: 177 on the base ref, 141 on this branch. No new failures, 36 fixed: DropFeatures, SelectByInformationValue and SelectBySingleFeaturePerformance now pass, as do some check_estimator, DropCorrelatedFeatures, SmartCorrelatedSelection and MRMR tests.
• tests/parametrize_with_checks_selection_v16.py has the same failures before and after.
• flake8 feature_engine tests is clean. mypy shows the same 2 errors as the base ref, in datetime_subtraction.py and log.py.

Expected failures fixed by the follow-up selector PRs

None: no test that passed on the base ref fails on this branch.

Needs decision

• feature_importances_ / feature_importances_std_ for non-pandas input (RFE, RFA, and find_feature_importance used by ProbeFeatureSelection). pandas users still get a pandas Series indexed by feature name. For polars I used a dict {feature: importance}: it keeps the feature names, needs no pandas, and matches the other dict attributes in the library. The alternatives are a polars DataFrame with feature and importance columns, or a dict for every backend, which would break code that relies on the pandas Series.

Pre-existing issues, not fixed

• check_numerical_variables(polars_df, ["Age", "Hola"]) raises narwhals ColumnNotFoundError for a column that is not in the dataframe, while pandas raises KeyError. The difference is in variable_handling, and _select_numerical_variables just passes it on.