Preserve weights through pickling and deepcopy - #313
Open
vahid-ahmadi wants to merge 1 commit into
Open
Conversation
weights was a plain instance attribute, so it was absent from the pickle
state:
pickle.loads(pickle.dumps(MicroSeries([1, 2], weights=[3, 4]))).sum()
# AttributeError: 'MicroSeries' object has no attribute 'weights'
Declaring _metadata = ["weights"] fixes it: pandas includes _metadata
attributes in the pickle state.
MicroDataFrame needed one more step. Its weighted aggregations are
installed as per-instance closures by override_df_functions, which only
runs in __init__ — a path unpickling skips. So an unpickled frame kept
its weights but mdf.sum() fell through to the unweighted pandas
implementation and returned 6 instead of 14, with no error. __setstate__
now reinstalls them.
This covers the pickle half of #300; the _constructor rework that would
stop sort_values/fillna/sample from dropping weights is still open there.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This was referenced Aug 31, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Partially addresses #300 (the pickle half — see the note at the end).
Problem
weightswas a plain instance attribute, so it was not part of the pickle state:That breaks
to_pickle/read_pickleand anything shipping a weighted object across a process boundary — joblib,multiprocessing, Dask.MicroDataFramehad a second, quieter failure underneath it. Its weighted aggregations are installed as per-instance closures byoverride_df_functions, which only runs in__init__— a path unpickling skips. So even once weights were restored, an unpickled frame'ssumwas no longer overridden and fell through to pandas' unweighted implementation:Change
_metadata = ["weights"]on both classes. pandas includes_metadataattributes in the pickle state, so weights round-trip.MicroDataFrame.__setstate__reinstalls the aggregation overrides after unpickling.Verification
Three new tests; all three fail on
main(AttributeErrorfor the Series,6 != 14for the frame) and pass with the change. Also verified by hand:pickleround-trip (MicroSeries)pickleround-trip (MicroDataFramesum)to_pickle/read_picklecopy.deepcopygroupby(...).sum()after unpickleNon-default indexes covered.
make test— 65 passed on pandas 3.0.5 and pandas 2.3.3.Scope
This is deliberately the low-risk half of #300. Declaring
_constructor/_constructor_sliced— which is what would stopsort_values(ascending=False),fillnaandsamplefrom returning weightless plain pandas objects — is a bigger change: every pandas-internal construction would then runMicroDataFrame.__init__, includingoverride_df_functions, so it needs a performance pass and a decision about how weights should be reindexed for row-changing operations. I've left #300 open for that, and noted there that the per-instance-closure design is what makes it awkward.