DAOS-17681 tools: fix cont clone of EC containers with holes (#18866) - #19089
Merged
Merged
Conversation
cont clone enumerated the extents of an akey with daos_obj_list_recx() and copied that range verbatim. Two problems: The listing and the fetch both ran at DAOS_TX_NONE, so they could be served at different epochs even though the tool creates a snapshot for the OIT. Read the source under a snapshot transaction instead. For an EC object the listing is served by a parity shard, where one parity block stands for a whole stripe, so the returned extents are an upper bound and can cover punched records. The fetch leaves those holes untouched and still reports the full length, because only trailing holes shorten it, so the sg_nr_out check could not detect it and uninitialized bytes were written to the destination. Fetch with an io map and copy the extents it reports. Also range check the extents and record size before deriving buffer offsets from them, and propagate a transaction close failure to the exit code. Document in daos_obj.h that daos_obj_list_recx() returns an upper bound for EC objects, that the fetch reply length cannot detect a partial result, and that the listing and the fetch should share a transaction. Test: daos_test -I -u 32,33,34,35 covers the copy and verifies the destination matches the source extent for extent. datamover/obj_ec.py runs the same over the real command on an EC container with punched extents. Adds IORequest.punch_array() and an object class and punch option to dataset_gen(). Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@hpe.com>
mchaarawi
added
the
clean-cherry-pick
|
Ticket title is 'daos container clone: daos_obj_fetch unexpectedly returns 0 sg_nr_out' |
github-actions
Bot
added
the
priority
mchaarawi
marked this pull request as ready for review
September 16, 2026 12:48
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
cont clone enumerated the extents of an akey with daos_obj_list_recx() and copied that range verbatim. Two problems:
The listing and the fetch both ran at DAOS_TX_NONE, so they could be served at different epochs even though the tool creates a snapshot for the OIT. Read the source under a snapshot transaction instead.
For an EC object the listing is served by a parity shard, where one parity block stands for a whole stripe, so the returned extents are an upper bound and can cover punched records. The fetch leaves those holes untouched and still reports the full length, because only trailing holes shorten it, so the sg_nr_out check could not detect it and uninitialized bytes were written to the destination. Fetch with an io map and copy the extents it reports.
Also range check the extents and record size before deriving buffer offsets from them, and propagate a transaction close failure to the exit code.
Document in daos_obj.h that daos_obj_list_recx() returns an upper bound for EC objects, that the fetch reply length cannot detect a partial result, and that the listing and the fetch should share a transaction.
Test: daos_test -I -u 32,33,34,35 covers the copy and verifies the destination matches the source extent for extent. datamover/obj_ec.py runs the same over the real command on an EC container with punched extents. Adds IORequest.punch_array() and an object class and punch option to dataset_gen().
Steps for the author:
After all prior steps are complete: