[python] Use conditional OSS puts for atomic metadata writes

[wangzhigang1999]

Purpose

Fixes #9714.

Concurrent PyPaimon writers to OSS can both pass the destination existence check and overwrite the same snapshot during temporary-file-and-rename publication. Both commits can return successfully while only one remains visible.

Introduce a thin OssFileIO(PyArrowFileIO) that overrides atomic metadata creation with OSS PutObject and x-oss-forbid-overwrite=true, following Java's approach in #8228. FileAlreadyExists returns False; other OSS failures propagate with their cause. Ordinary Arrow/Jindo operations remain inherited. Route the factory, REST token refresh and ResolvingFileIO through the OSS implementation, and migrate internal OSS callers.

Forward Java-aligned SSE options for these metadata PUTs, including KMS key/data encryption and the legacy algorithm fallback. Add oss2>=2.18,<3 to the optional oss and jindo extras; OSS atomic writes require explicit endpoint/credential options. The README documents setup and the encryption scope.

Conditional creation applies only to buckets that have never enabled versioning. Enabled/suspended/unknown states and GetBucketVersioning access denial fall back to the inherited write path with a warning, preserving legacy behavior without claiming concurrent-write protection or new SSE behavior for that fallback. Other query failures propagate. All writers must use conditional creation, and bucket versioning must remain disabled.

Tests

• After rebasing onto master 8f5ce6b84: 242 passed (208 related regression cases plus 34 OSS protocol cases), Flake8 and git diff --check passed. Local runtime: Python 3.12.6, PyArrow 19.0.1, oss2 2.19.1.
• The same 208 regression cases passed against a clean baseline and the implementation. The 34 protocol cases also passed with the minimum oss2 2.18.0. They exercise real SDK HTTP requests, concurrent creation/content preservation, lost responses, error propagation, versioning fallback, refreshed STS headers, URI handling and SSE validation.
• Real OSS, same settings per version: 64 independent processes × 300 rounds, 19,200 attempts with 64 KiB payloads. Baseline: 300 rounds with multiple successful creators and 7,545 overwritten successful writes. New: exactly one creator per round, zero overwritten successful writes.
• Real append commits: 16 processes × 10 commits × 2,000 rows, commit.max-retries=64 in both runs. Baseline: 160 completed calls, 232,000 rows and 116 snapshots. New: all 320,000 rows and 160 snapshots, no missing/duplicate rows. Lost-response recovery controls passed in both versions.
• Real OSS ordinary FileIO operations passed for both factory and Resolving routes. Seven SSE scenarios passed with the change (AES256, legacy AES256, KMS, KMS+SM4, SM4, option precedence and an explicit OSS-managed KMS key); encryption headers were absent in the baseline. Catalog/Resolving metadata writes also produced encrypted schema/snapshot objects.

The real OSS runs used a never-versioned bucket. Versioning/access-denied fallback and STS forwarding were validated locally; real Jindo, STS renewal, customer-managed KMS permissions and other object stores were not exercised. An extra Arrow read through ResolvingFileIO reproduced the same existing missing-filesystem AttributeError in both versions; reading the same committed data through the ordinary route succeeded. That reader issue is unchanged and outside this PR.

[Copilot]

🟡 Changes recommended

OssFileIO.try_to_write_atomic derives the OSS object key incorrectly for credential-in-URI OSS forms, which can write metadata to the wrong object path.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR fixes concurrent snapshot/metadata commit overwrites for OSS in PyPaimon by introducing an OSS-specific atomic-create path that uses the OSS SDK’s conditional PutObject (x-oss-forbid-overwrite=true) instead of the existing temp-file-and-rename publication flow.

Changes:

• Add OssFileIO (subclassing PyArrowFileIO) to implement conditional metadata writes via oss2, with bucket-versioning detection and a legacy fallback.
• Route OSS selection and REST token refresh / ResolvingFileIO atomic writes through the OSS implementation; update internal OSS call sites accordingly.
• Add OSS SSE option forwarding for conditional PUTs, extend optional extras to include oss2, document configuration, and add protocol-level tests.

File summaries

File	Description
paimon-python/setup.py	Adds oss2 to oss and jindo extras for OSS atomic metadata writes.
paimon-python/README.md	Documents OSS atomic metadata commits, versioning fallback semantics, and SSE option behavior.
paimon-python/pypaimon/tests/py36/ao_simple_test.py	Switches OSS initialization tests to use OssFileIO.
paimon-python/pypaimon/tests/oss_legacy_mode_test.py	Updates legacy-mode tests to target OssFileIO while keeping legacy assertions.
paimon-python/pypaimon/tests/oss_file_io_test.py	Updates OSS integration-style tests to instantiate OssFileIO.
paimon-python/pypaimon/tests/oss_atomic_write_test.py	New protocol tests exercising conditional PUT behavior, SSE headers, failures, and fallback.
paimon-python/pypaimon/tests/lance_utils_test.py	Uses FileIO.get(...) for OSS so the correct implementation is selected.
paimon-python/pypaimon/tests/file_io_test.py	Updates OSS-related unit tests to use OssFileIO.
paimon-python/pypaimon/sample/rest_catalog_blob_as_descriptor_sample.py	Uses FileIO.get(...) for external OSS IO selection.
paimon-python/pypaimon/filesystem/resolving_file_io.py	Forwards try_to_write_atomic to the scheme-resolved underlying FileIO.
paimon-python/pypaimon/filesystem/oss_file_io.py	Implements OSS conditional atomic creation and SSE header resolution.
paimon-python/pypaimon/common/options/config.py	Adds OSS SSE-related config options for atomic metadata PUTs.
paimon-python/pypaimon/common/file_io.py	Selects OssFileIO for oss:// URIs in FileIO.get(...).
paimon-python/pypaimon/catalog/rest/rest_token_file_io.py	Uses FileIO.get(...) so OSS paths pick up the OSS atomic-write implementation.

Review details

• Files reviewed: 14/14 changed files
• Comments generated: 2
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.