Antalya 26.6 - backport of 115603 - read row_lineage(iceberg v3) fields as virtual columns

[subkanthi]

Changelog category (leave one):

• New Feature

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Iceberg v3 has two new fields in manifest file - first_row_id and last_seq_num. This PR allows to read them as virtual columns from system tables(system.iceberg_files)

CI/CD Options

Exclude tests:

• Fast test
• Integration Tests
• Stateless tests
• Stateful tests
• Unit tests
• Performance tests
• Aarch64 tests
• All with ASAN
• All with TSAN
• All with MSAN
• All with UBSAN
• All with Coverage
• All Regression
• Disable CI Cache

Regression jobs to run:

• Fast suites (mostly <1h)
• Aggregate Functions (2h)
• Alter (1.5h)
• Benchmark (30m)
• CAS (content-addressed storage; Antalya only)
• ClickHouse Keeper (1h)
• Iceberg (2h)
• LDAP (1h)
• OAuth (5m)
• Parquet (1.5h)
• RBAC (1.5h)
• SSL Server (1h)
• S3 (2h)
• S3 Export (2h)
• Swarms (30m)
• Tiered Storage (2h)

[github-actions]

Workflow [PR], commit [40f8bfb]

[subkanthi]

@blau-ai

[blau-ai]

CI triage for #2308 (Iceberg v3 row_lineage backport)

Verdict: 7 failing checks — 0 caused by this PR. All are pre-existing base-branch failures, flaky/infra, or unrelated test-maintenance. The PR's own build & tests are green (all Builds, Fast test, Unit tests, gtest_iceberg_count_shortcuts, and stateless 04203_system_iceberg_files all pass; the added test_row_lineage*.py are not among the failures).

The strongest single piece of evidence: I compared against #2309 ("Bump to 26.6.4"), a trivial version bump that contains none of this Iceberg code. It reproduces the same integration, cas‑s3 stateless, and settings regression failures — so those are properties of the antalya-26.6 base/CI environment, not of this diff. The diff itself is entirely under src/Storages/ObjectStorage/DataLakes/Iceberg/*, ProtocolDefines.h, VirtualColumnUtils, and StorageObjectStorageSource — none of which the failing suites exercise.

---

1. Integration tests (amd_asan_ubsan, db disk, old analyzer, 4/8) — 12/1102 ❌ (blocking) → pre-existing, not PR-caused

All 12 failures are test_storage_iceberg_with_spark/test_schema_inference.py::test_schema_inference[...] (the s3/azure/local × 1/2 × True/False matrix). The first case crashes the server on SELECT * FROM <iceberg table> with ATTEMPT_TO_READ_AFTER_EOF (stack: throwReadAfterEOF → Connection::receivePacket); the remaining 11 then fail with Connection refused (…:9000) NETWORK_ERROR because the server is already dead.

• test_schema_inference.py is a pre-existing test (this PR only added test_row_lineage.py / test_row_lineage_pruning.py).
• Antalya 26.6: Bump to 26.6.4 #2309 fails the exact same 12 tests with the identical crash-then-connection-refused signature (only the version string 26.6.4 vs 26.6.2 and the minio IP differ). → This is a base-branch / spark-iceberg test-env instability on antalya-26.6, reproduced by any PR retargeted onto it.
• Next step: track separately as a base-branch flaky/crash (worth a server-log dig by whoever owns the iceberg-spark integration env); re-running won't clear it, but it is not a regression from this backport.

2. Stress test (amd_debug) — 1/5 ❌ (blocking) → not PR-caused (flaky/base instability)

Single failure: Lost connection to server (server crash, no stack in report; fatal.log empty). Stress test (amd_asan_ubsan) passed 0/4, and Stress (amd_debug) passes on #2309. No evidence tying it to the Iceberg diff.

• Next step: safe to re-run; if it recurs, pull clickhouse-server.err.log from the stress job to confirm it's the same base-branch crash.

3. Stateless tests (amd_asan_ubsan, cas s3 storage, parallel, 1/2) — Failed: 1 ❌ (non-blocking, do_not_block_pipeline_on_failure) → infra, not PR-caused

Failure is Connection refused (e.code=111) to the cas‑s3 backend during a plain CREATE TABLE … ENGINE=MergeTree (non-Iceberg test), which then took the server down. Flaky cas‑s3 storage backend; suite is explicitly non-blocking.

4. Stateless tests (amd_binary, cas s3 storage, parallel) — Failed: 2 ❌ (non-blocking) → flaky, not PR-caused

Both CI-labeled flaky:

• 02265_column_ttl — NETWORK_ERROR on a ReplicatedMergeTree part relink (13/15 same-settings reruns failed → flagged flaky by CI).
• 03629_duplicate_partition_keys_crash — S3 Connection refused to the cas_s3 bucket; all 6 reruns passed (transient).
  Neither touches Iceberg. Antalya 26.6: Bump to 26.6.4 #2309 shows the same cas‑s3 suites red with a shifting set of tests → environmental.

5. RegressionTestsRelease / CAS (selects) / cas_selects ❌ → effectively passed / flaky

The /selects module is OK (~99.9%); the only non-OK rows are 2 known XFails (/final/modifier "not implemented", "select union clause with alias" "need to add retry"). Zero real fails; the red status is a first-attempt/retry artifact. Passes on #2309.

6. RegressionTestsRelease / Common (settings) / settings ❌ → not PR-caused (snapshot maintenance)

SnapshotNotFoundError in default_values.py: the regression snapshot has no entry for a new server setting export_merge_tree_part_ignore_extra_source_columns (actual {"default":"0"}). That setting comes from the separate export … partition_by_name work, not from this Iceberg backport (this PR adds no server settings — it only bumps protocol-version constants in ProtocolDefines.h). Also fails on #2309.

• Next step (separate task, not this PR): regenerate the settings regression snapshot to include the new setting.

7. Regression release tiered_storage_minio ❌ (module errored) → infra teardown timeout

ExpectTimeoutError: Timeout 1000.000s in helpers/cluster.py __exit__ — the docker-compose cluster teardown hung after ~1h57m. All 90 executed scenarios passed (4 XFail). Passes on #2309.

• Next step: safe to re-run.

---

Recommendation

No code change to this PR is warranted — nothing in the red CI is attributable to the row_lineage diff, and the PR's own tests pass. The settings regression snapshot regen and the pre-existing test_schema_inference iceberg-spark crash on antalya-26.6 are real but belong to separate base-branch/test-maintenance tracks. The flaky/infra suites (stress, cas‑s3 stateless, cas_selects, tiered_storage_minio) are safe to re-run.

Analysis is evidence-first from the praktika S3 reports and a control comparison with #2309; I cannot build or run ClickHouse in this environment. Head SHA c9a0c51.

[subkanthi]

TESTING
Altinity/ice#214

ice insert --format-version=3 -p flowers.irisv3 file://iris.parquet

CH

 SELECT first_row_id,sequence_number FROM system.iceberg_files where database=currentDatabase() and table='flowers.irisv3';

SELECT
    first_row_id,
    sequence_number
FROM system.iceberg_files
WHERE (database = currentDatabase()) AND (`table` = 'flowers.irisv3')

Query id: c0012f98-d809-4a40-a3bc-799284d4a19a

   ┌─first_row_id─┬─sequence_number─┐
1. │            0 │               1 │
   └──────────────┴─────────────────┘

1 row in set. Elapsed: 0.075 sec. 

After inserting 150 rows, ice does Append

insert --format-version=3 -p flowers.irisv3 file://iris.parquet

SELECT first_row_id,sequence_number FROM system.iceberg_files where database=currentDatabase() and table='flowers.irisv3';

SELECT
    first_row_id,
    sequence_number
FROM system.iceberg_files
WHERE (database = currentDatabase()) AND (`table` = 'flowers.irisv3')

Query id: 8e405717-c67b-4944-945c-dbff2240ddf0

   ┌─first_row_id─┬─sequence_number─┐
1. │          150 │               2 │
2. │            0 │               1 │
   └──────────────┴─────────────────┘

[ianton-ru]

Code the same as in original PR, but I recommend also to keep the same format (indents, empty lines, brackets) to avoid additional conflicts in future. I did not mark all places with different formatting.

[ianton-ru]

This block a little bit reformatted (removed empty lines, brackets). This does not change something, but can create additional conflicts in future backports instead of automatic solving during git merge/git cherry-pick.

[ianton-ru]

The same as above - block is reformatted (different indent).

[ianton-ru]

Also - different indents.