Antalya 26.6 - Fixed refreshing of glue metadata after ALTER calls

[subkanthi]

#2085

Changelog category (leave one):

• Bug Fix (user-visible misbehavior in an official stable release)

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Fixed an issue where Glue DataLakeCatalog tables could retain stale schema metadata after ALTER TABLE operations.

CI/CD Options

Exclude tests:

• Fast test
• Integration Tests
• Stateless tests
• Stateful tests
• Performance tests
• Aarch64 tests
• All with ASAN
• All with TSAN
• All with MSAN
• All with UBSAN
• All with Coverage
• All Regression
• Disable CI Cache

Regression jobs to run:

• Fast suites (mostly <1h)
• Aggregate Functions (2h)
• Alter (1.5h)
• Benchmark (30m)
• CAS (content-addressed storage; Antalya only)
• ClickHouse Keeper (1h)
• Iceberg (2h)
• LDAP (1h)
• OAuth (5m)
• Parquet (1.5h)
• RBAC (1.5h)
• SSL Server (1h)
• S3 (2h)
• S3 Export (2h)
• Swarms (30m)
• Tiered Storage (2h)

[github-actions]

Workflow [PR], commit [61c7e4a]

[subkanthi]

@blau-ai

[blau-ai]

CI triage — PR #2272

Verdict: 0 failures. CI is not red — it just hasn't produced results yet.

The current head (15fbfac, the "Merge branch 'antalya-26.6' into …" commit) triggered a fresh run (32988139141) that started only a few minutes ago. As of now:

• Config Workflow → ✅ OK
• Fast test, Source upload → 🟡 running
• All builds / stateless / integration / unit / stress / fuzzer jobs → ⏳ pending (or SKIPPED for the arm/tsan/msan variants this PR's config excludes)
• DCO → ✅ pass

There are no failure/error check-runs, and the praktika report (result_pr.json) shows the workflow as RUNNING with nothing failed. The earlier commit 6c7b9aa has no uploaded report, so this is the first real CI pass for the branch — there's simply nothing to classify as PR-caused vs. pre-existing/flaky yet. Re-run this triage once the run finishes and I'll break down any failures.

---

One-paragraph health check of the change

The diff is tightly scoped to the reported bug (src/Databases/DataLake/GlueCatalog.cpp + one integration test). GlueCatalog::updateSchema previously only rewrote metadata_location and dropped new_schema on the floor (/*new_schema*/), which is exactly why Glue kept serving a stale StorageDescriptor.Columns after ALTER … ADD/DROP COLUMN; the fix now rebuilds the TableInput and pushes the columns via the new setGlueColumnsFromIcebergSchema / icebergTypeToGlueType helpers (with recursive array/map/struct handling and iceberg.field.* column parameters), and createTable now also stops discarding metadata_content. The change looks coherent and the added test_schema_evolution_show_create_and_drop is a real reproducer (SHOW CREATE + DROP COLUMN round-trip). It's gated behind the Glue integration test, which needs a full build + running server — so correctness will only be proven by CI on this run, not statically. Two things worth a glance while CI runs: (1) getCurrentSchemaFromMetadata calls metadata->getValue<Int32>(f_current_schema_id) / getArray(f_schemas) unguarded — fine for well-formed Iceberg metadata, but it will throw on a metadata object missing those keys; (2) updateSchema now constructs a fresh TableInput rather than read-modify-write, so make sure it isn't dropping any table parameters that the previous updateMetadata path preserved (it sets metadata_location + table_type, matching createTable, so this is likely intentional). Neither is a blocker — just verify the test_database_glue integration shard goes green.

Automated triage by @blau-ai — no build/test run locally; verdict is based on GitHub check-runs + the praktika S3 report for 15fbfac.

[subkanthi]

TESTING

alter table datalake_db.`db1.table22` add column column_c Nullable(String);

ALTER TABLE datalake_db.`db1.table22`
    (ADD COLUMN `column_c` Nullable(String))

Query id: 6d3a20fc-21f7-45e5-a87b-239aa14bad8c

Ok.

0 rows in set. Elapsed: 0.148 sec. 

Ubuntu-2404-noble-amd64-base :) show create table datalake_db.`db1.table22`;

SHOW CREATE TABLE datalake_db.`db1.table22`

Query id: 5b699d67-2300-49ea-9fe3-91199ff0cff1

   ┌─statement──────────────────────────────────────────────────────┐
1. │ CREATE TABLE datalake_db.`db1.table22`                        ↴│
   │↳(                                                             ↴│
   │↳    `name` Nullable(String),                                  ↴│
   │↳    `double` Nullable(Float64),                               ↴│
   │↳    `integer` Nullable(Int64),                                ↴│
   │↳    `column_a` Nullable(String),                              ↴│
   │↳    `column_b` Nullable(String),                              ↴│
   │↳    `column_c` Nullable(String)                               ↴│
   │↳)                                                             ↴│
   │↳ENGINE = Iceberg('http://localhost:9000/bucket1/db1/table22/') │
   └────────────────────────────────────────────────────────────────┘

1 row in set. Elapsed: 0.004 sec. 

[DimensionWieldr]

CI Failures Analysis

Verdict: no PR-caused regressions. Head 61c7e4a6, run 33412728540.

The PR’s own test test_database_glue/test.py::test_schema_evolution_show_create_and_drop PASSED in targeted integration.

Related to this PR

None.

Pre-existing / unrelated

• test_schema_inference (12 params, asan/ubsan) — known #2216. First param kills the server; the rest are cascades. Same fails on sibling PRs / MasterCI.
• test_cas_gcs/...::test_a_write_whose_response_carries_no_generation_is_refused — CAS GCS assertion, also on PR Antalya 26.6: Added support for alter add column with first/after #2289.
• Settings export_merge_tree_part_ignore_extra_source_columns / export_merge_tree_part_schema_match_mode — SnapshotNotFoundError after #2284; regression snapshots not updated yet. Same 2 fails on Antalya 26.6: Added support for alter add column with first/after #2289.
• Stress test (amd_debug) Cannot start clickhouse-server — Hybrid currentDatabase() restart bug, #1993. asan_ubsan stress passed.
• 04068_lwd_minmax_projection_recovery_rmt + Server died — CAS S3 asan LWD mutation (FILE_DOESNT_EXIST on dropRef). Cascade after that. amd_binary CAS S3 was green.

Flaky / infra

• /tiered storage/with cas/background move — other tiered jobs passed.
• cas_s3_cache_alter_attach_1 — 157 attach-partition fails; non-CAS alter_attach_1 passed.
• Cancelled long jobs (cas_alter_attach_1, LWD 2/2 shards, stateless CAS S3 1/2) — timeouts, not test failures.

Recommendations

1. Merge-ready from this PR’s Glue catalog changes.
2. Do not expect iceberg test_schema_inference to go green on rerun (antalya-26.6: test_schema_inference kills the server on amd_asan_ubsan — upstream Decimal-bounds overflow exposed by #2145 #2216).
3. Settings red clears once clickhouse-regression snapshots include the two export-part settings from Antalya 26.6 allow non matching schema export partition by name #2284.

[DimensionWieldr]

Ran https://github.com/Altinity/clickhouse-regression/blob/main/iceberg/tests/iceberg_engine/alter_support.py on Glue catalog, and all passed.

LGTM

[ianton-ru]

Comments from AI:

AI audit note: This review comment was generated by AI (Cursor Grok 4.6).

Audit update for PR #2272 (Glue DataLakeCatalog schema refresh after ALTER):

Confirmed defects:

Medium: `updateSchema` / `createTable` fail-open on schema conversion, then still commit Glue
    Impact: `ALTER`/`CREATE` can succeed while Glue `StorageDescriptor.Columns` stays stale or empty. That is the original bug (`SHOW CREATE` wrong, later `DROP COLUMN` can fail with `NOT_FOUND_COLUMN_IN_BLOCK`). Iceberg metadata on S3 is already written; Glue catalog metadata is left inconsistent.
    Anchor: `src/Databases/DataLake/GlueCatalog.cpp` / `GlueCatalog::updateSchema`, `GlueCatalog::createTable`, `updateTableInGlue`
    Trigger: Any exception in `icebergSchemaToGlueColumns` / `getCurrentSchemaFromMetadata` (malformed or unexpected Iceberg type JSON). `catch (...)` logs a warning, leaves `columns` empty, then `updateTableInGlue` skips `SetColumns` and still updates `metadata_location`.
    Why defect: ClickHouse treats Glue columns as the catalog schema for `SHOW CREATE` and `DatabaseDataLake::tryGetTable`. Swallowing conversion errors and continuing is fail-open on the path this PR claims to fix. REST `updateSchema` fails closed on non-retryable errors.
    Fix direction (short): Do not call Glue update/create after conversion failure; throw (or return false for retry). Prefer `metadata` / `parseTableSchemaV2Method` before giving up.
    Regression test direction (short): Inject a conversion failure (bad nested type or failpoint) on `ALTER ADD COLUMN` and assert the catalog commit fails, Glue columns unchanged, and `SHOW CREATE` does not report success with a stale schema.

Low: Iceberg metadata schema fallback is only used when Glue columns are empty
    Impact: After this PR writes a non-empty `StorageDescriptor`, a stale Glue schema is sticky. `SHOW CREATE` / `tryGetTable` will not re-read Iceberg metadata even though `metadata_location` is current. `SHOW TABLES` still skips empty-column tables (`getTablesForDatabase`), so a failed `createTable` conversion can hide a table that `SHOW CREATE` can still resolve via the new fallback.
    Anchor: `src/Databases/DataLake/GlueCatalog.cpp` / `GlueCatalog::tryGetTableMetadata` (columns empty vs non-empty branches), `getTablesForDatabase`
    Trigger: Conversion fail-open above, or any Glue row with outdated non-empty columns plus a newer `metadata_location`.
    Why defect: Two schema sources with different authority: non-empty Glue SD wins even when it disagrees with Iceberg metadata; listing uses a third rule (skip empty columns).
    Fix direction (short): If Glue columns are missing or conversion failed, fail closed, or always derive catalog schema from Iceberg metadata (REST-style) and treat Glue columns as display-only.
    Regression test direction (short): After a successful `ALTER`, point Glue columns at the old schema (or empty) and assert `SHOW CREATE` / `SHOW TABLES` / `DROP COLUMN` stay consistent with Iceberg metadata.

Coverage summary:

Scope reviewed: Glue catalog write path (`createTable`, `updateSchema`, `updateMetadata`/`updateTableInGlue`), Glue read path (`tryGetTableMetadata`, `SHOW CREATE`, `tryGetTable`), Iceberg `Mutations` `updateSchema` caller, REST parity, type conversion helpers.
Missed-class probes: 10/12 passed; failed: P2, P9; N/A: P3, P5, P12
Categories failed: protocol fail-open (`catch (...)`), dual-entrypoint schema authority (Glue SD vs Iceberg metadata vs `SHOW TABLES`)
Categories passed: P1 write/`ALTER` happy path (columns now pushed on success), P4 (no new settings), P6 (existing Glue ProfileEvents), P7 (Glue-only helpers), P8 (new integration test is a real reproducer), P10 (`new_schema` `fields` matches converter), P11 (metadata cache keyed by URI; new metadata file after `ALTER`), concurrency/lifetime of `CacheBase` + Glue client
Assumptions/limits: Static review of commit `61c7e4a`; no local Glue/moto run. AWS `UpdateTable` merge-vs-replace of omitted `Columns` was not executed; happy-path `ALTER` + `SHOW CREATE` is supported by the new test and reported Glue regression run.

[ianton-ru]

I found the same block in GlueCatalog::getActualTimestampType. Makes sense to move code in separate method and reuse in both places.