Skip to content

[core] Preserve timestamp values on changelog retraction records - #9691

Open
junmuz wants to merge 2 commits into
apache:masterfrom
junmuz:feature/preserve-sequence-retract
Open

[core] Preserve timestamp values on changelog retraction records#9691
junmuz wants to merge 2 commits into
apache:masterfrom
junmuz:feature/preserve-sequence-retract

Conversation

@junmuz

@junmuz junmuz commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

Purpose

  • Adds a new config option changelog-producer.preserve-sequence-on-retract that accepts a comma-separated list of column names whose values should be taken from the incoming event (rather than the stored row) when producing changelog retraction records (-U, -D)
  • This is useful when delete or update events carry an event timestamp that downstream consumers need — e.g., external systems like Cassandra that rely on WRITETIME for conflict resolution
  • Only supported with the lookup changelog producer
  • This also allows alignment with FlinkCDC that emits retract events with db event timestamps.

Tests

  • Unit tests for delete retraction preserving the event's sequence field value
  • Unit tests for update retraction (-U) carrying the new event's sequence field value
  • Unit tests confirming original behavior is preserved when the option is not set

Manual verification

  • Tested out after configuration, the table correctly produces the changelog events.
CREATE TABLE test.test_table ( 
    id INT, 
    name STRING, 
    updated_at TIMESTAMP, 
    op_type STRING
) TBLPROPERTIES (
    'primary-key' = 'id', 
    'changelog-producer' = 'lookup', 
    'write-only' = 'false',
    'changelog-producer.preserve-field-on-retract' = 'updated_at',
    'rowkind.field' = 'op_type'
)
+-------+---+-------+--------------------------+
|rowkind|id |name   |updated_at                |
+-------+---+-------+--------------------------+
|-D     |1  |one_new|2026-09-09 15:11:45.082928|
|+U     |1  |one_new|2026-09-09 15:11:03.804031|
|-U     |1  |one    |2026-09-09 15:11:03.804031|
|+I     |1  |one    |2026-09-09 15:10:31.101137|
+-------+---+-------+--------------------------+

@junmuz
junmuz force-pushed the feature/preserve-sequence-retract branch from 56365ee to ba28382 Compare September 8, 2026 14:17
@junmuz junmuz changed the title [core] Preserve sequence field values on changelog retraction records [core] Preserve timestamp values on changelog retraction records Sep 8, 2026
@junmuz
junmuz marked this pull request as ready for review September 8, 2026 16:55
@junmuz
junmuz marked this pull request as draft September 9, 2026 14:28
@junmuz
junmuz marked this pull request as ready for review September 9, 2026 15:27
@junmuz

junmuz commented Sep 9, 2026

Copy link
Copy Markdown
Contributor Author

@JingsongLi Can I get a review on the PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant