Skip to content

TEZ-4758: Add a buf configuration so Yetus bufcompat/buflint can run - #542

Merged
abstractdog merged 1 commit into
apache:masterfrom
deniskuzZ:TEZ-buf-config
Sep 30, 2026
Merged

abstractdog merged 1 commit into
apache:masterfrom
deniskuzZ:TEZ-buf-config

Conversation

@deniskuzZ

Copy link
Copy Markdown
Member

Yetus runs buf at the repository root with no config, so every cross-file import ("imported file does not exist" for import "DAGApiRecords.proto") fails and bufcompat votes -1 on the base branch before a patch is even applied. TEZ-4548 was the last proto change and predates the check; TEZ-4757 is the first patch to hit it

@tez-yetus

Copy link
Copy Markdown

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 14m 6s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
-1 ❌ test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ master Compile Tests _
+1 💚 mvninstall 6m 17s master passed
+1 💚 compile 4m 39s master passed
-1 ❌ bufcompat 0m 2s /branch-bufcompat-stderr.txt Error running buf. Please check buf stderr files.
_ Patch Compile Tests _
+1 💚 mvninstall 4m 56s the patch passed
+1 💚 codespell 2m 16s No new issues.
+1 💚 compile 4m 34s the patch passed
+1 💚 blanks 0m 1s The patch has no blanks issues.
+1 💚 buflint 0m 6s The patch generated 0 new + 0 unchanged - 28 fixed = 0 total (was 28)
-1 ❌ yamllint 0m 0s /results-yamllint.txt The patch generated 8 new + 0 unchanged - 0 fixed = 8 total (was 0)
_ Other Tests _
-1 ❌ unit 78m 1s /patch-unit-root.txt root in the patch passed.
-1 ❌ asflicense 0m 59s /results-asflicense.txt The patch generated 1 ASF License warnings.
117m 30s
Reason Tests
Failed junit tests tez.dag.app.TestSpeculation
Subsystem Report/Notes
Docker ClientAPI=1.56 ServerAPI=1.56 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-542/2/artifact/out/Dockerfile
Optional Tests dupname compile unit asflicense buflint bufcompat codespell detsecrets yamllint
uname Linux f259f2896d89 5.15.0-185-generic #195-Ubuntu SMP Fri Jun 19 17:11:50 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality tez-personality.sh
git revision master / 0e9f756
Test Results https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-542/2/testReport/
Max. process+thread count 1358 (vs. ulimit of 5500)
modules C: . U: .
Console output https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-542/2/console
versions git=2.43.0 maven=3.9.15 buf=1.68.2 codespell=2.4.1 yamllint=1.38.0
Powered by Apache Yetus 0.15.1 https://yetus.apache.org

This message was automatically generated.

- Yetus runs buf at the repository root with no config, so every cross-file import
  ("imported file does not exist" for import "DAGApiRecords.proto") fails and bufcompat
  votes -1 on the base branch before a patch is even applied. TEZ-4548 was the last proto
  change and predates the check; TEZ-4757 is the first patch to hit it
- One module per main proto directory, so tez-dag can import tez-api and tez-runtime-internals;
  the test fixture proto in tez-ext-service-tests is not a module, it has no compatibility contract
- breaking: WIRE. The payloads are internal wire messages; FILE would reject a same-number
  int32 -> int64 widening that is wire-compatible
- lint: MINIMAL minus PACKAGE_DEFINED. No Tez proto declares a package, and adding one
  renames every generated type and every message's fully-qualified name on the wire
- Verified with buf 1.50.0, the version Yetus rel/0.15.1 ships: build and lint clean on
  master, breaking clean for TEZ-4757 against master
@tez-yetus

Copy link
Copy Markdown

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 10m 51s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
-1 ❌ test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ master Compile Tests _
+1 💚 mvninstall 4m 59s master passed
+1 💚 compile 3m 58s master passed
-1 ❌ bufcompat 0m 2s /branch-bufcompat-stderr.txt Error running buf. Please check buf stderr files.
_ Patch Compile Tests _
+1 💚 mvninstall 4m 7s the patch passed
+1 💚 codespell 1m 47s No new issues.
+1 💚 compile 3m 57s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 buflint 0m 6s The patch generated 0 new + 0 unchanged - 28 fixed = 0 total (was 28)
+1 💚 yamllint 0m 0s No new issues.
_ Other Tests _
-1 ❌ unit 72m 57s /patch-unit-root.txt root in the patch passed.
+1 💚 asflicense 0m 57s The patch does not generate ASF License warnings.
105m 9s
Reason Tests
Failed junit tests tez.history.TestHistoryParser
Subsystem Report/Notes
Docker ClientAPI=1.56 ServerAPI=1.56 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-542/3/artifact/out/Dockerfile
Optional Tests dupname compile unit asflicense buflint bufcompat codespell detsecrets yamllint
uname Linux b6375bd02cd4 5.15.0-191-generic #201-Ubuntu SMP Fri Aug 7 18:39:04 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality tez-personality.sh
git revision master / 0e9f756
Test Results https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-542/3/testReport/
Max. process+thread count 1392 (vs. ulimit of 5500)
modules C: . U: .
Console output https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-542/3/console
versions git=2.43.0 maven=3.9.15 buf=1.68.2 codespell=2.4.1 yamllint=1.38.0
Powered by Apache Yetus 0.15.1 https://yetus.apache.org

This message was automatically generated.

@deniskuzZ

Copy link
Copy Markdown
Member Author

The bufcompat -1 is structural for this PR and cannot go green here. Yetus's bufcompat_preapply runs buf build on the unpatched base branch before the patch is applied; master has no buf.yaml, so every cross-file import fails (imported file does not exist for import "DAGApiRecords.proto", a same-directory import) and the plugin votes -1 before this change is even considered.
it needs to be merged over the vote once, after which the branch side builds and bufcompat runs for real on every later PR.

The check that does run on the patched tree is buflint, and it went from 28 errors to 0 on this PR — that's the config working.

Verified locally with buf 1.73 (CI runs 1.68.2): buf build and buf lint clean on master;

The unit failures are unrelated to a YAML file.

@abstractdog
abstractdog self-requested a review September 30, 2026 08:54
@abstractdog

Copy link
Copy Markdown
Contributor

The bufcompat -1 is structural for this PR and cannot go green here. Yetus's bufcompat_preapply runs buf build on the unpatched base branch before the patch is applied; master has no buf.yaml, so every cross-file import fails (imported file does not exist for import "DAGApiRecords.proto", a same-directory import) and the plugin votes -1 before this change is even considered. it needs to be merged over the vote once, after which the branch side builds and bufcompat runs for real on every later PR.

The check that does run on the patched tree is buflint, and it went from 28 errors to 0 on this PR — that's the config working.

Verified locally with buf 1.73 (CI runs 1.68.2): buf build and buf lint clean on master;

The unit failures are unrelated to a YAML file.

makes sense, ack

@abstractdog abstractdog left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TestHistoryParser flakiness is not related, so this LGTM

@abstractdog
abstractdog merged commit ae2756e into apache:master Sep 30, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants