Skip to content

Gate on full config_db table coverage - #2571

Merged
berendt merged 1 commit into
mainfrom
sonic-e2e-v2-coverage-gate
Sep 17, 2026
Merged

berendt merged 1 commit into
mainfrom
sonic-e2e-v2-coverage-gate

Conversation

@ideaship

@ideaship ideaship commented Aug 5, 2026 •

Copy link
Copy Markdown
Contributor

Part of the series tracked in #2562, which explains the ordering and what each PR covers. Based on the preceding PR in the stack, so review only the top commits here.

Thirty lines, and the reason the coverage number cannot quietly rot: a unit test
asserting that no table the generator can emit is left without a golden.

It closes the one coverage failure nothing else catches — a newly emitted
table arriving with no golden. Coverage lost the other way (a table that had a
golden becoming empty) already fails the golden comparison, because the golden
file itself changes.

It lives in the unit suite rather than the E2E job because it needs neither
NetBox nor a generated config, only the generator source and the committed
goldens. So it costs milliseconds and runs on every change, where the E2E job
runs on a file matcher and a 2400s budget. No .zuul.yaml change is needed.

It lands last because it can only pass once the golden set is complete — which
is what the PR below achieves.

@ideaship ideaship changed the title sonic e2e v2 coverage gate Gate on full config_db table coverage Aug 5, 2026
@berendt
berendt force-pushed the sonic-e2e-v2-coverage-gate branch from 369711b to 34a693a Compare August 5, 2026 15:10
@ideaship
ideaship force-pushed the sonic-e2e-v2-coverage-gate branch from 34a693a to bdf2899 Compare August 5, 2026 19:51
@berendt
berendt force-pushed the sonic-e2e-v2-coverage-gate branch from bdf2899 to 8793a02 Compare August 6, 2026 10:56
@ideaship
ideaship force-pushed the sonic-e2e-v2-coverage-gate branch from 8793a02 to f0dc3b3 Compare August 6, 2026 12:10
@ideaship
ideaship force-pushed the sonic-e2e-v2-coverage-gate branch from f0dc3b3 to 63b1f23 Compare August 6, 2026 12:19
@berendt
berendt force-pushed the sonic-e2e-v2-coverage-gate branch from 63b1f23 to c154634 Compare August 7, 2026 05:35
@ideaship
ideaship force-pushed the sonic-e2e-v2-coverage-gate branch from c154634 to f06dba9 Compare August 7, 2026 08:07
@berendt
berendt force-pushed the sonic-e2e-v2-coverage-gate branch from f06dba9 to eca8762 Compare August 7, 2026 08:59
@ideaship
ideaship force-pushed the sonic-e2e-v2-coverage-gate branch from eca8762 to 65b9ebc Compare August 25, 2026 13:55
@ideaship
ideaship force-pushed the sonic-e2e-v2-coverage-gate branch from 65b9ebc to b0b921e Compare September 16, 2026 14:19
@ideaship
ideaship removed this pull request from stack #2572 September 16, 2026 14:27
@ideaship
ideaship force-pushed the sonic-e2e-v2-coverage-gate branch from b0b921e to a849095 Compare September 16, 2026 15:02
@ideaship
ideaship force-pushed the sonic-e2e-v2-coverage-gate branch from a849095 to 0676f37 Compare September 16, 2026 15:11
@ideaship
ideaship added this pull request to stack #2704 September 16, 2026 15:50
@berendt
berendt force-pushed the sonic-e2e-v2-coverage-gate branch from 0676f37 to ba7c315 Compare September 16, 2026 16:03
@berendt
berendt force-pushed the sonic-e2e-v2-coverage-gate branch from ba7c315 to d94bba1 Compare September 16, 2026 18:05
@berendt
berendt force-pushed the sonic-e2e-v2-coverage-gate branch from d94bba1 to f5dc9be Compare September 16, 2026 19:28
@berendt
berendt force-pushed the sonic-e2e-v2-coverage-gate branch from f5dc9be to f2a9d25 Compare September 17, 2026 06:01
@berendt
berendt force-pushed the sonic-e2e-v2-coverage-gate branch from f2a9d25 to ab57053 Compare September 17, 2026 11:29
Base automatically changed from sonic-e2e-v2-evpn to main September 17, 2026 12:42
@berendt
berendt force-pushed the sonic-e2e-v2-coverage-gate branch from ab57053 to 5cef59a Compare September 17, 2026 12:42
@ideaship
ideaship marked this pull request as ready for review September 17, 2026 12:50
@ideaship
ideaship requested a review from berendt September 17, 2026 12:50

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've found 1 issue

Prompt for AI Agents
Please address the comments from this code review:

## Individual Comments

### Comment 1
<location path="tests/unit/e2e/test_golden_coverage.py" line_range="27-29" />
<code_context>
+        "the generator can emit these config_db tables, but they are empty in "
+        "every file under tests/e2e/golden/: " + ", ".join(missing) + ". Seed a "
+        "device that populates them under tests/e2e/scenario/resources/ and "
+        "rewrite the goldens with `make sonic-e2e-regen`, or -- if the table "
+        "cannot be reached by any fixture -- exclude it here with a comment "
+        "saying why."
+    )
</code_context>
<issue_to_address>
**issue (bug_risk):** The failure message instructs maintainers to exclude an unreachable table, but the test has no exclusion set or mechanism; adding a comment alone does not change `missing`, so the test remains permanently failing for any genuinely unreachable emitted table.

**Triggers:** When a table is detected by the static generator scan but cannot be populated by any available fixture.

**Suggested fix:** Add an explicit, reviewed exclusion set containing the table and enforce that every exclusion has a nearby reason, or remove the unsupported exclusion advice from the message.

```suggestion
        "rewrite the goldens with `make sonic-e2e-regen`."
```
</issue_to_address>

Sourcery assessment

Approval pending. 1 finding to address first.

Blocking findings: tests/unit/e2e/test_golden_coverage.py:29


Sourcery is free for open source - if you like our reviews please consider sharing them ✨

Comment thread tests/unit/e2e/test_golden_coverage.py Outdated
Comment on lines +27 to +29
"rewrite the goldens with `make sonic-e2e-regen`, or -- if the table "
"cannot be reached by any fixture -- exclude it here with a comment "
"saying why."

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

issue (bug_risk): The failure message instructs maintainers to exclude an unreachable table, but the test has no exclusion set or mechanism; adding a comment alone does not change missing, so the test remains permanently failing for any genuinely unreachable emitted table.

Triggers: When a table is detected by the static generator scan but cannot be populated by any available fixture.

Suggested fix: Add an explicit, reviewed exclusion set containing the table and enforce that every exclusion has a nearby reason, or remove the unsupported exclusion advice from the message.

Suggested change
"rewrite the goldens with `make sonic-e2e-regen`, or -- if the table "
"cannot be reached by any fixture -- exclude it here with a comment "
"saying why."
"rewrite the goldens with `make sonic-e2e-regen`."

@ideaship
ideaship marked this pull request as draft September 17, 2026 13:03
tests/e2e/coverage.py reports which config_db tables the golden set
reaches, but nothing consumes it: it is wired into neither
sonic_golden_test.sh nor the Zuul job, so the number is only seen by
someone who runs `make sonic-e2e-coverage` by hand. That is how the
"38 of 38" claim came to exist as an unverifiable one-off in the first
place, and leaving it human-run lets it decay the same way again.

Add a unit test asserting the set of emitted-but-uncovered tables is
empty. It closes the one coverage failure nothing else catches: a newly
emitted table arriving with no golden. Coverage lost in the other
direction -- a table that had a golden becoming empty -- already fails
the golden comparison, because the golden file itself changes, and the
regeneration path is covered separately by the coverage guard in
compare.py.

It belongs in the unit suite rather than the E2E job because it needs
neither NetBox nor a generated config, only the generator source and the
committed goldens. It therefore costs milliseconds and runs on every
change, where the E2E job runs on a file matcher and a 2400s budget.
This needs no .zuul.yaml change: coverage.py already exposes
emitted_tables() and covered_tables().

The assertion message names the missing tables and points at
`make sonic-e2e-regen`, because the cost of this gate is that adding a
generator table now obliges the same change to add golden coverage, and
that is a full regeneration cycle rather than a two-line edit. If a
table genuinely cannot be reached by any fixture, the message says to
subtract it from the missing set here with a stated reason -- one visible exception, not a
silently growing allowlist.

This lands after the last scenario because it can only pass once the
golden set is complete; the report itself lands with the first goldens,
where it is still useful at 30 of 38.

Assisted-by: Claude:claude-opus-5
Signed-off-by: Roger Luethi <luethi@osism.tech>
@ideaship
ideaship force-pushed the sonic-e2e-v2-coverage-gate branch from 5cef59a to 23edd8f Compare September 17, 2026 13:15
@ideaship
ideaship marked this pull request as ready for review September 17, 2026 13:24

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've reviewed your changes and they look great!

Sourcery assessment

Approved.


Sourcery is free for open source - if you like our reviews please consider sharing them ✨

@berendt
berendt merged commit 4aeb224 into main Sep 17, 2026
3 checks passed
@berendt
berendt deleted the sonic-e2e-v2-coverage-gate branch September 17, 2026 13:36
@github-project-automation github-project-automation Bot moved this from New to Done in Human Board Sep 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Archived in project

Development

Successfully merging this pull request may close these issues.

3 participants