From 23edd8fc2e162ae5162cfb6e0d36901819112e72 Mon Sep 17 00:00:00 2001 From: Roger Luethi Date: Wed, 5 Aug 2026 09:33:00 +0200 Subject: [PATCH] tests/e2e: gate on full config_db table coverage tests/e2e/coverage.py reports which config_db tables the golden set reaches, but nothing consumes it: it is wired into neither sonic_golden_test.sh nor the Zuul job, so the number is only seen by someone who runs `make sonic-e2e-coverage` by hand. That is how the "38 of 38" claim came to exist as an unverifiable one-off in the first place, and leaving it human-run lets it decay the same way again. Add a unit test asserting the set of emitted-but-uncovered tables is empty. It closes the one coverage failure nothing else catches: a newly emitted table arriving with no golden. Coverage lost in the other direction -- a table that had a golden becoming empty -- already fails the golden comparison, because the golden file itself changes, and the regeneration path is covered separately by the coverage guard in compare.py. It belongs in the unit suite rather than the E2E job because it needs neither NetBox nor a generated config, only the generator source and the committed goldens. It therefore costs milliseconds and runs on every change, where the E2E job runs on a file matcher and a 2400s budget. This needs no .zuul.yaml change: coverage.py already exposes emitted_tables() and covered_tables(). The assertion message names the missing tables and points at `make sonic-e2e-regen`, because the cost of this gate is that adding a generator table now obliges the same change to add golden coverage, and that is a full regeneration cycle rather than a two-line edit. If a table genuinely cannot be reached by any fixture, the message says to subtract it from the missing set here with a stated reason -- one visible exception, not a silently growing allowlist. This lands after the last scenario because it can only pass once the golden set is complete; the report itself lands with the first goldens, where it is still useful at 30 of 38. Assisted-by: Claude:claude-opus-5 Signed-off-by: Roger Luethi --- tests/unit/e2e/test_golden_coverage.py | 30 ++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) create mode 100644 tests/unit/e2e/test_golden_coverage.py diff --git a/tests/unit/e2e/test_golden_coverage.py b/tests/unit/e2e/test_golden_coverage.py new file mode 100644 index 000000000..0e302069c --- /dev/null +++ b/tests/unit/e2e/test_golden_coverage.py @@ -0,0 +1,30 @@ +# SPDX-License-Identifier: Apache-2.0 + +"""Gate on the golden set covering every config_db table the generator emits. + +tests/e2e/coverage.py is the human-facing report; this is the check that +actually fails. It lives in the unit suite rather than in the E2E job because +it needs neither NetBox nor a generated config -- only the generator source +and the committed goldens -- so it costs milliseconds and runs on every +change, instead of only when the E2E job's file matcher fires. + +What it catches is the one coverage failure nothing else does: a newly +emitted table that arrives with no golden. Coverage going the other way (a +table that had a golden becoming empty) already fails the golden comparison, +because the golden file itself changes. +""" + +from tests.e2e.coverage import covered_tables, emitted_tables + + +def test_every_emitted_table_has_golden_coverage(): + missing = sorted(emitted_tables() - covered_tables()) + + assert not missing, ( + "the generator can emit these config_db tables, but they are empty in " + "every file under tests/e2e/golden/: " + ", ".join(missing) + ". Seed a " + "device that populates them under tests/e2e/scenario/resources/ and " + "rewrite the goldens with `make sonic-e2e-regen`, or -- if no fixture " + "can reach the table -- subtract it from `missing` above with a " + "comment saying why." + )