Fix pipeline test triage feedback and history

[Amaury Levé (Evangelink)]

Summary

• make preliminary/final triage state detection use the durable gh-aw workflow marker and stable visible headings, including compatibility with existing comments
• avoid malformed Azure DevOps and empty history links in safe-output comments
• exclude slow-only history from pull-request analysis and download only matching historical artifact families
• record explicit history incompleteness reasons and stop collecting after a budget is exhausted
• add regression tests and a lightweight CI workflow for the pipeline triage helper

Validation

• gh aw compile pipeline-test-triage --strict
• python .github/scripts/test_pipeline_test_triage.py
• python -m py_compile .github/scripts/pipeline_test_triage.py .github/scripts/test_pipeline_test_triage.py
• python .github/scripts/check_action_pins.py
• verified state detection against the preliminary and final comments from [main] Bump System.Management from 9.0.4 to 10.0.12 #11140
• reviewed the generated lockfile and confirmed no uses: pins changed

[Copilot]

Copilot review overview

🟡 Changes recommended

The per-build state matcher misses an existing Markdown-linked heading format, which can misclassify overlapping analyses.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review tier: Balanced
Findings: 1

New issues introduced by this change (1)

Severity	Finding
	.github/​workflows/​pipeline-test-triage.md — The per-build matcher still misses the transitional format already present on PR #11140: a…

What changed in this PR

Improves pipeline-test triage state detection, historical evidence collection, and regression coverage.

Changes:

• Uses durable workflow markers and stable headings for comment-state detection.
• Filters historical artifacts and records incomplete-history reasons.
• Adds helper tests and CI validation.

File	Description
.github/​workflows/​test-pipeline-triage.yml	Adds lightweight Python test CI.
.github/​workflows/​pipeline-test-triage.md	Updates state and history handling.
.github/​workflows/​pipeline-test-triage.lock.yml	Regenerates the compiled workflow.
.github/​scripts/​test_pipeline_test_triage.py	Adds helper regression tests.
.github/​scripts/​pipeline_test_triage.py	Filters history and tracks collection gaps.
.github/​agents/​pipeline-test-triage-analyst.agent.md	Updates evidence and comment guidance.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Copilot review overview

🔵 Needs a closer look

Comment-state matching can misclassify final comments, and download-budget exhaustion is not reliably recorded or propagated.

Review tier: Balanced
Findings: None

Issues resolved since last review (1)

Severity	Finding
	.github/​workflows/​pipeline-test-triage.md — The per-build matcher still misses the transitional format already present on PR #11140: a… View resolved comment

Suppressed comments (3)

Previously missed (2) — in code that hasn't changed since the last review.

.github/scripts/pipeline_test_triage.py:791

• When an artifact exceeds the remaining global download allowance, download_artifact raises here after consuming that allowance. The counter is clamped to the global maximum, but budget_exhausted remains false and the reason is recorded only as a generic download failure; with no later artifact, download-budget-exhausted is never reported, and with later builds the collector performs extra API work before noticing. Detect the clamped maximum in this handler, record the budget reason, and stop outer collection.
  .github/workflows/pipeline-test-triage.md:210
• The state check is unanchored even though the playbook requires the state heading at the start of the comment. Since the preliminary branch runs first, a valid final comment that later quotes or links the prior preliminary heading is misclassified as preliminary, causing subsequent runs to keep treating it as unresolved. Anchor each recognized heading with startswith; retain contains only for the legacy final comment's separate build line.

This issue also appears on line 237 of the same file.

.github/workflows/pipeline-test-triage.md:238

• The fallback state scan also searches the entire body and evaluates preliminary before final. A conforming final comment can therefore become the latest “preliminary” state merely by quoting the previous preliminary heading. Match the required first heading with startswith so only the comment's actual state header controls this fallback.

                      (($body | contains("**Preliminary test-triage analysis — Azure Pipelines build ")) or
                       ($body | contains("**Preliminary analysis** of Azure Pipelines build ")))) then