Skip to content

feat(ci): emit metrics for E2E canary - #2508

Merged
Hweinstock merged 1 commit into
aws:refactorfrom
Hweinstock:feat/canary-workflow-metrics
Oct 2, 2026
Merged

Hweinstock merged 1 commit into
aws:refactorfrom
Hweinstock:feat/canary-workflow-metrics

Conversation

@Hweinstock

@Hweinstock Hweinstock commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Problem

We want to setup alarms on failing canary runs.

Solution

Verification

@github-actions github-actions Bot added the size/xs PR size: XS label Oct 2, 2026
@agentcore-devx-automation agentcore-devx-automation Bot added claude-security-reviewing Claude Code /security-review in progress agentcore-harness-reviewing AgentCore Harness review in progress labels Oct 2, 2026

@agentcore-devx-automation agentcore-devx-automation Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AgentCore Harness Review

Verdict: Changes requested

Two blocking issues with the new workflow wiring, plus a scope question on the cron change.

1. workflow-metrics.yml points at a personal fork, not the AWS org

File: .github/workflows/workflow-metrics.yml:13

uses: Hweinstock/agentcore-devx-devtools/.github/workflows/workflow-metrics.yml@c48c3f43c88b4f4266fd001d097bc50e63341d3a

The reusable workflow lives in a personal fork (Hweinstock/agentcore-devx-devtools). The sibling e2e-test.yml uses aws/agentcore-devx-devtools/..., and aws/agentcore-devx-devtools currently does not contain workflow-metrics.yml at all. This workflow assumes a CloudWatch writer role via a secrets reader role, so pointing trusted-CI infrastructure at an individual contributor's fork is a governance/supply‑chain concern even with a pinned SHA (the repo can be renamed, deleted, or re-permissioned at any time).

Please land workflow-metrics.yml in aws/agentcore-devx-devtools and update the uses: reference to point there (still SHA‑pinned).

2. Nested action SHA in the called workflow is malformed and will fail at runtime

Inside the referenced Hweinstock/agentcore-devx-devtools@c48c3f43... copy of workflow-metrics.yml, the inner step pins:

uses: aws/agentcore-devx-devtools/.github/actions/fetch-secrets@75989f6f7f193deaf83c237c36572d1a8f800b2

That ref is 39 hex chars — not a valid Git SHA — and GitHub's API returns 422 No commit found for it in aws/agentcore-devx-devtools. The correct SHA used in this repo's e2e-test.yml is 75989f65f7f193deaf83c237c36572d1a8f800b2 (note the missing 5 after 75989f6). The metrics job will fail to resolve this action every time it runs. Fix in whichever repo ends up hosting the reusable workflow, and verify the pin in this PR after.

3. Cron frequency change is out of scope / worth justifying

File: .github/workflows/canary.yml:4

The previous PR (#2503) explicitly introduced an hourly canary; this PR flips it to every 20 minutes while advertising only "emit metrics for E2E canary." Tripling canary frequency has real cost (CodeBuild runners, AWS calls in E2E tests) and, because concurrency.cancel-in-progress: false, overlapping runs will queue if a canary ever exceeds 20 minutes. Please either split this into its own PR with a rationale or call it out in the description — otherwise it's easy to miss in review.

@agentcore-devx-automation agentcore-devx-automation Bot removed the agentcore-harness-reviewing AgentCore Harness review in progress label Oct 2, 2026
@agentcore-devx-automation

Copy link
Copy Markdown
Contributor

Claude Security Review: no high-confidence findings. (run)

@agentcore-devx-automation agentcore-devx-automation Bot removed the claude-security-reviewing Claude Code /security-review in progress label Oct 2, 2026
@github-actions github-actions Bot added size/xs PR size: XS and removed size/xs PR size: XS labels Oct 2, 2026
@Hweinstock
Hweinstock force-pushed the feat/canary-workflow-metrics branch from 6d36c3b to 713b68e Compare October 2, 2026 16:26
@github-actions github-actions Bot added size/xs PR size: XS and removed size/xs PR size: XS labels Oct 2, 2026
@agentcore-devx-automation agentcore-devx-automation Bot added the claude-security-reviewing Claude Code /security-review in progress label Oct 2, 2026
@agentcore-devx-automation

Copy link
Copy Markdown
Contributor

Claude Security Review: no high-confidence findings. (run)

@agentcore-devx-automation agentcore-devx-automation Bot removed the claude-security-reviewing Claude Code /security-review in progress label Oct 2, 2026
@codecov-commenter

codecov-commenter commented Oct 2, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.40%. Comparing base (8e330f9) to head (d2850e1).

Additional details and impacted files
@@            Coverage Diff            @@
##           refactor    #2508   +/-   ##
=========================================
  Coverage     97.40%   97.40%           
=========================================
  Files           642      642           
  Lines         46865    46865           
=========================================
  Hits          45647    45647           
  Misses         1218     1218           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

nborges-aws
nborges-aws previously approved these changes Oct 2, 2026
@Hweinstock
Hweinstock force-pushed the feat/canary-workflow-metrics branch from 713b68e to d2850e1 Compare October 2, 2026 17:50
@github-actions github-actions Bot added size/xs PR size: XS and removed size/xs PR size: XS labels Oct 2, 2026
@Hweinstock

Copy link
Copy Markdown
Contributor Author

Note: harness reviewer is now outdated, swapped to version on main now that other PR is merged.

@github-actions github-actions Bot added size/xs PR size: XS and removed size/xs PR size: XS labels Oct 2, 2026
@agentcore-devx-automation agentcore-devx-automation Bot added the claude-security-reviewing Claude Code /security-review in progress label Oct 2, 2026
@agentcore-devx-automation

Copy link
Copy Markdown
Contributor

Claude Security Review: no high-confidence findings. (run)

@agentcore-devx-automation agentcore-devx-automation Bot removed the claude-security-reviewing Claude Code /security-review in progress label Oct 2, 2026
@Hweinstock
Hweinstock marked this pull request as ready for review October 2, 2026 18:24
@Hweinstock
Hweinstock merged commit f0943aa into aws:refactor Oct 2, 2026
22 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/xs PR size: XS

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants