Skip to content

feat(observability): publish standalone Dispatch dashboard and alerts #1255

Open

Description

@joryirving

Why

Make Dispatch's state/alert catalogue from design #1237 / PR #1239 operational without assuming Courier, one worker identity or a universal dashboard backend.

Work

  • Package an opt-in dashboard and executable evaluator/rule integration for the chosen compatible metrics backend; separate portable state/alert definitions from optional Prometheus/VictoriaMetrics assets, OTel transport, and deployment-owned notification routing.
  • Consume the bounded queue/ownership/execution/export metrics implemented in #1252. Standalone views for eligible/assigned/claimed/executing, PR-fix hand-out/ack/reclaim, blocked human vs operational park, grooming/sync freshness, and known/unknown worker availability.
  • Implement the correct identity/age-based semantics for stale assignments and overdue unacked hand-outs: continuous nonzero aggregate backlog during healthy throughput does not imply a stranded item.
  • For windowed DB failure gauges, use the current in-window counts rather than delta/counter-style operations that miss event replacement. A4 pipeline-idle alerts must exclude groomer/heartbeat emissions and be gated by fresh upstream sync.
  • Missing first successful sync, absent scrape, unsupported worker ownership and no meaningful traffic are distinct cases; healthy idle is not a page. Define exact thresholds, evidence, suppression and operator actions.
  • Keep UI/asset configuration independent of home-ops, worker credentials, courier metrics, Grafana provisioning or a mandatory Alertmanager.

Acceptance

  • Replayable fixtures or rule tests cover aged stranded claim, overlapping rapidly resolved assignments (no alert), prolonged unacked hand-out, zero-actionable queue, first-sync missing, failed sync, real execution outage, audit failures in a constant-size rolling window, and stopped telemetry.
  • A non-Courier consumer and completely idle Dispatch both render accurately. No privilege escalation or autonomous claim release via dashboards.
  • Document optional backend/evaluator matrix and installation guidance separately from upstream semantics.

Dependencies

Blocked by #1252 and accepted #1239. Do not conflate dashboard work with the PR-fix/ownership reaper implementation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions