alert-triage

Correlates Alertmanager alerts into incidents, attaches cluster evidence, and posts a short digest to Discord.

Alertmanager already groups identical alerts by alertname. What it cannot do is recognise that an NFS export going away, five pods crashlooping, and a Ceph warning are one event. This service fills that gap: it buffers a burst, groups the alerts by probable root cause, gathers supporting evidence from the cluster, and asks a model to explain what happened in a couple of sentences.

It is deliberately cluster-local — it reads only its own API server and keeps its own history — so a second instance in another cluster needs no shared credentials or state.

Pipeline

1. Ingest — POST /webhook accepts the Alertmanager v4 payload and returns 202 immediately. Resolved alerts are dropped.
2. Window — alerts buffer for FLUSH_DELAY after the first arrival, capped at MAX_WINDOW so a slow cascade still reports.
3. Correlate — a pure function groups alerts by, in order of precedence: a known failure signature (NFS, Ceph, node, DNS), a shared node, or a shared namespace with overlapping active windows. Signatures win because shared storage and DNS faults cross node boundaries.
4. Enrich — unhealthy pods, recent warning events, and recent Flux HelmRelease/Kustomization transitions for the involved namespaces.
5. History — every group is recorded to a JSONL file so the digest can say whether an incident is novel or routine.
6. Narrate — one chat completion per group via LiteLLM, which also judges where a fix would have to be made: git, partial, cluster, external or unknown. Only git and partial are actionable by editing the repository, which is the gate a future issue sink would use.
7. Deliver — one Discord embed per group.

Enrichment and narration are both optional: if the API server or the model is unreachable, the digest still ships with whatever is available.

Configuration

Variable	Default	Purpose
LISTEN_ADDR	:8080	HTTP listen address
WEBHOOK_TOKEN	—	Shared secret required on /webhook. Unset accepts everything
MAX_ALERTS	500	Cap on alerts buffered in one window
MAX_GROUPS	12	Cap on groups narrated per flush
LITELLM_URL	—	LiteLLM base URL, e.g. http://litellm.llm:4000/v1
LITELLM_API_KEY	—	Scoped LiteLLM key
MODEL	dsv4f	Model alias used for the narrative
DISCORD_WEBHOOK_URL	—	Digest channel webhook
HISTORY_PATH	/data/history.jsonl	History file (persist this)
FLUSH_DELAY	3m	Quiet period before reporting a burst
MAX_WINDOW	10m	Hard cap on buffering
CORRELATE_SLACK	5m	Overlap tolerance when grouping
EVIDENCE_WINDOW	30m	How far back to look for events and changes
RETENTION	168h	History retention
NARRATE_TIMEOUT	120s	Model call timeout
TRIAGE_LABEL	—	When set, only alerts whose <label>="true" is buffered. Empty triages everything.
CLUSTER	—	Cluster this instance serves, matching the alerts' cluster label. Empty enriches every group.

Opt-in triage

Triage costs a model call and a Discord message. Alerts that nobody reading this cluster can act on — upstream API quotas, third-party vendor outages, informational notices — still cost those today. The principle is that the alerting rules know whether a fire is worth investigating, so opt-in is by label: only PrometheusRules carrying <TRIAGE_LABEL>="true" get triaged.

The contract is enforced in two places:

• Alertmanager route (lives in joryirving/home-ops, not here): match on the label and route those alerts to the /webhook receiver. This is the primary contract — the filter below is a backstop for misroutes.
• Webhook (this service): when TRIAGE_LABEL is set, the handler drops any alert whose label is not exactly "true" and logs a single line per delivery with the cumulative drop count, so a label-rule typo is visible without checking metrics.

TRIAGE_LABEL defaults to empty (fail-open, matching WEBHOOK_TOKEN), so a fresh image triages every alert. Leave it unset until the rules in home-ops are labelled; flipping it on beforehand silently drops everything.

Rollout ordering:

1. Ship this code with TRIAGE_LABEL unset so behaviour is unchanged.
2. Label the PrometheusRules you want triaged in home-ops.
3. Add an Alertmanager route matcher for <label> = "true".
4. Set TRIAGE_LABEL on the deployment. The webhook filter then backstops anything the route missed.

Example rule with the opt-in label:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: home-cluster
spec:
  groups:
    - name: critical
      rules:
        - alert: NodeDown
          labels:
            triage: "true"
            severity: critical

The value is matched literally: triage="True", triage="yes", or an absent label are all dropped. This is deliberate — a typo on the rule side is what the backstop is for, and the cumulative drop log line is how you notice it.

Deploying

Images are published to ghcr.io/misospace/alert-triage on tag push. Point an Alertmanager webhook receiver at /webhook and give the pod a ServiceAccount with the RBAC below.

Securing the webhook

Set WEBHOOK_TOKEN and have Alertmanager send it. AlertmanagerConfig cannot set arbitrary headers on a webhook receiver, so use httpConfig.authorization, which sends Authorization: Bearer <token>; X-Webhook-Token is accepted too and is easier to send by hand.

- name: triage
  webhookConfigs:
    - url: http://alert-triage.observability.svc.cluster.local:8080/webhook
      sendResolved: false
      httpConfig:
        authorization:
          credentials:
            name: alert-triage-secret
            key: WEBHOOK_TOKEN

Leaving WEBHOOK_TOKEN unset accepts every request and logs a warning once, so that a new image cannot silently reject alerts before the receiver has the secret. Set the receiver up first, then the token.

A token is not a perimeter on its own: anything that can reach the pod can spend model budget and post to the digest channel. Pair it with a NetworkPolicy that admits only Alertmanager to port 8080.

RBAC

The enrichment client is read-only. Each row below is one API call the service makes on every window flush, the verb Kubernetes requires to authorise it, the shape of the read (list a collection, get a named object, or get a named subresource), and the enrichment path it backs. list is required for single-object GETs that the client walks by label: the API server maps both to the same URL pattern and authorises them with the collection's list verb.

A failed read degrades to absence and renders as an explicit negative ("all nodes Ready"), never as silence, so the model does not mistake missing data for an unprobed subject. Operators who do not run an optional integration should omit its rules and accept the corresponding gap in evidence.

Core Kubernetes

API group	Resource	Verbs	Read shape	What enrichment needs it for
""	nodes	get, list	List, cluster-wide	unhealthyNodes reports NotReady, under-pressure, or cordoned nodes; missing data renders as silence and the model reads that as "not checked".
""	pods	get, list	List, per namespace	The pod health sort that produces the "unhealthy pods" findings for a group, and the pod-to-node lookup that the node correlation key relies on.
""	pods/log	get	Get, per named pod (subresource)	Tail of the previous or current container log for failing pods; pods/log is a subresource and is not granted by pods alone.
""	events	get, list	List, cluster-wide and per namespace	Warning events that contextualise an alert; cluster-wide only when the alert names no namespace.
batch	jobs	get, list	Get, per named job (per alert)	KubeJobFailed resolution: the alert names the job by label but not the pod, so enrichment fetches the Job to walk ownerReferences and surface the failed pod as a subject.

Flux

API group	Resource	Verbs	Read shape	What enrichment needs it for
helm.toolkit.fluxcd.io	helmreleases	get, list	List, cluster-wide and per namespace	Flux activity feed: NotReady HelmReleases and namespace-scoped Ready transitions.
kustomize.toolkit.fluxcd.io	kustomizations	get, list	List (cluster-wide and per namespace) and get per named	Same Flux activity feed for Kustomizations, plus the workload-path and sourceRef lookup that drives the git/partial decision.
source.toolkit.fluxcd.io	gitrepositories	get	Get, per named, sourceRef-dependent	The Kustomization sourceRef is dereferenced to obtain the repository URL for commit classification; only the kinds a resolved Kustomization actually references are read.
source.toolkit.fluxcd.io	ocirepositories	get	Get, per named, sourceRef-dependent	As above, when sourceRef.kind is OCIRepository.
source.toolkit.fluxcd.io	buckets	get	Get, per named, sourceRef-dependent	As above, when sourceRef.kind is Bucket.
source.toolkit.fluxcd.io	helmcharts	get	Get, per named, sourceRef-dependent	As above, when sourceRef.kind is HelmChart.

Argo CD (optional)

API group	Resource	Verbs	Read shape	What enrichment needs it for
argoproj.io	applications	get	Get, per named instance, tried in the argocd namespace first and cluster-wide second	Fallback path: if an alert's instance label matches an Argo CD Application, enrichment reads its source.repoURL and source.path to anchor the workload. Omit these rules if the cluster does not run Argo CD.

Example ClusterRole

The deployment that materialises this ClusterRole lives in the GitOps repository, not here. The manifest below covers every read the service currently makes; trim the rules for integrations you do not run.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: alert-triage
rules:
  - apiGroups: [""]
    resources: ["nodes", "pods", "events"]
    verbs: ["get", "list"]
  - apiGroups: [""]
    resources: ["pods/log"]
    verbs: ["get"]
  - apiGroups: ["batch"]
    resources: ["jobs"]
    verbs: ["get", "list"]
  - apiGroups: ["helm.toolkit.fluxcd.io"]
    resources: ["helmreleases"]
    verbs: ["get", "list"]
  - apiGroups: ["kustomize.toolkit.fluxcd.io"]
    resources: ["kustomizations"]
    verbs: ["get", "list"]
  - apiGroups: ["source.toolkit.fluxcd.io"]
    resources: ["gitrepositories", "ocirepositories", "buckets", "helmcharts"]
    verbs: ["get"]
  - apiGroups: ["argoproj.io"]
    resources: ["applications"]
    verbs: ["get"]

Endpoints

• POST /webhook — Alertmanager receiver
• GET /healthz — liveness (unauthenticated, for probes)
• GET /recent — the last 20 delivered digests as JSON, including the evidence the model was given and the narrative it wrote
• GET /metrics — Prometheus counters for deliveries, narration, and alerts

/recent and /metrics require the same WEBHOOK_TOKEN as /webhook (Authorization: Bearer <token> or X-Webhook-Token: <token>); they return 401 without it. When WEBHOOK_TOKEN is unset they fail open, matching the webhook.

Reviewing what it said

Judging triage quality means reading the narratives, which otherwise only exist in a chat client. Two ways to get at them without one:

kubectl port-forward -n observability svc/alert-triage 8080:8080
TOKEN=$(kubectl -n observability get secret alert-triage -o jsonpath='{.data.WEBHOOK_TOKEN}' | base64 -d)
curl -s -H "Authorization: Bearer $TOKEN" localhost:8080/recent | jq '.[] | {key, title, narrative}'

Each digest is also logged as a single line (digest <key> [severity] <title> | <narrative>), so the history survives a restart wherever logs are shipped.