diff --git a/.claude-plugin/marketplace.json b/.claude-plugin/marketplace.json index 3c934ac..8009fec 100644 --- a/.claude-plugin/marketplace.json +++ b/.claude-plugin/marketplace.json @@ -6,7 +6,7 @@ }, "metadata": { "description": "Tsuga toolkit for AI coding agents: one plugin with the tsuga CLI driver, live-platform investigation, dashboards, incident workflows, OpenTelemetry instrumentation, Collector, signal-choice, telemetry debug, and audit skills.", - "version": "0.8.4" + "version": "0.9.0" }, "plugins": [ { diff --git a/AGENTS.md b/AGENTS.md index 91ad156..8a9f0e4 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -54,6 +54,22 @@ ${CLAUDE_PLUGIN_ROOT}/skills//references/foo.md **Never use `{{SKILLS_DIR}}/...`** — that's the `npx skills` install-time placeholder, NOT substituted by Claude Code or Codex. +## Served reference bundles + +Operational reference content lives in the product repo at `packages/documentation/agents/references/` and is served by Tsuga. Skills fetch it by exact path: + +``` +tsuga docs get references/technologies/postgres/overview | jq -r .content +``` + +Bundles in use: `references/cli/`, `references/dashboards/`, `references/incident-response/`, `references/technologies/`, `references/telemetry/`. + +- Do **not** re-add local `references/` dirs for this content. A local copy silently goes stale; the served copy is the source of truth. +- These pages are path-addressed only. They are deliberately absent from `tsuga docs search`, so any path a skill needs must be written out in that `SKILL.md`. +- Content edits go to the product repo, not here. + +The three bundle-authoring meta-skills (`build-incident-history`, `build-knowledge-company`, `check-skill-health`) keep local `references/` on purpose: they are authoring-time tooling and must work without a live Tsuga org. + ## Cross-skill references Skills can reference other skills by **name** (e.g. `tsuga-cli`, `otel-instrumentation`). Claude Code and Codex resolve skill names across installed plugins. diff --git a/plugins/tsuga/.claude-plugin/plugin.json b/plugins/tsuga/.claude-plugin/plugin.json index 3c07eea..0a0b768 100644 --- a/plugins/tsuga/.claude-plugin/plugin.json +++ b/plugins/tsuga/.claude-plugin/plugin.json @@ -1,7 +1,7 @@ { "name": "tsuga", "description": "Tsuga observability plugin: the `tsuga` CLI driver (commands, TQL syntax, aggregation bodies, counter math, deep links, cloud/k8s translators); live-platform investigation for service health, errors, latency, and monitor coverage; dashboard building; incident orchestration; OpenTelemetry SDK, Collector, OTTL, signal-choice, telemetry debug, and audit skills; and meta-skills for building and validating skill bundles.", - "version": "0.8.4", + "version": "0.9.0", "author": { "name": "Tsuga Engineering", "email": "engineering@tsuga.com" diff --git a/plugins/tsuga/skills/incident-investigation/SKILL.md b/plugins/tsuga/skills/incident-investigation/SKILL.md index aa76029..873889a 100644 --- a/plugins/tsuga/skills/incident-investigation/SKILL.md +++ b/plugins/tsuga/skills/incident-investigation/SKILL.md @@ -5,15 +5,17 @@ description: "Primary entry point for active-incident investigation, post-incide # Incident Investigation -Primary entry point. Read-only for everything **except the two standing deliverables** — the Tsuga investigation record and the proofs dashboard ([references/investigation-record.md](./references/investigation-record.md)) — which every concluded investigation publishes by default (skip conditions in step 10). All other mutations require an explicit user ask. +Primary entry point. Read-only for everything **except the two standing deliverables** — the Tsuga investigation record and the proofs dashboard (`references/incident-response/investigation-record`) — which every concluded investigation publishes by default (skip conditions in step 10). All other mutations require an explicit user ask. ## How to read this skill -This skill covers the full investigation loop — orchestrator-level workflow AND the detailed procedures for each parallel branch. Read selectively based on your role: +This skill covers the full investigation loop — orchestrator-level workflow AND the detailed procedures for each parallel branch. Read selectively based on your role. -- **Orchestrator (primary agent running the whole investigation):** read Anti-patterns → Workflow (steps 1-10) → Output contract → Evidence rules → Guardrails, plus [references/investigation-record.md](./references/investigation-record.md) for the two durable deliverables you publish at step 10. You spawn subagents at step 5 and synthesize their outputs at step 8/9. The branch procedures live in their own reference files; the subagents own those — you don't read them. -- **Telemetry-sweep subagent:** read [references/branch-telemetry-sweep.md](./references/branch-telemetry-sweep.md) — that file is everything you need. Skip the workflow, ledger, gate, verdict, and the change-correlation branch. -- **Change-correlation subagent:** read [references/branch-change-correlation.md](./references/branch-change-correlation.md) — that file is yours. +Every `references/incident-response/...` path below is a page served by Tsuga, not a local file. Fetch one with `tsuga docs get | jq -r .content`. These pages are path-addressed only and never show up in `tsuga docs search`. + +- **Orchestrator (primary agent running the whole investigation):** read Anti-patterns → Workflow (steps 1-10) → Output contract → Evidence rules → Guardrails, plus `references/incident-response/investigation-record` for the two durable deliverables you publish at step 10. You spawn subagents at step 5 and synthesize their outputs at step 8/9. The branch procedures live in their own reference pages; the subagents own those — you don't read them. +- **Telemetry-sweep subagent:** fetch `references/incident-response/branch-telemetry-sweep` — that page is everything you need. Skip the workflow, ledger, gate, verdict, and the change-correlation branch. +- **Change-correlation subagent:** fetch `references/incident-response/branch-change-correlation` — that page is yours. - **Codebase-grep subagent:** you're spawned with one verbatim signal (error string, metric name, log pattern). Grep the mounted codebases under `{{CODEBASES_DIR}}` for that literal string; return the `file:line` and ~5 lines of surrounding context (the enclosing function + nearby conditions that trigger the emission). Nothing else. You do not interpret — you locate. - **Challenger subagent:** you're spawned with the leading hypothesis and the current evidence. Name the single piece of evidence that would most cleanly falsify it, and say whether it's been checked. Do not build competing hypotheses; just falsify. @@ -27,7 +29,7 @@ Minimum viable case: - when it started (the incident's `declared_at` is your "now") - scope: service, cluster, customer, env, or monitor -Accept a human summary or a case manifest ([references/case-manifest.md](./references/case-manifest.md)). If scope is unclear, ask for the smallest missing fact — do not launch a generic sweep. +Accept a human summary or a case manifest (`references/incident-response/case-manifest`). If scope is unclear, ask for the smallest missing fact — do not launch a generic sweep. ## Time discipline (hard rule) @@ -90,7 +92,7 @@ Do not let `failing subsystem` silently replace `root cause`. For an active incident with ongoing impact, mitigation is the first question, not the last. Identify the fastest action that restores service — rollback, failover, scale, flag flip — and surface it early, in parallel with the RCA; never gate stopping the bleeding on a completed root cause. Mitigation actions postdate `declared_at`; that does not violate Time discipline — they document the response and never feed the causal chain. -Then open a Tsuga investigation record (beta) so progress is visible while you work. This is a default deliverable — create it without asking, unless the user opted out. Check the environment first (`tsuga config` — right key, right cluster; see the hygiene notes in [references/investigation-record.md](./references/investigation-record.md)): +Then open a Tsuga investigation record (beta) so progress is visible while you work. This is a default deliverable — create it without asking, unless the user opted out. Check the environment first (`tsuga config` — right key, right cluster; see the hygiene notes in `references/incident-response/investigation-record`): ```bash tsuga investigations create -d '{ @@ -101,7 +103,7 @@ tsuga investigations create -d '{ }' ``` -Keep `name` short — the app displays it everywhere; never restate it inside `contentMd`. Keep the returned `id` — you will update this record at checkpoints and finish it at step 10 with the structured document from [references/investigation-record.md](./references/investigation-record.md). Updates are full PUTs: always resend `name` and `owner` (omitted optional fields keep their current values). If the call returns 403 the key lacks the `investigations` permission: skip it silently and run the investigation as normal; never block on it. +Keep `name` short — the app displays it everywhere; never restate it inside `contentMd`. Keep the returned `id` — you will update this record at checkpoints and finish it at step 10 with the structured document from `references/incident-response/investigation-record`. Updates are full PUTs: always resend `name` and `owner` (omitted optional fields keep their current values). If the call returns 403 the key lacks the `investigations` permission: skip it silently and run the investigation as normal; never block on it. ### 3. Anchor from the broken monitor (if the case cites one) @@ -120,12 +122,12 @@ Then re-run that same query against the incident window AND a control window — One sentence of matching is enough to load one. Zero is fine. All-of-them-from-fear is not. -- DB / RDS / Postgres / MySQL / connections / replication → [playbooks/database.md](./references/playbooks/database.md) -- Kubernetes / pod / OOM / CrashLoopBackOff / node → [playbooks/kubernetes.md](./references/playbooks/kubernetes.md) -- Deploy / config drift / flag / IAM / key rotation → [playbooks/deploy-drift.md](./references/playbooks/deploy-drift.md) -- Queue / pub-sub / lag / backpressure → [playbooks/queue-backpressure.md](./references/playbooks/queue-backpressure.md) -- Cert / TLS / SSO / OAuth / credentials → [playbooks/auth-tls.md](./references/playbooks/auth-tls.md) -- Data quality / schema / upstream API / empty output → [playbooks/data-quality.md](./references/playbooks/data-quality.md) +- DB / RDS / Postgres / MySQL / connections / replication → `references/incident-response/playbooks/database` +- Kubernetes / pod / OOM / CrashLoopBackOff / node → `references/incident-response/playbooks/kubernetes` +- Deploy / config drift / flag / IAM / key rotation → `references/incident-response/playbooks/deploy-drift` +- Queue / pub-sub / lag / backpressure → `references/incident-response/playbooks/queue-backpressure` +- Cert / TLS / SSO / OAuth / credentials → `references/incident-response/playbooks/auth-tls` +- Data quality / schema / upstream API / empty output → `references/incident-response/playbooks/data-quality` If scope names a specific tech (Postgres, Redis, Kafka, …), the telemetry branch loads its `$knowledge-technology` reference. You don't need to orchestrate that. @@ -133,8 +135,8 @@ If scope names a specific tech (Postgres, Redis, Kafka, …), the telemetry bran Disjoint goals, run concurrently. Each branch is its own subagent; do not serialize. -- **telemetry sweep** — Tsuga evidence, monitor anchor, config-threshold preflight, surface verbatim signals. Procedure: [references/branch-telemetry-sweep.md](./references/branch-telemetry-sweep.md). -- **change correlation** — local git + `$gh`, strict mechanism fit (diff must touch emitter line). Procedure: [references/branch-change-correlation.md](./references/branch-change-correlation.md). +- **telemetry sweep** — Tsuga evidence, monitor anchor, config-threshold preflight, surface verbatim signals. Procedure: `references/incident-response/branch-telemetry-sweep`. +- **change correlation** — local git + `$gh`, strict mechanism fit (diff must touch emitter line). Procedure: `references/incident-response/branch-change-correlation`. - **history** — `$incident-history` (prior incident archive mining, if mounted). - **codebase-grep** — the emphasis branch. For every distinct verbatim signal the telemetry sweep surfaces (error string, log pattern, metric name, monitor filter), spawn **one subagent per signal** that greps the mounted codebases under `{{CODEBASES_DIR}}` to find the `file:line` where that signal is emitted. Output: file path, surrounding function context, nearby conditions that trigger the emission. 5 signals → 5 parallel greps — this scales linearly and each one returns a sharp pin, not a vague area. - **challenger** — attack the leading hypothesis. Its job is to name the single piece of evidence that would falsify the leader and say whether it's been checked. @@ -236,7 +238,7 @@ Hallucinated citations are worse than missing ones. When in doubt, demote. ### 10. Publish deliverables (default, not optional) -Verdict assigned → publish the two durable artifacts. Full spec and templates: [references/investigation-record.md](./references/investigation-record.md). +Verdict assigned → publish the two durable artifacts. Full spec and templates: `references/incident-response/investigation-record`. 1. **Proofs dashboard** — one graph per validated telemetry claim, assertion-style graph names, every query probe-verified before create, tagged with the incident id, owned by the affected team. 2. **Final investigation record** — replace the in-progress notes with the structured document (Summary / Key facts / Timeline / Symptoms / Contributing causes / Mitigation & action items / Open questions / Falsified along the way / Lessons learned draft). Deep-link every ID, use absolute time windows on evidence links, populate `linkedAssets` (dashboard, service, fired monitor). @@ -247,8 +249,8 @@ These ship by default — do not ask permission for them. The only reasons to sk Each parallel branch from step 5 has its own procedure file. The orchestrator does not read these inline — it spawns a subagent and points it at the file: -- **telemetry sweep** → [references/branch-telemetry-sweep.md](./references/branch-telemetry-sweep.md) — Tsuga evidence, monitor anchor, config-threshold preflight, verbatim-signal surfacing, completeness check. -- **change correlation** → [references/branch-change-correlation.md](./references/branch-change-correlation.md) — time-bounded git + `$gh`, strict `diff → emitter → observation` mechanism fit, candidate classification. +- **telemetry sweep** → `references/incident-response/branch-telemetry-sweep` — Tsuga evidence, monitor anchor, config-threshold preflight, verbatim-signal surfacing, completeness check. +- **change correlation** → `references/incident-response/branch-change-correlation` — time-bounded git + `$gh`, strict `diff → emitter → observation` mechanism fit, candidate classification. ## Output contract @@ -298,7 +300,7 @@ Deliverables: ### Finish the investigation record (beta) -End with one last clean update that replaces the in-progress notes with the **structured document** from [references/investigation-record.md](./references/investigation-record.md) — NOT a dump of the chat verdict. The name drops the "investigating" suffix and stays short (the app displays it; don't repeat it in `contentMd`): +End with one last clean update that replaces the in-progress notes with the **structured document** from `references/incident-response/investigation-record` — NOT a dump of the chat verdict. The name drops the "investigating" suffix and stays short (the app displays it; don't repeat it in `contentMd`): ```bash tsuga investigations update -d '{ @@ -335,9 +337,9 @@ Beta — the API will likely change. Any 403 means the key lacks the `investigat Load when needed: -- [references/branch-telemetry-sweep.md](./references/branch-telemetry-sweep.md) — telemetry-sweep subagent procedure -- [references/branch-change-correlation.md](./references/branch-change-correlation.md) — change-correlation subagent procedure -- [references/investigation-record.md](./references/investigation-record.md) — durable deliverables (record + proofs dashboard) spec + templates -- [references/case-manifest.md](./references/case-manifest.md) — JSON shape for structured case input -- [references/playbooks/](./references/playbooks/) — domain disambiguation guides -- [references/tsuga-rules.md](./references/tsuga-rules.md) — `tsuga` command patterns for the telemetry branch +- `references/incident-response/branch-telemetry-sweep` — telemetry-sweep subagent procedure +- `references/incident-response/branch-change-correlation` — change-correlation subagent procedure +- `references/incident-response/investigation-record` — durable deliverables (record + proofs dashboard) spec + templates +- `references/incident-response/case-manifest` — JSON shape for structured case input +- `references/incident-response/playbooks/` — domain disambiguation guides +- `references/incident-response/tsuga-rules` — `tsuga` command patterns for the telemetry branch diff --git a/plugins/tsuga/skills/incident-investigation/references/branch-change-correlation.md b/plugins/tsuga/skills/incident-investigation/references/branch-change-correlation.md deleted file mode 100644 index bdb4734..0000000 --- a/plugins/tsuga/skills/incident-investigation/references/branch-change-correlation.md +++ /dev/null @@ -1,79 +0,0 @@ -# Branch: change correlation - -You were spawned as a change-correlation subagent — this file is yours. Don't assign verdicts or synthesize root causes — produce the change timeline + candidate classifications (`mechanism_confirmed` / `mechanism_plausible` / `area_only` / `ruled_out`) for the orchestrator. - -Answer: what changed, was it deployed, and could it plausibly cause the symptom? - -## Inputs - -- incident window -- at least one repo slug or local repo path. The container mounts its codebases at `{{CODEBASES_DIR}}/` — check there first for subdirectories (each is a git repo you can inspect with `git log`, `git diff`, `git blame`). - -## Time-bound rule (hard) - -**Nothing past `declared_at` is admissible.** Restrict every `git log`, `git show`, and `gh pr` call to strictly-before the incident start. Every shell you run in this branch must include the time bound: - -- `git log --until="$DECLARED_AT" ...` -- `git log --before="$DECLARED_AT" ...` -- `gh pr list --search "merged:<$DECLARED_AT"` - -A post-incident PR titled "Fix " is the answer key leaking backward — do not use it, do not quote it, do not let it validate your leader. If you see one anyway (because a broad query returned it), drop it and rerun with the `--until` bound. Violating this invalidates the verdict. - -## Procedure - -1. **Map repos.** Case manifest > local paths > git remotes. Prioritize repos containing the affected service, cluster config, or incident hint. -2. **Collect the emitting `file:line` pins from codebase-grep.** The orchestrator spawns codebase-grep subagents for every verbatim signal in the telemetry output. Their results (one `file:line` per signal) are your highest-leverage inputs — each tells you exactly which source file a PR would need to touch to be a real candidate. -3. **Local git first — time-bounded.** Commits strictly before `declared_at`, files changed in config / helm / infra / auth / feature-flag / routing paths. Dirty working tree = not a safe proxy for the incident window. For each `file:line` from step 2, run `git log -L ,: --until=` to see which commits modified that specific line before the incident started. -4. **Then `$gh` — time-bounded.** Workflow runs, merged PRs, releases, commits, deployments with `merged:<` / `created:<` filters. A PR is a candidate only when it merged BEFORE `declared_at` AND a deploy completed between its merge and the incident start. -5. **Mechanism fit** per candidate — the strict version: - - Does the PR's diff touch the `file:line` that emits the observed signal? **If no → not a candidate**, regardless of timing. - - If yes: does the diff change the _condition that triggers emission_ or the _value being emitted_? Quote the relevant lines. - - Does the timing align (merge → deploy → incident start)? - - Is there a faster revert or verification step? -6. **Classify each candidate** as one of: - - `mechanism_confirmed` — diff → emitter → observation traces cleanly. - - `mechanism_plausible` — diff touches nearby code that could plausibly affect the observation; not directly on the emitter line. - - `area_only` — diff is in the right repo / service but doesn't touch the emitter. - - `ruled_out` — wrong surface or wrong timing. - -## Branch output - -``` -Most relevant changes: - - [evidence: gh_pr | gh_run | gh_release | gh_api | local_git] - emitting signal: "" - emitting file:line: : - diff touches emitter line?: yes | no - classification: mechanism_confirmed | mechanism_plausible | area_only | ruled_out -Strongest causal candidate: - — timestamp | artifact | surface | deploy status (deployed | merged only) | trace: diff→emitter→observation -Changes ruled out: - - — why it doesn't fit (wrong file, wrong surface, wrong timing, not deployed) -Best verification or rollback step: - -``` - -Deploy status unknown? Say so explicitly and lower candidate confidence. -`area_only` classification? Say so explicitly — don't promote to "strongest candidate" without a mechanism trace. - -## No-data fallback - -If no repos are mounted (`ENABLE_CODEBASES=0`, no paths given) AND `$gh` is unavailable or returns nothing useful: - -Return exactly: - -``` -Most relevant changes: (none — no repo / gh access) -Strongest causal candidate: (unavailable) -Best verification or rollback step: Operator should check deploy timestamps and config-change audit out-of-band. -``` - -Do not infer changes from telemetry alone. Do not cite PRs / SHAs / file paths that were not actually retrieved. - -## Branch guardrails - -- `merged` ≠ `deployed`. -- No blame without a traced mechanism: `diff → emitter → observation`. "Area match" is not a mechanism. -- Current checkout ≠ incident-window state. -- File paths, SHAs, PR numbers, run URLs over vague prose. -- Never skip codebase-grep. If the orchestrator didn't spawn it, spawn it yourself for the signals you're trying to explain before proposing a PR candidate. diff --git a/plugins/tsuga/skills/incident-investigation/references/branch-telemetry-sweep.md b/plugins/tsuga/skills/incident-investigation/references/branch-telemetry-sweep.md deleted file mode 100644 index 8f47c16..0000000 --- a/plugins/tsuga/skills/incident-investigation/references/branch-telemetry-sweep.md +++ /dev/null @@ -1,75 +0,0 @@ -# Branch: telemetry sweep - -You were spawned as a telemetry-sweep subagent — this file is everything you need. You don't synthesize a verdict; you surface facts + verbatim signals for codebase-grep + a completeness-check report. - -Read-only evidence gathering from Tsuga. This branch does not declare root cause on its own — it produces facts the orchestrator synthesizes. - -## Inputs - -Time window, scope hint (service / cluster / env / customer / monitor), reported symptom. Missing scope → return only the discovery steps needed to resolve it. - -## Procedure - -1. **Monitor anchor (if the case cites a monitor).** `tsuga monitors get ` FIRST. Read the monitor's filter, aggregation, threshold, and groupBy — this IS the exact telemetry shape that crossed. Re-run the same query against the incident window AND a control window (same weekday + hour, 7 days earlier). Record the crossed value, the control value, and the ratio. -2. **Normalize scope.** `tsuga services list|get`. Capture canonical name, env, team, versions, sources, 24h log/trace counts. -3. **Normalize session.** Check `tsuga config` (active key, default cluster). Always set explicit `--from`, `--to`, `--max-results`. For `tsuga aggregation`, convert windows to epoch seconds; on multi-cluster orgs include `"clusterId"` in the body. -4. **Load tech knowledge.** If scope names a known tech (Postgres, Redis, Kafka, …), load the matching `$knowledge-technology` reference to target the sweep. -5. **Config-threshold preflight (capacity-shaped symptoms).** If the reported symptom is capacity-shaped — queue lag, `CrashLoopBackOff`, `OOMKilled`, throttling, "too many", "insufficient" — spend one probe asking _"is there a single config knob that would fix this?"_ before any elaborate change-correlation. Grep mounted codebases / helm / Pulumi for patterns like `*BatchSize`, `*PoolSize`, `*Concurrency`, `*MaxConnections`, `*FailureThreshold`, `*InFlightBatches`, `*MemPoolSize` scoped to the affected service. -6. **Evidence sweep.** Prefer in order: - - `logs new-error-patterns` / `logs error-pattern-increases` (when team scope exists) - - `logs patterns` to cluster failure shapes - - `logs search` only after pattern discovery - - `traces search` for exact failing spans - - `aggregation scalar|timeseries` for counts, rates, comparisons - - `monitors list|get` for signal semantics (not live truth) - - `dashboards list|get` / `quality-reports list` as supporting context only -7. **Compare.** Bad window vs good control window. Affected entity vs sibling healthy entity when possible. -8. **Surface verbatim signals for codebase-grep.** As the sweep produces error strings, log patterns, metric names, and monitor filters, emit them as a distinct list at the end of the output — one per line. The orchestrator will spawn a codebase-grep subagent per entry to pin each signal to its emitting `file:line`. Do not try to explain what a signal _means_ until its emitting code is found. -9. **Write evidence matrix.** Four columns: symptom evidence | subsystem evidence | mechanism clues | unknowns. If evidence only supports subsystem diagnosis, say so. -10. **Sweep completeness check.** Before returning, tick these boxes — if any is unchecked and cheap to resolve, do it now: - - [ ] Service metadata resolved (canonical name, team, env, 24h counters) - - [ ] Monitor's own query pulled + replayed against bad + control windows (if case came from a monitor) - - [ ] Config-threshold preflight done (for capacity-shaped symptoms) - - [ ] Error-log patterns scanned (`new-error-patterns` OR `patterns`) - - [ ] Primary metric aggregated in bad window AND control window - - [ ] At least one trace from the failing path inspected (when traces exist) - - [ ] Recent-deploy correlation asked (even if the answer is "no data here — defer to change branch") - - [ ] Verbatim signals surfaced for the codebase-grep branch - -## No-data honesty - -A metric or log pattern being absent is NOT equivalent to its value being zero. Causes of absence include receiver scope / permission issues, feature not enabled, instrumentation gap, or scrape failure. - -When you checked and found nothing, say which: `(absent)` — did not appear in window `|` `(not instrumented)` — scope lacks the receiver `|` `(denied)` — permission error `|` `(empty)` — query ran, returned 0 rows. Never report silent absence as "metric is zero" in a Validated claim. - -## Branch output - -``` -Observed symptom: -Monitor anchor: -Confirmed failing subsystem: -Signals that support it: - - [evidence: tsuga_logs | tsuga_traces | tsuga_aggregation | tsuga_monitors | service_metadata] -Signals that do not yet support causality: - - -Control-window comparison: -Verbatim signals for codebase-grep: - - "exact error string 1" - - "exact error string 2" - - metric.name.to_grep - - log pattern -Config-threshold preflight result: -Best next non-Tsuga check: -``` - -Every claim carries `[evidence: …]`. No tag = hypothesis, belongs in the non-causal section. - -## Branch guardrails - -- `metrics list|get` is metadata. Use `aggregation` for values. -- Monitor definitions are clues, not live truth. -- Do not claim deploy or config causality from Tsuga alone. -- Exact counts and windows > prose summaries. -- Stop at `symptom diagnosis only` when you can't tie the subsystem to a trigger. - -More detail on `tsuga` command patterns: [tsuga-rules.md](./tsuga-rules.md). diff --git a/plugins/tsuga/skills/incident-investigation/references/case-manifest.md b/plugins/tsuga/skills/incident-investigation/references/case-manifest.md deleted file mode 100644 index 42ecdd3..0000000 --- a/plugins/tsuga/skills/incident-investigation/references/case-manifest.md +++ /dev/null @@ -1,43 +0,0 @@ -# Case Manifest Contract - -Use this when the user can provide paths and context up front. - -Minimal JSON shape: - -```json -{ - "incident_ref": "INC-1234", - "summary": "Short human summary of the incident", - "time_window": { - "from": "2026-04-10T09:30:00Z", - "to": "2026-04-10T11:30:00Z" - }, - "scope": { - "services": ["web-backend"], - "clusters": ["prod-mtc"], - "customers": [], - "teams": [], - "envs": ["prod"] - }, - "paths": { - "incident_archive": "/abs/path/to/incident-reports", - "codebases": [ - "/abs/path/to/repo-one", - "/abs/path/to/repo-two" - ] - }, - "repos": [ - "owner/repo-one", - "owner/repo-two" - ], - "known_hints": [ - "started after deploy", - "possible key rotation" - ], - "constraints": { - "read_only": true - } -} -``` - -Use the manifest when it exists. If it does not, reconstruct the same fields from the prompt and only ask for the missing minimum. diff --git a/plugins/tsuga/skills/incident-investigation/references/investigation-record.md b/plugins/tsuga/skills/incident-investigation/references/investigation-record.md deleted file mode 100644 index a65f867..0000000 --- a/plugins/tsuga/skills/incident-investigation/references/investigation-record.md +++ /dev/null @@ -1,100 +0,0 @@ -# Durable deliverables: investigation record + proofs dashboard - -Every investigation that reaches a verdict ships **three** artifacts: - -1. The chat verdict (the `## Output contract` in SKILL.md). -2. A Tsuga **investigation record** (`tsuga investigations create/update`) — the durable document. -3. A **proofs dashboard** (`tsuga dashboards create`) — one graph per validated telemetry claim. - -2 and 3 are **defaults, not extras**. Skip one ONLY when: - -- the user explicitly said not to create it, or -- the verdict is a fast-path close (`healthy` / `validation_noise` after one confirmation probe), or -- the API key lacks the permission (403 — skip silently, never block or retry), or -- (dashboard only) the verdict rests purely on code/config evidence with zero telemetry claims to graph, or -- rarely, your own judgment that the artifact adds nothing for this case — judgment skips must be the exception, not a habit. - -Whenever a deliverable is skipped, the chat verdict's `Deliverables:` line must say which one and why; an unexplained skip is a contract violation. - -## Environment hygiene (before any write) - -- Verify the CLI targets the intended environment: `tsuga config` (key + default cluster). On shared machines the active config may have been switched by another session — prefer explicit `--operation-api-key` / `--cluster` flags over mutating shared config. -- Multi-cluster orgs: aggregation bodies need `"clusterId"`, and every deep link needs `clusterId=` pinned. - -## Investigation record - -Open it at Workflow step 2 with the case board, update it at gate passes, and finish it by replacing the content with the structured document below. Updates are full PUTs: always resend `name` and `owner`. - -### `name` - -Short — `: `. The app displays it everywhere; do NOT restate it inside `contentMd` (no leading `# title`). - -### `contentMd` template - -```markdown -## Summary -3–5 lines: the finding(s), impact, status. Link the proofs dashboard here. - -## Key facts -| | | -|---|---| -| Status | investigating · root cause confirmed/fixes pending · mitigated · resolved | -| Severity | with one clause of justification | -| Reported | timestamp UTC + where (channel/monitor) + incident lead | -| Detected via | monitor that fired, or "human observation — no monitor fired" (a detection gap is itself a finding) | -| Symptom onset | when symptoms actually started (≠ reported); state evidence edge if unknown | -| Services | affected services (deep-linked) + relevant sidecars/callers | -| Teams | owner · deploy · other involved | -| Impact | quantified per dimension, or "none user-visible confirmed" | - -## Timeline (UTC) -| When | Event | -|---|---| -One row per event, chronological: relevant changes/deploys, symptom onsets, -detection, key diagnosis milestones, mitigations. End with current state. - -## Symptoms -What was observed, exact values, [evidence:] tags. Symptom ≠ cause. - -## Contributing causes -Numbered (plural — avoid single-root-cause framing). Each carries: -*Trigger:* (may honestly be "not pinned" + candidates) · *Mechanism:* with -file:line pins · *Verified:* how it was confirmed [evidence: tag]. - -## Mitigation & action items -**Mitigation:** what stopped the bleeding during the incident — the action, when it -took effect, and whether it's temporary (cause still live) or also the durable fix. -"None required" if impact self-resolved or was never user-visible. - -Then the durable corrective + preventive follow-ups: -| # | Action | Type | Owner | Ticket | Status | -Type = root-fix / preventive. Owner = team, not person. Ticket "—" until filed. -End with **Verify fixes:** the observable signal that proves each fix worked. - -## Open questions -Only genuinely open items + what would close each. - -## Falsified along the way -One line per dead hypothesis + the evidence that killed it. This is what -saves the next responder from re-walking dead ends. - -## Lessons learned *(draft — finalize at postmortem)* -Went well / went wrong / got lucky. Seed it; the team finalizes 48–72h later. -``` - -### Linking rules - -- Every ID in `contentMd` whose URL shape is known gets a deep link. URL catalog: `${CLAUDE_PLUGIN_ROOT}/skills/tsuga-cli/references/app-deep-links.md`. Never invent a URL shape. -- Evidence links use an **absolute** `timeRange` (the incident window, ms) so the view never drifts; only live views (service page) use relative presets. -- `linkedAssets`: always populate — proofs dashboard, affected service(s), the fired monitor if any. Allowed types: `dashboard`, `monitor`, `service`, `slo`, `log-route`. - -## Proofs dashboard - -- `name`: ` — (proofs)`; `owner`: the affected team's id; tag `{"key": "incident", "value": ""}`. -- **One graph per validated telemetry claim.** Graph names are assertions, not metric names: "Sidecar UNHEALTHY status (1 = failing, constant)", "In-flight requests — saturated since 06-03 08:00 UTC". -- **Probe-verify every query** (`tsuga aggregation scalar`) before creating the dashboard — never ship a graph whose query returns nothing, unless absence IS the claim (then say so in the graph name). -- Time preset must cover symptom onset plus a control stretch (e.g. `past-3-days`). -- Layout: two graphs per row (`w:6,h:4`). -- Link it from the record's Summary and in `linkedAssets`. - -These two creates (plus record updates) are the only sanctioned mutations in this skill — everything else stays read-only unless the user explicitly asks. diff --git a/plugins/tsuga/skills/incident-investigation/references/playbooks/auth-tls.md b/plugins/tsuga/skills/incident-investigation/references/playbooks/auth-tls.md deleted file mode 100644 index 79cafff..0000000 --- a/plugins/tsuga/skills/incident-investigation/references/playbooks/auth-tls.md +++ /dev/null @@ -1,48 +0,0 @@ -# Auth, TLS & Credentials Playbook - -For 401/403 spikes, token errors, cert warnings, SSO failures, or "it works for some customers." - -## Cert expiry - -- Hard symptom: TLS handshake failures, "certificate has expired", clients retry with same failure. -- Category: `infrastructure` (cert lifecycle) or `configuration_error` (auto-renewal misconfigured). -- Check cert `NotAfter` directly. In the past → cert is cause. Months away → not cert expiry; move on. -- Partial failures (some clients work) often = chain/intermediate issue, not leaf cert. - -## Token / key rotation - -- No overlap window → old-token holders all fail at rotation time. `configuration_error`. -- With overlap window → expect slow tail as cached tokens refresh. Dozens per hour = normal; thousands per minute = not. -- Service-account key rotation is the most common cause of "broke at 02:00, nobody deployed." - -## IAM policy drift - -- Policy change removing a permission → affected service fails on next call. `configuration_error`. -- Look for `AccessDenied` / `Forbidden` / 403 in logs; trace to exact API + role/policy name. -- Cross-account role changes break silently until an assume-role call fails. - -## Session / OAuth / SSO - -- OIDC provider outage → login + session-refresh failures. `dependency_failure`. -- Clock skew on verifier → JWT `iat` / `exp` fail. `infrastructure`. -- IDP key rollover without verifier picking up new JWKS → signature validation fails. Check verifier JWKS cache TTL. - -## "Works for some customers" - -- Tenant config: customer may have own OAuth client, webhook secret, or API key that rotated independently. -- Region / edge: CDN or edge rule affecting a subset. -- Feature flag: flipped for one cohort. - -Don't conclude "service broken" from a partial failure — find the partition first. - -## Misleading context - -- A 401 in logs is not proof of an auth fault. Misconfigured client can 401 against a healthy server. -- "TLS error" in an HTTP library can mask DNS or TCP. -- Cert-manager / renewal that "ran successfully" can still have failed to install the renewed cert on the listener. Check the listener, not the renewer. - -## Causal chain skeletons - -- Cert expiry: auto-renew hook disabled weeks ago → cert expires at NotAfter → handshake fails → all HTTPS clients error → 5xx. -- Token rotation: secret rotated in vault → consumer pods hold old token until restart → requests fail at refresh interval → partial outage following restart schedule. -- IAM drift: policy cleanup merged → role loses `s3:GetObject` on one bucket → first S3 read fails → empty response → downstream data-quality alert. diff --git a/plugins/tsuga/skills/incident-investigation/references/playbooks/data-quality.md b/plugins/tsuga/skills/incident-investigation/references/playbooks/data-quality.md deleted file mode 100644 index 39eb0ae..0000000 --- a/plugins/tsuga/skills/incident-investigation/references/playbooks/data-quality.md +++ /dev/null @@ -1,53 +0,0 @@ -# Data Quality & Upstream Change Playbook - -For symptoms about the **shape** or **content** of data: empty output, null surge, schema mismatch, row-count anomaly, decoded-but-wrong values, "the numbers look wrong." - -## Empty vs dropped rows - -- **Zero rows out** with nonzero input → filter / transform / auth rejecting everything. Upstream fine. -- **Partial rows** with nonzero input → schema/validation path dropping a subset. Find the discriminator (tenant, record type, field value). -- **Zero rows in** → problem is upstream. `dependency_failure` or `data_quality` at source. - -## Schema drift - -- New required field in upstream payload → consumer deserializer fails → all records drop. `data_quality`. -- Existing field type change (string→int, nullable→non-null) → decode errors, silent misparsing, or cast failures. -- Field renamed → hardest to spot; producer keeps sending, consumer maps old field to default. - -Check whether the upstream team shipped in the window before blaming your pipeline. - -## Null surge - -- Sudden NULL spike on previously non-null column → producer falling back silently, or ETL step losing the field. -- Pair with a trace: find the span populating the field, check for swallowed error tags. - -## Row-count anomaly - -- 10× spike or drop vs same hour last week → something changed. Verify input count at the same boundary before investigating processing. -- Seasonal effects (weekend, holiday, TZ rollover) explain many "anomalies." Check same hour, same day last week. - -## Upstream audit - -For pipelines consuming external APIs / feeds: -- S3 audit payloads, vendor audit logs, webhook delivery records are source of truth for what arrived. -- If they show a schema change, root cause is upstream even if the symptom is your error. `data_quality`, causal chain starts at the upstream change. - -## Silent-failure patterns - -- `try/except` → log warning → return None. Record lost, no error metric rises. Watch output-count drops even when error rates look flat. -- Fallback-to-default deserialization. Data "processed" but with zeros. Watch for anomalous floors in distributions. -- Dead-letter table nobody monitors. Pipeline looks healthy while dropping X% of records. - -Recovery from silent failure usually requires backfill, not just a deploy. - -## Misleading context - -- Flat-zero dashboard is LOUDEST signal, not quietest. Flat-zero is rarely natural traffic. -- Downstream monitor on data absence can mislead. Trace upstream: input → transform → output. -- Recent schema PR in your own repo may look like the cause but be a **response** to an earlier upstream change. - -## Causal chain skeletons - -- Upstream schema change: vendor adds new required enum → deserializer throws on unknown → handler returns None → zero rows written → downstream aggregate empty. -- Silent default: deserializer falls back to `0.0` on parse error → metrics "flat at zero" → no error count rise → dashboards quiet but wrong. -- TZ boundary: producer rolled into new day 8h before consumer expected → hourly partition empty → alert on absent data. diff --git a/plugins/tsuga/skills/incident-investigation/references/playbooks/database.md b/plugins/tsuga/skills/incident-investigation/references/playbooks/database.md deleted file mode 100644 index 9b9e2bc..0000000 --- a/plugins/tsuga/skills/incident-investigation/references/playbooks/database.md +++ /dev/null @@ -1,49 +0,0 @@ -# Database Playbook - -For Postgres / MySQL / RDS / Aurora / connection-pool symptoms. Common trap: treating the loudest metric (usually CPU) as the cause. - -Concrete metrics + queries in `$knowledge-technology/postgres.md`, `/mysql.md`, `/aws-rds.md`. This file is reasoning disambiguation only. - -## Connection exhaustion vs CPU saturation - -- `active_connections` ≥ 95% of `max_connections` → `resource_exhaustion` via connection exhaustion. High CPU = secondary symptom of accumulated idle sessions. -- Both near 100% → look for **one shared cause** (pool leak holding scan-heavy queries), not two problems. -- Single bad query driving CPU ≈ 100% with connections + storage healthy → `resource_exhaustion` via CPU saturation. Check missing indexes, full-table scans, ReadIOPS spike. - -## Storage exhaustion - -- `FreeStorageSpace → 0` blocks all writes; WriteIOPS collapses, WriteLatency spikes. -- Metric missing? Infer from: WriteIOPS collapse + WriteLatency spike + RDS event "ran out of storage space". - -## Checkpoint storm / VACUUM FREEZE - -High CPU + dominant `LWLock:BufferMapping` wait + massive WriteIOPS = I/O storm from checkpointing. Category `resource_exhaustion` (I/O), NOT `code_defect`. CPU is downstream. - -## Replication lag - -- Write-heavy primary exceeds replica replay → `ReplicaLag` spikes. Cause = write workload on primary, `resource_exhaustion`. -- A concurrent analytics query on the replica may cause high CPU but doesn't explain lag. Don't pin on CPU when the failing metric is `ReplicaLag`. -- `ReplicaLag` missing? Infer from RDS events ("exceeded 900s") + high `TransactionLogsGeneration`. - -## Compositional faults - -Two independent workloads causing two separate faults simultaneously (e.g. analytics SELECT → CPU saturation AND audit_log INSERT → storage exhaustion) — identify BOTH. Do NOT merge them into a single "IOPS fault" or single `resource_exhaustion` line. - -Protocol when compositional: -- Explicitly state "two independent, coincidental faults" in the verdict. -- Provide evidence for each cause separately (analytics query evidence, audit_log query evidence). -- Trace each causal chain separately in `Causal chain`. -- Category stays `resource_exhaustion` with a multi-part ROOT_CAUSE describing both. - -Connection growth during a blocked-writer incident = symptom of queued writes, NOT connection exhaustion. Similarly, `ReplicaLag` growth during a primary write burst is a downstream symptom, not an independent fault. NEVER diagnose `connection_exhaustion` when connections spike because writes are blocked. - -## Misleading context - -- RDS event timestamps: ignore historical events (maintenance, failover, promotion) from hours before the incident. -- Oscillating metrics within normal bounds (connections 55-65%, CPU 40-70%, no errors) = `healthy`. Briefly-crossed threshold that autoscaling recovered before investigation = stale alert, `healthy`. - -## Causal chain skeletons - -- Connection leak: new PR → driver skips release on early-return → idle sessions grow → max_connections hit. -- Missing index: new query path → full scan → ReadIOPS spike → CPU saturation → latency → upstream timeouts. -- Checkpoint storm: VACUUM FREEZE → massive WAL → checkpoint flush → I/O saturation → all queries slow. diff --git a/plugins/tsuga/skills/incident-investigation/references/playbooks/deploy-drift.md b/plugins/tsuga/skills/incident-investigation/references/playbooks/deploy-drift.md deleted file mode 100644 index 6078bc0..0000000 --- a/plugins/tsuga/skills/incident-investigation/references/playbooks/deploy-drift.md +++ /dev/null @@ -1,53 +0,0 @@ -# Deploy & Config Drift Playbook - -For "it was fine yesterday" incidents. Most are a change — code, config, secret, flag, IAM policy, quota, or schema. - -Concrete commands in `$gh`. This file is reasoning disambiguation only. - -## Merged ≠ deployed - -A merged PR is a candidate only if a deploy completed after merge and before incident start. - -Verify deployment status: -- GitHub Actions: `gh run list -R owner/repo --workflow deploy.yml --conclusion success --created ">="`. -- ArgoCD / Flux: check sync status and the image tag actually deployed. -- Vercel / similar: check production deployment URL + commit SHA. -- Manual deploys: ask. Don't assume. - -## Config vs code - -- Config change (env, ConfigMap, YAML, flag) → often reversible by redeploying old config. Lower blast radius. -- Code change → revert commit or roll back image. -- Secret rotation → rarely rolled back; usually forward-fix (rotate again, update consumers). - -Category: `configuration_error` for first, `code_defect` for second. - -## Feature flags - -- Check flag flip timestamps in the window, not just deploys. A flip is a production change with no commit. -- "Off in prod" can still be a trigger if the flag gates a safety path. - -## IAM and key rotation - -- Token rotation → who rotated, what consumers hold the old token, when the overlap window was supposed to expire. -- IAM policy change → what permission was removed, what service needed it, was the change intended. -- Cert rotation → expiry date, consumers validating old fingerprint. - -These are almost always `configuration_error` or `infrastructure`, not `code_defect`. - -## Schema / upstream API - -- New required field from upstream API, removed field, changed enum → `data_quality`, even if the symptom is your 5xx. -- Schema migration with long backfill → expect lock pressure, replica lag, write amplification. Correlate with migration-start timestamp, not PR merge. - -## Misleading context - -- Release tag ≠ shipped to every environment. Check per-env rollout. -- Green CI ≠ works in prod; only proves build + unit tests. -- Dependabot / vacation-time commits are still real changes — often cause incidents. - -## Causal chain skeletons - -- Config drift: env var renamed in repo → old env still set in prod → service reads empty → auth header empty → 401. -- Unsafe migration: migration PR merged → deploy runs migration → ALTER TABLE exclusive lock → writes blocked → queue grows → timeouts. -- Silent flag flip: ops flipped flag at 13:50 → new code path loads from S3 → bucket policy denies → fail open with zero rows. diff --git a/plugins/tsuga/skills/incident-investigation/references/playbooks/kubernetes.md b/plugins/tsuga/skills/incident-investigation/references/playbooks/kubernetes.md deleted file mode 100644 index 5ec0f2b..0000000 --- a/plugins/tsuga/skills/incident-investigation/references/playbooks/kubernetes.md +++ /dev/null @@ -1,64 +0,0 @@ -# Kubernetes Playbook - -For pods, deployments, nodes, or cluster-level symptoms. Trap: pinning cause on the restart/pod event when the restart is itself a symptom. - -Concrete metrics in `$knowledge-technology/kubernetes.md`. Remediation recipes per failure mode in `$knowledge-technology/kubernetes/runbooks.md` (CrashLoopBackOff, OOMKilled, ImagePullBackOff, Pending, NodeNotReady, DiskPressure, HighCPU). This file is reasoning disambiguation only — pick a runbook *after* the investigation validates the category, never before. - -## Symptom vs root cause - -- `CrashLoopBackOff` is symptom. Cause = bad config, missing secret, OOM, failed probe, broken dependency at startup, image pull failure, or corrupted volume. -- `OOMKilled` can be cause (leak, under-allocated limit) OR symptom (downstream returns huge response, payload explosion). Check memory trend + request/response size in the same window. -- Pod restart spike is not a root cause. Restarts are evidence something else failed — find that. - -## Scheduling and probes - -- `PodPending` + `FailedScheduling` → capacity or affinity problem (`resource_exhaustion` or `configuration_error`), not the pod. -- `Readiness probe failed` → pod started but can't serve. Check the probe target (usually `/health`); failure is upstream of probe config unless probe itself just changed. -- `Liveness probe failed` + restart loop → probe too strict (config) or service genuinely hung (code / dependency). - -## Validation failures: configuration error vs data quality - -When logs show the application validating incoming data against a configured list of required fields and a field in that list is missing from the data, the root cause is typically `configuration_error` — the required-fields list was misconfigured in the Job manifest, ConfigMap, or env vars — NOT `data_quality`. - -Data quality issues manifest as **malformed values** or **corrupted records** (bad JSON, wrong types, null in non-null column). A mismatch between a configured field list and the data schema means the configuration was wrong relative to what the producer reliably emits, so `configuration_error` on the consumer side. - -Counter-example: if the producer JUST changed its schema and the consumer's config is unchanged, it's `data_quality` (upstream API change). The distinction is "did the producer change, or did the config?" - -## Node vs pod vs shared infra - -- Many pods on same node failing → suspect node. Check `NodeReady`, disk pressure, kubelet health. -- Many pods across nodes in same deployment failing → suspect deployment (new image / config / secret). -- Many services on same node failing → suspect shared infra (node, network, control plane). - -## Deploy correlation - -- Pod failures right after a rollout → prime suspect is the new image or updated ConfigMap / Secret. Use `$incident-investigation` to find the PR / run. -- `ImagePullBackOff` post-rollout → registry auth, image tag typo, or registry outage. Check Deployment spec's image vs what's in the registry. - -## Misleading context - -- A single `OOMKilled` from 6h ago is noise if the current incident is a fresh 5xx spike. -- `Evicted` pods are node-level pressure symptom, not the application cause. -- `kubectl rollout restart` in the window is an operator response, not a trigger. Find who / why. - -## Causal chain skeletons - -- OOM: traffic spike OR payload growth → heap past limit → OOMKilled → restart → cold cache → latency → 5xx. -- Bad secret rotation: secret rotated → ConfigMap not redeployed → pods read stale → auth fails at startup → CrashLoopBackOff. -- Bad readiness probe: probe path changed in release → /health mismatch → pod never ready → rollout stuck → no new pods serve. - -## Runbooks (post-verdict only) - -Once the investigation has assigned a `category`, the matching runbook from `$knowledge-technology/kubernetes/runbooks.md` supplies concrete remediation steps. Map: - -| `category` | Runbook | -|---|---| -| `crashloop` | CrashLoopBackOff Recovery | -| `oom` | OOMKilled Recovery | -| `image_pull` | ImagePullBackOff Recovery | -| `pending` | Pending Pods Recovery | -| `node_not_ready` | Node Not Ready Recovery | -| `disk_pressure` | Disk Pressure Recovery | -| `high_cpu` | High CPU Usage Recovery | - -Pick a runbook only after the verdict; never let a runbook category force-fit the cause. Each step is tagged `Destructive` / `Human` — show the step + wait for explicit user confirmation before applying anything that changes cluster state. diff --git a/plugins/tsuga/skills/incident-investigation/references/playbooks/queue-backpressure.md b/plugins/tsuga/skills/incident-investigation/references/playbooks/queue-backpressure.md deleted file mode 100644 index 6f774e6..0000000 --- a/plugins/tsuga/skills/incident-investigation/references/playbooks/queue-backpressure.md +++ /dev/null @@ -1,47 +0,0 @@ -# Queue & Backpressure Playbook - -For Kafka / RabbitMQ / SQS / Pub/Sub / Kinesis / Celery / Sidekiq / bounded channels. - -Concrete metrics in `$knowledge-technology/kafka.md`, `/rabbitmq.md`, `/aws-sqs.md`, `/gcp-pubsub.md`. This file is reasoning disambiguation only. - -## Producer vs consumer - -- Lag growing + flat produce rate → consumer-side problem (slow processing, crash loop, DB dependency). -- Lag growing + spiking produce rate → producer-side burst or correct-but-insufficient capacity. -- Lag growing + falling produce rate + consume at zero → consumer is dead. Check worker health before assuming capacity. - -## Consumer: slow vs crashed vs stuck - -- Slow: consumer consuming < produce rate. Check per-message time + downstream (DB, HTTP call). -- Crashed: zero consumption. Check worker pod/process, lock contention, poison messages. -- Stuck: alive but waits forever (DB txn, HTTP with no timeout). Worse than crashed — monitoring often doesn't catch. - -## Poison message - -One unprocessable message blocking a partition / queue is a common trap: -- Kafka: partition stuck at one offset. Check per-partition lag, not aggregate. -- SQS: retry storm of one message. Check `ApproximateReceiveCount` outliers. -- DLQ growing → poison-message signal. - -Category: `data_quality` (bad input) or `code_defect` (bad handler), not `resource_exhaustion`. - -## Fan-out amplification - -One upstream event → N downstream jobs. A 10× upstream spike can be 100× downstream. Check amplification before blaming the worker. - -## Broker vs application - -- Broker healthy (low CPU, normal replication, no disconnects) + app lag growing → application problem. -- Broker unhealthy → infrastructure problem, not worker sizing. - -## Misleading context - -- Lag spike during deploy can be the expected rolling-restart pause, not an incident. Check deploy timestamps. -- Aggregate lag hides stuck partitions. Always drill per-partition. -- Queue "clearing" may be retention dropping messages (data loss), not workers catching up. - -## Causal chain skeletons - -- Slow consumer: new PR adds a DB call per message → DB P95 climbs → worker throughput halves → lag grows linearly. -- Poison message: producer schema change → deserializer throws → retry storm → worker CPU spikes → aggregate lag grows. -- Broker disk: broker disk fills → writes throttle → producers block → upstream queues fill → cascading timeout. diff --git a/plugins/tsuga/skills/incident-investigation/references/tsuga-rules.md b/plugins/tsuga/skills/incident-investigation/references/tsuga-rules.md deleted file mode 100644 index e00ec71..0000000 --- a/plugins/tsuga/skills/incident-investigation/references/tsuga-rules.md +++ /dev/null @@ -1,44 +0,0 @@ -# Tsuga Investigation Rules - -Use these defaults: - -1. Start with `tsuga services list|get`. - Resolve canonical service, env, team, versions, and whether logs/traces exist. - -2. Check `tsuga config` (active key, default cluster). - Override `--from`, `--to`, and `--max-results` explicitly. - On multi-cluster orgs, `aggregation` bodies need a `"clusterId"` field. - -3. Prefer high-signal commands first. - - `tsuga logs new-error-patterns` - - `tsuga logs error-pattern-increases` - - `tsuga logs patterns` - - `tsuga aggregation scalar|timeseries` - -4. Use `aggregation` for claims. - Counts, rates, latency, queue growth, and comparisons should come from aggregation, not from eyeballing raw search results. - -5. Remember the CLI limits. - - `metrics list|get` are metadata only - - monitor commands expose definitions, not live firing truth - - `aggregation` requires epoch seconds - - `quality-reports` are posture context, not real-time truth - -6. Compare a bad window against a control window whenever possible. - -7. Stop at subsystem diagnosis if the trigger is still missing. - -Read-only commands to prefer: -- `logs search` -- `logs patterns` -- `logs new-error-patterns` -- `logs error-pattern-increases` -- `traces search` -- `metrics list|get` -- `aggregation scalar|timeseries` -- `services list|get` -- `dashboards list|get` -- `monitors list|get` -- `teams list|get` -- `quality-reports list` -- `config` diff --git a/plugins/tsuga/skills/knowledge-technology/SKILL.md b/plugins/tsuga/skills/knowledge-technology/SKILL.md index 0a37022..1351b37 100644 --- a/plugins/tsuga/skills/knowledge-technology/SKILL.md +++ b/plugins/tsuga/skills/knowledge-technology/SKILL.md @@ -1,24 +1,21 @@ --- name: knowledge-technology -description: "Per-technology reference bundles with exact Tsuga metric names, incident shapes, derived signals, and log patterns for ~35 techs (postgres, mysql, redis, kafka, rabbitmq, cassandra, kubernetes, nginx, haproxy, envoy, istio, jvm, otel-collector, quickwit, aws-rds, aws-lambda, aws-ecs, aws-sqs, aws-dynamodb, aws-elasticache, gcp-pubsub, gcp-storage, …). Trigger before composing any `tsuga aggregation / tsuga logs / tsuga traces` query, or when an incident scope / error log / monitor name mentions a covered tech or a classic symptom (OOMKilled, CrashLoopBackOff, connection pool, deadlock, queue lag, compaction, throttle, replication lag, cold start, 5xx). Source-system metric names (CloudWatch CPUUtilization, etc.) do NOT work in Tsuga — use the `tsuga_metric_name` column in each tech's `metrics.csv` (AWS metrics register as `aws_rds_cpu_utilization`, `aws_lambda_errors`, …)." +description: 'Per-technology reference bundles with exact Tsuga metric names, incident shapes, derived signals, and log patterns for ~35 techs (postgres, mysql, redis, kafka, rabbitmq, cassandra, kubernetes, nginx, haproxy, envoy, istio, jvm, otel-collector, quickwit, aws-rds, aws-lambda, aws-ecs, aws-sqs, aws-dynamodb, aws-elasticache, gcp-pubsub, gcp-storage, …). Trigger before composing any `tsuga aggregation / tsuga logs / tsuga traces` query, or when an incident scope / error log / monitor name mentions a covered tech or a classic symptom (OOMKilled, CrashLoopBackOff, connection pool, deadlock, queue lag, compaction, throttle, replication lag, cold start, 5xx). Bundles are fetched from Tsuga with `tsuga docs get references/technologies//{overview,metrics,queries}`. Source-system metric names (CloudWatch CPUUtilization, etc.) do NOT work in Tsuga — use the `tsuga_metric_name` column of the `metrics` page (AWS metrics register as `aws_rds_cpu_utilization`, `aws_lambda_errors`, …).' --- # Knowledge — Technology -One folder per technology under `references/`. Each folder contains: +Three pages per technology, served by Tsuga and fetched by exact path: ``` -references// -├── README.md ← upstream integration bundle README (overview, concepts, glossary) -├── metrics.csv ← full metric inventory; column `tsuga_metric_name` = exact string to query in Tsuga -└── queries.md ← incident shapes, derived signals, log patterns, gotchas +references/technologies//overview ← overview, concepts, glossary +references/technologies//metrics ← CSV; column `tsuga_metric_name` = exact string to query in Tsuga +references/technologies//queries ← incident shapes, derived signals, log patterns, gotchas ``` -**Always use `tsuga_metric_name` from `metrics.csv` in actual `tsuga` queries.** Source-system names (e.g. CloudWatch `CPUUtilization`) do NOT work - Tsuga registers AWS metrics as `aws_rds_cpu_utilization`, etc. When in doubt, grep the CSV. +Fetch with `tsuga docs get `, which prints `{path, title, content}` JSON. Pipe through `jq -r .content` to get the raw page. These pages are path-addressed only and never appear in `tsuga docs search`, so use the covered-technologies list below to pick a path. -## Tool note — use `rtk` for scans - -The container ships `rtk` (https://github.com/rtk-ai/rtk), a CLI proxy that compresses `ls` / `read` / `grep` / `find` output to save tokens. Prefer `ls`, `cat`, `grep`, `find` over the bare commands when browsing this tree — the `references/` folder is big enough that raw `cat` / `ls -R` will eat context. `awk` / `sed` / `sort` / `jq` are not wrapped; keep them as-is. +**Always use `tsuga_metric_name` from the `metrics` page in actual `tsuga` queries.** Source-system names (e.g. CloudWatch `CPUUtilization`) do NOT work - Tsuga registers AWS metrics as `aws_rds_cpu_utilization`, etc. ## Generic investigation rule @@ -38,49 +35,42 @@ Compare the **bad window** (incident) against a **good control window** (same we Not listed? Fall back to a generic sweep via `$incident-investigation`. -## Shell commands - finding the right tech + metrics +## Shell commands - finding the right metrics -From the container shell (`${CLAUDE_PLUGIN_ROOT}/skills/knowledge-technology/` is `$TK` below): +Fetch a page once into a variable, then filter locally. Do not re-fetch per lookup. ```bash -TK=${CLAUDE_PLUGIN_ROOT}/skills/knowledge-technology/references - -# Which techs are covered? -ls "$TK" - -# Does this repo have a reference for ? -ls "$TK" | grep -i redis -grep -l -r -i "connection pool" "$TK"/*/README.md +# Grab a tech's metric catalog (CSV) once +RDS=$(tsuga docs get references/technologies/aws-rds/metrics | jq -r .content) -# Show the queries file (fastest read) -cat "$TK/postgres/queries.md" - -# List every tsuga_metric_name for a tech (exact strings for Tsuga queries) -awk -F, 'NR>1 {print $7}' "$TK/aws-rds/metrics.csv" | sort -u +# List every tsuga_metric_name (exact strings for Tsuga queries) +printf '%s\n' "$RDS" | awk -F, 'NR>1 {print $7}' | sort -u # Filter metrics by theme (Availability/Health, Capacity/Saturation, Performance/Latency, Errors/Failures, Throughput/Usage) -awk -F, 'NR>1 && $1=="Capacity/Saturation" {print $7}' "$TK/postgres/metrics.csv" +printf '%s\n' "$RDS" | awk -F, 'NR>1 && $1=="Capacity/Saturation" {print $7}' # Look up a metric's definition + aggregation + group_by -awk -F, 'NR>1 && $2=="postgresql.backends" {print "def:"$4"\nagg:"$9"\npost:"$10"\ngroup_by:"$11}' "$TK/postgres/metrics.csv" +printf '%s\n' "$RDS" | awk -F, 'NR>1 && $2=="FreeStorageSpace" {print "def:"$4"\nagg:"$9"\npost:"$10"\ngroup_by:"$11}' -# Source-name → tsuga-name lookup (critical for AWS) -awk -F, 'NR>1 {print $2" → "$7}' "$TK/aws-rds/metrics.csv" | grep -i cpu +# Source-name to tsuga-name lookup (critical for AWS) +printf '%s\n' "$RDS" | awk -F, 'NR>1 {print $2" -> "$7}' | grep -i cpu -# Cross-tech: which techs have a metric matching a pattern? -grep -l -r "replication" "$TK"/*/metrics.csv | sed 's|.*/references/||;s|/metrics.csv||' +# Incident shapes and query recipes for a tech (fastest read) +tsuga docs get references/technologies/postgres/queries | jq -r .content -# Glossary / concept lookup inside a tech's README -grep -B1 -A3 -i "bloom filter" "$TK/cassandra/README.md" +# Glossary / concept lookup inside a tech's overview +tsuga docs get references/technologies/cassandra/overview | jq -r .content | grep -B1 -A3 -i "bloom filter" ``` +Cross-tech search ("which techs expose a replication metric?") needs one fetch per tech, so it is expensive. Narrow to two or three candidates from the covered-technologies list first, then fetch only those. + ## Shell commands - composing a Tsuga query Once you have the exact `tsuga_metric_name`, compose via `$tsuga-cli`. Aggregation body uses `timeRange` + `dataSource` + `queries` (see `$tsuga-cli/SKILL.md` for the full schema). GNU `date -d` works on the container (Linux); avoid BSD-only flags like `-v-2H`. ```bash -METRIC=$(awk -F, 'NR>1 && $2=="FreeStorageSpace" {print $7; exit}' "$TK/aws-rds/metrics.csv") -echo "$METRIC" # → aws_rds_free_storage_space +METRIC=$(printf '%s\n' "$RDS" | awk -F, 'NR>1 && $2=="FreeStorageSpace" {print $7; exit}') +echo "$METRIC" # -> aws_rds_free_storage_space FROM_EPOCH=$(date -u -d '-2 hours' +%s) TO_EPOCH=$(date -u +%s) @@ -98,13 +88,14 @@ tsuga aggregation scalar -d "$(jq -n \ ## Boundary -- **This skill** - *where to look, what to query, exact metric names*. -- **`$tsuga-cli`** - *how to drive the CLI* (aggregation body syntax, flags). -- **`$incident-investigation/references/playbooks/`** - *how to reason* (disambiguation traps). +- **This skill** - _where to look, what to query, exact metric names_. +- **`$tsuga-cli`** - _how to drive the CLI_ (aggregation body syntax, flags). +- **`references/incident-response/playbooks/`** (via `tsuga docs get`) - _how to reason_ (disambiguation traps). ## Anti-patterns -- Do not guess metric names. Always pull from `tsuga_metric_name` column of `metrics.csv`. +- Do not guess metric names. Always pull from the `tsuga_metric_name` column of the `metrics` page. - Do not paste CloudWatch / OTel source names directly into `tsuga` - Tsuga registers them under different strings (see the CSV). -- Do not dump the whole catalog. Filter by theme or incident shape. Use `cat` / `grep` not `cat` / bare `grep` on the reference trees — otherwise a single file scan can blow 20k tokens. +- Do not dump a whole catalog into context. Capture it in a shell variable and `awk` it down by theme or incident shape; a raw `jq -r .content` on a large `metrics` page can blow 20k tokens. +- Do not re-fetch the same page. One `tsuga docs get` per tech per session, held in a variable. - Metric missing ≠ value zero. Receiver scope / permission issues look like absence. Say so explicitly. diff --git a/plugins/tsuga/skills/knowledge-technology/references/apache/README.md b/plugins/tsuga/skills/knowledge-technology/references/apache/README.md deleted file mode 100644 index 16ec0a9..0000000 --- a/plugins/tsuga/skills/knowledge-technology/references/apache/README.md +++ /dev/null @@ -1,352 +0,0 @@ -# Apache HTTP Server Integration Context Bundle - -## Metadata - -**Technology:** Apache HTTP Server -**Tech Slug:** apache-http-server -**Deployment:** self-hosted -**Environment:** prod -**Persona:** SRE, Dev, Ops -**Telemetry:** mixed (OTel apache receiver + mod_status) -**Scope:** core service only -**Use-case:** reliability and performance -**Where it runs:** Kubernetes (also on-prem bare metal / VM) -**Cloud provider:** N/A -**Tsuga context fields:** context.team, context.env -**Bundle created:** 2026-04-01 -**Stage:** 1 (bundle creation complete; Stage 2 discovery pending) - ---- - -## How to use this bundle - -1. **Stage 1 (this bundle):** Reference files 00–04 to understand the technology, confirm metric names, and plan the dashboard. -2. **Stage 2 (discovery):** Run metric discovery against a live Apache instance (or OTel Collector scrape endpoint) to validate exact attribute names, confirm temporality, and fill unknowns listed in `03_apache-http-server_state.yaml`. -3. **Stage 3 (dashboard build):** Use `02_apache-http-server_dashboard_plan.yaml` as the authoritative input to `_build_apache-http-server.py`. Run quality gates before push. -4. **Stage 4 (log intelligence):** Wire access log and error log parsing (see "Log intelligence" section below) to complement metric coverage — error rates come from logs, not from the apache receiver. - -The metrics CSV (`01_`) is the single source of truth for what to chart. The dashboard plan YAML (`02_`) is the authoritative layout spec. The state YAML (`03_`) tracks what is confirmed vs inferred. The memory file (`04_`) is the carry-forward summary for future sessions. - ---- - -## What it is and what "good" looks like - -Apache HTTP Server (httpd) is the most widely deployed open-source web server. It serves static files, proxies requests to upstream application servers, terminates TLS, and optionally runs embedded scripting (mod_php, mod_wsgi). In modern deployments it is commonly fronted by a load balancer and runs inside Kubernetes pods or bare-metal hosts. - -### What "good" looks like for Apache - -| Signal | Healthy baseline | Warning threshold | Critical threshold | -|---|---|---|---| -| Worker utilization | < 60 % busy | 60–80 % | > 80 % (saturation imminent) | -| Request rate | Stable or growing with traffic | Sudden drop (> 30 % from baseline) | Near-zero during expected traffic | -| Request time (apache.request.time rate) | < 200 ms p95 per worker | 200–500 ms | > 500 ms sustained | -| 5xx error rate (log-derived) | < 0.1 % of requests | 0.1–1 % | > 1 % | -| 4xx error rate (log-derived) | < 2 % of requests | 2–5 % | > 5 % sustained | -| Async keepalive connections | < 50 % of MaxRequestWorkers | 50–70 % | > 70 % (event MPM connection leak risk) | -| CPU load (apache.cpu.load) | < 50 % | 50–80 % | > 80 % | -| Server uptime | Increasing | Any restart = alert | Multiple restarts in window | - -Apache running the **event MPM** (default since Apache 2.4) separates connection handling from request processing. Keepalive connections are held by lightweight async threads, so `apache.connections.async{keepalive}` can be high without saturating workers. The correct saturation metric is `apache.workers{workers_state=busy}` / (`busy` + `idle`). - ---- - -## Key concepts - -### Glossary - -| Term | Definition | Operational meaning | Dashboard section | -|---|---|---|---| -| MPM | Multi-Processing Module. Controls how Apache handles concurrent connections. | Determines concurrency model — prefork (process), worker (thread), event (async). | saturation | -| event MPM | Apache's default MPM since 2.4. Uses async I/O for keepalive connections. | Idle async keepalives don't consume a worker slot; saturation = busy workers, not total connections. | saturation, connections | -| prefork MPM | One process per connection. No threads. | Higher memory usage; MaxRequestWorkers = max concurrent connections. | saturation | -| worker MPM | Hybrid: multiple processes, multiple threads per process. | Better concurrency than prefork with lower memory than one-process-per-connection. | saturation | -| MaxRequestWorkers | Hard limit on simultaneous requests Apache will serve. | The primary saturation ceiling. Formerly called MaxClients. | saturation | -| ServerLimit | Maximum number of processes (prefork) or threads (worker/event). | Absolute ceiling; requires httpd restart to change. | saturation | -| mod_status | Apache module exposing real-time server status at `/server-status`. | Source of all OTel apache receiver metrics. Must be enabled and accessible. | telemetry | -| Scoreboard | Internal Apache data structure tracking worker slot state. | Each slot maps to one possible connection. Slot states reveal what workers are doing. | scoreboard-detail | -| Busy worker | A worker slot actively processing a request (reading, sending, etc.). | Used in saturation calculation. Rises toward MaxRequestWorkers under load. | saturation, availability | -| Idle worker | A worker slot available to accept new requests. | Headroom metric. Should stay > 20 % of total under normal load. | saturation | -| Keepalive slot | Async connection waiting for next request on a persistent HTTP connection. | With event MPM, does not block a worker. High keepalive = normal. | connections | -| Scoreboard state: open | Slot not yet assigned to any connection. | Represents maximum additional concurrency available above current MaxRequestWorkers-in-use. | scoreboard-detail | -| Scoreboard state: waiting | Slot assigned but idle (listening for request). | Normal healthy idle state. | scoreboard-detail | -| Scoreboard state: reading | Slot reading the incoming request headers/body. | Short duration expected. Prolonged = slow clients or large uploads. | scoreboard-detail | -| Scoreboard state: sending | Slot writing response to client. | The primary "active work" state. High sustained value = throughput load. | scoreboard-detail | -| Scoreboard state: dnslookup | Slot performing DNS resolution. | Should be near zero. Non-zero = DNS latency impacting request handling. | scoreboard-detail | -| Scoreboard state: closing | Slot closing the connection. | Brief transition state. High value = connection teardown bottleneck. | scoreboard-detail | -| Scoreboard state: logging | Slot writing to access log. | Brief. High value = logging I/O bottleneck. | scoreboard-detail | -| Scoreboard state: finishing | Slot performing graceful shutdown cleanup. | Non-zero during graceful restarts. | scoreboard-detail | -| Scoreboard state: idle_cleanup | Slot being cleaned up after idle timeout. | Brief. High sustained = aggressive keepalive timeout cycling. | scoreboard-detail | -| apache.request.time | Cumulative monotonic counter of total milliseconds spent processing requests. | Rate gives average throughput time across all workers (not per-request latency). | throughput | -| apache.traffic | Cumulative bytes transferred. | Rate gives bandwidth in B/s. | throughput | -| apache.requests | Cumulative total requests handled. | Rate gives requests/second. | throughput | -| apache.cpu.load | Gauge: current CPU utilization percentage for the httpd process. | High CPU with low request rate = inefficiency. | availability | -| apache.connections.async | Async connection count by state (writing, keepalive, closing). | Event MPM specific. High keepalive is normal; high closing may indicate client issues. | connections | -| CLF | Common Log Format. Standard Apache access log format. | Used for log parsing to derive 4xx/5xx error rates. | errors | -| Combined Log Format | CLF + Referer + User-Agent fields. | Most common real-world format. Enables referrer/UA analysis. | errors | -| VirtualHost | Apache configuration block for a named site. | Multiple VirtualHosts on one Apache instance = per-site metrics not natively separated in OTel receiver. | cardinality | -| mod_proxy | Apache reverse proxy module. | Upstream errors appear as 5xx in access logs. Upstream health is not directly exposed via mod_status. | errors | - -### Concept Map - -``` -Client -> TCP connection -> event MPM listener (why: event MPM handles keepalives async) -event MPM listener -> worker thread assignment (why: only active requests consume worker slots) -worker thread -> request parsing -> scoreboard state: reading (why: slot transitions on request receive) -worker thread -> response generation -> scoreboard state: sending (why: slot transitions on response send) -worker thread -> access log write -> scoreboard state: logging (why: brief I/O state before slot release) -worker thread -> slot release -> scoreboard state: waiting (why: slot returns to idle pool) -apache.workers{busy} -> saturation numerator (why: busy = slots actively processing requests) -apache.workers{idle} -> saturation denominator complement (why: idle = headroom remaining) -apache.workers{busy} / (busy + idle) -> Worker Utilization % (why: primary saturation KPI) -Worker Utilization > 80% -> MaxRequestWorkers ceiling approached (why: no headroom for traffic spike) -MaxRequestWorkers -> ServerLimit (why: ServerLimit is the absolute ceiling requiring restart) -apache.scoreboard{open} -> unallocated capacity (why: open slots not yet assigned to workers) -apache.scoreboard{waiting} -> allocated idle capacity (why: worker exists but not processing) -apache.scoreboard{keepalive} -> async keepalive connections (why: event MPM holds these without blocking workers) -apache.scoreboard{dnslookup} -> DNS resolution in progress (why: spike = DNS degradation impacting latency) -apache.requests rate -> requests/second (why: primary throughput signal) -apache.traffic rate -> bytes/second (why: bandwidth consumption signal) -apache.request.time rate -> aggregate processing time rate (why: proxy for average request handling load) -apache.cpu.load -> process-level CPU % (why: complements request rate for efficiency analysis) -apache.connections.async{keepalive} -> persistent HTTP/1.1 + HTTP/2 clients (why: event MPM async count) -apache.connections.async{writing} -> connections currently writing response (why: active response I/O) -apache.connections.async{closing} -> connections in teardown (why: high = connection close bottleneck) -apache.uptime rate -> near-zero normal; spike to negative = restart detected (why: cumulative uptime resets on restart) -mod_status endpoint -> OTel apache receiver scrapes (why: all apache.* metrics originate here) -OTel apache receiver -> Tsuga ingest pipeline (why: telemetry path for apache.* metrics) -access log -> log shipper -> Tsuga log routes (why: 4xx/5xx errors come from logs not mod_status) -error log -> log shipper -> Tsuga log routes (why: startup errors, module failures in error log) -context.env -> global dashboard filter (why: separates prod/staging/dev instances) -context.team -> global dashboard filter (why: ownership routing for alert triage) -context.scope.name -> per-instance filter (why: server.address from OTel resource attributes) -VirtualHost -> single apache.* metric stream (why: OTel receiver does not disaggregate by vhost natively) -``` - -### Entities and dimensions - -| Entity / Dimension | Why useful | Cardinality risk | Safe top-N | -|---|---|---|---| -| Instance (server.address + server.port) | Per-host breakout essential for fleet debugging | Low — typically 1–20 instances per team | All | -| Environment (context.env) | Separates prod from staging/dev | Very low (2–4 values) | All | -| Team (context.team) | Ownership routing; alert assignment | Low (5–20 teams) | All | -| MPM type | Different saturation profiles per MPM | Static per host (1 value) | N/A | -| workers_state (busy/idle) | Core saturation split | 2 values — safe | All | -| scoreboard_state | Detailed slot state breakdown | 12 values — safe | All | -| connection_state (writing/keepalive/closing) | Async connection breakdown | 3 values — safe | All | -| cpu_level (self/children) | CPU attribution to parent/child processes | 2 values — safe | All | -| cpu_mode (system/user) | Kernel vs user CPU split | 2 values — safe | All | -| VirtualHost name | Per-site breakout | Medium — depends on config (1–50+) | Top 10 | -| HTTP status code (log-derived) | Error rate breakdown | Low for grouped (1xx/2xx/3xx/4xx/5xx) | All grouped | -| HTTP method (log-derived) | GET/POST/PUT/DELETE split | Very low (4–8 common values) | All | -| Request URI prefix (log-derived) | Top endpoints by traffic/error | High — unbounded URI space | Top 20 | - -### Tsuga field mapping - -| Vendor/exporter dimension | Recommended context.* key | Must-exist vs optional | -|---|---|---| -| server.address (OTel resource attribute) | context.scope.name or context.server.address | Must-exist (instance identification) | -| server.port (OTel resource attribute) | context.server.port | Optional (disambiguates multi-port) | -| deployment.environment (OTel resource attribute) | context.env | Must-exist | -| service.name (OTel resource attribute) | context.service.name | Must-exist | -| team label (K8s pod label or static config) | context.team | Must-exist | -| workers_state attribute | N/A — filter in query | N/A | -| scoreboard_state attribute | N/A — filter in query | N/A | -| connection_state attribute | N/A — filter in query | N/A | - ---- - -## Golden signals - -### Traffic (Throughput) - -- **Primary metric:** `apache.requests` (cumulative sum, rate → req/s) -- **Secondary metric:** `apache.traffic` (cumulative sum, rate → B/s) -- **Derived:** Request rate per instance; bandwidth per instance -- **What to look for:** Request rate drop > 30 % from baseline without a corresponding upstream change is a leading indicator of worker saturation or process crash. - -### Errors - -- **Primary source:** Access logs (4xx/5xx HTTP status codes) — NOT available from apache receiver metrics -- **Secondary source:** Apache error log (startup failures, module errors, misconfiguration) -- **Derived:** 5xx rate (%), 4xx rate (%), error rate trend -- **What to look for:** 5xx spike = upstream proxy failure or Apache internal error. 4xx spike = client request pattern change or auth/ACL misconfiguration. -- **Important:** The OTel apache receiver does NOT export per-status-code counters. Error observability requires log routing. - -### Latency - -- **Primary metric:** `apache.request.time` (cumulative sum ms, rate → ms consumed per second across all workers) -- **Derived:** Average request processing time = apache.request.time rate / apache.requests rate (ms/req) -- **What to look for:** Rising request time with stable request rate = upstream application slowing down. Rising request time AND rising request rate = normal load growth. -- **Limitation:** This is aggregate latency across all workers, not a per-request histogram. No P95/P99 available from mod_status alone. - -### Saturation - -- **Primary metric:** `apache.workers{workers_state=busy}` / (`apache.workers{busy}` + `apache.workers{idle}`) — Worker Utilization % -- **Secondary metric:** `apache.scoreboard{scoreboard_state=open}` — remaining unallocated capacity -- **Tertiary metric:** `apache.connections.async{connection_state=keepalive}` — async keepalive pressure (event MPM) -- **What to look for:** Worker utilization > 80 % sustained = approach to MaxRequestWorkers limit. New requests will queue or be refused when limit is hit. Open scoreboard slots near zero = absolute ceiling reached. - ---- - -## Telemetry sources - -| Source | Transport | Metrics | Logs | Traces | Notes | -|---|---|---|---|---|---| -| OTel apache receiver | HTTP pull from `/server-status?auto` | Yes (`apache.*`) | No | No | Requires `mod_status` enabled; `ExtendedStatus On` for cpu/request.time | -| Apache access log | File / stdout (K8s) | No | Yes (CLF / Combined) | No | Contains HTTP status codes, latency per-request, bytes, URI | -| Apache error log | File / stdout (K8s) | No | Yes | No | Contains startup errors, module failures, AH-prefixed error codes | -| mod_status HTML | Browser / synthetic check | Partial | No | No | Human-readable; not machine-parseable for OTel | -| OTel host metrics receiver | Host-level scrape | Partial (CPU, memory, disk) | No | No | Complements apache.* for host-level resource context | - ---- - -## Log intelligence (Stage 4 handoff) - -### Access log format - -**Common Log Format (CLF):** -``` -%h %l %u %t \"%r\" %>s %b -``` -Example: `192.168.1.1 - frank [10/Oct/2025:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326` - -**Combined Log Format (most common in production):** -``` -%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\" -``` - -**Extended with latency (recommended for Stage 4):** -``` -%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\" %D -``` -`%D` = time to serve request in microseconds. Parse and convert to ms for latency histograms. - -### Error log format - -``` -[DayOfWeek Mon DD HH:MM:SS.usec YYYY] [module:level] [pid PID] message -``` -Error codes use `AH` prefix (e.g., `AH00163`, `AH01215`). Filter on `[error]` or `[crit]` severity for alerting. - -### Candidate query filters - -| Precision | Filter string | -|---|---| -| Precise (service name) | `service.name:apache OR service.name:httpd` | -| Precise (scope name) | `scope.name:apache-http-server` | -| Fallback | `apache` | -| 5xx errors | `status:[500 TO 599]` (if status field extracted) | -| 4xx errors | `status:[400 TO 499]` | -| Error log only | `log.file.path:*error_log* OR log.file.path:*error.log*` | - -### Attribute mapping hints - -| Log field | OTel semantic convention | Tsuga field | -|---|---|---| -| %h (remote IP) | client.address | context.client.address | -| %u (auth user) | enduser.id | context.user.id | -| %r (request line) | http.request.method + url.path + network.protocol.version | context.http.method | -| %>s (status code) | http.response.status_code | context.http.status_code | -| %b (response bytes) | http.response.body.size | context.http.response.bytes | -| %{Referer}i | http.request.header.referer | context.http.referer | -| %D (microseconds) | http.server.request.duration | context.http.duration_us | - -### Parsing risks - -- Log format varies by VirtualHost and Apache version — validate format string against actual log lines before building parser rules. -- `%b` reports `-` (not `0`) for zero-byte responses (e.g., 304 Not Modified). Parser must handle null/dash. -- `%t` uses Apache's own time format (not ISO 8601). Requires custom time parsing: `[%d/%b/%Y:%H:%M:%S %z]`. -- Rotated logs (logrotate or piped logging) may cause brief gaps in log stream. -- K8s deployments typically write access log to stdout/stderr — ensure log shipper is configured to parse structured JSON or raw CLF from pod logs. - ---- - -## Caveats and footguns - -- **[availability]** `apache.uptime` is a monotonic cumulative sum. A restart resets the counter, causing a rate-spike and then near-zero rate. Do not alert on rate; alert on the raw value dropping near zero or resetting. -- **[availability]** `mod_status` must be enabled (`LoadModule status_module`) and `ExtendedStatus On` must be set for `apache.cpu.time`, `apache.cpu.load`, and `apache.request.time` to be populated. Without `ExtendedStatus`, these metrics return zero or are absent. -- **[saturation]** `MaxRequestWorkers` is NOT exposed as a metric by the apache receiver. You cannot compute worker utilization as a percentage of a known ceiling without querying the Apache config or setting it as a custom label/annotation. -- **[saturation]** Worker utilization denominator uses `busy + idle` (which equals current active workers), not `MaxRequestWorkers`. This means the saturation % can appear lower than reality if workers have not all been spawned yet (Apache spawns workers lazily up to MinSpareWorkers/MaxSpareWorkers). -- **[saturation]** With prefork MPM, `apache.workers{busy}` directly equals active requests. With event MPM, a single worker thread can handle many keepalive connections asynchronously — `apache.connections.async{keepalive}` reflects this, and busy workers remain low even under high keepalive load. -- **[connections]** `apache.current_connections` is a non-monotonic sum (cumulative but can decrease). Treat as a gauge for dashboarding. -- **[connections]** `apache.connections.async` is only meaningful with the event MPM. On prefork, this metric is zero. Including it in dashboards without MPM context can be misleading. -- **[scoreboard-detail]** `apache.scoreboard{scoreboard_state=open}` represents slots that have not yet been assigned to any worker — distinct from `waiting` (assigned but idle). `open` slots shrink as Apache spawns more workers up to MaxRequestWorkers. -- **[scoreboard-detail]** The attribute key is `scoreboard_state` (underscore), NOT `scoreboard.state` (dot). Verify in Stage 2 against live metric labels. -- **[scoreboard-detail]** The attribute key for workers is `workers_state` (underscore), NOT `workers.state` (dot). Verify in Stage 2 against live metric labels. -- **[throughput]** `apache.requests` and `apache.traffic` are monotonic cumulative sums. Always use `rate` post-function or `per-second` in Tsuga. Never plot raw cumulative values on a timeseries. -- **[throughput]** `apache.request.time` rate gives total milliseconds consumed per second across ALL workers collectively — not average per-request latency. Divide by `apache.requests` rate to get average request latency in ms. -- **[errors]** The apache receiver exports NO error rate metrics. 4xx and 5xx rates MUST come from access log routing. Building an "Errors" dashboard section without log intelligence wired up will result in empty panels. -- **[errors]** Apache error log entries are not in JSON by default. A log parsing rule (regex or grok) is required to extract severity, module, and error code fields. -- **[availability]** `apache.cpu.load` is a percentage of total CPU time consumed by httpd — it can exceed 100 % on multi-core hosts if Apache uses multiple cores. Not directly comparable to `top`-style per-core percentage. -- **[availability]** `apache.load.1`, `apache.load.5`, `apache.load.15` are system-level load averages from `/proc/loadavg` (Linux) as reported by Apache, NOT httpd-specific CPU load. They reflect total system load and should be interpreted as such. -- **[availability]** `apache.cpu.time` uses unit `{jiff}` (jiffy = 10ms on most Linux kernels). This is a cumulative counter; use rate for meaningful display. -- **[throughput]** `apache.traffic` counts bytes at the application layer. It does not include TLS overhead, HTTP/2 header compression savings, or network-level retransmissions. -- **[scoreboard-detail]** VirtualHost disaggregation is NOT natively available from mod_status or the OTel apache receiver. All apache.* metrics are aggregate across all VirtualHosts on the instance. -- **[saturation]** On K8s, each pod runs a separate Apache instance. Fleet-wide saturation requires summing across pods (use `sum` aggregation in Tsuga, then group by instance for breakout). -- **[availability]** A scrape failure by the OTel Collector (e.g., mod_status endpoint unreachable) will cause all apache.* metrics to go absent. Distinguish "server down" from "scrape misconfiguration" by cross-referencing OTel Collector health metrics. -- **[connections]** `apache.connections.async{writing}` overlaps conceptually with `apache.scoreboard{sending}` but counts at the TCP connection level vs scoreboard slot level. They are not identical and should not be summed. - ---- - -## Confirmed Tsuga prefixes - -| Prefix | Status | Source | Estimated metric count | -|---|---|---|---| -| `apache` | **CONFIRMED** | OTel Collector Contrib `apachereceiver` metadata.yaml | 12 metric names | - -Specific confirmed metric names (exact strings from metadata.yaml): -- `apache.uptime` -- `apache.current_connections` -- `apache.workers` -- `apache.requests` -- `apache.traffic` -- `apache.scoreboard` -- `apache.connections.async` -- `apache.cpu.load` -- `apache.cpu.time` -- `apache.request.time` -- `apache.load.1` -- `apache.load.5` -- `apache.load.15` - -Confirmed attribute names (exact strings from metadata.yaml): -- `workers_state` with values: `busy`, `idle` -- `scoreboard_state` with values: `open`, `waiting`, `starting`, `reading`, `sending`, `keepalive`, `dnslookup`, `closing`, `logging`, `finishing`, `idle_cleanup`, `unknown` -- `connection_state` with values: `writing`, `keepalive`, `closing` -- `cpu_level` with values: `self`, `children` -- `cpu_mode` with values: `system`, `user` - ---- - -## Discovery status - -**Stage 2 discovery: PENDING** - -Discovery has not been performed against a live Apache HTTP Server instance. All metric names and attribute values are sourced from the OTel Collector Contrib `apachereceiver/metadata.yaml` (confirmed) or inferred from Apache documentation (noted where applicable). - -Priority verification items for Stage 2: -1. Confirm `workers_state` vs `state` as actual label key on live metrics -2. Confirm `scoreboard_state` vs `state` as actual label key on live metrics -3. Confirm `apache.requests` and `apache.traffic` temporality (expected: cumulative monotonic) -4. Confirm `apache.uptime` temporality (expected: cumulative monotonic — resets on restart) -5. Confirm `apache.current_connections` behaves as non-monotonic sum in practice -6. Verify `ExtendedStatus On` is required for `apache.cpu.*` and `apache.request.time` -7. Verify whether `server.address` maps to `context.scope.name` or `context.server.address` in Tsuga -8. Check if mod_status VirtualHost disaggregation is available via `?server=vhost_name` parameter - ---- - -## Top sources - -1. **OTel apache receiver metadata.yaml** — https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/apachereceiver/metadata.yaml — authoritative source for all `apache.*` metric names, types, units, and attribute values -2. **OTel apache receiver README** — https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/apachereceiver — configuration reference for the receiver (endpoint, collection_interval, auth) -3. **Apache mod_status documentation** — https://httpd.apache.org/docs/current/mod/mod_status.html — authoritative source for worker scoreboard states and mod_status endpoint format -4. **Apache MPM documentation (event)** — https://httpd.apache.org/docs/current/mod/event.html — event MPM architecture, async connection handling, MaxRequestWorkers behavior -5. **Apache MPM documentation (prefork)** — https://httpd.apache.org/docs/current/mod/prefork.html — prefork process model; MaxRequestWorkers = max concurrent requests -6. **Apache core directives** — https://httpd.apache.org/docs/current/mod/core.html — MaxRequestWorkers, ServerLimit, StartServers, MinSpareThreads, MaxSpareThreads -7. **Apache log formats** — https://httpd.apache.org/docs/current/mod/mod_log_config.html — complete format string reference for access log parsing (CLF, Combined, custom) -8. **Apache error log format** — https://httpd.apache.org/docs/current/logs.html — error log structure, AH error codes, severity levels -9. **OpenTelemetry semantic conventions (HTTP)** — https://opentelemetry.io/docs/specs/semconv/http/ — attribute naming for HTTP request/response fields in log and trace data -10. **OTel Collector Contrib apachereceiver source** — https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/apachereceiver — full receiver implementation including scraper logic and metric construction -11. **Apache performance tuning guide** — https://httpd.apache.org/docs/current/misc/perf-tuning.html — MaxRequestWorkers tuning recommendations, keep-alive timeout guidance -12. **Prometheus apache_exporter** — https://github.com/Lusitaniae/apache_exporter — alternative metric source; useful for cross-referencing metric semantics with OTel receiver diff --git a/plugins/tsuga/skills/knowledge-technology/references/apache/metrics.csv b/plugins/tsuga/skills/knowledge-technology/references/apache/metrics.csv deleted file mode 100644 index 0557206..0000000 --- a/plugins/tsuga/skills/knowledge-technology/references/apache/metrics.csv +++ /dev/null @@ -1,19 +0,0 @@ -theme,name,confirmed,definition,unit,type,tsuga_metric_name,widget_suggestions,aggregation,post_function,group_by,required_context_fields,optional_context_fields,absence_means,cardinality_footguns,variants,citations -Availability,apache_up,true,Apache server availability as seen by the exporter. 1=up 0=down. Instant gauge — critical header KPI.,{},gauge,apache_up,query-value (threshold color 0=red 1=green),max,none,context.server.address:10,context.env,context.team,Exporter cannot reach Apache mod_status — check /server-status?auto endpoint and exporter logs,One value per instance — no cardinality risk,None,prometheus-community/apache_exporter (confirmed live) -Availability,apache_uptime_seconds_total,true,Cumulative total seconds since Apache last started. Monotonic counter — resets to zero on restart. Use increase() to display current uptime value or rate() to detect restarts (rate near-zero means just restarted).,s,counter,apache_uptime_seconds_total,query-value (increase for uptime display),sum,increase,context.server.address:10,context.env,context.team,Server not scraped or mod_status endpoint unreachable; distinguish from actual downtime via exporter health,Low cardinality — one value per instance,None,prometheus-community/apache_exporter (confirmed live) -Availability,apache_workers (busy),true,Number of worker threads/processes currently processing a request. Gauge.,{workers},gauge,apache_workers,timeseries + query-value,sum (filter state=busy),none,context.server.address:10 context.state:2,context.env,context.team,Workers metric absent = mod_status not enabled or scrape failure,state attribute has 2 values (busy/idle) — low cardinality,Filter: state=busy,prometheus-community/apache_exporter (confirmed live — state attribute key verified) -Availability,apache_workers (idle),true,Number of worker threads/processes available to accept new requests. Gauge.,{workers},gauge,apache_workers,timeseries + query-value,sum (filter state=idle),none,context.server.address:10 context.state:2,context.env,context.team,Workers metric absent = scrape failure,state attribute — low cardinality,Filter: state=idle,prometheus-community/apache_exporter (confirmed live — state attribute key verified) -Availability,apache_connections,true,Current open connections by state. Gauge. Has state attribute: closing/keepalive/total/writing.,{connections},gauge,apache_connections,timeseries + query-value (filter state=total for overall),sum,none,context.server.address:10 context.state:4,context.env,context.team,Connections metric absent = scrape failure or mod_status misconfigured,state has 4 values (closing/keepalive/total/writing) — low cardinality,Filter by state value,prometheus-community/apache_exporter (confirmed live) -Availability,apache_processes,true,Current number of Apache server processes. Gauge. Has state attribute (e.g. running/stopping).,{processes},gauge,apache_processes,timeseries + query-value,sum,none,context.server.address:10 context.state:5,context.env,context.team,Scrape failure or Apache not running,state attribute — low cardinality,None,prometheus-community/apache_exporter (confirmed live) -Throughput,apache_accesses_total,true,Cumulative total HTTP requests handled since last restart. Monotonic counter — use rate() for req/s.,{requests},counter,apache_accesses_total,timeseries (rate) + query-value,sum,rate,context.server.address:10,context.env,context.team,No traffic OR scrape failure OR server restarted (rate will spike then reset),One value per instance — no cardinality risk,None,prometheus-community/apache_exporter (confirmed live) -Throughput,apache_sent_kilobytes_total,true,Cumulative kilobytes of data sent by Apache since last restart. Monotonic counter. Unit is KiB (1024 bytes) NOT bytes. Use rate() for KiB/s bandwidth.,KiBy,counter,apache_sent_kilobytes_total,timeseries (rate) + query-value,sum,rate,context.server.address:10,context.env,context.team,No traffic OR scrape failure — cross-reference with apache_accesses_total,One value per instance — no cardinality risk,None,prometheus-community/apache_exporter (confirmed live — unit is KiB not bytes) -Throughput,apache_duration_ms_total,true,Cumulative total request processing time in milliseconds since last restart. Monotonic counter. rate(duration_ms_total) / rate(accesses_total) = avg request latency in ms/req. Requires ExtendedStatus On.,ms,counter,apache_duration_ms_total,timeseries (rate / accesses rate = avg latency),sum,rate,context.server.address:10,context.env,context.team,ExtendedStatus not enabled in httpd.conf — metric will be 0 or absent,One value per instance — no cardinality risk,None,prometheus-community/apache_exporter (confirmed live) -Throughput,apache_cpu_time_ms_total,true,Cumulative CPU time consumed by Apache in milliseconds. Monotonic counter. Has type attribute (system/user or self/children — verify live values). Use rate() for CPU consumption trend. Requires ExtendedStatus On.,ms,counter,apache_cpu_time_ms_total,timeseries (rate),sum,rate,context.server.address:10 context.type:4,context.env,context.team,ExtendedStatus not enabled OR non-Linux host,type attribute — low cardinality (verify values in Stage 3),None,prometheus-community/apache_exporter (confirmed live — context.type attribute key verified) -Saturation,apache_cpuload,true,Current Apache CPU load as a percentage of one CPU. Gauge. High values indicate CPU-bound request processing. Requires ExtendedStatus On. Can exceed 100% on multi-core systems.,%,gauge,apache_cpuload,timeseries + query-value,avg,none,context.server.address:10,context.env,context.team,ExtendedStatus not enabled — metric will be 0 or absent,One value per instance,None,prometheus-community/apache_exporter (confirmed live) -Saturation,apache_load,true,System load average by time interval. Gauge. Has interval attribute with values 1 / 5 / 15 (minutes). System-level metric — NOT Apache-specific CPU. Reflects total host load from /proc/loadavg.,dimensionless,gauge,apache_load,timeseries (group by interval),avg,none,context.server.address:10 context.interval:3,context.env,context.team,/proc/loadavg unavailable (non-Linux) or ExtendedStatus Off,interval attribute (1/5/15) — low cardinality,Filter: interval=1 for 1-minute load,prometheus-community/apache_exporter (confirmed live — context.interval attribute key verified) -Saturation,apache_scoreboard,true,Apache request processing slots by state from mod_status scoreboard. Gauge. Has state attribute with values: open/waiting/starting/reading/sending/keepalive/logging/finishing/idle_cleanup/closing/dnslookup/unknown.,{slots},gauge,apache_scoreboard,timeseries + pie (by state),sum,none,context.server.address:10 context.state:12,context.env,context.team,Scrape failure or mod_status disabled,state has 12 values per instance — low cardinality,Filter by state value,prometheus-community/apache_exporter (confirmed live — state attribute key verified) -Saturation,apache_generation,true,Current server configuration generation count. Gauge. Has type attribute. Increments on graceful restart — use to detect configuration reloads.,dimensionless,gauge,apache_generation,query-value + timeseries,max,none,context.server.address:10 context.type:2,context.env,context.team,Scrape failure,type attribute — low cardinality,None,prometheus-community/apache_exporter (confirmed live — context.type attribute key verified) -Availability,apache_exporter_build_info,true,Prometheus exporter metadata (version goversion goos goarch). Not a service metric — info metric always equals 1. Has build context attributes: branch/goarch/goos/goversion/revision/version.,N/A,meta,apache_exporter_build_info,N/A — meta/info metric only,N/A,N/A,N/A,N/A,N/A,meta/info only — not actionable as dashboard signal,N/A,N/A,prometheus-community/apache_exporter (confirmed live) -Availability,apache_version,true,Apache server version as a label metric. Always 1. Not a service metric.,N/A,meta,apache_version,N/A — meta/info metric only,N/A,N/A,N/A,N/A,N/A,meta/info only — not actionable as dashboard signal,N/A,N/A,prometheus-community/apache_exporter (confirmed live) -Availability,apache_info,true,Apache server information metric with mpm label. Always 1. Has mpm and version attributes useful for discovery but not dashboarding.,N/A,meta,apache_info,N/A — meta/info metric only. mpm attribute useful for documentation,N/A,N/A,N/A,N/A,N/A,meta/info only — mpm attribute tells which MPM is in use but value is always 1,N/A,N/A,prometheus-community/apache_exporter (confirmed live) -Cost,NOT_APPLICABLE,true,Apache HTTP Server is self-hosted on-prem / Kubernetes. No per-request cloud cost signal is available from the apache exporter or mod_status. Cost observability (CPU hours / GB memory per pod) requires host-level or Kubernetes metrics outside this integration scope.,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A — self-hosted deployment has no per-request cost metric in scope,N/A,N/A,N/A diff --git a/plugins/tsuga/skills/knowledge-technology/references/apache/queries.md b/plugins/tsuga/skills/knowledge-technology/references/apache/queries.md deleted file mode 100644 index 7fdcf37..0000000 --- a/plugins/tsuga/skills/knowledge-technology/references/apache/queries.md +++ /dev/null @@ -1,53 +0,0 @@ -# Apache HTTP Server - -Classic web server / reverse proxy. Healthy: `apache_up=1`, busy workers below total, idle dominant, low error rate. - -## Incident shapes - -- **Worker exhaustion** — busy workers near total → queue / refusal -- **CPU-bound workers** — cpuload climbs with heavy request mix (often PHP-FPM or mod_php) -- **Process storm** — scoreboard shows many `S` / `G` states → bad restart cycle -- **Upstream slowness** (reverse proxy) — workers held waiting on backend -- **Exporter scrape failure** — `apache_up=0` could be either Apache down OR stub_status unreachable - -## Key metrics - -| Metric | Unit | Signal | -|---|---|---| -| `apache_up` | 0/1 | Exporter reachability | -| `apache_uptime_seconds_total` | seconds | Reset = restart | -| `apache_workers` (state=busy) | count | Busy workers | -| `apache_workers` (state=idle) | count | Idle workers | -| `apache_connections` | count | Total open connections | -| `apache_processes` | count | Server processes | -| `apache_accesses_total` | count | Cumulative request count | -| `apache_sent_kilobytes_total` | KiB | Cumulative bytes sent | -| `apache_duration_ms_total` | ms | Cumulative total duration | -| `apache_cpu_time_ms_total` | ms | Cumulative worker CPU | -| `apache_cpuload` | % | CPU load estimate | -| `apache_scoreboard` | count/state | Per-state worker counts | - -## Derived signals - -- `busy / (busy + idle)` — worker saturation. > 0.85 sustained = request queueing. -- `Δaccesses_total / Δt` — request rate. -- `Δduration_ms_total / Δaccesses_total` — avg request duration. -- `Δcpu_time_ms_total / Δaccesses_total` — avg CPU time per request. Rising = heavier workload. - -## Log patterns - -- `[mpm_event:error] AH00484: server reached MaxRequestWorkers setting` — worker cap hit -- `[proxy:error] AH00940: HTTP: disabled connection` — upstream repeatedly failing -- `[proxy:error] AH01102: error reading status line` — backend hangup -- `[core:error] AH00045: child process %d still did not exit` — hung worker -- `[ssl:error] SSL Library Error` — TLS issue -- `[authz_core:error] AH01630: client denied` — ACL denying -- Segfault / signal errors — worker crash - -## Gotchas - -- Scoreboard string encodes per-slot state. Collected as separate counters, slot detail is lost. Use raw `server-status?refresh=1` for pathological cases. -- Keep-alive workers (state) are held for idle connections. `workers{state=keepalive}` dominant = new requests may queue even when workers "look idle." -- MPM modes (prefork / worker / event) have different worker semantics; tuning assumptions change. -- Access-log-only analysis misses hung workers, SSL errors, and backend issues. Use error log for those. -- Behind LB/CDN, `X-Forwarded-For` is real client IP; direct IP is the proxy. diff --git a/plugins/tsuga/skills/knowledge-technology/references/aws-api-gateway/README.md b/plugins/tsuga/skills/knowledge-technology/references/aws-api-gateway/README.md deleted file mode 100644 index fd4dc0e..0000000 --- a/plugins/tsuga/skills/knowledge-technology/references/aws-api-gateway/README.md +++ /dev/null @@ -1,259 +0,0 @@ -# AWS API Gateway Integration Context Bundle - -## Metadata -**Technology:** AWS API Gateway -**Deployment:** managed -**Environment:** prod -**Persona:** SRE Dev and ops -**Telemetry preference:** mixed -**Integration scope:** core service only -**Primary use-case:** reliability and performance - -## How to use this bundle -Read `01_aws-api-gateway_metrics.csv` first for the Stage 1 source of truth on API Gateway request, error, latency, and cache metrics. Read `02_aws-api-gateway_dashboard_plan.yaml` next for the concrete dashboard structure, widget intent, derived signals, explanation notes, triage chains, and playbooks. - -Use `03_aws-api-gateway_state.yaml` for machine-readable assumptions, unknowns, metric-prefix status, and Stage 4 log-intel handoff. Use `04_aws-api-gateway_memory.md` for the narrative handoff that tells Stage 2 what to verify first. Stage 2 will create `05_aws-api-gateway_metric_catalog.csv` as the discovered Tsuga metric catalog for reconciliation and coverage checks. Stage 4 should read this file's `Log intelligence (Stage 4 handoff)` section and the `03_aws-api-gateway_state.yaml` `log_intel` block before attempting log-route creation. - -## What it is and what "good" looks like -### Confirmed by sources -Amazon API Gateway is AWS's managed API front door for REST, HTTP, and WebSocket APIs. The metric family documented in the `AWS/ApiGateway` namespace exposes stage and method execution signals such as total request count, 4XX and 5XX errors, total latency, integration latency, and cache hits or misses when stage caching is enabled. API Gateway can also emit execution logs, custom access logs, detailed metrics, and X-Ray traces at the stage level. [Metrics and dimensions](https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-metrics-and-dimensions.html), [Set up logging](https://docs.aws.amazon.com/apigateway/latest/developerguide/set-up-logging.html), [Set up X-Ray](https://docs.aws.amazon.com/apigateway/latest/developerguide/apigateway-enabling-xray.html) - -Good operational posture for API Gateway means request volume stays near expected baseline, 5XX errors remain near zero, 4XX increases are explainable by caller behavior instead of platform changes, and end-to-end latency tracks integration latency closely without unexplained gateway overhead. If caching is enabled, cache hit ratio should be stable for cacheable routes and latency should improve without masking backend failures. [Metrics and dimensions](https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-metrics-and-dimensions.html), [Cache settings](https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-caching.html) - -The three highest-value incident shapes are: -1. Traffic normal but 5XX rises: start in `apigw-server-faults`. -2. Traffic stable but latency rises: start in `apigw-latency-path`. -3. Traffic drop or route skew: start in `apigw-traffic-demand`. - -Execution logging and access logging are independent knobs. Detailed metrics are optional and matter because method and resource-level breakdowns are far more useful than a single stage aggregate during incident triage. [Set up logging](https://docs.aws.amazon.com/apigateway/latest/developerguide/set-up-logging.html) - -### Best-practice inference -This bundle intentionally centers the REST API execution surface because the documented `AWS/ApiGateway` metrics that map cleanly to the user-provided namespace hint (`aws_apigateway_latency`) are the stage and method metrics used for REST API monitoring. If the workspace later shows HTTP API or WebSocket-specific families, Stage 2 should widen the catalog, but Stage 1 should not invent them. - -A healthy first-response dashboard for API Gateway should separate three ownership domains quickly: caller mistakes (4XX), gateway or integration failures (5XX), and latency spent inside API Gateway versus latency spent waiting on the backend. That split is more actionable than a single generic "availability" card because it tells the on-call whether to page application owners, client owners, or platform operators. - -## Key concepts -### Glossary -| Term | Definition | Operational meaning | Dashboard section | -|---|---|---|---| -| API Gateway stage | Named deployment environment such as `prod` or `staging` | Smallest reliable first-response scope for most incidents | apigw-traffic-demand | -| REST API execution metric | CloudWatch metric emitted for request processing at API or method level | Core signal source for RED dashboarding | apigw-traffic-demand | -| Count | Number of API requests in a period | Primary traffic denominator for all error ratios | apigw-traffic-demand | -| 4XXError | Client-visible requests ending in 4xx | Caller contract, auth, throttling, or validation symptom | apigw-client-faults | -| 5XXError | Requests ending in 5xx | Gateway, integration, or backend failure signal | apigw-server-faults | -| Latency | End-to-end API Gateway request time in milliseconds | User-perceived latency headline | apigw-latency-path | -| IntegrationLatency | Time API Gateway waits on backend integration in milliseconds | Backend latency component | apigw-latency-path | -| Gateway overhead | Latency not spent in the backend integration | Mapping, auth, request/response handling, or gateway-side work | apigw-latency-path | -| Detailed metrics | Optional per-stage, per-method, per-resource metrics | Enables route-level and method-level breakdowns | apigw-traffic-demand | -| Stage cache | Optional API Gateway response cache | Can reduce backend calls and user latency | apigw-cache-efficiency | -| Method dimension | HTTP method such as `GET` or `POST` | Helps separate read-heavy from write-heavy route behavior | apigw-traffic-demand | -| Resource dimension | API resource path or route template | Best route-level breakdown when available | apigw-traffic-demand | -| ApiName dimension | Human-readable API name | Primary multi-API grouping key | apigw-traffic-demand | -| Stage dimension | Deployment stage name | Safe filter and grouping key for isolating prod incidents | apigw-traffic-demand | -| Access log | Custom log line built from `$context` variables | Best request-level log substrate for Stage 4 | apigw-server-faults | -| Execution log | API Gateway-managed CloudWatch Logs execution traces | Best for request lifecycle debugging and auth/mapping failures | apigw-server-faults | -| Extended request ID | API Gateway-generated unique request identifier | Strong correlation handle for logs and support cases | apigw-server-faults | -| Integration status | Backend or Lambda proxy integration status code in access logs | Distinguishes gateway wrapper status from backend status | apigw-server-faults | -| Canary release | Small-percent stage deployment receiving a subset of traffic | Important because metrics and logs may split between base and canary | apigw-traffic-demand | -| Active tracing | X-Ray tracing enabled on a stage | Adds latency-path context outside metrics | apigw-latency-path | -| Lambda proxy integration | Common API Gateway integration mode for Lambda-backed APIs | Changes interpretation of integration status fields | apigw-server-faults | -| Throttling | Request rejection due to quotas, usage plans, or backend protection | Often appears to callers as a 4xx pattern rather than 5xx | apigw-client-faults | - -### Concept Map -Client -> sends request to -> API Gateway stage (why: every incident starts with whether the stage is accepting work) -API Gateway stage -> exposes -> Count (why: traffic baseline is the denominator for health ratios) -API Gateway stage -> emits -> 4XXError (why: caller-visible failure can rise even when the platform is healthy) -API Gateway stage -> emits -> 5XXError (why: server-side failure is the clearest outage signal) -API Gateway stage -> measures -> Latency (why: this is closest to user-perceived request time) -API Gateway stage -> measures -> IntegrationLatency (why: this isolates backend wait time inside the end-to-end request) -Latency -> minus -> IntegrationLatency (why: the difference approximates gateway-side overhead) -API Gateway stage -> optionally enables -> Detailed metrics (why: route and method drilldown is needed for triage) -Detailed metrics -> add dimensions -> ApiName/Stage/Resource/Method (why: these are the safe bounded breakdown keys) -API Gateway stage -> optionally enables -> Stage cache (why: cache hits reduce backend traffic and latency) -API Gateway stage -> forwards request to -> Integration target (why: backends own most of IntegrationLatency) -Integration target -> returns status/bytes to -> API Gateway (why: backend behavior influences 5xx and latency patterns) -API Gateway mapping/auth/request processing -> contributes to -> Gateway overhead (why: overhead growth without backend growth suggests gateway-side work) -Canary request routing -> splits traffic between -> base stage and canary path (why: metric and log interpretation can differ during rollout) -API Gateway stage -> writes -> Execution logs (why: execution logs explain request lifecycle and mapping/auth problems) -API Gateway stage -> writes -> Access logs (why: access logs are the cleanest request-level Stage 4 source) -Access logs -> include -> requestId and extendedRequestId (why: request-level correlation depends on a stable ID) -Access logs -> include -> integrationStatus and integrationLatency (why: backend root cause can be separated from gateway wrapper behavior) -X-Ray tracing -> correlates -> API Gateway and downstream services (why: latency-path analysis is stronger with traces) -ApiName -> maps to -> context.apiname (why: multi-API estates need a bounded primary ownership key) -Stage -> maps to -> context.stage (why: prod and staging must be split cleanly) -Method -> maps to -> context.method (why: GET and POST often have different latency and cache behavior) -Resource -> maps to -> context.resource (why: one hot route can dominate error or latency patterns) -Environment and team tags -> map to -> context.env and context.team (why: global dashboard filters need tenant and owner scoping) - -### Entities and dimensions -| Entity/Dimension | Why useful | Cardinality risk | Safe top-N suggestion | Do NOT group-by guidance | -|---|---|---|---|---| -| `context.apiname` | Primary API ownership and grouping key | Low | 20 | Use first before method or resource splits | -| `context.stage` | Separates prod, staging, or canary-adjacent environments | Low | 10 | Prefer as a global filter before chart splits | -| `context.resource` | Best path-level hot spot isolation | Medium | 20 | Avoid raw URL paths if the pipeline uses untemplated values | -| `context.method` | Distinguishes reads from writes and mutation-heavy traffic | Low | 8 | Use alongside API or resource, not alone, for context | -| `context.apiid` | Stable identifier when names are duplicated | Low | 20 | Keep as fallback when `context.apiname` is absent | -| `context.cloud.region` | Multi-region isolation | Low | 10 | Better as a global filter than a chart series in single-region estates | -| `context.cloud.account.id` | Multi-account ownership and blast radius | Low | 10 | Prefer filter over timeseries split | -| `context.env` | Environment isolation | Low | 5 | Keep as a dashboard filter | -| `context.team` | Ownership boundary | Low | 10 | Keep as a dashboard filter | -| `context.domainname` | Useful when custom domains front multiple APIs | Medium | 10 | Do not rely on it unless Stage 2 confirms consistent tagging | -| `context.requestid` | Request-level correlation key | Very high | 0 | Never use in metric group-bys | -| `context.extendedrequestid` | Stronger API Gateway-generated request identifier | Very high | 0 | Never use in metric group-bys | -| `context.canary` | Distinguishes canary and base traffic during rollout | Low | 2 | Only use if Stage 2 confirms the field exists | - -### Tsuga field mapping -#### Confirmed by sources -| Vendor/exporter dimension | Recommended context.* key | Must-exist vs optional | -|---|---|---| -| ApiName | `context.apiname` | Must-exist | -| Stage | `context.stage` | Must-exist | -| Method | `context.method` | Strongly preferred | -| Resource | `context.resource` | Strongly preferred | -| API identifier | `context.apiid` | Optional | -| Environment tag | `context.env` | Must-exist | -| Team tag | `context.team` | Must-exist | - -AWS documents `ApiName`, `Stage`, `Resource`, and `Method` as dimensions for the `AWS/ApiGateway` metric family, especially when detailed metrics are enabled. [Metrics and dimensions](https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-metrics-and-dimensions.html) - -#### Best-practice inference -If the Tsuga pipeline uses standard semantic keys instead of AWS dimension names, `context.resource` may appear as a normalized route key and `context.method` may align to an HTTP semantic field. If `context.apiname` is not present, `context.apiid` or `context.scope.name` is the safest fallback. Stage 2 must confirm the actual field inventory before dashboard filters are frozen. - -## Golden signals -### Confirmed by sources -| Signal | What it means for API Gateway | Typical causes when it degrades | Best telemetry sources | What people page on | Section questions | -|---|---|---|---|---|---| -| Traffic | Request demand reaching the stage and each route | upstream traffic drop, client outage, DNS/routing issue, canary split effects | `Count`, access logs | sudden request collapse or skewed route concentration | Is traffic arriving? Which API, stage, or route changed shape? | -| Errors | Caller-visible 4xx and platform/backend 5xx failures | auth failures, usage-plan throttling, contract drift, backend outage, mapping failures | `4XXError`, `5XXError`, execution logs, access logs | sustained 5xx, sudden 4xx wave after deploy, one-route failure concentration | Are failures client-driven or server-driven? Which route owns them? | -| Latency | Total request time and backend contribution | slow backend, cold dependency, increased gateway processing, auth/mapping changes, cache regression | `Latency`, `IntegrationLatency`, X-Ray, access logs | p95/p99-like latency growth, gateway overhead drift | Is the backend slow, or is API Gateway adding overhead? | -| Saturation | API Gateway has no single CPU widget here, so saturation is inferred from rising latency, growing 4xx/5xx, and cache misses under load | quota pressure, throttling, route hot spots, backend strain, canary imbalance | `Count`, errors, latency, cache metrics, execution logs | traffic stable but latency and 5xx rising, backend amplified by cache misses | Is the request path keeping up with demand? Is cache loss amplifying load? | - -### Best-practice inference -For API Gateway, RED is the right primary dashboard story. Saturation is not a first-class direct metric in this family, so the dashboard should treat saturation as a compound pattern: same or rising request rate, rising latency, and rising server-side faults. In this workspace the cache metrics are absent, so cache-driven saturation analysis must remain gated rather than assumed. - -## Telemetry sources -### Confirmed by sources -| Source type | How collected | What it provides | Pros/cons | Common pitfalls | -|---|---|---|---|---| -| CloudWatch execution metrics | Native `AWS/ApiGateway` namespace | Count, 4xx, 5xx, latency, integration latency, cache hit and miss counts | Canonical request KPI source; low setup overhead | Detailed metrics are optional, so route breakdowns may be absent | -| Detailed stage metrics | Stage setting in API Gateway | ApiName, Stage, Resource, Method-level breakdowns | Best metric drilldown for incidents | If disabled, you get only coarse aggregate visibility | -| Execution logs | Stage logging to CloudWatch Logs | Request lifecycle and execution-path troubleshooting | Best for mapping, auth, and integration errors | Data tracing can expose sensitive payloads and is not recommended for prod | -| Custom access logs | Stage access-log format built from `$context` variables | Structured request records with status, request IDs, integration metadata | Best Stage 4 log substrate | Must include requestId or extendedRequestId; schema is only as good as the configured template | -| Stage cache metrics | Native metrics when caching is enabled | Cache hit and miss metrics | Useful for cache-efficiency analysis | Absent entirely when caching is off | -| X-Ray traces | Optional stage active tracing | End-to-end latency path and downstream trace linkage | Best latency-path evidence beyond metrics | REST API only; not all stages enable it | - -Detailed metrics, execution logs, access logs, stage cache, and X-Ray are all optional per-stage capabilities, so "no data" frequently means "feature disabled" rather than "zero events." [Set up logging](https://docs.aws.amazon.com/apigateway/latest/developerguide/set-up-logging.html), [Cache settings](https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-caching.html), [Set up X-Ray](https://docs.aws.amazon.com/apigateway/latest/developerguide/apigateway-enabling-xray.html) - -### Best-practice inference -If a workspace only exports `aws_apigateway_latency` and not sibling count or error metrics, the dashboard should degrade to a latency-centric health surface and explicitly call out the missing denominator problem. A latency-only API Gateway dashboard is still useful, but it cannot honestly answer RED questions by itself. - -## Log intelligence (Stage 4 handoff) -### Confirmed by sources -1. Log sources matrix - -| Source | Access path | Typical format | Structured vs unstructured | Evidence | -|---|---|---|---|---| -| Execution logs | CloudWatch Logs group `API-Gateway-Execution-Logs_{rest-api-id}/{stage_name}` | API Gateway execution trace lines | Semi-structured text | [Set up logging](https://docs.aws.amazon.com/apigateway/latest/developerguide/set-up-logging.html) | -| Access logs | Custom CloudWatch Logs destination chosen on the stage | JSON, CLF, XML, or CSV depending on template | Structured if JSON, otherwise semi-structured | [Set up logging](https://docs.aws.amazon.com/apigateway/latest/developerguide/set-up-logging.html) | -| Canary access and execution logs | Same as base logs with `/Canary` suffix behavior | Same as base format, split for canary traffic | Same as configured format | [Canary release](https://docs.aws.amazon.com/apigateway/latest/developerguide/canary-release.html) | - -2. Known log formats - -- `API Gateway access log (JSON template)` - - Sample line: `{"requestId":"$context.requestId","extendedRequestId":"$context.extendedRequestId","apiId":"$context.apiId","stage":"$context.stage","resourcePath":"$context.resourcePath","httpMethod":"$context.httpMethod","status":"$context.status","integrationStatus":"$context.integration.status","integrationLatency":"$context.integrationLatency","responseLatency":"$context.responseLatency"}` - - Delimiter and shape notes: JSON object if the stage uses a JSON template; exact keys depend on the selected `$context` variables. - - Timestamp pattern: configurable if included in the template. - - Quoting behavior: valid JSON quoting if the template is authored correctly. - - Optional fields: canary, authorizer, integration, and identity fields depend on configuration and request path. [Access logging variables](https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-variables-for-access-logging.html), [Set up logging](https://docs.aws.amazon.com/apigateway/latest/developerguide/set-up-logging.html) - -- `API Gateway execution log` - - Sample line: `Extended Request Id: API Key: Stage: Method request path: Method completed with status: ` - - Delimiter and shape notes: line-oriented text emitted by API Gateway internals; shape varies by event type. - - Timestamp pattern: CloudWatch Logs event timestamp. - - Quoting behavior: free-form text, not guaranteed JSON. - - Optional fields: data tracing adds request or response payload details and should be treated as sensitive. [Set up logging](https://docs.aws.amazon.com/apigateway/latest/developerguide/set-up-logging.html) - -3. Candidate query filters for Stage 4 - -- Precise: `context.service.name:"aws-apigateway" AND context.stage:* AND (context.apiname:* OR context.apiid:*)` - - Rationale: targets normalized API Gateway request logs while requiring stage scope. - - Risk: depends on the actual service-name normalization in the current Tsuga pipeline. -- Fallback: `"API-Gateway-Execution-Logs" OR ("integrationLatency" AND "requestId" AND "stage")` - - Rationale: catches both raw execution logs and common JSON access-log templates. - - Risk: can overmatch setup or synthetic log lines if the workspace has multiple API Gateway logging conventions. - -4. Attribute mapping hints - -| Raw field | Suggested Tsuga key | Confidence | Notes | -|---|---|---|---| -| `$context.requestId` | `context.requestid` | High | Required in access log format or use extendedRequestId | -| `$context.extendedRequestId` | `context.extendedrequestid` | High | Better unique API Gateway identifier | -| `$context.apiId` | `context.apiid` | High | Stable API identifier | -| `$context.apiId` or API name tag | `context.apiname` | Medium | Depends on whether API name is logged directly or enriched later | -| `$context.stage` | `context.stage` | High | Best first log filter | -| `$context.resourcePath` | `context.resource` | High | Safer than raw URL when available | -| `$context.httpMethod` | `context.method` | High | Bounded and highly useful | -| `$context.status` | `context.status_code` | High | Client-visible response code | -| `$context.integration.status` | `context.integration_status` | Medium | Important for Lambda proxy and backend separation | -| `$context.integrationLatency` | `context.integration_latency_ms` | High | Maps directly to latency-path analysis | -| `$context.responseLatency` | `context.latency_ms` | High | Useful when request-level latency logs are present | -| `$context.isCanaryRequest` | `context.canary` | Medium | Present only when canary is enabled | - -5. Parsing risks - -- Access log shape is entirely template-defined, so one team's JSON schema may not match another team's CSV or CLF schema. -- Execution logs are semi-structured and event-type-dependent, so a single parser is unlikely to fit every line without split processors. -- `$context.requestId` can be overridden by clients; use `$context.extendedRequestId` when uniqueness matters. -- Data tracing may log sensitive request or response details; Stage 4 should avoid assuming payload fields are safe to retain. -- Canary deployments can duplicate log destinations with `/Canary` suffix behavior and split apparent traffic unexpectedly. - -### Best-practice inference -If both access logs and execution logs are present, Stage 4 should prefer access logs as the primary structured route and treat execution logs as a secondary troubleshooting stream. Execution logs are better for lifecycle debugging, but access logs produce more stable attribute extraction for dashboards and log pivots. - -## Caveats and footguns -- **[apigw-traffic-demand]** `Count` is the denominator for most RED formulas; if it is missing, error-rate and success-rate widgets should gate instead of backfilling from errors alone. (Inference) -- **[apigw-traffic-demand]** Detailed metrics are optional, so missing `Method` or `Resource` breakdowns often mean stage settings are coarse rather than route traffic being zero. ([Set up logging](https://docs.aws.amazon.com/apigateway/latest/developerguide/set-up-logging.html)) -- **[apigw-traffic-demand]** A canary release can split traffic and logs between base and canary paths, which can make a normal traffic shift look like a production drop. ([Canary release](https://docs.aws.amazon.com/apigateway/latest/developerguide/canary-release.html)) -- **[apigw-traffic-demand]** `ApiName` is human-readable and can be renamed; keep `apiId` available as a fallback identity key. (Inference) -- **[apigw-client-faults]** `4XXError` is not purely "bad clients"; usage-plan throttling, auth failures, or request-validation mistakes can all look like caller faults at the metric layer. ([Metrics and dimensions](https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-metrics-and-dimensions.html)) -- **[apigw-client-faults]** A deployment that tightens auth or request validation can create a clean 4xx spike with no backend incident at all. (Inference) -- **[apigw-client-faults]** If route-level dimensions are absent, a global 4xx spike can hide that only one method or one resource changed behavior. (Inference) -- **[apigw-client-faults]** Treat client-fault ratios carefully during sharp traffic drops because a few repeated bad requests can dominate the percentage. (Inference) -- **[apigw-server-faults]** `5XXError` is the clearest outage signal in this family and should not be averaged away into a low-resolution widget. (Inference) -- **[apigw-server-faults]** Lambda proxy integrations can return both integration status and function status details; do not assume one field alone tells the full backend story. ([Access logging variables](https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-variables-for-access-logging.html)) -- **[apigw-server-faults]** Execution logs and access logs are independent; having one does not imply the other exists. ([Set up logging](https://docs.aws.amazon.com/apigateway/latest/developerguide/set-up-logging.html)) -- **[apigw-server-faults]** `$context.requestId` can be client-overridden, so request-level investigations should prefer `$context.extendedRequestId` when possible. ([Set up logging](https://docs.aws.amazon.com/apigateway/latest/developerguide/set-up-logging.html)) -- **[apigw-latency-path]** `Latency` includes the full API Gateway request path, while `IntegrationLatency` covers backend wait time; subtracting them is useful, but it is still an approximation of gateway overhead rather than a first-class AWS metric. (Inference) -- **[apigw-latency-path]** A flat `IntegrationLatency` with rising `Latency` points to gateway-side work, policy changes, or request transformation overhead, not necessarily a backend outage. (Inference) -- **[apigw-latency-path]** X-Ray tracing is optional and REST-only, so a missing trace view does not prove tracing is broken. ([Set up X-Ray](https://docs.aws.amazon.com/apigateway/latest/developerguide/apigateway-enabling-xray.html)) -- **[apigw-latency-path]** If only average latency is present in Tsuga, tail latency incidents can be partially hidden. Stage 2 should verify whether percentiles are available on the discovered metric type. (Inference) -- **[apigw-cache-efficiency]** Cache metrics exist only when stage caching is enabled. Missing cache metrics do not mean zero cache usage; they usually mean the feature is off. ([Cache settings](https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-caching.html)) -- **[apigw-cache-efficiency]** Cache hit ratio only makes sense for cacheable methods and resources; mixing write routes into the denominator can make healthy caches look ineffective. (Inference) -- **[apigw-cache-efficiency]** A cache flush or deployment can produce a temporary miss spike without any backend regression. ([Cache settings](https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-caching.html)) -- **[apigw-cache-efficiency]** Canary traffic can use separate cache behavior, so cache hit changes during rollout may reflect split cache warming rather than broad regression. ([Canary release](https://docs.aws.amazon.com/apigateway/latest/developerguide/canary-release.html)) -- **[apigw-traffic-demand, apigw-client-faults, apigw-server-faults]** The current bundle standardizes Count and error widgets on `sum + per-second` so every rate-style panel uses the same function choice. (Current bundle decision) -- **[apigw-server-faults, apigw-latency-path]** Data tracing is tempting during incidents, but AWS explicitly advises against using it in production because sensitive data can be logged. ([Set up logging](https://docs.aws.amazon.com/apigateway/latest/developerguide/set-up-logging.html)) - -## Confirmed Tsuga prefixes -- `aws_apigateway_` - **CONFIRMED** (5/5 live metrics present in Tsuga: `aws_apigateway_count`, `aws_apigateway_4xx_error`, `aws_apigateway_5xx_error`, `aws_apigateway_latency`, `aws_apigateway_integration_latency`) - -## Discovery status -Discovery complete in Stage 2. - -- Confirmed live metric count: 5 -- Confirmed live metrics: `aws_apigateway_count`, `aws_apigateway_4xx_error`, `aws_apigateway_5xx_error`, `aws_apigateway_latency`, `aws_apigateway_integration_latency` -- Cache metrics for stage caching are currently absent from this workspace. -- Confirmed context fields on all live metrics: `context.apiname`, `context.stage`, `context.method`, `context.resource`, `context.env`, `context.team`, `context.cloud.region`, `context.cloud.account.id` -- Live Tsuga metadata reports all five metrics as `summary` with `cumulative` temporality and `avg|sum|count|min|max` capabilities - -## Top sources -- https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-metrics-and-dimensions.html - Canonical metric names, dimensions, and cache-metric semantics for the `AWS/ApiGateway` namespace. -- https://docs.aws.amazon.com/apigateway/latest/developerguide/set-up-logging.html - Primary source for execution logging, access logging, detailed metrics, and log-group behavior. -- https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-variables-for-access-logging.html - Canonical list of `$context` fields used to design Stage 4 access-log parsing. -- https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-caching.html - Defines stage caching behavior for API Gateway and why cache metrics require explicit stage caching. -- https://docs.aws.amazon.com/apigateway/latest/developerguide/apigateway-enabling-xray.html - Explains stage-level X-Ray support and its operational prerequisites. -- https://docs.aws.amazon.com/xray/latest/devguide/xray-services-apigateway.html - Cross-check for API Gateway active tracing behavior and the REST-only caveat. -- https://docs.aws.amazon.com/apigateway/latest/developerguide/canary-release.html - Important for canary-specific logs, metrics interpretation, and cache behavior during rollout. -- https://docs.aws.amazon.com/apigateway/latest/developerguide/create-canary-deployment.html - Additional canary-stage operational detail used to frame rollout-related traffic shifts. -- https://docs.aws.amazon.com/apigateway/latest/developerguide/how-to-deploy-api.html - Deployment and stage lifecycle context that informs stage-based scoping and incident interpretation. -- https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-logging.html - Useful contrast source to keep the bundle honest about REST versus HTTP API logging surfaces. diff --git a/plugins/tsuga/skills/knowledge-technology/references/aws-api-gateway/metrics.csv b/plugins/tsuga/skills/knowledge-technology/references/aws-api-gateway/metrics.csv deleted file mode 100644 index 4f96c59..0000000 --- a/plugins/tsuga/skills/knowledge-technology/references/aws-api-gateway/metrics.csv +++ /dev/null @@ -1,7 +0,0 @@ -theme,name,confirmed,definition,unit,type,tsuga_metric_name,widget_suggestions,aggregation,post_function,group_by,required_context_fields,optional_context_fields,absence_means,cardinality_footguns,variants,citations -Throughput/Usage,Count,true,"Total API requests in a period; the primary traffic baseline and denominator for most RED formulas.",{request},counter,aws_apigateway_count,"timeseries|query-value|top-list|bar",sum,per-second,"context.apiname:20|context.stage:10|context.resource:20|context.method:8",context.env|context.team|context.apiname|context.stage,context.resource|context.method|context.cloud.region|context.cloud.account.id,"Metric absent in the current workspace, detailed metrics disabled for route splits, or no requests in the selected window.","Use API, stage, resource, and method only. Never group by request identifiers.",cloudwatch=Count|tsuga=aws_apigateway_count,https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-metrics-and-dimensions.html -Errors/Failures,4XXError,true,"Requests ending in 4xx. High values usually indicate caller, auth, usage-plan, or validation problems rather than API Gateway infrastructure failure.",{error},counter,aws_apigateway_4xx_error,"timeseries|query-value|top-list|bar",sum,per-second,"context.apiname:20|context.stage:10|context.resource:20|context.method:8",context.env|context.team|context.apiname|context.stage,context.resource|context.method|context.cloud.region|context.cloud.account.id,"Metric absent in the current workspace, detailed metrics disabled for route splits, or no recent 4xx responses.","Do not group by request IDs or high-cardinality identity fields.",cloudwatch=4XXError|tsuga=aws_apigateway_4xx_error,https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-metrics-and-dimensions.html -Errors/Failures,5XXError,true,"Requests ending in 5xx. Highest-signal server-side failure metric for API Gateway request handling.",{error},counter,aws_apigateway_5xx_error,"timeseries|query-value|top-list|bar",sum,per-second,"context.apiname:20|context.stage:10|context.resource:20|context.method:8",context.env|context.team|context.apiname|context.stage,context.resource|context.method|context.cloud.region|context.cloud.account.id,"Metric absent in the current workspace, detailed metrics disabled for route splits, or no recent 5xx responses.","Keep separate from 4xx because the ownership domain is different.",cloudwatch=5XXError|tsuga=aws_apigateway_5xx_error,https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-metrics-and-dimensions.html -Performance/Latency,Latency,true,"End-to-end request latency observed by API Gateway in milliseconds. Closest metric to user-perceived API latency in this core family.",ms,summary,aws_apigateway_latency,"timeseries|query-value|top-list",average|max,none,"context.apiname:20|context.stage:10|context.resource:20|context.method:8",context.env|context.team|context.apiname|context.stage,context.resource|context.method|context.cloud.region|context.cloud.account.id,"Metric absent in the current workspace or no requests in the selected window.","The live Tsuga catalog exposes avg|sum|count|min|max only; do not assume percentile support.",cloudwatch=Latency|tsuga=aws_apigateway_latency,https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-metrics-and-dimensions.html -Performance/Latency,IntegrationLatency,true,"Milliseconds API Gateway waits for the backend integration to respond. Best backend-latency component in the core metric family.",ms,summary,aws_apigateway_integration_latency,"timeseries|query-value|top-list",average|max,none,"context.apiname:20|context.stage:10|context.resource:20|context.method:8",context.env|context.team|context.apiname|context.stage,context.resource|context.method|context.cloud.region|context.cloud.account.id,"Metric absent in the current workspace or no requests in the selected window.","Interpret together with Latency; on its own it cannot distinguish gateway overhead.",cloudwatch=IntegrationLatency|tsuga=aws_apigateway_integration_latency,https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-metrics-and-dimensions.html -Cost,NOT_APPLICABLE,true,"The core API Gateway CloudWatch metric family does not expose direct cost. Use Count, Latency, and IntegrationLatency as cost-adjacent proxies until billing or usage-plan spend data is integrated.",Unknown,Unknown,Unknown,note,Unknown,none,Unknown,context.env|context.team,context.apiname|context.stage|context.cloud.account.id,"Not applicable in the core execution-metric family.","Do not invent spend widgets from request KPIs alone.",proxy=Count|proxy=IntegrationLatency,https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-metrics-and-dimensions.html diff --git a/plugins/tsuga/skills/knowledge-technology/references/aws-api-gateway/queries.md b/plugins/tsuga/skills/knowledge-technology/references/aws-api-gateway/queries.md deleted file mode 100644 index 104129f..0000000 --- a/plugins/tsuga/skills/knowledge-technology/references/aws-api-gateway/queries.md +++ /dev/null @@ -1,48 +0,0 @@ -# AWS API Gateway - -Managed HTTP / REST / WebSocket edge. Fronts Lambda, HTTP backends, VPC-link targets. Healthy: low 4xx + 5xx, latency bounded, integration latency ≪ total. - -## Incident shapes - -- **Integration failure** — `aws_apigateway_5xx_error` spikes, `aws_apigateway_integration_latency` high → backend sick -- **Client errors** — `aws_apigateway_4xx_error` spikes → client bug, schema change, auth, or 429 throttle -- **aws_apigateway_latency regression** — `aws_apigateway_latency` p95 climbs; split into `aws_apigateway_integration_latency` (backend) vs gateway overhead -- **Throttling** — 429s from stage/account throttle settings (infer from 4xx detail) -- **Authorizer timeouts** — custom Lambda authorizer slow → 5xx with low aws_apigateway_integration_latency but high aws_apigateway_latency - -## Key metrics - -| Metric | Unit | Signal | -|---|---|---| -| `aws_apigateway_count` | requests | Total rate | -| `aws_apigateway_4xx_error` | count | Client errors (incl. 429) | -| `aws_apigateway_5xx_error` | count | Server errors (usually integration / authorizer) | -| `aws_apigateway_latency` | ms | End-to-end (gateway + integration) | -| `aws_apigateway_integration_latency` | ms | Integration (backend) only | - -Metric set is small; most investigation goes through access logs. - -## Derived signals - -- `(aws_apigateway_4xx_error + aws_apigateway_5xx_error) / aws_apigateway_count` — error rate. Baseline < 0.01 typically. -- `aws_apigateway_latency - aws_apigateway_integration_latency` — gateway overhead. Normally < 20ms; spike here without backend movement = authorizer / gateway-side. -- `p99(aws_apigateway_latency) / p50(aws_apigateway_latency)` — > 10 = long-tail regression. - -## Log patterns - -API Gateway access logs (CloudWatch): - -- `"status":"5\d\d"` — 5xx response -- `"errorMessage":` — integration-level error -- `"integrationStatus":"504"` — integration timeout -- `"authorizerError":` — authorizer failure -- `"throttled":true` — stage/account throttle -- `"integrationLatency":` very high — backend slow - -## Gotchas - -- `aws_apigateway_4xx_error` lumps 429 throttles, 403 auth failures, and 400 bad-request. Don't interpret a 4xx spike as "client bug" without checking log status codes. -- `aws_apigateway_integration_latency` is only populated when there's an integration. MOCK integrations or authorizer rejections leave it absent. -- Per-stage and per-account throttles coexist. Stage limits are in stage config; account limits are soft. -- Caching hides misses: `aws_apigateway_count` increments on cached response but `aws_apigateway_integration_latency` doesn't. `aws_apigateway_count` vs backend call rate divergence = cache hit rate. -- Websocket and REST APIs have different metric semantics; confirm API type before reasoning. diff --git a/plugins/tsuga/skills/knowledge-technology/references/aws-docdb/README.md b/plugins/tsuga/skills/knowledge-technology/references/aws-docdb/README.md deleted file mode 100644 index cf866f9..0000000 --- a/plugins/tsuga/skills/knowledge-technology/references/aws-docdb/README.md +++ /dev/null @@ -1,282 +0,0 @@ -# AWS DocDB Integration Context Bundle - -## Metadata -**Technology:** AWS DocDB -**Deployment:** managed -**Environment:** prod -**Persona:** SRE Dev and ops -**Telemetry preference:** mixed -**Integration scope:** core service only -**Primary use-case:** reliability and performance - -## How to use this bundle -- Use `01_aws-docdb_metrics.csv` as the source of truth for DocDB metric names, units, and safe widget/query patterns. -- Use `02_aws-docdb_dashboard_plan.yaml` for the proposed overview and deep-dive sections, derived signals, notes, and playbooks. -- Use `03_aws-docdb_state.yaml` for machine-readable stage state, assumptions, namespace hints, and unresolved unknowns. -- Use `04_aws-docdb_memory.md` for the human-readable Stage 1 handoff and Stage 2 verification priorities. -- Stage 2 will add `05_aws-docdb_metric_catalog.csv` as the discovered Tsuga inventory and reconciliation record. -- Stage 4 should read this file's `Log intelligence (Stage 4 handoff)` section and `03.log_intel` before authoring any log routes. - -## What it is and what "good" looks like - -### Confirmed by sources -- Amazon DocumentDB is a managed, MongoDB-compatible document database whose baseline health is expressed through CloudWatch instance and cluster metrics, with deeper engine load exposed through Performance Insights when enabled. [S1][S2][S3] -- The core operational surfaces are instance resource pressure, query and document operation volume, cache effectiveness, replica lag, and storage-side pressure. [S1][S2] -- "Good" for DocDB means writer and readers are reachable, client connections stay below instance limits, replica lag remains low, cache hit ratios stay high enough that volume I/O does not become the bottleneck, and low-memory throttling stays at zero. [S1][S2] -- Change streams, profiler logs, and audit logs are optional but operationally important. They add context for write amplification, slow operations, and access/security investigations. [S5][S6][S7] -- Paging intent should first separate: writer or cluster distress, memory/cache collapse, storage latency, and replication/change-stream backlog. - -### Best-practice inference -- Incident shape 1: **Cluster health regression**. Start in `cluster-health` to decide whether the problem is broad, reader-only, or isolated to one instance. -- Incident shape 2: **Throughput or query-path regression**. Start in `traffic-operations` to determine whether command/query load, returned document volume, or connection concurrency changed first. -- Incident shape 3: **Latency driven by cache or storage**. Start in `latency-storage` and `cache-memory`; cache misses and billed volume I/O matter more than raw CPU for DocDB read-path triage. -- Incident shape 4: **Replication or change-stream risk**. Start in `replication-change-streams`; replica lag and change-stream log growth are the fastest signals of data freshness or cost risk. -- A useful dashboard must be explicit about feature-gated surfaces: Performance Insights, audit logs, profiler logs, NVMe-backed metrics, and T3 CPU credit metrics are not universal. [S1][S3][S5][S6] - -## Key concepts - -### Glossary - -| Term | Definition | Operational meaning | Dashboard section | -|---|---|---|---| -| Cluster | The Amazon DocumentDB deployment boundary containing one writer and zero or more readers | Main blast radius for storage, change streams, and aggregate replication state | cluster-health | -| Writer | Primary instance accepting writes | First place to check for write-path saturation and memory pressure | cluster-health | -| Reader | Replica instance serving read traffic and replication apply | Reader lag or poor cache state often causes stale or slow reads | replication-change-streams | -| DBClusterIdentifier | CloudWatch cluster dimension | Best cluster-wide filter when storage or replica metrics are used | all | -| DBInstanceIdentifier | CloudWatch instance dimension | Main per-instance drilldown key | all | -| Role | CloudWatch cluster role dimension (`WRITER` or `READER`) | Safest split for top-level writer vs reader comparisons | cluster-health | -| DatabaseConnections | Number of client-initiated connections on an instance | Early signal for pool leaks, reconnect storms, or hot instances | cluster-health | -| DatabaseConnectionsLimit | Max allowed concurrent connections on an instance | Required denominator for safe connection-utilization reasoning | cluster-health | -| DatabaseCursors | Number of open cursors | Useful for cursor leaks and fan-out query patterns | traffic-operations | -| Opcounters | Per-minute counts of MongoDB-compatible operation classes | Fastest way to separate read-heavy, write-heavy, and command-heavy demand | traffic-operations | -| DocumentsReturned | Returned document count over the minute | Useful to detect fan-out or unbounded queries | traffic-operations | -| BufferCacheHitRatio | Percent of requests served from buffer cache | Most important read-path efficiency signal after basic health | cache-memory | -| IndexBufferCacheHitRatio | Percent of index requests served from cache | Falling values often precede billed read I/O growth | cache-memory | -| LowMemThrottleQueueDepth | Queue depth of requests throttled for low memory | Clear memory distress signal with direct user impact | cache-memory | -| LowMemNumOperationsThrottled | Count of operations throttled because memory is low | Best "this is hurting users now" memory symptom | cache-memory | -| FreeableMemory | Available RAM in bytes | Headroom indicator; low values plus cache misses usually mean pressure | cache-memory | -| FreeLocalStorage | Temporary local storage available for logs and temp tables | Important for spill-heavy work and local temp exhaustion | cache-memory | -| SwapUsage | Swap in use on the instance | Non-zero or growing swap is a late memory distress signal | cache-memory | -| ReadLatency | Average time per disk read I/O | One of the clearest storage stress indicators | latency-storage | -| WriteLatency | Average time per disk write I/O | Important for write stalls and storage saturation | latency-storage | -| VolumeReadIOPs | Billed read I/O operations at cluster volume level | Better cost and miss-path signal than instance `ReadIOPS` alone | latency-storage | -| VolumeWriteIOPs | Billed write I/O operations at cluster volume level | Indicates storage-layer cost and replication/write amplification | latency-storage | -| DBClusterReplicaLagMaximum | Max lag between writer and replicas | Top cluster-level freshness risk signal | replication-change-streams | -| DBInstanceReplicaLag | Lag for a specific replica instance | Best drilldown signal for one bad reader | replication-change-streams | -| ChangeStreamLogSize | Storage consumed by retained change stream log | Direct cost and backlog indicator for CDC-driven use cases | replication-change-streams | -| AvailableMVCCIds | Remaining write IDs before the cluster becomes read-only | Rare but critical exhaustion signal on write-heavy or GC-impaired clusters | capacity-cost | -| TransactionsOpen | Current open transactions | Useful for long-running transactions and lock/memory amplification | traffic-operations | -| TransactionsAborted | Aborted transactions in the minute | Strong symptom of application retries or contention | traffic-operations | -| NVMeStorageCacheHitRatio | Cache hit ratio for tiered cache on NVMe-backed instances | Optional high-value signal for NVMe-capable families | latency-storage | -| CPUCreditBalance | Remaining burst credits on T3 instances | Needed only for burstable classes; absent elsewhere | capacity-cost | - -[S1][S2][S3][S7] - -### Concept Map -Client application -> opens -> Database connection (why: demand first appears as connection concurrency) -Connection pool -> concentrates on -> Writer or reader instance (why: one hot app tier can overload one instance before the cluster) -Writer -> replicates to -> Reader instances (why: freshness and read scaling depend on replica apply health) -Cluster -> exposes -> Cluster-level lag metrics (why: replica health is not visible from one instance alone) -Instance -> emits -> CPU, memory, cursor, and operation metrics (why: instance distress is often the first operational boundary) -Query workload -> drives -> Opcounters and DocumentsReturned (why: traffic shape matters more than request count alone) -DocumentsReturned -> amplifies -> Storage reads (why: fan-out queries pull more pages and can inflate billed reads) -Buffer cache -> reduces -> VolumeReadIOPs (why: cache hits avoid storage access) -Index cache -> protects -> Query latency (why: missed index pages force expensive storage fetches) -FreeableMemory -> constrains -> Cache residency (why: shrinking memory usually means poorer cache hit ratios) -Low memory -> triggers -> Operation throttling (why: DocDB explicitly queues and throttles work under low memory) -Low memory throttling -> increases -> Client-visible latency (why: requests wait before execution) -FreeLocalStorage -> supports -> Temporary tables and logs (why: some operations need local scratch space) -Swap usage -> follows -> Severe memory pressure (why: swap is a late, expensive fallback) -ReadLatency -> reflects -> Storage read path health (why: misses and storage pressure show up here) -WriteLatency -> reflects -> Storage write path health (why: commits and replication materialization depend on writes) -VolumeReadIOPs -> depends on -> Cache miss behavior (why: billed reads occur when data is not in buffer cache) -VolumeWriteIOPs -> depends on -> Write volume plus replication/storage behavior (why: write amplification affects cost and latency) -Change streams -> consume -> ChangeStreamLogSize (why: longer retention or slow consumers retain more log) -ChangeStreamLogSize -> increases -> Storage cost and pressure (why: it is a subset of cluster storage) -TransactionsOpen -> increases -> Memory and lock footprint (why: long-running transactions hold resources) -TransactionsAborted -> signals -> contention or application retries (why: not all failed work appears as simple error counts) -Performance Insights -> emits -> DBLoad and wait split (why: engine saturation is more meaningful than CPU alone) -DBLoadNonCPU -> usually indicates -> waits on memory, locks, or storage (why: CPU is not the only bottleneck) -Audit logs -> capture -> auth, DDL, DML, and authorization events (why: security and access investigations need logs) -Profiler logs -> capture -> slow operations and plan summaries (why: metrics tell you something is wrong; profiler shows which operations) -context.env/context.team -> scope -> ownership and environment routing (why: dashboards must support operational boundaries) -context.dbclusteridentifier/context.dbinstanceidentifier -> anchor -> bounded group-by strategy (why: these are safe, useful drilldown keys) - -### Entities and dimensions - -| Entity/Dimension | Why useful | Cardinality risk | Safe top-N | Do NOT group-by guidance | -|---|---|---|---|---| -| `context.env` | Fast prod/non-prod split | Low | 5 | Use as global filter rather than on every chart | -| `context.team` | Ownership boundary | Low | 10 | Avoid using it as the first technical split | -| `context.dbclusteridentifier` | Best cluster-wide filter and top-list key | Low | 10 | Prefer this over account-wide or tenant-like keys | -| `context.dbinstanceidentifier` | Main per-instance attribution | Low | 20 | Safe default instance group-by | -| `context.role` | Writer vs reader separation | Low | 4 | Avoid stacking with too many other dimensions | -| `context.cloud.region` | Regional failure-domain split | Low | 10 | Prefer region over AZ unless AZ is confirmed in Tsuga | -| `context.cloud.account.id` | Multi-account scoping | Medium | 10 | Skip if the workspace is single-account | -| `context.collection` | Useful for hot collection investigation | High | 10 | Keep out of overview widgets unless Stage 2 proves bounded coverage | -| `context.database` | Helpful if multiple logical databases are tagged | Medium | 12 | Do not assume it exists on all metrics | -| `context.operation` | Useful for command or workload class analysis | Medium | 10 | Avoid if values are sparse or unnormalized | -| `context.change_stream` | Useful for CDC troubleshooting | Medium | 8 | Only use if Stage 2 confirms a bounded field | -| `context.instance.class` | Explains different limits and temp storage behavior | Medium | 10 | Do not mix with instance id in every widget | -| `context.engine.version` | Important for feature availability | Medium | 8 | Use for notes/gating, not routine timeseries splits | - -### Tsuga field mapping - -| Vendor/exporter dimension | Recommended context.* key | Must-exist vs optional | -|---|---|---| -| `DBClusterIdentifier` | `context.dbclusteridentifier` | Must-exist | -| `DBInstanceIdentifier` | `context.dbinstanceidentifier` | Must-exist | -| `Role` | `context.role` | Must-exist on cluster-role surfaces | -| AWS region enrichment | `context.cloud.region` | Optional | -| AWS account enrichment | `context.cloud.account.id` | Confirmed in Stage 2 | -| Environment tag enrichment | `context.env` | Must-exist | -| Team tag enrichment | `context.team` | Must-exist | -| Collection name from profiler/audit enrichment | `context.collection` | Optional | -| Logical database name from profiler/audit enrichment | `context.database` | Optional | -| Operation name from profiler/audit enrichment | `context.operation` | Optional | - -### Confirmed by sources -- CloudWatch exposes `DBClusterIdentifier`, `DBClusterIdentifier, Role`, and `DBInstanceIdentifier` dimensions for DocumentDB metrics. [S1] -- Audit logs are JSON documents exported to CloudWatch Logs; profiler logs include command, plan summary, timestamp, and client metadata. [S5][S6] - -### Best-practice inference -- Stage 2 should prefer flattened `context.dbclusteridentifier`, `context.dbinstanceidentifier`, and `context.role` keys over deeply nested or source-specific aliases. -- Collection-, database-, and operation-level keys should be treated as log/profiler-only until discovered in Tsuga metrics. - -## Golden signals - -### Confirmed by sources -| Signal | What it means for AWS DocDB | Typical degradation causes | Best telemetry sources | What people page on | Section questions | -|---|---|---|---|---|---| -| Traffic | Whether clients are connecting and issuing healthy read/write/command volume | Reconnect storms, hot reader, unbounded queries, retry loops | `DatabaseConnections`, `Opcounters*`, `DocumentsReturned`, network throughput [S1] | Connections surge, ops skew sharply, or document-return volume spikes unexpectedly | Is demand normal? Is load read-heavy, write-heavy, or pathological? | -| Errors | Not classic HTTP errors, but aborted work, timeouts, throttling, and auth failures | Low memory throttling, aborted transactions, auth failures, audit events | `LowMemNumOperationsThrottled`, `TransactionsAborted`, audit logs [S1][S6] | Throttled or aborted work rising, failed authentication activity, or repeated denied operations | Is user work failing now, and is the cause memory, auth, or contention? | -| Latency | How expensive it is to serve storage-backed or wait-heavy work | Cache miss storm, storage pressure, replica lag, slow operations | `ReadLatency`, `WriteLatency`, `VolumeReadIOPs`, profiler logs, PI load [S1][S3][S5] | Read/write latency increases with cache drop or storage I/O growth | Is the slowness in storage, memory/cache, or engine waits? | -| Saturation | Whether memory, cache, storage, or replication headroom is running out | Low free memory, swap use, low local storage, change-stream growth, MVCC pressure | `FreeableMemory`, `FreeLocalStorage`, `SwapUsage`, `LowMemThrottleQueueDepth`, `ChangeStreamLogSize`, `AvailableMVCCIds`, replica lag [S1][S7] | Memory throttling, shrinking storage headroom, lag growth, or MVCC runway collapse | Which resource is closest to failing first? | - -### Best-practice inference -- For DocDB, cache effectiveness and low-memory throttling are more actionable than CPU alone because poor cache residency converts ordinary read traffic into storage latency and cost. -- Replica lag and change-stream log growth matter more than generic throughput when the workload depends on readers, CDC pipelines, or near-real-time projections. - -## Telemetry sources - -### Confirmed by sources -| Source type | How collected | What it provides | Pros/cons | Common pitfalls | -|---|---|---|---|---| -| CloudWatch DocDB metrics | Native service metrics in namespace `AWS/DocDB` | Core instance, cluster, operation, latency, throughput, cache, and cost-adjacent signals | Always the baseline; broad coverage | Some metrics are cluster-level, some instance-level, and some are optional by instance family [S1] | -| Performance Insights CloudWatch metrics | PI-published `DBLoad*` metrics | Engine load and CPU vs non-CPU wait split | Best saturation signal when enabled | Published only when PI is enabled and there is load on the instance [S3] | -| CloudWatch audit logs | Optional audit export | Auth, DDL, DML, authorization, and user-management events | Best for access/security and correctness context | Disabled by default; DML payload fields are truncated at 1 KB [S6] | -| CloudWatch profiler logs | Optional profiler export | Slow operation details, command, plan summary, time, client metadata | Best for slow-query style investigation | Disabled by default and adds resource overhead; threshold selection matters [S5] | -| Change streams | App-level CDC feature plus change-stream retention metric | CDC semantics and backlog/cost context | Important when downstream consumers exist | Not every deployment uses change streams; backlog may look like storage growth if unlabeled [S1][S7] | -| Event subscriptions | Control-plane notifications | Maintenance, failover, and cluster event context | Good incident timeline context | Not a replacement for continuous metrics and not always ingested into Tsuga [S8] | - -### Best-practice inference -- "No data" on audit or profiler widgets usually means the feature is disabled, not that no risky events exist. -- "No data" on NVMe or T3 metrics usually means the instance family does not support that surface. -- Stage 2 confirmed the live export is `aws_docdb_*` and does not currently include replica-lag or Performance Insights `DBLoad*` metrics in this workspace, so those surfaces must stay gated. - -## Log intelligence (Stage 4 handoff) - -### Confirmed by sources -1. **Log sources matrix** - -| Source | Access path | Typical format | Structured vs unstructured | Evidence | -|---|---|---|---|---| -| Audit log | CloudWatch Logs export from the cluster | JSON documents | Structured | [S6] | -| Profiler log | CloudWatch Logs export from the cluster | JSON-like operation records with command metadata | Structured/semi-structured | [S5] | -| Control-plane event notifications | SNS/Event Subscription destinations | Event payloads, not engine logs | Structured | [S8] | - -2. **Known log formats** -- **Audit events**: JSON documents containing event category, event type, auth/authorization results, and DML or DDL context. DML `param` values are truncated at 1 KB. [S6] -- **Profiler records**: operation-oriented records containing execution time, profiled command, timestamp, plan summary, and client metadata. Exported only when the profiler is enabled and the threshold is met. [S5] - -3. **Candidate query filters for Stage 4** -- Precise: `source:docdb AND (context.log.type:audit OR context.log.type:profiler) AND context.dbclusteridentifier:*` - Rationale: directly targets the two known log surfaces. - Risk: `context.log.type` may not exist yet. -- Fallback: `source:docdb AND context.dbclusteridentifier:*` - Rationale: safer if only the cluster identifier is normalized. - Risk: may mix audit and profiler events into one stream. - -4. **Attribute mapping hints** - -| Raw field | Suggested Tsuga key | Confidence | Notes | -|---|---|---|---| -| Cluster identifier | `context.dbclusteridentifier` | High | Strongest routing key for DocDB logs | -| Instance identifier | `context.dbinstanceidentifier` | Medium | Helpful when profiler records carry instance context | -| Event type | `context.operation` | Medium | Good fit for audit event names like `authenticate`, `find`, `update` | -| Database name | `context.database` | Medium | Likely present on many audit/profiler events | -| Collection name | `context.collection` | Medium | High value for hot collection triage | -| Auth result / error code | `context.result.code` | Medium | Useful for auth failure investigations | -| Client metadata | `context.client` | Low | Valuable but may be high-cardinality | - -5. **Parsing risks** -- Audit and profiler surfaces are optional and disabled by default, so empty result sets may reflect config rather than health. [S5][S6] -- Audit JSON truncates large DML parameter values after 1 KB, so parsers must not assume full payload fidelity. [S6] -- Profiler thresholds materially change event volume and semantics; one cluster may log only very slow operations while another logs more aggressively. [S5] -- Event-subscription notifications are not a substitute for query or engine logs and should not be mixed into the same parsing route. - -### Best-practice inference -- Stage 4 should build separate parsing paths for audit vs profiler streams if log type can be detected. -- `context.operation`, `context.database`, and `context.collection` are the highest-value target fields for investigative search, but they should be added only if sample events confirm stable shapes. - -## Caveats and footguns -- **[cluster-health]** `DatabaseConnections` counts client-initiated connections only; engine/internal health checks can make engine-level commands show slightly more. [S1] -- **[cluster-health]** `DatabaseConnectionsMax` is a per-minute maximum, not an always-current gauge. [S1] -- **[cluster-health]** `DatabaseConnectionsLimit` varies by instance class, so absolute counts without the limit can mislead. (Inference) -- **[traffic-operations]** Idle clusters still show non-zero opcounter values because DocumentDB performs periodic health checks and internal work. [S1] -- **[traffic-operations]** `DocumentsReturned` can explode for one bad query pattern even when query counts stay flat. (Inference) -- **[traffic-operations]** Cursor counts are useful, but grouping by raw client metadata would be too high-cardinality for a default dashboard. (Inference) -- **[traffic-operations]** `TransactionsOpen` without `TransactionsOpenLimit` context can cause false concern on large instances. (Inference) -- **[traffic-operations]** `TransactionsAborted` is a symptom, not a root cause; pair it with profiler or audit context. (Inference) -- **[latency-storage]** `VolumeReadIOPs` and `VolumeWriteIOPs` are storage-layer metrics reported at 5-minute intervals and can repeat the same datapoint across the period. [S1] -- **[latency-storage]** Volume I/O is aggregated at cluster storage level and should not be treated as a per-instance real-time throughput metric. [S1] -- **[latency-storage]** `ReadLatency` and `WriteLatency` are averages, not percentiles. [S1] -- **[latency-storage]** NVMe metrics exist only on NVMe-backed instance families; absence is not equivalent to healthy zero. [S1] -- **[cache-memory, latency-storage]** A drop in `IndexBufferCacheHitRatio` can transiently exceed 100 percent right after dropping an index, collection, or database. [S1] -- **[cache-memory]** `FreeLocalStorage` is temporary scratch and log space, not the cluster volume. [S1] -- **[cache-memory]** `SwapUsage` is already a late-stage signal; waiting for swap before paging is too late. (Inference) -- **[cache-memory]** `LowMemThrottleQueueDepth` and `LowMemNumOperationsThrottled` are higher-signal user-impact metrics than CPU during memory crises. (Inference) -- **[cache-memory]** Cache-hit ratios are percentages and should not be re-normalized with rate functions. [S1] -- **[replication-change-streams]** `DBClusterReplicaLagMaximum` can hide one unhealthy reader behind a fleet view; pair with `DBInstanceReplicaLag` for drilldown. [S1] -- **[replication-change-streams]** `ChangeStreamLogSize` is a subset of `VolumeBytesUsed`; do not sum them and call it total storage. [S1][S7] -- **[replication-change-streams]** No change-stream consumer metric is guaranteed in CloudWatch, so backlog cause often has to be inferred from app context or logs. (Inference) -- **[capacity-cost]** `AvailableMVCCIds` is rare but critical: when it reaches zero, the cluster becomes read-only until IDs are reclaimed. [S1] -- **[capacity-cost]** T3 CPU credit metrics apply only to T3-backed instances; keep those widgets gated. [S1] -- **[capacity-cost]** Backup- and snapshot-storage metrics are cost surfaces, not immediate availability surfaces. [S1] - -## Confirmed Tsuga prefixes -- `aws_docdb_` - **CONFIRMED** (65 live metrics present in Tsuga over the last 24 hours; this is the real namespace family used in this workspace). -- `aws_docdb.*` - **INFERRED** (useful as a human alias, but Tsuga normalizes the live family to `aws_docdb_` rather than dot notation). - -## Discovery status -- Discovery completed in Stage 2 and was rerun after the Tsuga MCP update; direct MCP catalog listing and the repo discovery scripts both confirmed the same 65 live `aws_docdb_*` metrics. -- Live Tsuga discovery found **65** `aws_docdb_*` metrics. -- Reconciliation outcome: **58 confirmed** in `01`, **6 missing** from the Stage 1 seed, and **7 live-only metrics** kept in `05` without promotion to `01`. -- Key naming corrections: cluster volume metrics live as `aws_docdb_volume_read_io_ps` and `aws_docdb_volume_write_io_ps`, not the Stage 1 `..._iops` placeholders. -- Key field corrections: the live account field is `context.cloud.account.id`; `context.dbclusteridentifier`, `context.dbinstanceidentifier`, `context.role`, `context.cloud.region`, `context.env`, `context.team`, and `context.source` are the reusable discovery anchors. -- Missing documented surfaces in this workspace: replica-lag metrics, `DBLoad*` Performance Insights metrics, and NVMe cache-hit metrics. -- Spot-check data validation by scalar aggregation was **inconclusive** because Tsuga returned `500 INTERNAL_ERROR` even for confirmed live metrics, so Stage 2 relied on catalog and metadata evidence rather than zero-count assumptions. - -## Top sources -1. https://docs.aws.amazon.com/documentdb/latest/developerguide/cloud_watch.html - Authoritative CloudWatch metric, dimension, and unit reference for DocumentDB. -2. https://docs.aws.amazon.com/documentdb/latest/developerguide/performance-insights.html - Best source for why DB load matters and when PI should influence dashboard design. -3. https://docs.aws.amazon.com/documentdb/latest/developerguide/performance-insights-cloudwatch.html - Exact PI CloudWatch metrics (`DBLoad`, `DBLoadCPU`, `DBLoadNonCPU`) and publication behavior. -4. https://docs.aws.amazon.com/documentdb/latest/developerguide/performance-insights-metrics.html - PI API semantics and decomposition dimensions useful for later enrichment decisions. -5. https://docs.aws.amazon.com/documentdb/latest/developerguide/profiling.html - Profiler log shape, enablement model, and operational tradeoffs. -6. https://docs.aws.amazon.com/documentdb/latest/developerguide/event-auditing.html - Audit event categories, JSON log export behavior, and truncation caveats. -7. https://docs.aws.amazon.com/documentdb/latest/developerguide/change_streams.html - Change-stream semantics and retention behavior behind `ChangeStreamLogSize`. -8. https://docs.aws.amazon.com/documentdb/latest/developerguide/event-subscriptions.subscribe.html - Event-subscription model for control-plane context and operational timeline enrichment. -9. https://docs.aws.amazon.com/documentdb/latest/developerguide/best_practices.html - General DocumentDB operational guidance useful for section ordering and footguns. -10. https://aws.amazon.com/documentdb/pricing/ - Pricing context for why `VolumeBytesUsed`, backup storage, and change-stream retention deserve cost-aware coverage. - -## Source Legend -- `[S1]` CloudWatch metrics and dimensions -- `[S2]` Performance Insights overview -- `[S3]` Performance Insights CloudWatch metrics -- `[S4]` Performance Insights API metrics -- `[S5]` Profiler logs -- `[S6]` Audit logs -- `[S7]` Change streams -- `[S8]` Event subscriptions -- `[S9]` Best practices -- `[S10]` Pricing diff --git a/plugins/tsuga/skills/knowledge-technology/references/aws-docdb/metrics.csv b/plugins/tsuga/skills/knowledge-technology/references/aws-docdb/metrics.csv deleted file mode 100644 index 96e96d5..0000000 --- a/plugins/tsuga/skills/knowledge-technology/references/aws-docdb/metrics.csv +++ /dev/null @@ -1,65 +0,0 @@ -theme,name,confirmed,definition,unit,type,tsuga_metric_name,widget_suggestions,aggregation,post_function,group_by,required_context_fields,optional_context_fields,absence_means,cardinality_footguns,variants,citations -Availability/Health,CPUUtilization,true,Instance CPU utilization; useful but secondary to cache and memory distress for DocDB triage.,%,summary,aws_docdb_cpu_utilization,query-value|timeseries|top-list,max,none,context.dbinstanceidentifier:20|context.role:4,context.env|context.team,context.dbclusteridentifier|context.cloud.region|context.cloud.account.id,Metric absent or namespace mismatch,Stay at instance or role granularity,cloudwatch=CPUUtilization,https://docs.aws.amazon.com/documentdb/latest/developerguide/cloud_watch.html -Availability/Health,DatabaseConnections,true,Client-initiated active and idle connections on an instance.,count,summary,aws_docdb_database_connections,query-value|timeseries|top-list,max,none,context.dbinstanceidentifier:20|context.role:4,context.env|context.team,context.dbclusteridentifier|context.cloud.region,No client traffic or namespace mismatch,Do not group by client metadata in default dashboards,cloudwatch=DatabaseConnections,https://docs.aws.amazon.com/documentdb/latest/developerguide/cloud_watch.html -Availability/Health,DatabaseConnectionsMax,true,One-minute maximum of open connections on an instance.,count,summary,aws_docdb_database_connections_max,query-value|timeseries|top-list,max,none,context.dbinstanceidentifier:20|context.role:4,context.env|context.team,context.dbclusteridentifier|context.cloud.region,Metric absent or namespace mismatch,Interpret with DatabaseConnectionsLimit,cloudwatch=DatabaseConnectionsMax,https://docs.aws.amazon.com/documentdb/latest/developerguide/cloud_watch.html -Availability/Health,DatabaseConnectionsLimit,true,Maximum allowed concurrent connections on an instance.,count,summary,aws_docdb_database_connections_limit,query-value|timeseries,max,none,context.dbinstanceidentifier:20,context.env|context.team,context.dbclusteridentifier|context.cloud.region,Metric absent or namespace mismatch,Low-cardinality and safe,cloudwatch=DatabaseConnectionsLimit,https://docs.aws.amazon.com/documentdb/latest/developerguide/cloud_watch.html -Availability/Health,EngineUptime,true,Time the instance engine has been running; useful to spot restarts or failovers.,s,summary,aws_docdb_engine_uptime,query-value|timeseries,min,none,context.dbinstanceidentifier:20,context.env|context.team,context.dbclusteridentifier|context.role|context.cloud.region,"Restart, failover, or namespace mismatch",Do not over-split a monotonic uptime metric,cloudwatch=EngineUptime,https://docs.aws.amazon.com/documentdb/latest/developerguide/cloud_watch.html -Traffic/Operations,DatabaseCursors,true,Number of open cursors on an instance.,count,summary,aws_docdb_database_cursors,query-value|timeseries|top-list,max,none,context.dbinstanceidentifier:20,context.env|context.team,context.dbclusteridentifier|context.role,Metric absent or namespace mismatch,Cursors by raw client is too high-cardinality,cloudwatch=DatabaseCursors,https://docs.aws.amazon.com/documentdb/latest/developerguide/cloud_watch.html -Traffic/Operations,DatabaseCursorsTimedOut,true,Timed-out cursors in the one-minute period.,count,summary,aws_docdb_database_cursors_timed_out,query-value|timeseries|top-list,sum,none,context.dbinstanceidentifier:20,context.env|context.team,context.dbclusteridentifier|context.role,No timed-out cursors or namespace mismatch,Keep at instance-level,cloudwatch=DatabaseCursorsTimedOut,https://docs.aws.amazon.com/documentdb/latest/developerguide/cloud_watch.html -Traffic/Operations,OpcountersQuery,true,Queries issued in the one-minute period; idle baseline is non-zero due to internal work.,count,summary,aws_docdb_opcounters_query,query-value|timeseries|top-list|bar,sum,none,context.dbinstanceidentifier:20|context.role:4,context.env|context.team,context.dbclusteridentifier|context.cloud.region,Idle workload or namespace mismatch,Do not over-interpret tiny non-zero baselines,cloudwatch=OpcountersQuery,https://docs.aws.amazon.com/documentdb/latest/developerguide/cloud_watch.html -Traffic/Operations,OpcountersCommand,true,Commands issued in the one-minute period.,count,summary,aws_docdb_opcounters_command,query-value|timeseries|top-list|bar,sum,none,context.dbinstanceidentifier:20|context.role:4,context.env|context.team,context.dbclusteridentifier|context.cloud.region,Idle workload or namespace mismatch,Command mix detail is log/profiler work not metric work,cloudwatch=OpcountersCommand,https://docs.aws.amazon.com/documentdb/latest/developerguide/cloud_watch.html -Traffic/Operations,OpcountersInsert,true,Insert operations issued in the one-minute period.,count,summary,aws_docdb_opcounters_insert,query-value|timeseries|bar,sum,none,context.dbinstanceidentifier:20|context.role:4,context.env|context.team,context.dbclusteridentifier|context.cloud.region,No insert activity or namespace mismatch,Keep bounded to instance or role,cloudwatch=OpcountersInsert,https://docs.aws.amazon.com/documentdb/latest/developerguide/cloud_watch.html -Traffic/Operations,OpcountersUpdate,true,Update operations issued in the one-minute period.,count,summary,aws_docdb_opcounters_update,query-value|timeseries|bar,sum,none,context.dbinstanceidentifier:20|context.role:4,context.env|context.team,context.dbclusteridentifier|context.cloud.region,No update activity or namespace mismatch,Useful with change streams and transactions,cloudwatch=OpcountersUpdate,https://docs.aws.amazon.com/documentdb/latest/developerguide/cloud_watch.html -Traffic/Operations,OpcountersDelete,true,Delete operations issued in the one-minute period.,count,summary,aws_docdb_opcounters_delete,query-value|timeseries|bar,sum,none,context.dbinstanceidentifier:20|context.role:4,context.env|context.team,context.dbclusteridentifier|context.cloud.region,No delete activity or namespace mismatch,Keep bounded to instance or role,cloudwatch=OpcountersDelete,https://docs.aws.amazon.com/documentdb/latest/developerguide/cloud_watch.html -Traffic/Operations,DocumentsReturned,true,Documents returned in the one-minute period; good fan-out and read-amplification proxy.,count,summary,aws_docdb_documents_returned,query-value|timeseries|top-list,sum,none,context.dbinstanceidentifier:20|context.role:4,context.env|context.team,context.dbclusteridentifier|context.cloud.region,No reads or namespace mismatch,Collection-level grouping should be deferred to logs,cloudwatch=DocumentsReturned,https://docs.aws.amazon.com/documentdb/latest/developerguide/cloud_watch.html -Traffic/Operations,TransactionsOpen,true,Open transactions on an instance.,count,summary,aws_docdb_transactions_open,query-value|timeseries|top-list,max,none,context.dbinstanceidentifier:20,context.env|context.team,context.dbclusteridentifier|context.role,No open transactions or namespace mismatch,Useful only with bounded instance grouping,cloudwatch=TransactionsOpen,https://docs.aws.amazon.com/documentdb/latest/developerguide/cloud_watch.html -Traffic/Operations,TransactionsAborted,true,Transactions aborted in the one-minute period.,count,summary,aws_docdb_transactions_aborted,query-value|timeseries|top-list,sum,none,context.dbinstanceidentifier:20,context.env|context.team,context.dbclusteridentifier|context.role,No aborted transactions or namespace mismatch,Pair with profiler/audit before conclusions,cloudwatch=TransactionsAborted,https://docs.aws.amazon.com/documentdb/latest/developerguide/cloud_watch.html -Latency/Storage,ReadLatency,true,Average disk read I/O latency.,ms,summary,aws_docdb_read_latency,query-value|timeseries|top-list,max,none,context.dbinstanceidentifier:20|context.role:4,context.env|context.team,context.dbclusteridentifier|context.cloud.region,Metric absent or namespace mismatch,Average not percentile,cloudwatch=ReadLatency,https://docs.aws.amazon.com/documentdb/latest/developerguide/cloud_watch.html -Latency/Storage,WriteLatency,true,Average disk write I/O latency.,ms,summary,aws_docdb_write_latency,query-value|timeseries|top-list,max,none,context.dbinstanceidentifier:20|context.role:4,context.env|context.team,context.dbclusteridentifier|context.cloud.region,Metric absent or namespace mismatch,Average not percentile,cloudwatch=WriteLatency,https://docs.aws.amazon.com/documentdb/latest/developerguide/cloud_watch.html -Latency/Storage,ReadIOPS,true,Average disk read I/O operations per second at the instance layer.,count/s,summary,aws_docdb_read_iops,query-value|timeseries|top-list,sum,none,context.dbinstanceidentifier:20|context.role:4,context.env|context.team,context.dbclusteridentifier|context.cloud.region,Read path idle or namespace mismatch,Use with cache-hit ratios and volume reads,cloudwatch=ReadIOPS,https://docs.aws.amazon.com/documentdb/latest/developerguide/cloud_watch.html -Latency/Storage,WriteIOPS,true,Average disk write I/O operations per second at the instance layer.,count/s,summary,aws_docdb_write_iops,query-value|timeseries|top-list,sum,none,context.dbinstanceidentifier:20|context.role:4,context.env|context.team,context.dbclusteridentifier|context.cloud.region,Write path idle or namespace mismatch,Use with write latency and replica lag,cloudwatch=WriteIOPS,https://docs.aws.amazon.com/documentdb/latest/developerguide/cloud_watch.html -Latency/Storage,VolumeReadIOPs,true,Cluster volume billed read operations aggregated over 5-minute intervals.,count/5m,summary,aws_docdb_volume_read_io_ps,query-value|timeseries|top-list,sum,none,context.dbclusteridentifier:10,context.env|context.team,context.cloud.region|context.role,Metric absent or namespace mismatch,Cluster-level only; reported as the live VolumeReadIOPs surface with io_ps naming,cloudwatch=VolumeReadIOPs,https://docs.aws.amazon.com/documentdb/latest/developerguide/cloud_watch.html -Latency/Storage,VolumeWriteIOPs,true,Cluster volume billed write operations aggregated over 5-minute intervals.,count/5m,summary,aws_docdb_volume_write_io_ps,query-value|timeseries|top-list,sum,none,context.dbclusteridentifier:10,context.env|context.team,context.cloud.region|context.role,Metric absent or namespace mismatch,Cluster-level only; reported as the live VolumeWriteIOPs surface with io_ps naming,cloudwatch=VolumeWriteIOPs,https://docs.aws.amazon.com/documentdb/latest/developerguide/cloud_watch.html -Latency/Storage,StorageNetworkReceiveThroughput,true,Network bytes per second received from the cluster storage volume by each instance.,By/s,summary,aws_docdb_storage_network_receive_throughput,query-value|timeseries|top-list,sum,none,context.dbinstanceidentifier:20,context.env|context.team,context.dbclusteridentifier|context.role,Metric absent or namespace mismatch,"Interpret as storage path, not client path",cloudwatch=StorageNetworkReceiveThroughput,https://docs.aws.amazon.com/documentdb/latest/developerguide/cloud_watch.html -Latency/Storage,StorageNetworkTransmitThroughput,true,Network bytes per second sent to the cluster storage volume by each instance.,By/s,summary,aws_docdb_storage_network_transmit_throughput,query-value|timeseries|top-list,sum,none,context.dbinstanceidentifier:20,context.env|context.team,context.dbclusteridentifier|context.role,Metric absent or namespace mismatch,"Interpret as storage path, not client path",cloudwatch=StorageNetworkTransmitThroughput,https://docs.aws.amazon.com/documentdb/latest/developerguide/cloud_watch.html -Latency/Storage,NVMeStorageCacheHitRatio,false,Percent of requests served by the tiered cache on NVMe-backed instances.,%,summary,aws_docdb_nvme_storage_cache_hit_ratio,query-value|timeseries,max,none,context.dbinstanceidentifier:20,context.env|context.team,context.dbclusteridentifier|context.role,Instance family does not support NVMe tiered cache or namespace mismatch,Gate by instance family support,cloudwatch=NVMeStorageCacheHitRatio,https://docs.aws.amazon.com/documentdb/latest/developerguide/cloud_watch.html -Cache/Memory,BufferCacheHitRatio,true,Percent of requests served from the buffer cache.,%,summary,aws_docdb_buffer_cache_hit_ratio,query-value|timeseries|top-list,max,none,context.dbinstanceidentifier:20|context.role:4,context.env|context.team,context.dbclusteridentifier|context.cloud.region,Metric absent or namespace mismatch,Percentage metric; do not rate or sum across unlike scopes,cloudwatch=BufferCacheHitRatio,https://docs.aws.amazon.com/documentdb/latest/developerguide/cloud_watch.html -Cache/Memory,IndexBufferCacheHitRatio,true,Percent of index requests served from cache.,%,summary,aws_docdb_index_buffer_cache_hit_ratio,query-value|timeseries|top-list,max,none,context.dbinstanceidentifier:20|context.role:4,context.env|context.team,context.dbclusteridentifier|context.cloud.region,Metric absent or namespace mismatch,Can briefly spike above 100 percent after drops,cloudwatch=IndexBufferCacheHitRatio,https://docs.aws.amazon.com/documentdb/latest/developerguide/cloud_watch.html -Cache/Memory,FreeableMemory,true,Available memory in bytes on the instance.,By,summary,aws_docdb_freeable_memory,query-value|timeseries|top-list,min,none,context.dbinstanceidentifier:20,context.env|context.team,context.dbclusteridentifier|context.role,Metric absent or namespace mismatch,Use instance or role only,cloudwatch=FreeableMemory,https://docs.aws.amazon.com/documentdb/latest/developerguide/cloud_watch.html -Cache/Memory,FreeLocalStorage,true,Available local temporary/log storage per instance.,By,summary,aws_docdb_free_local_storage,query-value|timeseries|top-list,min,none,context.dbinstanceidentifier:20,context.env|context.team,context.dbclusteridentifier|context.role,Metric absent or namespace mismatch,This is not cluster volume free space,cloudwatch=FreeLocalStorage,https://docs.aws.amazon.com/documentdb/latest/developerguide/cloud_watch.html -Cache/Memory,SwapUsage,true,Swap space in use on the instance.,By,summary,aws_docdb_swap_usage,query-value|timeseries|top-list,max,none,context.dbinstanceidentifier:20,context.env|context.team,context.dbclusteridentifier|context.role,No swap use or namespace mismatch,Late-stage memory distress signal,cloudwatch=SwapUsage,https://docs.aws.amazon.com/documentdb/latest/developerguide/cloud_watch.html -Cache/Memory,LowMemThrottleQueueDepth,true,Queue depth for requests throttled by low memory.,count,summary,aws_docdb_low_mem_throttle_queue_depth,query-value|timeseries|top-list,max,none,context.dbinstanceidentifier:20,context.env|context.team,context.dbclusteridentifier|context.role,No throttling or namespace mismatch,Stay at instance-level,cloudwatch=LowMemThrottleQueueDepth,https://docs.aws.amazon.com/documentdb/latest/developerguide/cloud_watch.html -Cache/Memory,LowMemNumOperationsThrottled,true,Requests throttled due to low available memory in the one-minute period.,count,summary,aws_docdb_low_mem_num_operations_throttled,query-value|timeseries|top-list,sum,none,context.dbinstanceidentifier:20,context.env|context.team,context.dbclusteridentifier|context.role,No throttled operations or namespace mismatch,Pair with FreeableMemory and cache-hit ratios,cloudwatch=LowMemNumOperationsThrottled,https://docs.aws.amazon.com/documentdb/latest/developerguide/cloud_watch.html -Replication/Change Streams,DBClusterReplicaLagMaximum,false,Maximum replica lag in milliseconds across the cluster.,ms,summary,aws_docdb_db_cluster_replica_lag_maximum,query-value|timeseries|top-list,max,none,context.dbclusteridentifier:10|context.role:4,context.env|context.team,context.cloud.region,"No replicas, no lag, or namespace mismatch",Cluster-level metric can hide one bad reader,cloudwatch=DBClusterReplicaLagMaximum,https://docs.aws.amazon.com/documentdb/latest/developerguide/cloud_watch.html -Replication/Change Streams,DBInstanceReplicaLag,false,Replica lag in milliseconds for a specific replica instance.,ms,summary,aws_docdb_db_instance_replica_lag,query-value|timeseries|top-list,max,none,context.dbinstanceidentifier:20|context.role:4,context.env|context.team,context.dbclusteridentifier|context.cloud.region,Metric absent on writer-only or namespace mismatch,Best per-reader drilldown signal,cloudwatch=DBInstanceReplicaLag,https://docs.aws.amazon.com/documentdb/latest/developerguide/cloud_watch.html -Replication/Change Streams,ChangeStreamLogSize,true,Storage used by retained change stream log; subset of cluster storage and a cost/backlog signal.,MiB,summary,aws_docdb_change_stream_log_size,query-value|timeseries|top-list,max,none,context.dbclusteridentifier:10,context.env|context.team,context.cloud.region,Change streams disabled or namespace mismatch,Do not sum with VolumeBytesUsed,cloudwatch=ChangeStreamLogSize,https://docs.aws.amazon.com/documentdb/latest/developerguide/cloud_watch.html -Capacity/Cost,VolumeBytesUsed,true,Cluster storage used in bytes.,By,summary,aws_docdb_volume_bytes_used,query-value|timeseries|top-list,max,none,context.dbclusteridentifier:10,context.env|context.team,context.cloud.region,Metric absent or namespace mismatch,Cluster-level cost and capacity signal,cloudwatch=VolumeBytesUsed,https://docs.aws.amazon.com/documentdb/latest/developerguide/cloud_watch.html -Capacity/Cost,AvailableMVCCIds,true,Remaining write operations before the cluster becomes read-only.,count,summary,aws_docdb_available_mvcc_ids,query-value|timeseries,min,none,context.dbclusteridentifier:10,context.env|context.team,context.cloud.region,Metric absent or namespace mismatch,Critical runway metric; keep ungroupped or cluster-level only,cloudwatch=AvailableMVCCIds,https://docs.aws.amazon.com/documentdb/latest/developerguide/cloud_watch.html -Capacity/Cost,BackupRetentionPeriodStorageUsed,true,Backup storage used inside the retention window.,By,summary,aws_docdb_backup_retention_period_storage_used,query-value|timeseries|top-list,max,none,context.dbclusteridentifier:10,context.env|context.team,context.cloud.region,Metric absent or namespace mismatch,Cost signal not immediate health signal,cloudwatch=BackupRetentionPeriodStorageUsed,https://docs.aws.amazon.com/documentdb/latest/developerguide/cloud_watch.html -Capacity/Cost,TotalBackupStorageBilled,true,Total billed backup storage for the cluster.,By,summary,aws_docdb_total_backup_storage_billed,query-value|timeseries|top-list,max,none,context.dbclusteridentifier:10,context.env|context.team,context.cloud.region,Metric absent or namespace mismatch,Cost signal not immediate health signal,cloudwatch=TotalBackupStorageBilled,https://docs.aws.amazon.com/documentdb/latest/developerguide/cloud_watch.html -Capacity/Cost,CPUCreditBalance,true,Remaining burst credits for T3 instances.,count,summary,aws_docdb_cpu_credit_balance,query-value|timeseries,min,none,context.dbinstanceidentifier:20,context.env|context.team,context.dbclusteridentifier|context.role,Instance class not T3 or namespace mismatch,Gate by instance family,cloudwatch=CPUCreditBalance,https://docs.aws.amazon.com/documentdb/latest/developerguide/cloud_watch.html -Performance Insights,DBLoad,false,Average active sessions from Performance Insights; only published when PI is enabled and there is load.,count,summary,aws_docdb_db_load,query-value|timeseries|top-list,average,none,context.dbinstanceidentifier:20,context.env|context.team,context.dbclusteridentifier|context.role|context.cloud.region,PI disabled or no recent load,Do not assume presence in every deployment,cloudwatch=DBLoad,https://docs.aws.amazon.com/documentdb/latest/developerguide/performance-insights-cloudwatch.html -Performance Insights,DBLoadCPU,false,Average active sessions whose wait type is CPU.,count,summary,aws_docdb_db_load_cpu,query-value|timeseries|top-list,average,none,context.dbinstanceidentifier:20,context.env|context.team,context.dbclusteridentifier|context.role|context.cloud.region,PI disabled or no recent load,Interpret only with DBLoad or DBLoadNonCPU,cloudwatch=DBLoadCPU,https://docs.aws.amazon.com/documentdb/latest/developerguide/performance-insights-cloudwatch.html -Performance Insights,DBLoadNonCPU,false,Average active sessions whose wait type is not CPU.,count,summary,aws_docdb_db_load_non_cpu,query-value|timeseries|top-list,average,none,context.dbinstanceidentifier:20,context.env|context.team,context.dbclusteridentifier|context.role|context.cloud.region,PI disabled or no recent load,Wait-heavy load without context can mislead,cloudwatch=DBLoadNonCPU,https://docs.aws.amazon.com/documentdb/latest/developerguide/performance-insights-cloudwatch.html -Latency/Storage,DiskQueueDepth,true,Queued disk operations waiting on the storage path for a DocumentDB instance.,count,summary,aws_docdb_disk_queue_depth,query-value|timeseries|top-list,max,none,context.dbinstanceidentifier:20|context.role:4,context.env|context.team,context.dbclusteridentifier|context.cloud.region,Metric absent or namespace mismatch,Keep at instance or role granularity,live=aws_docdb_disk_queue_depth,https://docs.aws.amazon.com/documentdb/latest/developerguide/cloud_watch.html -Traffic/Operations,DocumentsInserted,true,Documents inserted during the one-minute period on the instance.,count,summary,aws_docdb_documents_inserted,query-value|timeseries|bar,sum,none,context.dbinstanceidentifier:20|context.role:4,context.env|context.team,context.dbclusteridentifier|context.cloud.region,No insert activity or namespace mismatch,Keep bounded to instance or role,live=aws_docdb_documents_inserted,https://docs.aws.amazon.com/documentdb/latest/developerguide/cloud_watch.html -Traffic/Operations,DocumentsUpdated,true,Documents updated during the one-minute period on the instance.,count,summary,aws_docdb_documents_updated,query-value|timeseries|bar,sum,none,context.dbinstanceidentifier:20|context.role:4,context.env|context.team,context.dbclusteridentifier|context.cloud.region,No update activity or namespace mismatch,Keep bounded to instance or role,live=aws_docdb_documents_updated,https://docs.aws.amazon.com/documentdb/latest/developerguide/cloud_watch.html -Traffic/Operations,DocumentsDeleted,true,Documents deleted during the one-minute period on the instance.,count,summary,aws_docdb_documents_deleted,query-value|timeseries|bar,sum,none,context.dbinstanceidentifier:20|context.role:4,context.env|context.team,context.dbclusteridentifier|context.cloud.region,No delete activity or namespace mismatch,Keep bounded to instance or role,live=aws_docdb_documents_deleted,https://docs.aws.amazon.com/documentdb/latest/developerguide/cloud_watch.html -Traffic/Operations,TTLDeletedDocuments,true,Documents deleted by the TTL monitor during the one-minute period.,count,summary,aws_docdb_ttl_deleted_documents,query-value|timeseries|bar,sum,none,context.dbinstanceidentifier:20|context.role:4,context.env|context.team,context.dbclusteridentifier|context.cloud.region,No TTL deletes or namespace mismatch,Usually low signal until TTL cleanup surges,live=aws_docdb_ttl_deleted_documents,https://docs.aws.amazon.com/documentdb/latest/developerguide/cloud_watch.html -Traffic/Operations,OpcountersGetmore,true,getMore operations issued during the one-minute period as cursors continue returning results.,count,summary,aws_docdb_opcounters_getmore,query-value|timeseries|bar,sum,none,context.dbinstanceidentifier:20|context.role:4,context.env|context.team,context.dbclusteridentifier|context.cloud.region,No getMore activity or namespace mismatch,Useful for cursor-heavy read patterns,live=aws_docdb_opcounters_getmore,https://docs.aws.amazon.com/documentdb/latest/developerguide/cloud_watch.html -Traffic/Operations,NetworkReceiveThroughput,true,Client network bytes per second received by the instance.,By/s,summary,aws_docdb_network_receive_throughput,query-value|timeseries|top-list,sum,none,context.dbinstanceidentifier:20|context.role:4,context.env|context.team,context.dbclusteridentifier|context.cloud.region|context.cloud.account.id,Metric absent or namespace mismatch,Prefer instance or role over raw clients,live=aws_docdb_network_receive_throughput,https://docs.aws.amazon.com/documentdb/latest/developerguide/cloud_watch.html -Traffic/Operations,NetworkThroughput,true,Total client network bytes per second in and out of the instance.,By/s,summary,aws_docdb_network_throughput,query-value|timeseries|top-list,sum,none,context.dbinstanceidentifier:20|context.role:4,context.env|context.team,context.dbclusteridentifier|context.cloud.region|context.cloud.account.id,Metric absent or namespace mismatch,Use with directional network metrics for interpretation,live=aws_docdb_network_throughput,https://docs.aws.amazon.com/documentdb/latest/developerguide/cloud_watch.html -Traffic/Operations,NetworkTransmitThroughput,true,Client network bytes per second sent by the instance.,By/s,summary,aws_docdb_network_transmit_throughput,query-value|timeseries|top-list,sum,none,context.dbinstanceidentifier:20|context.role:4,context.env|context.team,context.dbclusteridentifier|context.cloud.region|context.cloud.account.id,Metric absent or namespace mismatch,Prefer instance or role over raw clients,live=aws_docdb_network_transmit_throughput,https://docs.aws.amazon.com/documentdb/latest/developerguide/cloud_watch.html -Latency/Storage,ReadThroughput,true,Bytes read from disk per second by the instance.,By/s,summary,aws_docdb_read_throughput,query-value|timeseries|top-list,sum,none,context.dbinstanceidentifier:20|context.role:4,context.env|context.team,context.dbclusteridentifier|context.cloud.region,Read path idle or namespace mismatch,Use with ReadIOPS to interpret average request size,live=aws_docdb_read_throughput,https://docs.aws.amazon.com/documentdb/latest/developerguide/cloud_watch.html -Latency/Storage,WriteThroughput,true,Bytes written to disk per second by the instance.,By/s,summary,aws_docdb_write_throughput,query-value|timeseries|top-list,sum,none,context.dbinstanceidentifier:20|context.role:4,context.env|context.team,context.dbclusteridentifier|context.cloud.region,Write path idle or namespace mismatch,Use with WriteIOPS and change-stream growth for context,live=aws_docdb_write_throughput,https://docs.aws.amazon.com/documentdb/latest/developerguide/cloud_watch.html -Latency/Storage,EBSByteBalance,true,Remaining EBS throughput credits for supporting instance families.,%,summary,aws_docdb_ebs_byte_balance,query-value|timeseries,min,none,context.dbinstanceidentifier:20,context.env|context.team,context.dbclusteridentifier|context.cloud.region,Unsupported instance family or namespace mismatch,Keep gated because not every family emits it,live=aws_docdb_ebs_byte_balance,https://docs.aws.amazon.com/documentdb/latest/developerguide/cloud_watch.html -Latency/Storage,EBSIOBalance,true,Remaining EBS I/O credits for supporting instance families.,%,summary,aws_docdb_ebsio_balance,query-value|timeseries,min,none,context.dbinstanceidentifier:20,context.env|context.team,context.dbclusteridentifier|context.cloud.region,Unsupported instance family or namespace mismatch,Keep gated because not every family emits it,live=aws_docdb_ebsio_balance,https://docs.aws.amazon.com/documentdb/latest/developerguide/cloud_watch.html -Cache/Memory,LowMemNumOperationsTimedOut,true,Operations that timed out under low-memory pressure during the one-minute period.,count,summary,aws_docdb_low_mem_num_operations_timed_out,query-value|timeseries|top-list,sum,none,context.dbinstanceidentifier:20,context.env|context.team,context.dbclusteridentifier|context.role|context.cloud.region,No timed-out operations or namespace mismatch,Use with throttled ops and queue depth,live=aws_docdb_low_mem_num_operations_timed_out,https://docs.aws.amazon.com/documentdb/latest/developerguide/cloud_watch.html -Cache/Memory,LowMemThrottleMaxQueueDepth,true,Highest low-memory throttle queue depth observed within the minute.,count,summary,aws_docdb_low_mem_throttle_max_queue_depth,query-value|timeseries|top-list,max,none,context.dbinstanceidentifier:20,context.env|context.team,context.dbclusteridentifier|context.role|context.cloud.region,No low-memory queue spikes or namespace mismatch,Use with LowMemThrottleQueueDepth rather than alone,live=aws_docdb_low_mem_throttle_max_queue_depth,https://docs.aws.amazon.com/documentdb/latest/developerguide/cloud_watch.html -Cache/Memory,LongestActiveGCRuntime,true,Duration of the longest currently active garbage-collection process on the instance.,s,summary,aws_docdb_longest_active_gc_runtime,query-value|timeseries|top-list,max,none,context.dbinstanceidentifier:20,context.env|context.team,context.dbclusteridentifier|context.role|context.cloud.region,No active GC runtime metric or namespace mismatch,Interpret alongside memory pressure and write activity,live=aws_docdb_longest_active_gc_runtime,https://docs.aws.amazon.com/documentdb/latest/developerguide/cloud_watch.html -Traffic/Operations,DatabaseCursorsLimit,true,Maximum number of cursors allowed on an instance at a point in time.,count,summary,aws_docdb_database_cursors_limit,query-value|timeseries,max,none,context.dbinstanceidentifier:20,context.env|context.team,context.dbclusteridentifier|context.role|context.cloud.region,Metric absent or namespace mismatch,"Best used as a denominator or guardrail, not alone",live=aws_docdb_database_cursors_limit,https://docs.aws.amazon.com/documentdb/latest/developerguide/cloud_watch.html -Traffic/Operations,DatabaseCursorsMax,true,Peak open cursors seen during the one-minute period on the instance.,count,summary,aws_docdb_database_cursors_max,query-value|timeseries|top-list,max,none,context.dbinstanceidentifier:20,context.env|context.team,context.dbclusteridentifier|context.role|context.cloud.region,Metric absent or namespace mismatch,Interpret with current cursor count and getMore activity,live=aws_docdb_database_cursors_max,https://docs.aws.amazon.com/documentdb/latest/developerguide/cloud_watch.html -Traffic/Operations,TransactionsStarted,true,Transactions started during the one-minute period on the instance.,count,summary,aws_docdb_transactions_started,query-value|timeseries|top-list,sum,none,context.dbinstanceidentifier:20,context.env|context.team,context.dbclusteridentifier|context.role|context.cloud.region,No transaction starts or namespace mismatch,Use with commits and aborts for transaction health,live=aws_docdb_transactions_started,https://docs.aws.amazon.com/documentdb/latest/developerguide/cloud_watch.html -Traffic/Operations,TransactionsCommitted,true,Transactions committed during the one-minute period on the instance.,count,summary,aws_docdb_transactions_committed,query-value|timeseries|top-list,sum,none,context.dbinstanceidentifier:20,context.env|context.team,context.dbclusteridentifier|context.role|context.cloud.region,No committed transactions or namespace mismatch,Use with started and aborted counts,live=aws_docdb_transactions_committed,https://docs.aws.amazon.com/documentdb/latest/developerguide/cloud_watch.html -Traffic/Operations,TransactionsOpenLimit,true,Maximum number of concurrent open transactions allowed on the instance.,count,summary,aws_docdb_transactions_open_limit,query-value|timeseries,max,none,context.dbinstanceidentifier:20,context.env|context.team,context.dbclusteridentifier|context.role|context.cloud.region,Metric absent or namespace mismatch,"Best used as a denominator or guardrail, not alone",live=aws_docdb_transactions_open_limit,https://docs.aws.amazon.com/documentdb/latest/developerguide/cloud_watch.html -Traffic/Operations,TransactionsOpenMax,true,Peak open transactions seen during the one-minute period on the instance.,count,summary,aws_docdb_transactions_open_max,query-value|timeseries|top-list,max,none,context.dbinstanceidentifier:20,context.env|context.team,context.dbclusteridentifier|context.role|context.cloud.region,Metric absent or namespace mismatch,Interpret with current open transactions and aborts,live=aws_docdb_transactions_open_max,https://docs.aws.amazon.com/documentdb/latest/developerguide/cloud_watch.html diff --git a/plugins/tsuga/skills/knowledge-technology/references/aws-docdb/queries.md b/plugins/tsuga/skills/knowledge-technology/references/aws-docdb/queries.md deleted file mode 100644 index 0a9e5de..0000000 --- a/plugins/tsuga/skills/knowledge-technology/references/aws-docdb/queries.md +++ /dev/null @@ -1,54 +0,0 @@ -# AWS DocumentDB - -Managed MongoDB-compatible document DB. Healthy: connections under ceiling, cursors bounded, bounded latency, replication lag near zero, cache hit ratio high. - -## Incident shapes - -- **Connection exhaustion** — `aws_docdb_database_connections` near `aws_docdb_database_connections_limit` -- **Long-running cursors** — `aws_docdb_database_cursors` + `aws_docdb_database_cursors_timed_out` climb → slow scans -- **Storage / IO saturation** — `aws_docdb_read_iops` / `aws_docdb_write_iops` at instance ceiling -- **Replication lag** — primary → replica drift; reads stale -- **Transaction contention** — `aws_docdb_transactions_aborted` climbs under load -- **Cache-miss storm** — `aws_docdb_nvme_storage_cache_hit_ratio` drops → reads fall through to storage - -## Key metrics - -| Metric | Unit | Signal | -|---|---|---| -| `aws_docdb_cpu_utilization` | % | Instance CPU | -| `aws_docdb_database_connections` | count | Current connections | -| `aws_docdb_database_connections_limit` | count | Ceiling | -| `aws_docdb_database_cursors` | count | Open cursors; climb = leak | -| `aws_docdb_database_cursors_timed_out` | count | Cursor timeouts | -| `aws_docdb_engine_uptime` | seconds | Reset = restart | -| `aws_docdb_opcounters_query/Command/Insert/Update/Delete` | ops/s | Workload mix | -| `aws_docdb_documents_returned` | docs/s | Read work | -| `aws_docdb_transactions_open` | count | Open multi-doc txns | -| `aws_docdb_transactions_aborted` | count | Abort rate | -| `aws_docdb_read_latency` / `aws_docdb_write_latency` | ms | Per-op latency | -| `aws_docdb_read_iops` / `aws_docdb_write_iops` | ops/s | Storage IO | -| `aws_docdb_nvme_storage_cache_hit_ratio` | ratio | Local NVMe cache hits | -| `StorageNetworkReceive/Transmit Throughput` | bytes/s | Cluster storage network | - -## Derived signals - -- `aws_docdb_database_connections / aws_docdb_database_connections_limit` — connection utilization. > 0.85 sustained = alert. -- Derivative of `aws_docdb_database_cursors` — leak indicator. Steady positive slope = leak. -- `aws_docdb_transactions_aborted / (aws_docdb_transactions_open + Aborted)` — abort rate. > 0.05 sustained = contention. -- `1 - aws_docdb_nvme_storage_cache_hit_ratio` — cache-miss pressure. - -## Log patterns - -- `Error Code 50 (MaxTimeMSExpired)` — op exceeded client timeout -- `Error Code 11000 (DuplicateKey)` — unique-index violation -- `WriteConflict` — optimistic-concurrency conflict in txn -- `NetworkTimeout` / `server selection timed out` — client-side connectivity -- `Authentication failed` — IAM / password rotation -- Audit log `type: slowop` — ops above `slowOpThresholdMs` - -## Gotchas - -- DocumentDB isn't 100% MongoDB-compatible (`$lookup` complex pipelines, change streams differ). Don't assume Mongo guides apply. -- Replicas only serve reads when `readPreference` is explicit. Default-to-primary drivers get no read scaling. -- Multi-doc txn concurrency limits are lower than equivalent MongoDB; `aws_docdb_transactions_aborted` rising is often the cluster limit. -- `aws_docdb_nvme_storage_cache_hit_ratio` is instance-class specific; smaller instances show lower ratios under the same workload. diff --git a/plugins/tsuga/skills/knowledge-technology/references/aws-dynamodb/README.md b/plugins/tsuga/skills/knowledge-technology/references/aws-dynamodb/README.md deleted file mode 100644 index 81d02a2..0000000 --- a/plugins/tsuga/skills/knowledge-technology/references/aws-dynamodb/README.md +++ /dev/null @@ -1,262 +0,0 @@ -# AWS DynamoDB Integration Context Bundle - -## Metadata -**Technology:** AWS DynamoDB -**Deployment:** managed -**Environment:** prod -**Persona:** SRE Dev and ops -**Telemetry preference:** mixed -**Integration scope:** core service only -**Primary use-case:** reliability and performance - -## How to use this bundle -- Use `01_aws-dynamodb_metrics.csv` as the source of truth for DynamoDB metric names, units, and safe aggregation patterns. -- Use `02_aws-dynamodb_dashboard_plan.yaml` for the dashboard structure, derived signals, explanation notes, and playbooks. -- Use `03_aws-dynamodb_state.yaml` for machine-readable assumptions, prefix status, log intelligence status, and unresolved unknowns. -- Use `04_aws-dynamodb_memory.md` for the concise Stage 1 handoff and Stage 2 verification order. -- Stage 2 will create `05_aws-dynamodb_metric_catalog.csv` as the discovered Tsuga metric inventory and reconciliation memory. -- Stage 4 should read this file's `Log intelligence (Stage 4 handoff)` section and `03.log_intel` before proposing any log routes. - -## What it is and what "good" looks like - -### Confirmed by sources -- Amazon DynamoDB is a fully managed key-value and document database where operational health is mainly a function of request latency, throughput consumption, throttling, and replication state for global tables or Kinesis-backed change data capture. [S1][S4][S5] -- DynamoDB exposes service metrics through CloudWatch for tables, global secondary indexes, operations, account-level limits, online index builds, and global-table replication. [S1][S4] -- "Good" for DynamoDB means request latency is stable, consumed capacity remains comfortably below the active provisioned or on-demand ceiling, throttling is rare, and any replication backlog remains low and short-lived. [S1][S2][S4] -- CloudWatch Contributor Insights is the fastest built-in path for finding hot keys and throttled keys when the aggregate metrics say there is a hotspot but do not identify the key. [S2][S3] -- CloudTrail provides API-level history for table mutations, scaling changes, IAM access patterns, and data-plane events if enabled, but it is not a substitute for steady-state service metrics. [S6] - -### Best-practice inference -- Incident shape 1: **Latency or customer-visible slowness**. Start in `request-health` to decide whether latency moved first or whether throttling and capacity pressure explain it. -- Incident shape 2: **Capacity or throttle regression**. Start in `capacity-mode` and `throttling-hotspots` to separate provisioned exhaustion, on-demand ceilings, account quotas, and hot partitions. -- Incident shape 3: **Global table or CDC lag**. Start in `replication-cdc`; replication latency and oldest unreplicated age matter more than generic request throughput when cross-Region freshness is the concern. -- Incident shape 4: **Index build or schema change side effects**. Start in `index-operations`; online index progress and index-build throttle counters tell you whether a planned change is the reason baseline traffic is degrading. -- Paging intent should prioritize user-visible request failure or sustained throttling over raw consumption percentages. DynamoDB can run hot without user pain, but repeated throttle events, growing replication age, or an index build colliding with production traffic are direct operational risks. - -## Key concepts - -### Glossary - -| Term | Definition | Operational meaning | Dashboard section | -|---|---|---|---| -| Table | Primary DynamoDB storage object | Main unit for traffic, latency, and throttle attribution | request-health | -| Global secondary index (GSI) | Alternate query surface with separate capacity behavior | A table can look healthy while one GSI is the actual bottleneck | index-operations | -| Partition key | Primary key component used for data placement | Hot keys create partition-level throttling before total table capacity is exhausted | throttling-hotspots | -| Sort key | Optional second key component used for ordered access | Drives item-shape and query behavior, but is usually too high-cardinality for default metric group-bys | request-health | -| Provisioned mode | Capacity mode with explicit RCU/WCU allocation | Requires utilization and scaling-headroom tracking | capacity-mode | -| On-demand mode | Capacity mode that auto-scales request units | Still subject to per-table, per-index, and account ceilings | capacity-mode | -| Consumed read capacity units | Read throughput actually used over the period | Best baseline for read demand and one denominator for read saturation | capacity-mode | -| Consumed write capacity units | Write throughput actually used over the period | Best baseline for write demand and one denominator for write saturation | capacity-mode | -| Throttled request | Request rejected because throughput or quota limits were exceeded | User-visible symptom that matters more than raw capacity percentage | throttling-hotspots | -| Read throttle events | Read-side throttled events at table or GSI scope | Separates read pain from write pain | throttling-hotspots | -| Write throttle events | Write-side throttled events at table or GSI scope | Useful when write amplification or index fan-out is the issue | throttling-hotspots | -| Key range throughput exceeded | Partition-level limit breach | Strong sign of hot partition or poor key distribution | throttling-hotspots | -| Account limit exceeded | Regional account quota breach | Means local tuning will not fully help until quotas or traffic shape change | capacity-mode | -| Successful request latency | Latency for successful operations | Best user-facing performance surface when segmented by operation | request-health | -| User errors | HTTP 400-class request issues | Often client misuse, bad requests, or transaction conflicts rather than service failure | latency-errors | -| System errors | HTTP 500-class service-side failures | Rare and high signal when non-zero | latency-errors | -| Conditional check failure | Write rejected because a condition expression evaluated false | Usually expected contention or optimistic locking behavior, not a platform outage | latency-errors | -| Transaction conflict | Rejected item-level transactional request due to concurrency conflict | Critical for transactional workloads that appear slow despite low throttle counts | latency-errors | -| Returned bytes | Response payload bytes returned | Useful to separate "slow because large result sets" from "slow because throttled" | request-health | -| Returned item count | Items returned by reads | High values often indicate inefficient query or scan patterns | request-health | -| Replication latency | Cross-Region lag for global tables | Direct freshness risk for failover and multi-Region reads | replication-cdc | -| Pending replication count | Updates not yet applied to another replica | Backlog indicator for global table propagation | replication-cdc | -| Age of oldest unreplicated record | Time since the oldest not-yet-replicated record arrived | Better urgency signal than backlog size alone | replication-cdc | -| Consumed change data capture units | Change-data-capture work consumed for Kinesis integration | Helps quantify CDC cost or downstream pressure | replication-cdc | -| Online index percentage progress | Progress of a new GSI build | Explains planned operational noise during schema evolution | index-operations | -| Contributor Insights | CloudWatch top-N analysis for accessed or throttled keys | Best way to pivot from aggregate throttle metrics to hot-key evidence | throttling-hotspots | - -### Concept Map -Client request -> targets -> DynamoDB table (why: every incident starts at table or index scope) -Table request -> may route through -> Global secondary index (why: a GSI can throttle independently of the base table) -Provisioned table -> bounded by -> ProvisionedReadCapacityUnits and ProvisionedWriteCapacityUnits (why: utilization above this drives provisioned throttling) -On-demand table -> bounded by -> OnDemandMaxReadRequestUnits and OnDemandMaxWriteRequestUnits (why: user-configured or service limits can still cap burst traffic) -Request mix -> drives -> ConsumedReadCapacityUnits and ConsumedWriteCapacityUnits (why: these are the cleanest demand baselines) -High consumed capacity -> does not always imply -> user pain (why: adaptive capacity and burst behavior can absorb short spikes) -Hot partition key -> causes -> ReadKeyRangeThroughputThrottleEvents or WriteKeyRangeThroughputThrottleEvents (why: partition limits trip before table totals) -Hot partition -> identified by -> Contributor Insights throttled keys reports (why: aggregate table metrics hide which keys are hottest) -ThrottledRequests -> summarizes -> request-level impact (why: it answers whether callers were rejected at all) -ReadThrottleEvents and WriteThrottleEvents -> explain -> event-level scope of rejection (why: one request can create multiple throttled events) -SuccessfulRequestLatency -> shaped by -> operation type and payload size (why: Query and Scan behavior can differ sharply from point reads) -ReturnedItemCount -> amplifies -> ReturnedBytes (why: larger result sets often explain rising successful latency) -Conditional writes -> increment -> ConditionalCheckFailedRequests (why: contention can look like errors without being platform failure) -Concurrent transactions -> increment -> TransactionConflict (why: item-level contention is a workload-shape issue, not usually a service outage) -SystemErrors -> indicate -> DynamoDB-side failure (why: they are rarer and higher-severity than client-side errors) -Global table replica -> receives -> replicated writes from source Region (why: freshness depends on cross-Region apply) -PendingReplicationCount -> raises -> ReplicationLatency and AgeOfOldestUnreplicatedRecord risk (why: backlog that lasts long enough becomes stale data) -Kinesis CDC integration -> consumes -> ConsumedChangeDataCaptureUnits (why: downstream change-stream usage adds cost and replication-style pressure) -FailedToReplicateRecordCount -> signals -> dropped or stuck CDC delivery (why: this is stronger than a backlog warning) -Online GSI build -> consumes -> OnlineIndexConsumedWriteCapacity (why: schema changes compete with steady-state traffic) -Online index build -> exposes -> OnlineIndexPercentageProgress (why: operators need to distinguish planned backfill from unexplained saturation) -context.env and context.team -> scope -> ownership and paging boundaries (why: dashboards must support service ownership, not only raw AWS dimensions) -context.tablename and context.globalsecondaryindexname -> anchor -> safe default group-bys (why: table and GSI are useful and bounded) -CloudTrail data events -> capture -> API caller and request context (why: they help explain who changed or accessed what when metrics show a symptom) - -### Entities and dimensions - -| Entity/Dimension | Why useful | Cardinality risk | Safe top-N suggestion | Do NOT group-by guidance | -|---|---|---|---|---| -| `context.env` | Separates prod from lower-risk environments | Low | 5 | Prefer as a global filter, not repeated series on every chart | -| `context.team` | Ownership and escalation boundary | Low | 10 | Use as filter first, not as the primary technical split | -| `context.cloud.region` | Failure domain and replica boundary | Low | 10 | Use Region before Availability Zone unless AZ is actually present | -| `context.cloud.account.id` | Multi-account scope control | Medium | 10 | Skip if the workspace is single-account | -| `context.tablename` | Primary table attribution | Low to medium | 20 | Safe default table split for most charts | -| `context.globalsecondaryindexname` | Reveals index-specific hotspots | Medium | 20 | Do not combine with high-cardinality request parameters | -| `context.operation` | Helps split `Query`, `Scan`, `GetItem`, `PutItem`, and transactional operations | Medium | 12 | Confirm existence in Stage 2 before making it a dashboard-wide filter | -| `context.receivingregion` | Needed for global-table backlog analysis | Low | 10 | Use only in replication sections | -| `context.delegatedoperation` | Required for CDC metrics to distinguish Kinesis replication work | Medium | 10 | Keep to CDC sections only | -| `context.billingmode` | Distinguishes on-demand from provisioned behavior | Low | 4 | Treat as optional enrichment, not guaranteed metric metadata | -| `context.source` | Helpful to distinguish cloudwatch vs enriched pipelines | Low | 4 | Not useful as an end-user default filter | -| `context.aws.resource.arn` | Useful for drilldown or links, especially for GSI and table resources | Medium | 10 | Avoid chart legends based on full ARN strings | -| `context.partitionkey` | Useful only when Contributor Insights or logs expose it | Very high | 25 max when explicitly investigating | Never use as a default group-by in service dashboards | - -### Tsuga field mapping - -| Vendor/exporter dimension | Recommended context.* key | Must-exist vs optional | -|---|---|---| -| `TableName` | `context.tablename` | Must-exist | -| `GlobalSecondaryIndexName` | `context.globalsecondaryindexname` | Optional but strongly preferred | -| `Operation` | `context.operation` | Optional | -| `ReceivingRegion` | `context.receivingregion` | Optional | -| `DelegatedOperation` | `context.delegatedoperation` | Optional | -| AWS Region enrichment | `context.cloud.region` | Must-exist | -| AWS account enrichment | `context.cloud.account.id` | Optional | -| Environment tag enrichment | `context.env` | Must-exist | -| Team tag enrichment | `context.team` | Must-exist | -| Contributor Insights hot key payload | `context.partitionkey` | Optional and high-risk | -| Billing mode metadata | `context.billingmode` | Optional | -| Resource ARN enrichment | `context.aws.resource.arn` | Optional | - -### Confirmed by sources -- The CloudWatch metrics reference explicitly documents `TableName`, `GlobalSecondaryIndexName`, `Operation`, `ReceivingRegion`, and `DelegatedOperation` as DynamoDB metric dimensions depending on metric family. [S1] -- CloudTrail events for DynamoDB include API action, actor, time, source IP, and request details, which makes them the most realistic Stage 4 log source for service-generated operational history. [S6] - -### Best-practice inference -- Stage 2 should prefer flattened keys such as `context.tablename`, `context.globalsecondaryindexname`, `context.operation`, and `context.receivingregion` if multiple source-specific aliases exist. -- Partition-key level fields should remain investigation-only fields. They are valuable when present, but they are too sensitive and high-cardinality for default dashboard filters. - -## Golden signals - -### Confirmed by sources -| Signal | What it means for AWS DynamoDB | Typical degradation causes | Best telemetry sources | What people page on | Section questions | -|---|---|---|---|---|---| -| Traffic | Read and write request volume, result-set size, and capacity consumption shape | Bursty clients, index backfill overlap, scans, retry storms | `ConsumedReadCapacityUnits`, `ConsumedWriteCapacityUnits`, `ReturnedBytes`, `ReturnedItemCount` [S1] | Consumption climbs sharply or result sizes balloon before latency degrades | Is demand higher, or is the workload simply less efficient? | -| Errors | Client-side misuse, concurrency conflicts, and service-side failures | Bad request patterns, conditional contention, transactional conflicts, rare platform errors | `UserErrors`, `ConditionalCheckFailedRequests`, `TransactionConflict`, `SystemErrors` [S1][S8] | System errors or transaction conflicts rise enough to affect successful work | Are failures due to callers, concurrency, or the service itself? | -| Latency | Time to complete successful requests | Large result sets, scans, retries, throttle retries, hot partitions, cross-Region lag | `SuccessfulRequestLatency`, `ReturnedBytes`, `ReturnedItemCount`, replication metrics [S1][S4] | Sustained latency increase, especially when throttling or payload growth also rises | Is slowness request-shape-driven, throttle-driven, or replica-driven? | -| Saturation | Nearness to throughput ceilings, quotas, and hotspot boundaries | Provisioned under-sizing, on-demand max limits, account quotas, hot partitions, GSI backfill | Consumed vs provisioned or on-demand limit metrics, throttle metrics, Contributor Insights [S1][S2][S3] | Repeated throttle events, partition-level throttle indicators, quota-limit throttle counts | Which limit is actually binding: table, index, partition, on-demand max, or account quota? | - -### Best-practice inference -- For DynamoDB, throttle classification matters more than CPU-style host metrics because the service hides host internals and instead exposes the limit surface directly. A good dashboard should answer "what kind of throttle is this?" before it asks "how busy is the service?" -- Returned bytes and returned item count matter more than a generic request-count graph because many DynamoDB latency incidents are query-shape or scan-shape problems rather than pure request volume problems. -- Replication backlog should be treated like a fifth golden signal when global tables or Kinesis CDC are in use because stale data is an application-level outage even if the base table is still serving requests. - -## Telemetry sources - -### Confirmed by sources -| Source type | How collected | What it provides | Pros/cons | Common pitfalls | -|---|---|---|---|---| -| CloudWatch DynamoDB metrics | Native service metrics in the DynamoDB namespace | Throughput, throttling, errors, latency, global-table, CDC, and index-build signals | Broadest coverage and best default source | Many metrics are scoped differently by table, GSI, operation, or receiving Region [S1] | -| CloudWatch Contributor Insights | Enabled per table or GSI | Top accessed and throttled keys | Best hotspot evidence without custom logs | Not enabled everywhere; key data can be sensitive; max 25 contributors in reports [S3] | -| CloudTrail management events | Enabled by default in account history, or via trails | Table creation, scaling changes, policy changes, stream settings, GSI changes | Best control-plane change history | Event history is short unless trails are retained centrally [S6] | -| CloudTrail data events for DynamoDB | Optional CloudTrail data event logging | API-level reads and writes with actor and request context | Best Stage 4 service log candidate | Often disabled because of cost and volume [S6] | -| DynamoDB Streams / Kinesis CDC metrics | Native metrics when change data capture is enabled | Replication backlog, oldest age, failed replication counts, CDC unit consumption | Best freshness and CDC cost context | Absent if the feature is not enabled [S1][S5] | -| DescribeTable / control-plane inspection | AWS API or inventory tooling | Billing mode, GSI status, stream settings, replica layout | Explains why some metric families appear or are absent | Not a timeseries source; must be combined with metrics [S10] | - -### Best-practice inference -- "No data" on replication or CDC widgets usually means global tables or Kinesis integration are not enabled, not that replication is perfectly healthy. -- "No data" on online-index widgets usually means no active GSI build is happening, which is a healthy steady-state default. -- "No data" on Contributor Insights derived pivots usually means the feature was never enabled or is not ingested into Tsuga. -- Stage 2 should treat capacity-mode-specific metrics carefully: provisioned and on-demand surfaces can coexist in documentation, but a given table often makes only one family operationally relevant at a time. -- Stage 2 confirmed that this workspace currently exposes only provisioned-capacity and account-quota metrics for DynamoDB, so any dashboard built from this bundle must be explicitly framed as capacity-only rather than full service health. - -## Log intelligence (Stage 4 handoff) - -### Confirmed by sources -1. **Log sources matrix** - -| Source | Access path | Typical format | Structured vs unstructured | Evidence | -|---|---|---|---|---| -| CloudTrail management events | CloudTrail Event History, S3 trail delivery, or CloudWatch Logs integration | CloudTrail JSON event | Structured | [S6] | -| CloudTrail data events for DynamoDB | Optional CloudTrail data-event logging for tables | CloudTrail JSON event with request and identity context | Structured | [S6] | -| Application consumers of DynamoDB Streams or Kinesis CDC | App logs around stream processors | App-specific JSON or text, not a DynamoDB-native format | Mixed | [S5] | - -2. **Known log formats** -- **CloudTrail DynamoDB event**: JSON event with fields such as `eventSource`, `eventName`, `awsRegion`, `userIdentity`, `sourceIPAddress`, and request details. Typical event source is `dynamodb.amazonaws.com`. [S6] -- **CloudTrail data event for table access**: JSON event similar to management events, but focused on data-plane operations such as item reads and writes when data-event logging is enabled. Request shape and response elements vary by API operation. [S6] - -3. **Candidate query filters for Stage 4** -- Precise: `eventSource:dynamodb.amazonaws.com AND requestParameters.tableName:*` - Rationale: targets CloudTrail records that clearly belong to DynamoDB and have a table target. - Risk: field names may be normalized differently in Tsuga. -- Fallback: `(message:*dynamodb.amazonaws.com* OR message:*DynamoDB*) AND (GetItem OR PutItem OR UpdateItem OR Query OR Scan)` - Rationale: works when CloudTrail JSON is not fully normalized. - Risk: could mix application logs mentioning DynamoDB with true CloudTrail events. - -4. **Attribute mapping hints** - -| Raw field | Suggested Tsuga key | Confidence | Notes | -|---|---|---|---| -| `awsRegion` | `context.cloud.region` | High | Stable CloudTrail field | -| `requestParameters.tableName` | `context.tablename` | High | Best Stage 4 table filter | -| `requestParameters.indexName` | `context.globalsecondaryindexname` | Medium | Present only for index-scoped operations | -| `eventName` | `context.operation` | High | Maps naturally to DynamoDB API operation | -| `recipientAccountId` | `context.cloud.account.id` | High | Good ownership and quota boundary | -| `userIdentity.arn` | `context.actor.arn` | Medium | High-cardinality, useful for investigations but not defaults | -| `sourceIPAddress` | `context.source.ip` | Medium | Useful for anomalous access paths | - -5. **Parsing risks** -- CloudTrail data events for DynamoDB are optional and can be absent even in production workloads. [S6] -- Different DynamoDB APIs emit different request payload shapes, so parsers must handle missing `indexName`, transaction substructures, and batch request arrays. [S6][S8] -- Streams and Kinesis consumers are not DynamoDB-native logs; their formats come from the application or integration layer, so Stage 4 should not infer one canonical parser from DynamoDB docs alone. [S5] -- Contributor Insights exposes key-level evidence in CloudWatch graphs, but that is not a raw log stream and should not be treated as a parsable log source. [S3] - -### Best-practice inference -- Stage 4 should prioritize CloudTrail if the goal is who-did-what change analysis, and application stream-processor logs if the goal is CDC failure root cause. -- If Tsuga already normalizes CloudTrail JSON fields, `context.tablename`, `context.globalsecondaryindexname`, and `context.operation` should be the first extracted keys because they align directly with the dashboard group-bys. - -## Caveats and footguns -- **[table-capacity]** `SuccessfulRequestLatency` covers successful requests only. A retry storm can make clients slow even if the service latency metric looks stable. (Inference) -- **[table-capacity]** `ReturnedItemCount` and `ReturnedBytes` can rise because callers switched from point reads to broad `Query` or `Scan` patterns, not because throughput capacity is broken. [S1] -- **[table-capacity]** `SuccessfulRequestLatency` is often segmented by `Operation`; if Stage 2 does not find `context.operation`, the dashboard must not promise operation-sliced latency. [S1] -- **[table-capacity]** `ConsumedReadCapacityUnits` and `ConsumedWriteCapacityUnits` are period totals. For per-second intuition, divide the summed value by the interval length. [S1] -- **[table-capacity]** Short spikes can be muted in CloudWatch minute rollups, so "utilization looked fine" does not prove the workload never burst. [S1] -- **[account-capacity, table-capacity]** Provisioned and on-demand limit metrics are not equally relevant for every table. Showing both without mode context can confuse operators. [S1][S9] -- **[table-capacity]** Transactional APIs consume double read or write units per item compared with non-transactional operations, so a transaction-heavy rollout can look like unexplained capacity inflation. [S8] -- **[table-capacity]** Adaptive capacity can mask hotspot pain temporarily, so near-flat table-level utilization does not rule out partition pressure. [S2][S9] -- **[table-capacity]** `ThrottledRequests` increments at request level, while `ReadThrottleEvents` and `WriteThrottleEvents` can increment multiple times per request because tables and GSIs are counted separately. [S1] -- **[table-capacity]** Batch requests only increment `ThrottledRequests` when every request in the batch is throttled, so request-level throttles can understate partial pain. [S1] -- **[table-capacity]** Partition-limit throttles (`KeyRangeThroughput`) usually require key-distribution fixes, not only more table capacity. [S2][S7] -- **[table-capacity]** Contributor Insights may expose sensitive partition-key values. Do not make those default legends or dashboard filters. [S3] -- **[table-capacity]** GSI throttles can be invisible if you only chart table-level dimensions. Always preserve the optional index dimension when available. [S1][S10] -- **[table-capacity]** `ConditionalCheckFailedRequests` is often normal application behavior for optimistic locking. Treat it as correctness or contention context, not a platform outage signal by default. [S1] -- **[table-capacity]** `UserErrors` aggregates many HTTP 400 cases, including `TransactionConflict`, but excludes throughput-exceeded and conditional-check failures. The metric is broad and must be interpreted with the more specific counters. [S1] -- **[table-capacity]** `SystemErrors` is Region-account scoped and usually sparse. One spike is high-signal, but low sustained values can still matter during customer-facing incidents. [S1] -- **[account-capacity]** `PendingReplicationCount` is documented for legacy global tables. Stage 2 must verify whether it exists in this workspace before promising it in the UI. [S1][S4] -- **[account-capacity]** Replication lag metrics can be absent simply because global tables are not enabled. Missing data does not always mean telemetry failure. [S4] -- **[account-capacity]** `ConsumedChangeDataCaptureUnits`, `FailedToReplicateRecordCount`, and `ThrottledPutRecordCount` only matter if DynamoDB-to-Kinesis change data capture is enabled. [S1][S5] -- **[account-capacity]** Online index metrics are intentionally quiet outside active index builds. A blank chart here is often healthy. [S1][S10] -- **[account-capacity]** `OnlineIndexConsumedWriteCapacity` and `OnlineIndexThrottleEvents` are expected to stay at zero during index builds in current AWS docs, so `OnlineIndexPercentageProgress` is the more actionable signal. [S1] -- **[account-capacity]** GSI creation does not use Application Auto Scaling to speed up the build, so increasing minimum autoscaling targets will not shorten build time. [S10] - -## Confirmed Tsuga prefixes -- `aws_dynamodb*` - **CONFIRMED** (12/12 live matches during Stage 2 preflight and catalog bootstrap) -- `aws_dynamodb_` - **CONFIRMED** (12/12 live matches; the exported family is underscore-normalized in Tsuga) - -## Discovery status -- Stage 2 discovery completed: 12 DynamoDB metrics confirmed in Tsuga, all `summary` with `cumulative` temporality. -- Confirmed live families are limited to account-level provisioned-capacity ceilings and utilization plus table-level consumed and provisioned capacity. -- The following Stage 1 families are absent in this workspace: latency, throttling, user or system errors, replication or CDC, TTL, and online-index metrics. -- The live context registry is narrow: `context.env`, `context.team`, `context.cloud.region`, `context.cloud.account.id`, and `context.tablename` on table-scoped capacity metrics. -- Scalar spot-checks through Tsuga aggregation returned internal server errors, so recent-data validation is inconclusive rather than zero. - -## Top sources -- [S1] https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/metrics-dimensions.html - Primary DynamoDB CloudWatch metric catalog, dimensions, and statistics. -- [S2] https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/throttling-diagnosing-workflow.html - Best official mapping from throttle reason to the right metric family. -- [S3] https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/contributorinsights_HowItWorks.html - Explains hot-key and throttled-key investigation behavior and caveats. -- [S4] https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/globaltables_monitoring.html - Official guidance for global-table replication health and lag interpretation. -- [S5] https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/streamsmain.html - Official change-data-capture options and feature boundaries. -- [S6] https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/logging-using-cloudtrail.html - Best source for Stage 4 log-source reality and event fields. -- [S7] https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/bp-partition-key-sharding.html - Official hot-partition mitigation guidance. -- [S8] https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/transaction-apis.html - Transaction semantics, conflict handling, and double-capacity caveats. -- [S9] https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/capacity-mode.html - Capacity-mode behavior and why provisioned vs on-demand must be separated. -- [S10] https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/GSI.OnlineOps.html - Official GSI build behavior, monitoring, and build-time operational caveats. diff --git a/plugins/tsuga/skills/knowledge-technology/references/aws-dynamodb/metrics.csv b/plugins/tsuga/skills/knowledge-technology/references/aws-dynamodb/metrics.csv deleted file mode 100644 index a82ffaa..0000000 --- a/plugins/tsuga/skills/knowledge-technology/references/aws-dynamodb/metrics.csv +++ /dev/null @@ -1,43 +0,0 @@ -theme,name,confirmed,definition,unit,type,tsuga_metric_name,widget_suggestions,aggregation,post_function,group_by,required_context_fields,optional_context_fields,absence_means,cardinality_footguns,variants,citations -Capacity/Saturation,AccountMaxReads,true,Maximum read capacity units the account can use in the Region for provisioned tables.,count,summary,aws_dynamodb_account_max_reads,query-value|timeseries,max,none,context.cloud.region:10,context.env|context.team,context.cloud.account.id,Metric absent or namespace mismatch,Account-scoped metric so table group-by is invalid,live=aws_dynamodb_account_max_reads,https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/metrics-dimensions.html -Capacity/Saturation,AccountMaxTableLevelReads,true,Maximum read capacity units any single table or GSI in the account can use in the Region.,count,summary,aws_dynamodb_account_max_table_level_reads,query-value|timeseries,max,none,context.cloud.region:10,context.env|context.team,context.cloud.account.id,Metric absent or namespace mismatch,Account-scoped metric so table group-by is invalid,live=aws_dynamodb_account_max_table_level_reads,https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/metrics-dimensions.html -Capacity/Saturation,AccountMaxTableLevelWrites,true,Maximum write capacity units any single table or GSI in the account can use in the Region.,count,summary,aws_dynamodb_account_max_table_level_writes,query-value|timeseries,max,none,context.cloud.region:10,context.env|context.team,context.cloud.account.id,Metric absent or namespace mismatch,Account-scoped metric so table group-by is invalid,live=aws_dynamodb_account_max_table_level_writes,https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/metrics-dimensions.html -Capacity/Saturation,AccountMaxWrites,true,Maximum write capacity units the account can use in the Region for provisioned tables.,count,summary,aws_dynamodb_account_max_writes,query-value|timeseries,max,none,context.cloud.region:10,context.env|context.team,context.cloud.account.id,Metric absent or namespace mismatch,Account-scoped metric so table group-by is invalid,live=aws_dynamodb_account_max_writes,https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/metrics-dimensions.html -Capacity/Saturation,AccountProvisionedReadCapacityUtilization,true,Percentage of provisioned read capacity used across the account in a Region; quota proximity context.,%,summary,aws_dynamodb_account_provisioned_read_capacity_utilization,query-value|timeseries,max,none,context.cloud.region:10,context.env|context.team,context.cloud.account.id,No provisioned tables in region or namespace mismatch,Account scope is too broad for table drilldowns,cloudwatch=AccountProvisionedReadCapacityUtilization,https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/metrics-dimensions.html -Capacity/Saturation,AccountProvisionedWriteCapacityUtilization,true,Percentage of provisioned write capacity used across the account in a Region; quota proximity context.,%,summary,aws_dynamodb_account_provisioned_write_capacity_utilization,query-value|timeseries|max,none,context.cloud.region:10,context.env|context.team,context.cloud.account.id,No provisioned tables in region or namespace mismatch,Account scope is too broad for table drilldowns,cloudwatch=AccountProvisionedWriteCapacityUtilization,https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/metrics-dimensions.html -Throughput/Usage,ConsumedReadCapacityUnits,true,Read capacity actually consumed over the time period; best baseline for read demand and one denominator for read saturation.,count,summary,aws_dynamodb_consumed_read_capacity_units,query-value|timeseries|top-list,sum,none,context.tablename:20,context.env|context.team,context.cloud.region|context.cloud.account.id,No reads or namespace mismatch,Keep default grouping at table scope because GSI context is not present in this workspace,cloudwatch=ConsumedReadCapacityUnits,https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/metrics-dimensions.html -Throughput/Usage,ConsumedWriteCapacityUnits,true,Write capacity actually consumed over the time period; best baseline for write demand and one denominator for write saturation.,count,summary,aws_dynamodb_consumed_write_capacity_units,query-value|timeseries|top-list,sum,none,context.tablename:20,context.env|context.team,context.cloud.region|context.cloud.account.id,No writes or namespace mismatch,Keep default grouping at table scope because GSI context is not present in this workspace,cloudwatch=ConsumedWriteCapacityUnits,https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/metrics-dimensions.html -Capacity/Saturation,MaxProvisionedTableReadCapacityUtilization,true,Highest provisioned read-capacity utilization observed on any single table or GSI in the account during the interval.,%,summary,aws_dynamodb_max_provisioned_table_read_capacity_utilization,query-value|timeseries,max,none,context.cloud.region:10,context.env|context.team,context.cloud.account.id,No provisioned tables in region or namespace mismatch,Account-scoped metric so it cannot identify the hot table by itself,live=aws_dynamodb_max_provisioned_table_read_capacity_utilization,https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/metrics-dimensions.html -Capacity/Saturation,MaxProvisionedTableWriteCapacityUtilization,true,Highest provisioned write-capacity utilization observed on any single table or GSI in the account during the interval.,%,summary,aws_dynamodb_max_provisioned_table_write_capacity_utilization,query-value|timeseries,max,none,context.cloud.region:10,context.env|context.team,context.cloud.account.id,No provisioned tables in region or namespace mismatch,Account-scoped metric so it cannot identify the hot table by itself,live=aws_dynamodb_max_provisioned_table_write_capacity_utilization,https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/metrics-dimensions.html -Capacity/Saturation,ProvisionedReadCapacityUnits,true,Provisioned read capacity configured for a table; denominator for read headroom in this workspace.,count,summary,aws_dynamodb_provisioned_read_capacity_units,query-value|timeseries|top-list,max,none,context.tablename:20,context.env|context.team,context.cloud.region|context.cloud.account.id,Table is on-demand or namespace mismatch,Use table grouping only because GSI context is not present in this workspace,cloudwatch=ProvisionedReadCapacityUnits,https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/metrics-dimensions.html -Capacity/Saturation,ProvisionedWriteCapacityUnits,true,Provisioned write capacity configured for a table; denominator for write headroom in this workspace.,count,summary,aws_dynamodb_provisioned_write_capacity_units,query-value|timeseries|top-list,max,none,context.tablename:20,context.env|context.team,context.cloud.region|context.cloud.account.id,Table is on-demand or namespace mismatch,Use table grouping only because GSI context is not present in this workspace,cloudwatch=ProvisionedWriteCapacityUnits,https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/metrics-dimensions.html -Capacity/Saturation,OnDemandMaxReadRequestUnits,false,Configured or effective read ceiling for on-demand read request units at table or GSI level.,count,summary,aws_dynamodb_on_demand_max_read_request_units,query-value|timeseries|max,none,context.tablename:20|context.globalsecondaryindexname:20,context.env|context.team,context.cloud.region|context.cloud.account.id,Table is provisioned or metric not ingested in this workspace,Useful only for on-demand or mixed fleets,cloudwatch=OnDemandMaxReadRequestUnits,https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/metrics-dimensions.html -Capacity/Saturation,OnDemandMaxWriteRequestUnits,false,Configured or effective write ceiling for on-demand write request units at table or GSI level.,count,summary,aws_dynamodb_on_demand_max_write_request_units,query-value|timeseries|max,none,context.tablename:20|context.globalsecondaryindexname:20,context.env|context.team,context.cloud.region|context.cloud.account.id,Table is provisioned or metric not ingested in this workspace,Useful only for on-demand or mixed fleets,cloudwatch=OnDemandMaxWriteRequestUnits,https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/metrics-dimensions.html -Errors/Failures,ThrottledRequests,false,Request-level throttling indicator; increments when a request is rejected by throughput limits.,count,summary,aws_dynamodb_throttled_requests,query-value|timeseries|top-list,sum,none,context.tablename:20|context.operation:12,context.env|context.team,context.globalsecondaryindexname|context.cloud.region,Metric not present in the live Tsuga export for this workspace,Request-level and event-level throttles are not interchangeable,cloudwatch=ThrottledRequests,https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/metrics-dimensions.html -Errors/Failures,ReadThrottleEvents,false,Read-side throttle events for tables or GSIs; stronger root-cause detail than request-level throttles.,count,summary,aws_dynamodb_read_throttle_events,query-value|timeseries|top-list,sum,none,context.tablename:20|context.globalsecondaryindexname:20,context.env|context.team,context.cloud.region|context.operation,Metric not present in the live Tsuga export for this workspace,One request can create multiple throttle events,cloudwatch=ReadThrottleEvents,https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/metrics-dimensions.html -Errors/Failures,WriteThrottleEvents,false,Write-side throttle events for tables or GSIs; shows table or index write pain.,count,summary,aws_dynamodb_write_throttle_events,query-value|timeseries|top-list,sum,none,context.tablename:20|context.globalsecondaryindexname:20,context.env|context.team,context.cloud.region|context.operation,Metric not present in the live Tsuga export for this workspace,One request can create multiple throttle events,cloudwatch=WriteThrottleEvents,https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/metrics-dimensions.html -Errors/Failures,ReadProvisionedThroughputThrottleEvents,false,Read requests throttled due to provisioned throughput limits.,count,summary,aws_dynamodb_read_provisioned_throughput_throttle_events,query-value|timeseries|top-list,sum,none,context.tablename:20|context.globalsecondaryindexname:20,context.env|context.team,context.cloud.region,Metric not present in the live Tsuga export for this workspace,Interpret only when provisioned mode is active,cloudwatch=ReadProvisionedThroughputThrottleEvents,https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/metrics-dimensions.html -Errors/Failures,WriteProvisionedThroughputThrottleEvents,false,Write requests throttled due to provisioned throughput limits.,count,summary,aws_dynamodb_write_provisioned_throughput_throttle_events,query-value|timeseries|top-list,sum,none,context.tablename:20|context.globalsecondaryindexname:20,context.env|context.team,context.cloud.region,Metric not present in the live Tsuga export for this workspace,Interpret only when provisioned mode is active,cloudwatch=WriteProvisionedThroughputThrottleEvents,https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/metrics-dimensions.html -Errors/Failures,ReadKeyRangeThroughputThrottleEvents,false,Read requests throttled due to partition-level key range limits.,count,summary,aws_dynamodb_read_key_range_throughput_throttle_events,query-value|timeseries|top-list,sum,none,context.tablename:20|context.globalsecondaryindexname:20,context.env|context.team,context.cloud.region,Metric not present in the live Tsuga export for this workspace,Usually requires key-distribution investigation not only more capacity,cloudwatch=ReadKeyRangeThroughputThrottleEvents,https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/metrics-dimensions.html -Errors/Failures,WriteKeyRangeThroughputThrottleEvents,false,Write requests throttled due to partition-level key range limits.,count,summary,aws_dynamodb_write_key_range_throughput_throttle_events,query-value|timeseries|top-list,sum,none,context.tablename:20|context.globalsecondaryindexname:20,context.env|context.team,context.cloud.region,Metric not present in the live Tsuga export for this workspace,Usually requires key-distribution investigation not only more capacity,cloudwatch=WriteKeyRangeThroughputThrottleEvents,https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/metrics-dimensions.html -Errors/Failures,ReadAccountLimitThrottleEvents,false,Read requests throttled due to account-level limits.,count,summary,aws_dynamodb_read_account_limit_throttle_events,query-value|timeseries|top-list,sum,none,context.tablename:20,context.env|context.team,context.cloud.region|context.cloud.account.id,Metric not present in the live Tsuga export for this workspace,Table grouping is useful but the fix is often quota-level,cloudwatch=ReadAccountLimitThrottleEvents,https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/metrics-dimensions.html -Errors/Failures,WriteAccountLimitThrottleEvents,false,Write requests throttled due to account-level limits.,count,summary,aws_dynamodb_write_account_limit_throttle_events,query-value|timeseries|top-list,sum,none,context.tablename:20,context.env|context.team,context.cloud.region|context.cloud.account.id,Metric not present in the live Tsuga export for this workspace,Table grouping is useful but the fix is often quota-level,cloudwatch=WriteAccountLimitThrottleEvents,https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/metrics-dimensions.html -Errors/Failures,ReadMaxOnDemandThroughputThrottleEvents,false,Read requests throttled due to on-demand maximum throughput settings.,count,summary,aws_dynamodb_read_max_on_demand_throughput_throttle_events,query-value|timeseries|top-list,sum,none,context.tablename:20|context.globalsecondaryindexname:20,context.env|context.team,context.cloud.region,Metric not present in the live Tsuga export for this workspace,Relevant only when on-demand max throughput is configured,cloudwatch=ReadMaxOnDemandThroughputThrottleEvents,https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/metrics-dimensions.html -Errors/Failures,WriteMaxOnDemandThroughputThrottleEvents,false,Write requests throttled due to on-demand maximum throughput settings.,count,summary,aws_dynamodb_write_max_on_demand_throughput_throttle_events,query-value|timeseries|top-list,sum,none,context.tablename:20|context.globalsecondaryindexname:20,context.env|context.team,context.cloud.region,Metric not present in the live Tsuga export for this workspace,Relevant only when on-demand max throughput is configured,cloudwatch=WriteMaxOnDemandThroughputThrottleEvents,https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/metrics-dimensions.html -Performance/Latency,SuccessfulRequestLatency,false,Latency of successful DynamoDB requests; best user-facing service metric when segmented by operation.,ms,summary,aws_dynamodb_successful_request_latency,query-value|timeseries|top-list,max,none,context.operation:12|context.tablename:20,context.env|context.team,context.globalsecondaryindexname|context.cloud.region,Metric not present in the live Tsuga export for this workspace,Do not average unlike operations if operation labels exist,cloudwatch=SuccessfulRequestLatency,https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/metrics-dimensions.html -Performance/Latency,ReturnedBytes,false,Bytes returned by DynamoDB operations; useful to explain query or scan result growth.,By,summary,aws_dynamodb_returned_bytes,query-value|timeseries|top-list,sum,none,context.tablename:20|context.operation:12,context.env|context.team,context.globalsecondaryindexname|context.cloud.region,Metric not present in the live Tsuga export for this workspace,Large result sets can dominate latency interpretation,cloudwatch=ReturnedBytes,https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/metrics-dimensions.html -Performance/Latency,ReturnedItemCount,false,Number of items returned by DynamoDB operations; useful for query-shape efficiency.,count,summary,aws_dynamodb_returned_item_count,query-value|timeseries|top-list,sum,none,context.tablename:20|context.operation:12,context.env|context.team,context.globalsecondaryindexname|context.cloud.region,Metric not present in the live Tsuga export for this workspace,Use with ReturnedBytes rather than alone,cloudwatch=ReturnedItemCount,https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/metrics-dimensions.html -Errors/Failures,ConditionalCheckFailedRequests,false,Conditional writes rejected because their condition evaluated false; often expected contention signal.,count,summary,aws_dynamodb_conditional_check_failed_requests,query-value|timeseries|top-list,sum,none,context.tablename:20,context.env|context.team,context.cloud.region|context.operation,Metric not present in the live Tsuga export for this workspace,Do not equate directly to service failure,cloudwatch=ConditionalCheckFailedRequests,https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/metrics-dimensions.html -Errors/Failures,TransactionConflict,false,Rejected item-level requests due to transactional conflicts between concurrent requests on the same items.,count,summary,aws_dynamodb_transaction_conflict,query-value|timeseries|top-list,sum,none,context.tablename:20,context.env|context.team,context.cloud.region,Metric not present in the live Tsuga export for this workspace,High signal only for transactional workloads,cloudwatch=TransactionConflict,https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/metrics-dimensions.html -Errors/Failures,UserErrors,false,Aggregate HTTP 400-class request errors for DynamoDB or DynamoDB Streams in the Region and account.,count,summary,aws_dynamodb_user_errors,query-value|timeseries|top-list,sum,none,context.cloud.region:10,context.env|context.team,context.cloud.account.id|context.operation,Metric not present in the live Tsuga export for this workspace,Broad bucket; pair with specific metrics before escalation,cloudwatch=UserErrors,https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/metrics-dimensions.html -Errors/Failures,SystemErrors,false,Aggregate HTTP 500-class service-side request errors for DynamoDB in the Region and account.,count,summary,aws_dynamodb_system_errors,query-value|timeseries|top-list,sum,none,context.cloud.region:10,context.env|context.team,context.cloud.account.id|context.operation,Metric not present in the live Tsuga export for this workspace,Usually sparse and high-signal,cloudwatch=SystemErrors,https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/metrics-dimensions.html -Replication/Change Data Capture,ReplicationLatency,false,Cross-Region replication latency for global tables; direct freshness signal.,ms,summary,aws_dynamodb_replication_latency,query-value|timeseries|top-list,max,none,context.tablename:20|context.receivingregion:10,context.env|context.team,context.cloud.region|context.cloud.account.id,Metric not present in the live Tsuga export for this workspace,Use receiving Region only in replication section,cloudwatch=ReplicationLatency,https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/metrics-dimensions.html -Replication/Change Data Capture,PendingReplicationCount,false,Number of updates not yet written to another replica in legacy global tables.,count,summary,aws_dynamodb_pending_replication_count,query-value|timeseries|top-list,sum,none,context.tablename:20|context.receivingregion:10,context.env|context.team,context.cloud.region|context.cloud.account.id,Metric not present in the live Tsuga export for this workspace,Legacy-surface metric; keep gated until discovered,cloudwatch=PendingReplicationCount,https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/metrics-dimensions.html -Replication/Change Data Capture,AgeOfOldestUnreplicatedRecord,false,Elapsed time since the oldest not-yet-replicated record first appeared for Kinesis CDC.,ms,summary,aws_dynamodb_age_of_oldest_unreplicated_record,query-value|timeseries|top-list,max,none,context.tablename:20|context.delegatedoperation:10,context.env|context.team,context.cloud.region|context.cloud.account.id,Metric not present in the live Tsuga export for this workspace,Use in CDC section only,cloudwatch=AgeOfOldestUnreplicatedRecord,https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/metrics-dimensions.html -Replication/Change Data Capture,FailedToReplicateRecordCount,false,Records DynamoDB failed to replicate to the Kinesis data stream.,count,summary,aws_dynamodb_failed_to_replicate_record_count,query-value|timeseries|top-list,sum,none,context.tablename:20|context.delegatedoperation:10,context.env|context.team,context.cloud.region|context.cloud.account.id,Metric not present in the live Tsuga export for this workspace,High-value if present and non-zero,cloudwatch=FailedToReplicateRecordCount,https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/metrics-dimensions.html -Replication/Change Data Capture,ConsumedChangeDataCaptureUnits,false,Consumed change data capture units for DynamoDB CDC workflows.,count,summary,aws_dynamodb_consumed_change_data_capture_units,query-value|timeseries|top-list,sum,none,context.tablename:20|context.delegatedoperation:10,context.env|context.team,context.cloud.region|context.cloud.account.id,Metric not present in the live Tsuga export for this workspace,Interpret with workload changes rather than as a standalone health alarm,cloudwatch=ConsumedChangeDataCaptureUnits,https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/metrics-dimensions.html -Replication/Change Data Capture,ThrottledPutRecordCount,false,Records throttled while DynamoDB attempts to put CDC records into the Kinesis data stream.,count,summary,aws_dynamodb_throttled_put_record_count,query-value|timeseries|top-list,sum,none,context.tablename:20|context.delegatedoperation:10,context.env|context.team,context.cloud.region|context.cloud.account.id,Metric not present in the live Tsuga export for this workspace,Only relevant for Kinesis CDC,cloudwatch=ThrottledPutRecordCount,https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/metrics-dimensions.html -Index Operations,OnlineIndexPercentageProgress,false,Relative progress of a new GSI build while the index is being added to a table.,%,summary,aws_dynamodb_online_index_percentage_progress,query-value|timeseries|top-list,max,none,context.tablename:20|context.globalsecondaryindexname:20,context.env|context.team,context.cloud.region|context.cloud.account.id,Metric not present in the live Tsuga export for this workspace,Blank usually means no active build,cloudwatch=OnlineIndexPercentageProgress,https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/metrics-dimensions.html -Index Operations,OnlineIndexConsumedWriteCapacity,false,Metric historically associated with GSI build write-capacity consumption; expected to show zero in current docs during index builds.,count,summary,aws_dynamodb_online_index_consumed_write_capacity,query-value|timeseries|top-list,sum,none,context.tablename:20|context.globalsecondaryindexname:20,context.env|context.team,context.cloud.region|context.cloud.account.id,Metric not present in the live Tsuga export for this workspace,Low confidence for operational value; interpret with caution,cloudwatch=OnlineIndexConsumedWriteCapacity,https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/metrics-dimensions.html -Index Operations,OnlineIndexThrottleEvents,false,Metric historically associated with throttles during a GSI build; expected to show zero in current docs during index builds.,count,summary,aws_dynamodb_online_index_throttle_events,query-value|timeseries|top-list,sum,none,context.tablename:20|context.globalsecondaryindexname:20,context.env|context.team,context.cloud.region|context.cloud.account.id,Metric not present in the live Tsuga export for this workspace,Low confidence for operational value; interpret with caution,cloudwatch=OnlineIndexThrottleEvents,https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/metrics-dimensions.html -Throughput/Usage,TimeToLiveDeletedItemCount,false,Number of items deleted by TTL; useful to distinguish expected automated deletion churn from user write load.,count,summary,aws_dynamodb_time_to_live_deleted_item_count,query-value|timeseries|top-list,sum,none,context.tablename:20,context.env|context.team,context.cloud.region|context.cloud.account.id,Metric not present in the live Tsuga export for this workspace,Low-signal outside TTL-heavy workloads,cloudwatch=TimeToLiveDeletedItemCount,https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/metrics-dimensions.html diff --git a/plugins/tsuga/skills/knowledge-technology/references/aws-dynamodb/queries.md b/plugins/tsuga/skills/knowledge-technology/references/aws-dynamodb/queries.md deleted file mode 100644 index 1ee43ab..0000000 --- a/plugins/tsuga/skills/knowledge-technology/references/aws-dynamodb/queries.md +++ /dev/null @@ -1,57 +0,0 @@ -# AWS DynamoDB - -Managed NoSQL KV + document store. Provisioned or On-Demand capacity. Healthy: no throttling, bounded read/write latency, no hot-partition skew, GSI in sync. - -## Incident shapes - -- **Throttling** — `aws_dynamodb_throttled_requests` spikes → capacity-mode limit hit -- **Hot partition / key range** — `aws_dynamodb_read_key_range_throughput_throttle_events` ≠ aggregate throttles → skew onto one partition -- **On-demand scaling lag** — sudden 5-10x burst exceeds adaptive rate; brief throttles during scale -- **Provisioned under-sizing** — `aws_dynamodb_account_provisioned_read_capacity_utilization → 1.0` -- **System errors** — `aws_dynamodb_system_errors > 0` → DynamoDB-side failure (rare) -- **GSI divergence** — GSI throttled while base table fine → GSI reads stale - -## Key metrics - -| Metric | Unit | Signal | -|---|---|---| -| `aws_dynamodb_consumed_read_capacity_units` / `aws_dynamodb_consumed_write_capacity_units` | RCU / WCU | Capacity used | -| `aws_dynamodb_provisioned_read_capacity_units` / `WriteCapacityUnits` | RCU / WCU | Ceiling (provisioned) | -| `aws_dynamodb_throttled_requests` | count | Total throttled | -| `aws_dynamodb_read_throttle_events` / `aws_dynamodb_write_throttle_events` | count | Per-op throttling | -| `aws_dynamodb_read_key_range_throughput_throttle_events` | count | Hot-partition signal | -| `aws_dynamodb_write_key_range_throughput_throttle_events` | count | Hot-partition on write | -| `aws_dynamodb_read_provisioned_throughput_throttle_events` | count | Provisioned-mode throttling | -| `aws_dynamodb_read_max_on_demand_throughput_throttle_events` | count | On-demand ceiling throttling | -| `aws_dynamodb_successful_request_latency` (operation) | ms | Per-op latency | -| `aws_dynamodb_system_errors` | count | Provider 5xx | -| `aws_dynamodb_user_errors` | count | 4xx (validation, condition-check-failed) | -| `aws_dynamodb_account_provisioned_read_capacity_utilization` | % | Account utilization | -| `aws_dynamodb_max_provisioned_table_read_capacity_utilization` | % | Hottest table | - -## Derived signals - -- `aws_dynamodb_throttled_requests / (aws_dynamodb_throttled_requests + Consumed)` — throttle rate. Any sustained = capacity issue. -- `aws_dynamodb_read_key_range_throughput_throttle_events > 0` while account util < 0.8 = hot partition (skew, not saturation). -- p99 / p50 Latency divergence per operation under stable rate = hot keys or GSI pressure. -- GSI `ConsumedWriteCapacity` / base-table write capacity — fan-out projection ratio. - -## Log patterns - -DynamoDB has no direct logs; SDK-side exceptions: - -- `ProvisionedThroughputExceededException` — throttling -- `ThrottlingException` — rate-limit -- `ConditionalCheckFailedException` — optimistic-concurrency conflict (often intentional) -- `ResourceNotFoundException` — table/index missing (config bug) -- `ValidationException` — bad query shape -- `InternalServerError` — provider-side failure -- `TransactionCanceledException: Reason: aws_dynamodb_transaction_conflict` — txn contention - -## Gotchas - -- Eventually consistent reads = 0.5 RCU; strongly consistent = 1 RCU. `Consumed` aggregates both. -- `aws_dynamodb_throttled_requests = 0` can hide hot-partition throttling on individual keys. Always check `*KeyRangeThroughputThrottleEvents`. -- GSI throttling silently stales reads from GSI while base table is healthy. Your app returns old data with no error. -- Auto-scaling alarms are separate from throttling alarms; auto-scaling lags by minutes. During a burst, throttling is the fast signal. -- DynamoDB Streams / DAX throttling aren't in this metric set; check their own namespaces. diff --git a/plugins/tsuga/skills/knowledge-technology/references/aws-ecs/README.md b/plugins/tsuga/skills/knowledge-technology/references/aws-ecs/README.md deleted file mode 100644 index cde6a79..0000000 --- a/plugins/tsuga/skills/knowledge-technology/references/aws-ecs/README.md +++ /dev/null @@ -1,657 +0,0 @@ -# AWS ECS Integration Context Bundle - -**Technology:** AWS ECS (Elastic Container Service) -**Deployment:** Managed (AWS) -**Environment:** prod -**Persona:** SRE Dev and ops -**Telemetry preference:** mixed -**Integration scope:** core service only -**Primary use-case:** reliability and performance - ---- - -## How to use this bundle - -- **Dashboard plan:** `07_aws-ecs_dashboard_plan.yaml` — section structure, widget specs, gating rules, coverage map -- **Metric truth:** `05_aws-ecs_metric_inventory.csv` — source of truth for all metrics, units, temporality, Tsuga mapping, aggregations -- **Notes & playbooks:** `09_aws-ecs_section_notes_and_playbooks.md` — all note prose, triage chains, operational playbooks -- **Reconciliation:** `12_aws-ecs_discovery_reconciliation.md` — Stage 2 discovery results, confirmed/missing/unexpected metrics - ---- - -## Confirmed Tsuga prefixes - -- `aws_ecs_*` — **CONFIRMED** (11 metrics found; maps CloudWatch AWS/ECS namespace: cpu_utilization, memory_utilization, request_count, target_response_time, Service Connect metrics) -- `ecs_containerinsights_*` — **CONFIRMED** (25 metrics found; maps CloudWatch ECS/ContainerInsights namespace: running_task_count, cpu_utilized, memory_utilized, network_rx_bytes, plus 6 Enhanced Observability metrics) - -> **Naming convention:** Tsuga uses underscore-separated snake_case for all ECS metrics. No dots or PascalCase. -> **Context fields:** All lowercase joined — `context.servicename`, `context.clustername`, `context.taskdefinitionfamily`, `context.discoveryname`, `context.targetdiscoveryname`. -> **Counter temporality:** All metrics are `summary` type with `cumulative` temporality. Counters use `sum` + `rate`; gauges use `avg` + `none`. - ---- - -## Discovery status - -Discovery: **COMPLETE** (Stage 2 performed 2026-02-10) - -- **36 metrics confirmed** in Tsuga (11 aws_ecs + 25 ecs_containerinsights) -- **11 metrics not found** — EC2-only (CPUReservation, MemoryReservation, GPUReservation), EBS (EBSFilesystemUtilization, EBSFilesystemSize, EBSFilesystemUtilized), HTTP error codes (3XX, 4XX, 5XX), TLS errors (Client, Target). All gated in dashboard plan. -- **6 unexpected Enhanced Observability metrics** added — ContainerCpuUtilization, ContainerMemoryUtilization, UnHealthyContainerHealthStatus, TaskCpuUtilization, TaskMemoryUtilization, TaskEphemeralStorageUtilization -- **Account profile:** Fargate-only (no EC2 reservation metrics), Service Connect enabled, Enhanced Observability enabled - ---- - -## Bundle files - -| # | Filename | Purpose | -|---|----------|---------| -| 00 | `00_aws-ecs_cover.md` | This file — metadata, prefixes, sources, navigation | -| 01 | `01_aws-ecs_executive_overview.md` | What "good" looks like, incident shapes, dashboard intent | -| 02 | `02_aws-ecs_key_concepts.md` | Glossary (24 terms), concept map (28 lines), entities (16), context.* mapping | -| 03 | `03_aws-ecs_golden_signals.md` | Traffic/Errors/Latency/Saturation mapping to section questions | -| 04 | `04_aws-ecs_telemetry_sources.md` | Source matrix, optional features, "no data" meanings | -| 05 | `05_aws-ecs_metric_inventory.csv` | 47 metrics: names, units, types, Tsuga mapping, aggregations, group-bys | -| 06 | `06_aws-ecs_derived_signals.csv` | 14 derived signals: formulas, inputs, gating, interpretation | -| 07 | `07_aws-ecs_dashboard_plan.yaml` | 6 sections, ~60 widgets, coverage map, gating rules | -| 09 | `09_aws-ecs_section_notes_and_playbooks.md` | Mission note, 6 section notes, 22 triage chains, 8 playbooks | -| 10 | `10_aws-ecs_caveats_footguns.md` | 23 caveats tagged to sections | -| 11 | `11_aws-ecs_unknowns_verify_next.yaml` | Remaining unknowns after Stage 2 discovery | -| 12 | `12_aws-ecs_discovery_reconciliation.md` | Stage 2 discovery results, reconciliation details | - ---- - -## Top sources - -1. **[AWS ECS CloudWatch Metrics](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/available-metrics.html)** — Definitive reference for AWS/ECS namespace metrics (CPUUtilization, MemoryUtilization, Service Connect metrics), dimensions, and statistics. -2. **[AWS Container Insights Metrics for ECS](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-metrics-ECS.html)** — Complete list of ECS/ContainerInsights namespace metrics, dimensions, and enablement requirements. -3. **[Container Insights Enhanced Observability](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-enhanced-observability-metrics-ECS.html)** — Container-level and task-level metrics available with enhanced observability tier. -4. **[AWS ECS Developer Guide](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/Welcome.html)** — Architecture, concepts, task definitions, services, deployments, capacity providers. -5. **[How Amazon ECS Manages CPU and Memory](https://aws.amazon.com/blogs/containers/how-amazon-ecs-manages-cpu-and-memory-resources/)** — Deep dive on CPU soft/hard limits, burst behavior, and memory semantics across EC2 and Fargate. -6. **[AWS ECS Stopped Task Error Codes](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/stopped-task-error-codes.html)** — Complete list of stop codes (OOM, TaskFailedToStart, etc.) and their meanings. -7. **[Setting up Container Insights on ECS](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/deploy-container-insights-ECS-cluster.html)** — How to enable Container Insights standard and enhanced tiers. -8. **[AWS Fargate for ECS](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/AWS_Fargate.html)** — Fargate-specific constraints: CPU/memory combinations, networking, ephemeral storage, platform versions. -9. **[ECS Task State Change Events](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs_task_events.html)** — EventBridge integration for task lifecycle, stop codes, and failure diagnostics. -10. **[AWS ECS Prescriptive Guidance](https://docs.aws.amazon.com/prescriptive-guidance/latest/implementing-logging-monitoring-cloudwatch/ecs-metrics.html)** — AWS best practices for ECS monitoring and metrics selection. - - ---- - -# AWS ECS — Executive Overview - -## What it is -AWS Elastic Container Service (ECS) is a fully managed container orchestration service that runs Docker containers on AWS infrastructure. Workloads run on either **Fargate** (serverless, AWS-managed compute) or **EC2** (self-managed instances). ECS manages the lifecycle of services (long-running tasks with desired count), standalone tasks, and scheduled tasks across clusters. - -## What "good" looks like -- **RunningTaskCount == DesiredTaskCount** for every service (no pending or failed tasks). -- **CPU and memory utilization** between 30-70% at service level — enough headroom for spikes, not wasting capacity. -- **PendingTaskCount near zero** — tasks launch quickly and transition to RUNNING. -- **DeploymentCount == 1** per service — no stuck or rolling-back deployments. -- **RestartCount stable** — not climbing, indicating container stability. -- Zero sustained 5xx errors from Service Connect (if enabled). - -## Paging intent -- Service unable to maintain desired task count (tasks failing to start or being killed). -- CPU or memory utilization sustained above 85% with no auto-scaling headroom. -- Deployment stuck: DeploymentCount > 1 for extended period with RunningTaskCount < DesiredTaskCount. - -## Top 3 incident shapes - -| Incident | First dashboard section | -|----------|----------------------| -| **Tasks failing to start / OOM kills** — RunningTaskCount drops below DesiredTaskCount, PendingTaskCount spikes, RestartCount climbs. Common causes: OOM, image pull failures, resource exhaustion. | Task Lifecycle & Deployments | -| **CPU/memory saturation** — CPUUtilization or MemoryUtilization sustained near 100%. Containers throttled, latency increases, potential OOM kills. | Resource Utilization | -| **Deployment rollback / stuck deployment** — DeploymentCount > 1, new tasks not reaching RUNNING state. Circuit breaker may trigger automatic rollback. | Task Lifecycle & Deployments | - ---- - -### Confirmed by sources -- Task lifecycle states, stop codes, and deployment circuit breaker behavior: [AWS ECS Developer Guide](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-event-messages.html) -- CPU/memory utilization metric definitions and Fargate vs EC2 differences: [AWS ECS CloudWatch Metrics](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/available-metrics.html) -- Container Insights task count and health metrics: [AWS Container Insights Metrics](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-metrics-ECS.html) - -### Best-practice inference -- The 30-70% utilization target range is industry-standard SRE guidance, not AWS-specific documentation. -- "DeploymentCount == 1 means healthy" is inferred from how ECS manages rolling deployments (old deployment persists until new one stabilizes). - - ---- - -# AWS ECS — Key Concepts - -## Glossary (24 terms) - -| Term | Definition | Operational Meaning | Dashboard Section | -|------|-----------|---------------------|-------------------| -| **Cluster** | Logical grouping of tasks and services. Provides isolation and capacity infrastructure. | Top-level scope for all dashboards. Global filter target. | All sections | -| **Service** | Maintains a desired count of task instances with deployment management and LB integration. | Primary operational unit. RunningTaskCount vs DesiredTaskCount is the health signal. | Task Lifecycle & Deployments | -| **Task** | A running instantiation of a task definition. Contains one or more containers. | The unit of work. Lifecycle state (PENDING → RUNNING → STOPPED) drives health metrics. | Task Lifecycle & Deployments | -| **Task Definition** | Blueprint (JSON) specifying container images, CPU/memory limits, networking, IAM roles. Versioned as family:revision. | Family name is the primary group-by dimension for per-workload breakdowns. | Resource Utilization | -| **Container** | Individual Docker container within a task. Essential containers stop the whole task on exit. | Container-level metrics require Enhanced Container Insights. | Resource Utilization | -| **Fargate** | Serverless launch type. AWS manages compute. Fixed CPU/memory combinations. No burst. | No cluster-level reservation metrics. Ephemeral storage metrics available. | Resource Utilization, Storage | -| **EC2 Launch Type** | Self-managed EC2 instances registered with the cluster. Allows GPU, flexible CPU/memory. | Adds CPUReservation/MemoryReservation and instance-level metrics. | Resource Utilization, Capacity | -| **Capacity Provider** | Abstraction for compute supply: FARGATE, FARGATE_SPOT, or Auto Scaling Group. | Determines available compute and cost model. Capacity provider name is a useful group-by. | Capacity & Scaling | -| **Desired Count** | Target number of running tasks for a service. Set manually or by auto-scaling. | DesiredTaskCount is the baseline. RunningTaskCount < DesiredTaskCount means trouble. | Task Lifecycle & Deployments | -| **Deployment** | A change to a service (new revision, count change). Rolling update with min/max healthy percent. | DeploymentCount > 1 = active deployment in progress. Sustained > 1 = stuck. | Task Lifecycle & Deployments | -| **Circuit Breaker** | Auto-rollback mechanism when tasks repeatedly fail to reach RUNNING state. | Prevents bad deployments from fully rolling out. Creates rollback deployment events. | Task Lifecycle & Deployments | -| **Service Connect** | Service mesh proxy (Envoy-based) for ECS service-to-service communication. | Provides traffic metrics (RequestCount, latency, HTTP status codes). Feature-gated. | Service Connect Traffic | -| **Container Insights** | CloudWatch feature providing cluster/service/task-level telemetry for ECS. Must be explicitly enabled. | Required for most Container Insights metrics. Standard vs Enhanced tiers. | All Container Insights sections | -| **Enhanced Observability** | Container Insights tier adding container-level and task-level (TaskId) breakdown metrics. | Adds ContainerCpuUtilization, ContainerMemoryUtilization, UnHealthyContainerHealthStatus. | Resource Utilization (container-level) | -| **CPUUtilization** | Percentage of CPU units in use. Service-level (Fargate+EC2) or cluster-level (EC2 only). | Primary saturation signal. 100% = throttling on Fargate. | Resource Utilization | -| **MemoryUtilization** | Percentage of memory in use. Service-level (Fargate+EC2) or cluster-level (EC2 only). | Approaching 100% = OOM kill risk. | Resource Utilization | -| **CPUReservation** | Percentage of registered CPU units reserved by running tasks (EC2 only). | High reservation = cluster needs more instances. | Capacity & Scaling | -| **MemoryReservation** | Percentage of registered memory reserved by running tasks (EC2 only). | Often the binding constraint for task placement on EC2. | Capacity & Scaling | -| **RunningTaskCount** | Tasks in RUNNING state for a service. | Must equal DesiredTaskCount. Gap = problem. | Task Lifecycle & Deployments | -| **PendingTaskCount** | Tasks in PENDING state for a service (waiting for resources or scheduling). | Should be near zero. Sustained pending = placement failure or resource shortage. | Task Lifecycle & Deployments | -| **RestartCount** | Cumulative container restart count. Monotonically increasing counter. | Rising slope = containers crashing and restarting. | Task Lifecycle & Deployments | -| **OOM Kill** | Out Of Memory kill. Container exceeds memory hard limit. Stop code: OutOfMemoryError. | Memory utilization near 100% preceding a task stop. | Resource Utilization | -| **Stop Code** | Reason a task was stopped (OutOfMemoryError, TaskFailedToStart, SpotInterruptionError, etc.). | Key diagnostic dimension for failed task analysis. Available via EventBridge, not as a metric dimension. | Task Lifecycle & Deployments | -| **awsvpc Network Mode** | Each task gets its own ENI and private IP. Required for Fargate. Optional for EC2. | Enables network metrics (NetworkRxBytes, NetworkTxBytes). Other modes have limited visibility. | Network I/O | - ---- - -## Concept Map (28 lines) - -``` -Cluster -> contains -> Service (why: services are the operational unit within a cluster) -Cluster -> contains -> Standalone Task (why: batch jobs or one-off tasks run without a service) -Cluster -> has -> Capacity Provider (why: determines compute supply — Fargate, EC2, Spot) -Service -> maintains -> Desired Task Count (why: defines the expected steady state) -Service -> creates -> Deployment (why: every change triggers a new deployment) -Service -> registers with -> Load Balancer (why: distributes traffic to task targets) -Deployment -> launches -> Task (why: tasks are the runtime instantiation of a deployment) -Deployment -> monitored by -> Circuit Breaker (why: auto-rollback on repeated failures) -Task -> defined by -> Task Definition (why: blueprint for containers, resources, IAM) -Task -> contains -> Container (why: one or more containers share resources and network) -Task -> has -> Launch Type: Fargate or EC2 (why: determines compute model and metric availability) -Task -> has -> Lifecycle State (why: PENDING → RUNNING → STOPPED drives health signals) -Task -> has -> Stop Code (why: explains why a task stopped — OOM, Spot interruption, etc.) -Container -> has -> CPU Limit (why: hard limit = throttling on Fargate, soft limit = burstable on EC2) -Container -> has -> Memory Limit (why: exceeding hard limit = OOM kill) -Container -> may have -> Health Check (why: UnHealthyContainerHealthStatus metric) -CPUUtilization -> approximates -> CPU Saturation (why: 100% = throttled on Fargate, may burst on EC2) -MemoryUtilization -> predicts -> OOM Risk (why: approaching 100% precedes OOM kills) -CPUReservation -> measures -> Cluster Capacity Used (why: EC2 only; high = need more instances) -MemoryReservation -> measures -> Cluster Capacity Used (why: often binding constraint for EC2 placement) -RunningTaskCount -> compared to -> DesiredTaskCount (why: gap = unhealthy service) -PendingTaskCount -> indicates -> Placement Failures (why: sustained pending = resource shortage) -RestartCount -> indicates -> Container Instability (why: monotonically increasing = crash loops) -NetworkRxBytes -> measures -> Inbound Traffic (why: requires awsvpc or bridge network mode) -NetworkTxBytes -> measures -> Outbound Traffic (why: requires awsvpc or bridge network mode) -Service Connect -> provides -> Request Metrics (why: L7 traffic visibility — counts, latency, HTTP codes) -Container Insights -> enables -> Task/Service Metrics (why: must be explicitly enabled on the cluster) -Enhanced Observability -> enables -> Container-level Metrics (why: adds per-container and per-TaskId breakdown) -``` - ---- - -## Entities and Dimensions (16) - -| Dimension | Vendor Name | Tsuga `context.*` Mapping | Cardinality | Safe Top-N | Notes | -|-----------|-------------|--------------------------|-------------|-----------|-------| -| Cluster Name | `ClusterName` | `context.cluster_name` | Low (1-20) | 10 | **Must-exist.** Top-level scope for all dashboards. | -| Service Name | `ServiceName` | `context.service_name` | Low-Medium (5-100) | 25 | **Must-exist.** Primary operational unit. | -| Task Definition Family | `TaskDefinitionFamily` | `context.task_definition_family` | Low-Medium (5-100) | 25 | Must-exist. Group workloads without service coupling. | -| Container Name | `ContainerName` | `context.container_name` | Low (1-10 per task) | 10 | Optional. Enhanced Observability only. | -| Task ID | `TaskId` | `context.task_id` | **HIGH** (hundreds-thousands) | 10 | **Do NOT group-by** in dashboards. Use only for drill-down filters. Enhanced Observability only. | -| Instance ID | `InstanceId` / `EC2InstanceId` | `context.instance_id` | Medium (10-500) | 25 | EC2 only. Instance-level metrics. | -| Container Instance ID | `ContainerInstanceId` | `context.container_instance_id` | Medium (10-500) | 25 | EC2 only. ECS agent on the instance. | -| Capacity Provider | `CapacityProviderName` | `context.capacity_provider_name` | Very Low (1-5) | 5 | Useful for Fargate vs Spot vs EC2 breakdown. | -| Discovery Name | `DiscoveryName` | `context.discovery_name` | Low (5-50) | 15 | Service Connect only. Service mesh endpoint name. | -| Target Discovery Name | `TargetDiscoveryName` | `context.target_discovery_name` | Low (5-50) | 15 | Service Connect only. Target endpoint name. | -| Volume Name | `VolumeName` | `context.volume_name` | Low (1-10 per task) | 10 | EBS volume metrics only. | -| Region | (AWS tag) | `context.cloud.region` | Very Low (1-10) | 10 | Standard cloud dimension. | -| Account ID | (AWS tag) | `context.cloud.account.id` | Very Low (1-10) | 10 | Multi-account deployments. | -| Environment | (custom tag) | `context.env` | Very Low (2-5) | 5 | **Must-exist.** Global filter. | -| Team | (custom tag) | `context.team` | Low (5-20) | 10 | **Must-exist.** Global filter. | -| Launch Type | `LaunchType` | `context.launch_type` | Very Low (2-3) | 3 | Unknown if available as a tag in Tsuga. Useful for Fargate vs EC2 breakdowns. | - -### Do NOT group-by -- **`context.task_id`** — Very high cardinality (hundreds to thousands of unique values). Creates unreadable charts and high query cost. Use only as a filter for targeted investigation. -- **`context.container_instance_id`** — Prefer `context.instance_id` for EC2 instance breakdown; container instance ID is an ECS-internal reference less useful for operators. - ---- - -## Tsuga Field Mapping Table - -| Vendor/CloudWatch Dimension | Recommended `context.*` Key | Status | Must-exist? | -|-----------------------------|----------------------------|--------|-------------| -| `ClusterName` | `context.cluster_name` | **Unknown** — verify in Stage 2 | Yes | -| `ServiceName` | `context.service_name` | **Unknown** — verify in Stage 2 | Yes | -| `TaskDefinitionFamily` | `context.task_definition_family` | **Unknown** — verify in Stage 2 | Yes | -| `ContainerName` | `context.container_name` | **Unknown** — verify in Stage 2 (Enhanced only) | No | -| `TaskId` | `context.task_id` | **Unknown** — verify in Stage 2 (Enhanced only) | No | -| `InstanceId` / `EC2InstanceId` | `context.instance_id` | **Unknown** — verify in Stage 2 (EC2 only) | No | -| `ContainerInstanceId` | `context.container_instance_id` | **Unknown** — verify in Stage 2 (EC2 only) | No | -| `CapacityProviderName` | `context.capacity_provider_name` | **Unknown** — verify in Stage 2 | No | -| `DiscoveryName` | `context.discovery_name` | **Unknown** — verify in Stage 2 (Service Connect only) | No | -| `TargetDiscoveryName` | `context.target_discovery_name` | **Unknown** — verify in Stage 2 (Service Connect only) | No | -| `VolumeName` | `context.volume_name` | **Unknown** — verify in Stage 2 | No | -| (AWS tag) | `context.cloud.region` | **Inferred** — standard AWS tag | No | -| (AWS tag) | `context.cloud.account.id` | **Inferred** — standard AWS tag | No | -| (custom) | `context.env` | **Inferred** — from .env config | Yes | -| (custom) | `context.team` | **Inferred** — from .env config | Yes | -| `LaunchType` | `context.launch_type` | **Unknown** — may or may not be a tag | No | - -> All `context.*` field names are Unknown until Stage 2 discovery confirms them. The names above follow Tsuga naming conventions observed in other AWS integrations. - ---- - -### Confirmed by sources -- ECS architecture (clusters, services, tasks, task definitions, capacity providers): [AWS ECS Developer Guide](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/Welcome.html) -- CloudWatch metric dimensions: [AWS ECS Available Metrics](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/available-metrics.html) -- Container Insights dimensions and enhanced observability: [AWS Container Insights Metrics](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-metrics-ECS.html) -- Fargate vs EC2 capability differences: [AWS Fargate Documentation](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/AWS_Fargate.html) - -### Best-practice inference -- `context.*` field name mappings are inferred from Tsuga naming conventions. Actual field names may differ (e.g., `context.ecs.cluster_name` vs `context.cluster_name`). Stage 2 discovery will resolve. -- Top-N suggestions (25 for services, 10 for clusters) are based on typical deployment sizes. -- The "Do NOT group-by" guidance for task_id is based on cardinality risk analysis, not Tsuga-specific documentation. - - ---- - -# AWS ECS — Golden Signals - -## Traffic - -**What it means for ECS:** -Traffic measures the volume of work flowing through ECS — task execution, network I/O, and (with Service Connect) HTTP/gRPC request throughput. Unlike request-serving systems, ECS itself is an orchestration layer, so "traffic" is primarily measured at the network and service-mesh level, plus the number of tasks being scheduled and executed. - -**Typical causes when it degrades:** -- Upstream callers reducing load (planned or unplanned) -- Auto-scaling reducing desired count during low-traffic periods -- Network connectivity issues between tasks and load balancers -- Service Connect proxy failures silently dropping traffic - -**Best telemetry sources:** -- `NetworkRxBytes` / `NetworkTxBytes` (Container Insights) — network throughput per task/service -- `RequestCount` / `ProcessedBytes` (Service Connect) — L7 request volume -- `RunningTaskCount` — proxy for compute capacity handling traffic - -**What people page on:** -- Network throughput drops to zero or near-zero on a service that normally has steady traffic -- Service Connect RequestCount drops significantly without a corresponding desired count change -- RunningTaskCount drops below DesiredTaskCount (capacity to serve traffic is reduced) - -**Section questions:** -1. Is the cluster handling expected network traffic? (Network I/O) -2. What is the request volume through Service Connect? (Service Connect Traffic) - ---- - -## Errors - -**What it means for ECS:** -Errors in ECS manifest as tasks failing to start, containers being OOM-killed, health check failures, and (with Service Connect) HTTP 4xx/5xx responses. Task-level errors are the most critical — they indicate workloads are not running as expected. - -**Typical causes when it degrades:** -- **OOM kills**: Container exceeds memory hard limit (OutOfMemoryError stop code) -- **Image pull failures**: Bad image tag, ECR authentication expired, registry unreachable (CannotPullContainer) -- **Resource exhaustion**: No capacity for task placement on EC2 clusters -- **Application errors**: 5xx responses through Service Connect indicate application failures -- **TLS errors**: Service Connect TLS negotiation failures (misconfigured certificates) - -**Best telemetry sources:** -- `RestartCount` (Container Insights) — container crash loops -- `UnHealthyContainerHealthStatus` (Enhanced Observability) — health check failures -- `HTTPCode_Target_5XX_Count` / `4XX_Count` (Service Connect) — application errors -- `RunningTaskCount` vs `DesiredTaskCount` gap — task failures -- `ClientTLSNegotiationErrorCount` / `TargetTLSNegotiationErrorCount` (Service Connect) — TLS errors - -**What people page on:** -- RestartCount climbing rapidly across multiple containers (crash loop) -- Sustained gap between RunningTaskCount and DesiredTaskCount (tasks failing to start) -- 5xx error rate exceeding a threshold through Service Connect -- UnHealthyContainerHealthStatus showing unhealthy containers - -**Section questions:** -1. Are tasks running successfully or failing? (Task Lifecycle & Deployments) -2. Are Service Connect endpoints returning errors? (Service Connect Traffic) - ---- - -## Latency - -**What it means for ECS:** -Latency in ECS is primarily visible through Service Connect's `TargetResponseTime` metric, which measures proxy-to-target response time. Without Service Connect, ECS itself doesn't expose latency metrics — applications must instrument their own. Task startup time (PENDING → RUNNING duration) is an infrastructure latency signal. - -**Typical causes when it degrades:** -- CPU throttling causing application slowdown (CPUUtilization at 100% on Fargate) -- Memory pressure causing garbage collection pauses -- Network congestion between services -- Slow image pulls during task startup (cold start latency) -- ENI provisioning delays for Fargate tasks in awsvpc mode - -**Best telemetry sources:** -- `TargetResponseTime` (Service Connect) — proxy-to-target latency -- `CPUUtilization` — indirect: high CPU correlates with increased latency -- `PendingTaskCount` duration — how long tasks stay in PENDING (no direct metric; inferred from count trends) - -**What people page on:** -- TargetResponseTime p95/p99 spiking above SLO thresholds -- CPUUtilization sustained at 100% with latency complaints from consumers -- PendingTaskCount elevated for extended period (task startup delay) - -**Section questions:** -1. How fast are Service Connect endpoints responding? (Service Connect Traffic) - ---- - -## Saturation - -**What it means for ECS:** -Saturation measures how close ECS resources are to their limits. This includes CPU and memory utilization at the task/service level, cluster-level reservation (EC2 only), storage consumption, and the gap between running and desired task counts (which indicates scheduling pressure). - -**Typical causes when it degrades:** -- Workload growth exceeding allocated CPU/memory per task -- Auto-scaling not keeping up with demand -- EC2 cluster running out of registered instance capacity (high reservation) -- Ephemeral storage filling up (Fargate) -- EBS volume reaching filesystem capacity limits -- Too many tasks competing for limited EC2 instance resources - -**Best telemetry sources:** -- `CPUUtilization` / `MemoryUtilization` (AWS/ECS) — service and cluster level -- `CpuUtilized` vs `CpuReserved`, `MemoryUtilized` vs `MemoryReserved` (Container Insights) — absolute values -- `CPUReservation` / `MemoryReservation` (AWS/ECS, EC2 only) — cluster capacity -- `EphemeralStorageUtilized` vs `EphemeralStorageReserved` (Container Insights, Fargate only) -- `EBSFilesystemUtilization` (AWS/ECS) — EBS storage pressure -- `StorageReadBytes` / `StorageWriteBytes` (Container Insights) — I/O volume - -**What people page on:** -- CPUUtilization or MemoryUtilization sustained above 85% with no auto-scaling response -- CPUReservation or MemoryReservation above 80% on EC2 clusters (can't place new tasks) -- EBSFilesystemUtilization approaching 90%+ (disk full risk) -- MemoryUtilization approaching 100% (OOM kill imminent) - -**Section questions:** -1. How utilized are CPU and memory across services? (Resource Utilization) -2. Is the cluster running out of capacity? (Capacity & Scaling — EC2 only) -3. How is storage being consumed? (Storage & I/O) - ---- - -### Confirmed by sources -- CloudWatch metric definitions and dimensions: [AWS ECS CloudWatch Metrics](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/available-metrics.html) -- Container Insights metric definitions: [AWS Container Insights Metrics](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-metrics-ECS.html) -- Service Connect metrics: [AWS ECS Service Connect Metrics](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/available-metrics.html) -- Stopped task error codes and lifecycle: [AWS ECS Stopped Tasks](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/stopped-task-error-codes.html) -- Fargate CPU/memory limits: [AWS Fargate Documentation](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/AWS_Fargate.html) - -### Best-practice inference -- Task startup latency (PENDING → RUNNING) is a known operational concern but has no direct CloudWatch metric — inferred from PendingTaskCount trends. -- The 85% utilization paging threshold is industry SRE convention, not AWS-specific. -- Latency correlation with CPUUtilization is standard observability practice, not documented by AWS for ECS specifically. - - ---- - -# AWS ECS — Section Notes & Playbooks - ---- - -## Part 1: Overview Mission Note - -**AWS Elastic Container Service (ECS)** — managed container orchestration running workloads on Fargate (serverless) or EC2 instances. - -**Scope:** Covers all ECS clusters, services, and tasks reporting via CloudWatch and Container Insights. Service Connect traffic metrics require Service Connect to be configured. EC2 cluster capacity metrics require EC2 launch type. - -**Links:** -- [AWS ECS Developer Guide](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/Welcome.html) -- [Container Insights Metrics Reference](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-metrics-ECS.html) -- [Deep Dive Dashboard →](#) - ---- - -## Part 2: Section Explanation Notes - -### # Tasks & Deployments - Are services running the expected number of tasks? - -#### So what? -Healthy services maintain **RunningTaskCount == DesiredTaskCount** with **PendingTaskCount near zero**. The key signal is the **Task Deficit** — any sustained gap means tasks are failing to start or being killed. **DeploymentCount > 1** during a rolling update is normal, but if it persists beyond the expected rollout window, the deployment is stuck or rolling back (circuit breaker). **Restart Rate > 0** means containers are crashing — correlate with memory utilization for OOM kills. Watch out: PendingTaskCount > 0 for a few seconds during scale-out is expected, not an incident. - -#### Now what? -Check **Task Deficit** → if positive, check **Pending Tasks by Service** for which services are affected → check **Restart Rate by Service** for crash loops → if restarts are climbing with high memory, likely OOM — increase task memory limits or investigate memory leaks. - ---- - -### # Resource Utilization - How utilized are CPU and memory across services? - -#### So what? -**CPU and Memory Utilization** between 30-70% indicates healthy headroom. **CPUUtilization** at 100% on Fargate means hard throttling (no burst). On EC2 it can exceed 100% if no hard CPU limit is set — this is normal burst behavior, not an error. **Memory approaching 100%** is always concerning: on Fargate it means OOM kill is imminent. Watch out: `CpuUtilized`/`CpuReserved` are in **CPU units** (not percent); `MemoryUtilized`/`MemoryReserved` are in **MiB**. Don't confuse with the percentage-based utilization metrics. - -#### Now what? -Check **Top Services by CPU/Memory** → identify which services are hottest → compare **Utilized vs Reserved** to assess over/under-provisioning → if utilization is high and tasks are restarting, likely need to increase task resource limits or scale out the service. - ---- - -### # Network - Is the cluster handling expected network traffic? - -#### So what? -**NetworkRxBytes** and **NetworkTxBytes** show per-service throughput. A drop to zero on a normally active service signals connectivity failure. A sudden spike may indicate traffic flood or batch processing. These metrics are already rates (Bytes/Second) — do NOT apply per-second post-function. Watch out: Network metrics only work with **awsvpc** and **bridge** network modes. `host` mode reports zero — that's expected, not broken. - -#### Now what? -Check **Network In/Out by Service** → identify which services have unusual patterns → if traffic drops, correlate with **RunningTaskCount** (are tasks down?) and **Service Connect Request Rate** (are requests failing?) → if spikes, check if auto-scaling is responding. - ---- - -### # Storage - How is storage being consumed and how fast is I/O? - -#### So what? -**Storage I/O Rate** shows combined read+write throughput. High write rates may indicate heavy logging or temp file activity. **Ephemeral Storage Utilization** (Fargate only) above 80% risks hitting the storage limit — Fargate tasks that exceed ephemeral storage are stopped. **EBS Filesystem Utilization** above 90% means disk-full is imminent. Watch out: Ephemeral storage metrics are Fargate-only (platform v1.4.0+). EBS metrics only exist if EBS volumes are attached. - -#### Now what? -Check **Storage Read vs Write** for I/O pattern → if writes dominate, check logging configuration or temp file cleanup → check **Ephemeral Storage %** for Fargate services → if approaching limit, increase ephemeral storage in task definition or reduce disk usage. - ---- - -### # Capacity - Is the EC2 cluster running out of room? - -#### So what? -**CPU/Memory Reservation** shows what percentage of registered EC2 cluster capacity is reserved by running tasks. **Headroom** below 20% means new tasks may fail to place. Memory is often the binding constraint — it runs out before CPU. Watch out: Reservation is **EC2 only**. If you use Fargate exclusively, these metrics will be 0% or absent — that's expected. Also, newly launching instances aren't counted until they register as ACTIVE, causing a brief lag in headroom calculations. - -#### Now what? -Check **CPU & Memory Reservation Over Time** → if trending up, verify capacity provider auto-scaling is active → check **Container Instances Over Time** to see if instances are being added → if reservation is high but instances aren't scaling, check capacity provider configuration or ASG limits. - ---- - -### # Service Connect - What is the request volume and error rate? - -#### So what? -**Request Rate** and **HTTP Error Rate** are the primary application health signals when using Service Connect. **5xx Error Rate > 1%** sustained indicates application failures. **4xx errors** are usually client-side issues (broken callers, API changes). **Response Time** shows proxy-to-target latency — spikes correlate with CPU throttling or memory pressure. Watch out: All Service Connect metrics require Service Connect to be configured. HTTP metrics additionally require `appProtocol` in port mappings. If these metrics are absent, your services may use ALB instead — that's expected. - -#### Now what? -Check **HTTP 5xx Error Rate** → if elevated, check **Response Time** for latency spikes → check **CPU/Memory Utilization** for resource pressure → check **Requests by Discovery Name** to isolate which endpoint is failing → check **TLS Errors** if connection failures appear. - ---- - -## Part 3: Cause-Effect Triage Chains (22 chains) - -1. If **Task Availability (%)** drops below 100% → check **Task Deficit** → check **Pending Tasks by Service** → likely causes: resource shortage, OOM kills, image pull failures → scale out service or fix task definition. (Confirmed) -2. If **Pending Tasks (count)** sustained > 0 → check **CPU/Memory Reservation** (EC2) → likely causes: cluster capacity exhausted, placement constraints unsatisfiable → add capacity or relax constraints. (Confirmed) -3. If **Restart Rate (/min)** climbing → check **Memory Utilization (%)** → if near 100%, OOM kills → increase task memory limits. (Confirmed) -4. If **Restart Rate (/min)** climbing but memory is fine → check application logs → likely causes: application crash, health check timeout, dependency failure. (Inference) -5. If **Active Deployments > 1** sustained → check **Task Deficit** → check **Restart Rate** → likely cause: new task revision failing to start, circuit breaker may trigger rollback. (Confirmed) -6. If **CPU Utilization (%)** sustained > 90% → check **Response Time** (Service Connect) → likely cause: CPU throttling causing latency → scale out service or increase CPU limits. (Mixed) -7. If **Memory Utilization (%)** > 85% sustained → check **Restart Rate** → if restarts start climbing, OOM kills are occurring → increase memory limits or investigate leaks. (Mixed) -8. If **CPU Used/Reserved (%)** low but **CPU Utilization (%)** high → tasks are using more than reserved (EC2 burst) → may need to set hard CPU limit or increase reservation. (Inference) -9. If **Memory Used/Reserved (%)** consistently > 90% → check **Restart Rate** → OOM imminent → increase memory hard limit in task definition. (Mixed) -10. If **Network In (MB/s)** drops to zero → check **Running Tasks** → if tasks are up, check network configuration (awsvpc mode, security groups, VPC endpoints). (Inference) -11. If **Network Out (MB/s)** spikes unexpectedly → check **Request Rate** (Service Connect) → likely causes: traffic surge, data transfer job, misconfigured egress. (Inference) -12. If **Storage I/O Rate (MB/s)** spiking → check **Ephemeral Storage (%)** → if ephemeral filling up, check log volume or temp files → likely cause: excessive logging or cache accumulation. (Inference) -13. If **Ephemeral Storage (%)** > 80% → check which services are affected → likely cause: log files not rotated, temp files not cleaned, large container images → increase ephemeral storage or fix cleanup. (Inference) -14. If **EBS Filesystem (%)** > 90% → check **EBS Filesystem Size** vs **EBS Utilized** → likely cause: persistent volume filling up → expand volume or clean data. (Inference) -15. If **CPU Reservation (%)** > 80% → check **Container Instances Over Time** → if instances not scaling, check ASG/capacity provider limits → add instances or enable capacity provider scaling. (Mixed) -16. If **Memory Reservation (%)** > 80% but CPU Reservation low → memory is the binding constraint → right-size tasks (reduce memory reservations on over-provisioned tasks) or add instances. (Inference) -17. If **HTTP 5xx Error Rate (%)** > 1% → check **Response Time** → check **CPU/Memory Utilization** → likely causes: application error, resource exhaustion, dependency failure. (Mixed) -18. If **HTTP 4xx Error Rate (%)** rising → check **Requests by Discovery Name** → likely causes: client-side bugs, API contract change, broken routing. (Inference) -19. If **Response Time (ms)** p95 spiking → check **CPU Utilization** → check **Active Connections** → likely causes: CPU throttling, connection exhaustion, downstream dependency slow. (Mixed) -20. If **Active Connections** spiking → check **New Connection Count** → if new connections also spiking, traffic surge → if new connections flat but active growing, connection leak. (Inference) -21. If **TLS Negotiation Errors** > 0 → check certificate expiry → check Service Connect TLS configuration → likely cause: expired or misconfigured certificates. (Inference) -22. If **Cluster Total Tasks** dropping but **Running Tasks** per service stable → standalone tasks or batch jobs completing → check if scheduled tasks are finishing normally. (Inference) - ---- - -## Part 4: Operational Playbooks (8 playbooks) - -### Playbook 1: Service Failing to Maintain Task Count -**Trigger:** Task Availability (%) < 100% sustained, Task Deficit > 0 -**Decision rule:** If deficit persists > 5 minutes, investigate. -**Steps:** -1. Check **Task Availability (%)** and **Task Deficit** for scope (which services) -2. Check **Pending Tasks by Service** to find affected services -3. Check **Restart Rate by Service** — if climbing, containers are crash-looping -4. Check **Memory Utilization by Service** — if near 100%, likely OOM -5. Check **CPU Utilization by Service** — if near 100%, may be stuck -6. Check **Active Deployments** — if > 1, new deployment may be failing -7. Check **CPU/Memory Reservation** (EC2) — if near 100%, no room for new tasks -**Likely causes:** OOM kills, image pull failures, resource exhaustion, bad task definition, placement constraints -**Next actions:** Increase task memory/CPU limits, fix container image, add cluster capacity, check deployment circuit breaker status -**Label:** Mixed - -### Playbook 2: Container Crash Loop -**Trigger:** Restart Rate (/min) > 0 sustained -**Decision rule:** Any sustained restarts warrant investigation. -**Steps:** -1. Check **Restart Rate by Service** to identify affected services -2. Check **Memory Utilization by Service** for the affected service -3. If memory high → OOM kills → increase task memory limits -4. If memory normal → application crash → check application logs -5. Check **Active Deployments** — may correlate with a bad deployment -6. Check **Task Deficit** — are tasks being replaced? -**Likely causes:** OOM, application bugs, dependency failures, health check timeouts -**Next actions:** Increase memory, roll back bad deployment, fix application, adjust health check timing -**Label:** Mixed - -### Playbook 3: CPU Throttling / High Utilization -**Trigger:** CPU Utilization (%) > 90% sustained on a service -**Decision rule:** If correlated with latency increase or restarts, act. -**Steps:** -1. Check **Top Services by CPU (%)** to identify hottest services -2. Check **CPU: Utilized vs Reserved** for over/under-provisioning -3. Check **Response Time** (Service Connect) for latency impact -4. Check **Running vs Desired Tasks** — auto-scaling may already be responding -5. Check if this is Fargate (hard limit) vs EC2 (burst possible) -**Likely causes:** Under-provisioned CPU, traffic surge, inefficient code, memory GC pressure -**Next actions:** Increase task CPU limits, scale out service (more tasks), optimize application -**Label:** Mixed - -### Playbook 4: EC2 Cluster Capacity Exhaustion -**Trigger:** CPU Reservation (%) or Memory Reservation (%) > 80% -**Decision rule:** If reservation trending up and capacity provider not scaling, investigate. -**Steps:** -1. Check **CPU & Memory Reservation Over Time** for trend -2. Check **Container Instances Over Time** — are instances being added? -3. Check **CPU Headroom (%)** and **Memory Headroom (%)** — which is lower? -4. Identify binding constraint (usually memory) -5. Check if capacity provider auto-scaling is configured -6. Check ASG max size — may have hit the ceiling -**Likely causes:** Traffic growth, over-reserved tasks, ASG max limit reached, capacity provider misconfiguration -**Next actions:** Increase ASG max, right-size task reservations, add capacity provider strategy -**Label:** Mixed - -### Playbook 5: Deployment Stuck or Rolling Back -**Trigger:** Active Deployments > 1 sustained for longer than expected rollout time -**Decision rule:** If deployment count > 1 for > 15 minutes (adjust per service), investigate. -**Steps:** -1. Check **Active Deployments** — confirm > 1 -2. Check **Task Deficit** — new tasks not reaching RUNNING? -3. Check **Restart Rate** — new revision crashing? -4. Check **Memory Utilization** — new revision OOM? -5. Check deployment circuit breaker status (ECS console / events) -6. Check **Running vs Desired** — is old deployment still serving? -**Likely causes:** Bad container image, insufficient resources, health check failing, dependency missing -**Next actions:** Roll back manually if circuit breaker hasn't triggered, fix image/config, increase resources -**Label:** Mixed - -### Playbook 6: Service Connect High Error Rate -**Trigger:** HTTP 5xx Error Rate (%) > 1% sustained -**Decision rule:** If error rate is service-wide and sustained, escalate. -**Steps:** -1. Check **HTTP Response Codes Over Time** for pattern (gradual vs sudden) -2. Check **Requests by Discovery Name** to isolate affected endpoints -3. Check **Response Time** — errors often correlate with timeouts -4. Check **CPU/Memory Utilization** — resource exhaustion may cause errors -5. Check **Active Connections** — connection pool exhaustion? -6. Check **Restart Rate** — are backend tasks dying? -**Likely causes:** Application bugs, dependency failures, resource exhaustion, configuration errors -**Next actions:** Check application logs, roll back recent deployment, increase resources, check downstream dependencies -**Label:** Mixed - -### Playbook 7: Network Throughput Anomaly -**Trigger:** Network In/Out (MB/s) drops to zero or spikes >3x baseline -**Decision rule:** If traffic pattern deviates significantly from normal, investigate. -**Steps:** -1. Check **Network Throughput (Rx vs Tx)** for pattern -2. Check **Running Tasks** — are tasks still up? -3. Check **Network In/Out by Service** to isolate affected service -4. Check **Request Rate** (Service Connect) — are requests still flowing? -5. Check security groups and network ACLs (outside dashboard scope) -**Likely causes:** Network misconfiguration, security group change, service outage, traffic rerouting -**Next actions:** Verify network configuration, check VPC flow logs, verify load balancer health -**Label:** Inference (relies on external data sources for root cause) - -### Playbook 8: Storage Filling Up -**Trigger:** Ephemeral Storage (%) > 80% or EBS Filesystem (%) > 90% -**Decision rule:** If storage utilization is trending up toward limits, act before it hits 100%. -**Steps:** -1. Check **Ephemeral Storage: Utilized vs Reserved** for Fargate services -2. Check **EBS Filesystem (%)** for EBS-attached tasks -3. Check **Storage Read vs Write** — is write rate excessive? -4. Check **Storage I/O by Service** to identify which service -5. Check if log rotation is configured, temp files are being cleaned -**Likely causes:** Unrotated logs, temp file accumulation, cache growth, large artifacts -**Next actions:** Increase ephemeral storage in task definition, enable log rotation, clean temp files, expand EBS volume -**Label:** Inference - - ---- - -# AWS ECS — Caveats & Footguns - -## High-Cardinality Dimensions to Avoid - -- **[task-lifecycle-deployments, resource-utilization]** `TaskId` is very high cardinality (hundreds to thousands of unique values across a cluster). Never use as a group-by in timeseries or top-list widgets. Use `ServiceName` or `TaskDefinitionFamily` instead. ([AWS Container Insights Enhanced Observability](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-enhanced-observability-metrics-ECS.html)) -- **[resource-utilization]** `ContainerInstanceId` is an ECS-internal reference. Prefer `InstanceId` (EC2InstanceId) for instance-level breakdowns — more recognizable in AWS console. (Inference) -- **[service-connect-traffic]** Combining `DiscoveryName` and `ServiceName` as simultaneous group-by levels can create cartesian explosion if many services talk to many endpoints. Use one dimension at a time. (Inference) - -## Misleading Metrics and Wrong Aggregations - -- **[resource-utilization]** `CPUUtilization` can **exceed 100%** on EC2 launch type when no hard CPU limit (`cpu` at task level) is set. Containers burst beyond their reservation. This is normal on EC2 but alarming if you expect 100% to be the ceiling. On Fargate, 100% is the hard ceiling. ([AWS Blog: How ECS manages CPU and memory](https://aws.amazon.com/blogs/containers/how-amazon-ecs-manages-cpu-and-memory-resources/)) -- **[resource-utilization]** `MemoryUtilization` approaching 100% has different consequences per launch type. On Fargate it means OOM kill is imminent (hard limit). On EC2 with `memoryReservation` (soft limit) + `memory` (hard limit), the container can use memory up to the hard limit before being killed. Without a hard limit, the Linux kernel OOM killer decides. ([AWS ECS Task Definition Params](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_definition_parameters.html)) -- **[capacity-scaling]** `CPUReservation` and `MemoryReservation` only count instances in ACTIVE or DRAINING status. Instances being launched by capacity providers are not counted until they register and become ACTIVE, causing a brief lag. (Inference based on [AWS ECS CloudWatch Metrics](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/available-metrics.html)) -- **[resource-utilization]** Do NOT use `sum` aggregation for `CPUUtilization` or `MemoryUtilization` — these are already percentages. Summing them produces meaningless values. Use `average`, `min`, or `max`. ([AWS ECS CloudWatch Metrics](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/available-metrics.html)) -- **[task-lifecycle-deployments]** `RestartCount` is monotonically increasing. Display it as a rate (per-second) not as a raw value — the raw value only goes up and provides no visual signal. (Inference) - -## Unit Pitfalls - -- **[resource-utilization]** `CpuUtilized` and `CpuReserved` are in **CPU units** (1024 units = 1 vCPU), NOT percentage. Don't confuse with `CPUUtilization` which IS a percentage. Display CPU units or convert to vCPU by dividing by 1024. ([AWS Container Insights Metrics](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-metrics-ECS.html)) -- **[resource-utilization]** `MemoryUtilized` and `MemoryReserved` are in **MiB** (mebibytes). Don't confuse with `MemoryUtilization` which is a percentage. Label charts clearly. ([AWS Container Insights Metrics](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-metrics-ECS.html)) -- **[network-io]** `NetworkRxBytes` and `NetworkTxBytes` from Container Insights are reported as **Bytes/Second** (already a rate). Do NOT apply `per-second` post-function — that would double-rate them. Use `average` aggregation with `none` post-function. ([AWS Container Insights Metrics](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-metrics-ECS.html)) -- **[storage-io]** `EphemeralStorageReserved` and `EphemeralStorageUtilized` are in **GB** (gigabytes), while `MemoryUtilized`/`MemoryReserved` are in MiB. Don't mix units in the same chart. ([AWS Container Insights Metrics](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-metrics-ECS.html)) -- **[service-connect-traffic]** `TargetResponseTime` is in **milliseconds**. Verify this in Stage 2 — some integrations may normalize to seconds. (Inference) - -## Sampling/Temporality Pitfalls - -- **[task-lifecycle-deployments, service-connect-traffic]** Counter metrics from CloudWatch (RestartCount, RequestCount, HTTPCode_*, StorageReadBytes, etc.) may be ingested into Tsuga as either delta or cumulative counters depending on the integration pipeline. This affects whether to use `per-second` (delta) or `rate` (cumulative) post-function. **Stage 2 discovery must verify temporality.** (Inference) -- **[resource-utilization]** CloudWatch `CPUUtilization` emits ~3 samples per minute (one every 20 seconds). The Average statistic is the most meaningful. Using Minimum or Maximum can be misleading for short-lived spikes within a single reporting period. ([AWS ECS CloudWatch Metrics](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/available-metrics.html)) -- **[task-lifecycle-deployments]** `RunningTaskCount` and `DesiredTaskCount` are sampled at reporting intervals, not event-driven. Brief task failures between samples may not appear in the metrics. Use EventBridge for exact task lifecycle events. ([AWS ECS Task State Change Events](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs_task_events.html)) - -## "This Looks Bad But Isn't" - -- **[capacity-scaling]** `CPUReservation` or `MemoryReservation` at 0% on a cluster does NOT mean no work is running — it means no **EC2** tasks are running. Fargate tasks don't contribute to reservation metrics. If you use Fargate exclusively, reservation will always be 0% or absent. ([AWS ECS CloudWatch Metrics](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/available-metrics.html)) -- **[task-lifecycle-deployments]** `DeploymentCount > 1` during a rolling update is normal. It only signals a problem if it persists well beyond the expected deployment duration. ([AWS ECS Service Deployments](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/deployment-types.html)) -- **[task-lifecycle-deployments]** `PendingTaskCount > 0` during scale-out or deployment is normal. It's only concerning when sustained for minutes (placement failure or resource shortage). (Inference) -- **[network-io]** `NetworkRxBytes` = 0 for a service using `host` network mode is expected — the metric only works with `awsvpc` and `bridge` modes. Not an outage. ([AWS Container Insights Metrics](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-metrics-ECS.html)) - -## Optional-Feature Traps - -- **[task-lifecycle-deployments, resource-utilization, network-io, storage-io]** Container Insights is **disabled by default**. Without it, most Container Insights metrics (RunningTaskCount, PendingTaskCount, NetworkRxBytes, StorageReadBytes, CpuUtilized, etc.) are simply absent. This is the single most common reason for "missing data" on ECS dashboards. ([Setting up Container Insights on ECS](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/deploy-container-insights-ECS-cluster.html)) -- **[service-connect-traffic]** All Service Connect metrics require Service Connect to be configured. If services use ALB + Cloud Map instead, the entire Service Connect section will have no data. This is expected, not broken. ([AWS ECS Service Connect](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-connect.html)) -- **[service-connect-traffic]** HTTP status code metrics (`HTTPCode_Target_*`) and `RequestCount` require `appProtocol` to be set in the task definition port mapping. Without it, Service Connect still works but only provides byte-level and connection-level metrics. ([AWS ECS Available Metrics](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/available-metrics.html)) -- **[storage-io]** `EphemeralStorageReserved` and `EphemeralStorageUtilized` are **Fargate only** (Linux platform 1.4.0+). On EC2 launch type these metrics will not exist. ([AWS Container Insights Metrics](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-metrics-ECS.html)) -- **[capacity-scaling]** `GPUReservation` only appears on EC2 clusters with GPU instances. If no GPU instances are registered, this metric will be absent. ([AWS ECS CloudWatch Metrics](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/available-metrics.html)) - - ---- - diff --git a/plugins/tsuga/skills/knowledge-technology/references/aws-ecs/metrics.csv b/plugins/tsuga/skills/knowledge-technology/references/aws-ecs/metrics.csv deleted file mode 100644 index 38d314d..0000000 --- a/plugins/tsuga/skills/knowledge-technology/references/aws-ecs/metrics.csv +++ /dev/null @@ -1,49 +0,0 @@ -theme,name,confirmed,definition,unit,type,tsuga_metric_name,widget_suggestions,aggregation,post_function,group_by,required_context_fields,optional_context_fields,absence_means,cardinality_footguns,variants,citations -Availability/Health,RunningTaskCount,true,Number of tasks in RUNNING state for a service. Primary health signal — must equal DesiredTaskCount.,1,summary,ecs_containerinsights_running_task_count,timeseries|query-value|top-list,avg,none,context.servicename:25|context.clustername:10,context.env|context.team,context.cloud.region|context.cloud.account.id,Container Insights not enabled or no services in cluster,Safe,cw=ECS/ContainerInsights/RunningTaskCount,https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-metrics-ECS.html -Availability/Health,PendingTaskCount,true,Number of tasks in PENDING state for a service. Sustained non-zero indicates placement failures or resource shortage.,1,summary,ecs_containerinsights_pending_task_count,timeseries|query-value|top-list,avg,none,context.servicename:25|context.clustername:10,context.env|context.team,context.cloud.region|context.cloud.account.id,Container Insights not enabled or all tasks running,Safe,cw=ECS/ContainerInsights/PendingTaskCount,https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-metrics-ECS.html -Availability/Health,DesiredTaskCount,true,Target number of running tasks for a service. Set by operator or auto-scaling policy.,1,summary,ecs_containerinsights_desired_task_count,timeseries|query-value,avg,none,context.servicename:25|context.clustername:10,context.env|context.team,context.cloud.region|context.cloud.account.id,Container Insights not enabled,Safe,cw=ECS/ContainerInsights/DesiredTaskCount,https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-metrics-ECS.html -Availability/Health,DeploymentCount,true,Number of active deployments for a service. >1 means a deployment is in progress or stuck.,1,summary,ecs_containerinsights_deployment_count,query-value|timeseries,max,none,context.servicename:25|context.clustername:10,context.env|context.team,context.cloud.region|context.cloud.account.id,Container Insights not enabled,Safe,cw=ECS/ContainerInsights/DeploymentCount,https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-metrics-ECS.html -Availability/Health,RestartCount,true,"Cumulative container restart count. Rising slope indicates container crash loops or health check failures.",1,summary,ecs_containerinsights_restart_count,timeseries|query-value,sum,rate,context.servicename:25|context.clustername:10|context.taskdefinitionfamily:25,context.env|context.team,context.cloud.region|context.cloud.account.id|context.containername,No restarts (healthy) or Container Insights not enabled,Safe,cw=ECS/ContainerInsights/RestartCount,https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-metrics-ECS.html -Availability/Health,TaskCount,true,Total number of tasks running in the cluster (all services + standalone tasks).,1,summary,ecs_containerinsights_task_count,query-value|timeseries,avg,none,context.clustername:10,context.env|context.team,context.cloud.region,Container Insights not enabled,Safe,cw=ECS/ContainerInsights/TaskCount,https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-metrics-ECS.html -Availability/Health,ServiceCount,true,Number of services in the cluster.,1,summary,ecs_containerinsights_service_count,query-value,avg,none,context.clustername:10,context.env|context.team,context.cloud.region,Container Insights not enabled,Safe,cw=ECS/ContainerInsights/ServiceCount,https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-metrics-ECS.html -Availability/Health,ContainerInstanceCount,true,Number of EC2 container instances registered with the cluster. EC2 launch type only.,1,summary,ecs_containerinsights_container_instance_count,query-value|timeseries,avg,none,context.clustername:10,context.env|context.team,context.cloud.region,EC2 launch type not used or Container Insights not enabled,Safe,cw=ECS/ContainerInsights/ContainerInstanceCount,https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-metrics-ECS.html -Performance/Latency,CPUUtilization,true,"Percentage of CPU units in use. Service-level available for Fargate+EC2. Cluster-level available for EC2 only. Can exceed 100% on EC2 if no hard CPU limit set.",percent,summary,aws_ecs_cpu_utilization,timeseries|query-value|top-list,avg,none,context.servicename:25|context.clustername:10,context.env|context.team,context.cloud.region|context.cloud.account.id,No tasks running in service/cluster,Safe,cw=AWS/ECS/CPUUtilization,https://docs.aws.amazon.com/AmazonECS/latest/developerguide/available-metrics.html -Performance/Latency,MemoryUtilization,true,"Percentage of memory in use. Service-level available for Fargate+EC2. Cluster-level available for EC2 only. Approaching 100% = OOM kill risk.",percent,summary,aws_ecs_memory_utilization,timeseries|query-value|top-list,avg,none,context.servicename:25|context.clustername:10,context.env|context.team,context.cloud.region|context.cloud.account.id,No tasks running in service/cluster,Safe,cw=AWS/ECS/MemoryUtilization,https://docs.aws.amazon.com/AmazonECS/latest/developerguide/available-metrics.html -Capacity/Saturation,CpuUtilized,true,CPU units used by tasks. Absolute value (not percentage). Useful alongside CpuReserved for utilization ratio.,CPU units,summary,ecs_containerinsights_cpu_utilized,timeseries|query-value,avg,none,context.servicename:25|context.clustername:10|context.taskdefinitionfamily:25,context.env|context.team,context.cloud.region|context.cloud.account.id|context.taskid,Container Insights not enabled or no tasks running,Safe,cw=ECS/ContainerInsights/CpuUtilized,https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-metrics-ECS.html -Capacity/Saturation,CpuReserved,true,CPU units reserved by tasks. Absolute value. Denominator for CPU utilization ratio.,CPU units,summary,ecs_containerinsights_cpu_reserved,timeseries|query-value,avg,none,context.servicename:25|context.clustername:10|context.taskdefinitionfamily:25,context.env|context.team,context.cloud.region|context.cloud.account.id|context.taskid,Container Insights not enabled or no tasks running,Safe,cw=ECS/ContainerInsights/CpuReserved,https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-metrics-ECS.html -Capacity/Saturation,MemoryUtilized,true,Memory in MiB used by tasks. Absolute value. Compare with MemoryReserved to compute utilization.,MiB,summary,ecs_containerinsights_memory_utilized,timeseries|query-value,avg,none,context.servicename:25|context.clustername:10|context.taskdefinitionfamily:25,context.env|context.team,context.cloud.region|context.cloud.account.id|context.taskid,Container Insights not enabled or no tasks running,Safe,cw=ECS/ContainerInsights/MemoryUtilized,https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-metrics-ECS.html -Capacity/Saturation,MemoryReserved,true,Memory in MiB reserved by tasks. Absolute value. Denominator for memory utilization ratio.,MiB,summary,ecs_containerinsights_memory_reserved,timeseries|query-value,avg,none,context.servicename:25|context.clustername:10|context.taskdefinitionfamily:25,context.env|context.team,context.cloud.region|context.cloud.account.id|context.taskid,Container Insights not enabled or no tasks running,Safe,cw=ECS/ContainerInsights/MemoryReserved,https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-metrics-ECS.html -Capacity/Saturation,CPUReservation,false,Percentage of total registered CPU units reserved by running tasks. EC2 launch type only. NOT FOUND in Tsuga — likely Fargate-only account.,percent,summary,aws_ecs_cpu_reservation,timeseries|query-value,avg,none,context.clustername:10,context.env|context.team,context.cloud.region|context.cloud.account.id,EC2 launch type not used or no tasks running,Safe,cw=AWS/ECS/CPUReservation,https://docs.aws.amazon.com/AmazonECS/latest/developerguide/available-metrics.html -Capacity/Saturation,MemoryReservation,false,Percentage of total registered memory reserved by running tasks. EC2 launch type only. NOT FOUND in Tsuga — likely Fargate-only account.,percent,summary,aws_ecs_memory_reservation,timeseries|query-value,avg,none,context.clustername:10,context.env|context.team,context.cloud.region|context.cloud.account.id,EC2 launch type not used or no tasks running,Safe,cw=AWS/ECS/MemoryReservation,https://docs.aws.amazon.com/AmazonECS/latest/developerguide/available-metrics.html -Capacity/Saturation,GPUReservation,false,Percentage of total available GPUs reserved by running tasks. EC2 launch type with GPU instances only. NOT FOUND in Tsuga.,percent,summary,aws_ecs_gpu_reservation,query-value|timeseries,avg,none,context.clustername:10,context.env|context.team,context.cloud.region,EC2 launch type not used or no GPU instances,Safe,cw=AWS/ECS/GPUReservation,https://docs.aws.amazon.com/AmazonECS/latest/developerguide/available-metrics.html -Throughput/Usage,NetworkRxBytes,true,"Bytes received per second by tasks. Available only for awsvpc and bridge network modes. Reported as a rate (Bytes/Second).",By/s,summary,ecs_containerinsights_network_rx_bytes,timeseries|query-value,avg,none,context.servicename:25|context.clustername:10|context.taskdefinitionfamily:25,context.env|context.team,context.cloud.region|context.taskid,Network mode does not support metric or Container Insights not enabled,Safe,cw=ECS/ContainerInsights/NetworkRxBytes,https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-metrics-ECS.html -Throughput/Usage,NetworkTxBytes,true,"Bytes transmitted per second by tasks. Available only for awsvpc and bridge network modes. Reported as a rate (Bytes/Second).",By/s,summary,ecs_containerinsights_network_tx_bytes,timeseries|query-value,avg,none,context.servicename:25|context.clustername:10|context.taskdefinitionfamily:25,context.env|context.team,context.cloud.region|context.taskid,Network mode does not support metric or Container Insights not enabled,Safe,cw=ECS/ContainerInsights/NetworkTxBytes,https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-metrics-ECS.html -Throughput/Usage,StorageReadBytes,true,Bytes read from storage devices by tasks.,By,summary,ecs_containerinsights_storage_read_bytes,timeseries|query-value,sum,rate,context.servicename:25|context.clustername:10|context.taskdefinitionfamily:25,context.env|context.team,context.cloud.region|context.taskid,No storage reads or Container Insights not enabled,Safe,cw=ECS/ContainerInsights/StorageReadBytes,https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-metrics-ECS.html -Throughput/Usage,StorageWriteBytes,true,Bytes written to storage devices by tasks.,By,summary,ecs_containerinsights_storage_write_bytes,timeseries|query-value,sum,rate,context.servicename:25|context.clustername:10|context.taskdefinitionfamily:25,context.env|context.team,context.cloud.region|context.taskid,No storage writes or Container Insights not enabled,Safe,cw=ECS/ContainerInsights/StorageWriteBytes,https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-metrics-ECS.html -Capacity/Saturation,EphemeralStorageReserved,true,"Ephemeral storage in GB reserved for Fargate tasks. Fargate Linux platform 1.4.0+ only.",GB,summary,ecs_containerinsights_ephemeral_storage_reserved,query-value|timeseries,avg,none,context.servicename:25|context.clustername:10|context.taskdefinitionfamily:25,context.env|context.team,context.cloud.region|context.taskid,Not Fargate or platform version < 1.4.0 or Container Insights not enabled,Safe,cw=ECS/ContainerInsights/EphemeralStorageReserved,https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-metrics-ECS.html -Capacity/Saturation,EphemeralStorageUtilized,true,"Ephemeral storage in GB used by Fargate tasks. Fargate Linux platform 1.4.0+ only.",GB,summary,ecs_containerinsights_ephemeral_storage_utilized,query-value|timeseries,avg,none,context.servicename:25|context.clustername:10|context.taskdefinitionfamily:25,context.env|context.team,context.cloud.region|context.taskid,Not Fargate or platform version < 1.4.0 or Container Insights not enabled,Safe,cw=ECS/ContainerInsights/EphemeralStorageUtilized,https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-metrics-ECS.html -Capacity/Saturation,EBSFilesystemUtilization,false,"Percentage of EBS filesystem used. NOT FOUND in Tsuga — no EBS volumes attached to tasks in this account.",percent,summary,aws_ecs_ebs_filesystem_utilization,query-value|timeseries,avg,none,context.servicename:25|context.clustername:10,context.env|context.team,context.cloud.region,No EBS volumes attached to tasks,Safe,cw=AWS/ECS/EBSFilesystemUtilization,https://docs.aws.amazon.com/AmazonECS/latest/developerguide/available-metrics.html -Throughput/Usage,RequestCount,true,"Inbound requests processed by Service Connect proxies. Requires appProtocol in port mapping. Service Connect only.",1,summary,aws_ecs_request_count,timeseries|query-value|top-list,sum,rate,context.discoveryname:15|context.servicename:25|context.clustername:10,context.env|context.team,context.cloud.region|context.cloud.account.id,Service Connect not configured or appProtocol not set,"discoveryname cardinality typically low (5-50), safe",cw=AWS/ECS/RequestCount,https://docs.aws.amazon.com/AmazonECS/latest/developerguide/available-metrics.html -Throughput/Usage,ProcessedBytes,true,Inbound bytes processed by Service Connect proxies. Service Connect only.,By,summary,aws_ecs_processed_bytes,timeseries|query-value,sum,rate,context.discoveryname:15|context.servicename:25|context.clustername:10,context.env|context.team,context.cloud.region|context.cloud.account.id,Service Connect not configured,Safe,cw=AWS/ECS/ProcessedBytes,https://docs.aws.amazon.com/AmazonECS/latest/developerguide/available-metrics.html -Throughput/Usage,ActiveConnectionCount,true,Number of concurrent active connections to Service Connect proxies. Service Connect only.,1,summary,aws_ecs_active_connection_count,timeseries|query-value,avg,none,context.discoveryname:15|context.servicename:25|context.clustername:10,context.env|context.team,context.cloud.region|context.cloud.account.id,Service Connect not configured,Safe,cw=AWS/ECS/ActiveConnectionCount,https://docs.aws.amazon.com/AmazonECS/latest/developerguide/available-metrics.html -Throughput/Usage,NewConnectionCount,true,New connections established via Service Connect proxies. Service Connect only.,1,summary,aws_ecs_new_connection_count,timeseries|query-value,sum,rate,context.discoveryname:15|context.servicename:25|context.clustername:10,context.env|context.team,context.cloud.region|context.cloud.account.id,Service Connect not configured,Safe,cw=AWS/ECS/NewConnectionCount,https://docs.aws.amazon.com/AmazonECS/latest/developerguide/available-metrics.html -Errors/Failures,HTTPCode_Target_2XX_Count,true,HTTP 2xx (success) responses from Service Connect targets.,1,summary,aws_ecs_http_code_target_2xx_count,timeseries|query-value,sum,rate,context.servicename:25|context.clustername:10,context.env|context.team,context.cloud.region|context.cloud.account.id,Service Connect not configured or appProtocol not set,Safe,cw=AWS/ECS/HTTPCode_Target_2XX_Count,https://docs.aws.amazon.com/AmazonECS/latest/developerguide/available-metrics.html -Errors/Failures,HTTPCode_Target_3XX_Count,false,HTTP 3xx (redirect) responses from Service Connect targets. NOT FOUND in Tsuga — no 3xx responses in this account.,1,summary,aws_ecs_http_code_target_3xx_count,timeseries|query-value,sum,rate,context.servicename:25|context.clustername:10,context.env|context.team,context.cloud.region|context.cloud.account.id,Service Connect not configured or no 3xx responses,Safe,cw=AWS/ECS/HTTPCode_Target_3XX_Count,https://docs.aws.amazon.com/AmazonECS/latest/developerguide/available-metrics.html -Errors/Failures,HTTPCode_Target_4XX_Count,false,HTTP 4xx (client error) responses from Service Connect targets. NOT FOUND in Tsuga — no 4xx responses in this account.,1,summary,aws_ecs_http_code_target_4xx_count,timeseries|query-value,sum,rate,context.servicename:25|context.clustername:10,context.env|context.team,context.cloud.region|context.cloud.account.id,Service Connect not configured or no 4xx responses,Safe,cw=AWS/ECS/HTTPCode_Target_4XX_Count,https://docs.aws.amazon.com/AmazonECS/latest/developerguide/available-metrics.html -Errors/Failures,HTTPCode_Target_5XX_Count,false,HTTP 5xx (server error) responses from Service Connect targets. NOT FOUND in Tsuga — no 5xx responses in this account.,1,summary,aws_ecs_http_code_target_5xx_count,timeseries|query-value,sum,rate,context.servicename:25|context.clustername:10,context.env|context.team,context.cloud.region|context.cloud.account.id,Service Connect not configured or no 5xx responses,Safe,cw=AWS/ECS/HTTPCode_Target_5XX_Count,https://docs.aws.amazon.com/AmazonECS/latest/developerguide/available-metrics.html -Performance/Latency,TargetResponseTime,true,"Latency from Service Connect proxy to target response. Supports average min max. Service Connect only.",ms,summary,aws_ecs_target_response_time,timeseries|query-value,avg,none,context.targetdiscoveryname:15|context.servicename:25|context.clustername:10,context.env|context.team,context.cloud.region|context.cloud.account.id,Service Connect not configured,Safe,cw=AWS/ECS/TargetResponseTime,https://docs.aws.amazon.com/AmazonECS/latest/developerguide/available-metrics.html -Throughput/Usage,TargetProcessedBytes,true,Bytes processed by Service Connect proxies toward targets. Service Connect only.,By,summary,aws_ecs_target_processed_bytes,timeseries|query-value,sum,rate,context.targetdiscoveryname:15|context.servicename:25|context.clustername:10,context.env|context.team,context.cloud.region|context.cloud.account.id,Service Connect not configured,Safe,cw=AWS/ECS/TargetProcessedBytes,https://docs.aws.amazon.com/AmazonECS/latest/developerguide/available-metrics.html -Throughput/Usage,RequestCountPerTarget,true,Average requests per target for Service Connect. Useful for per-instance scaling decisions.,1,summary,aws_ecs_request_count_per_target,timeseries|query-value,avg,none,context.targetdiscoveryname:15|context.servicename:25|context.clustername:10,context.env|context.team,context.cloud.region|context.cloud.account.id,Service Connect not configured,Safe,cw=AWS/ECS/RequestCountPerTarget,https://docs.aws.amazon.com/AmazonECS/latest/developerguide/available-metrics.html -Errors/Failures,ClientTLSNegotiationErrorCount,false,TLS connection failures on the client side of Service Connect. NOT FOUND in Tsuga — TLS not configured.,1,summary,aws_ecs_client_tls_negotiation_error_count,timeseries|query-value,sum,rate,context.discoveryname:15|context.servicename:25|context.clustername:10,context.env|context.team,context.cloud.region,Service Connect TLS not configured,Safe,cw=AWS/ECS/ClientTLSNegotiationErrorCount,https://docs.aws.amazon.com/AmazonECS/latest/developerguide/available-metrics.html -Errors/Failures,TargetTLSNegotiationErrorCount,false,TLS failures for target connections. NOT FOUND in Tsuga — TLS not configured.,1,summary,aws_ecs_target_tls_negotiation_error_count,timeseries|query-value,sum,rate,context.targetdiscoveryname:15|context.servicename:25|context.clustername:10,context.env|context.team,context.cloud.region,Service Connect TLS not configured,Safe,cw=AWS/ECS/TargetTLSNegotiationErrorCount,https://docs.aws.amazon.com/AmazonECS/latest/developerguide/available-metrics.html -Errors/Failures,GrpcRequestCount,true,gRPC inbound requests processed by Service Connect. Requires appProtocol: GRPC. Service Connect only.,1,summary,aws_ecs_grpc_request_count,timeseries|query-value,sum,rate,context.discoveryname:15|context.servicename:25|context.clustername:10,context.env|context.team,context.cloud.region|context.cloud.account.id,Service Connect not configured or appProtocol not GRPC,Safe,cw=AWS/ECS/GrpcRequestCount,https://docs.aws.amazon.com/AmazonECS/latest/developerguide/available-metrics.html -Availability/Health,TaskSetCount,true,Number of task sets in a service. Relevant for external deployment controllers.,1,summary,ecs_containerinsights_task_set_count,query-value,max,none,context.servicename:25|context.clustername:10,context.env|context.team,context.cloud.region,Container Insights not enabled or using rolling deployment (not external),Safe,cw=ECS/ContainerInsights/TaskSetCount,https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-metrics-ECS.html -Capacity/Saturation,EBSFilesystemSize,false,Total EBS filesystem size in GB. NOT FOUND in Tsuga — no EBS volumes attached.,GB,summary,ecs_containerinsights_ebs_filesystem_size,query-value|timeseries,max,none,context.taskdefinitionfamily:25|context.clustername:10,context.env|context.team,,No EBS volumes attached or Container Insights not enabled,Safe,cw=ECS/ContainerInsights/EBSFilesystemSize,https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-metrics-ECS.html -Capacity/Saturation,EBSFilesystemUtilized,false,EBS filesystem used in GB. NOT FOUND in Tsuga — no EBS volumes attached.,GB,summary,ecs_containerinsights_ebs_filesystem_utilized,query-value|timeseries,max,none,context.taskdefinitionfamily:25|context.clustername:10,context.env|context.team,,No EBS volumes attached or Container Insights not enabled,Safe,cw=ECS/ContainerInsights/EBSFilesystemUtilized,https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-metrics-ECS.html -Capacity/Saturation,ContainerCpuUtilization,true,UNEXPECTED — Container-level CPU utilization percentage. Enhanced Observability only.,percent,summary,ecs_containerinsights_container_cpu_utilization,timeseries|query-value|top-list,avg,none,context.containername:10|context.servicename:25|context.clustername:10|context.taskdefinitionfamily:25,context.env|context.team,context.cloud.region|context.taskid,Enhanced Observability not enabled,"containername x taskid can be high cardinality; prefer containername alone",cw=ECS/ContainerInsights/ContainerCpuUtilization,https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-enhanced-observability-metrics-ECS.html -Capacity/Saturation,ContainerMemoryUtilization,true,UNEXPECTED — Container-level memory utilization percentage. Enhanced Observability only.,percent,summary,ecs_containerinsights_container_memory_utilization,timeseries|query-value|top-list,avg,none,context.containername:10|context.servicename:25|context.clustername:10|context.taskdefinitionfamily:25,context.env|context.team,context.cloud.region|context.taskid,Enhanced Observability not enabled,"containername x taskid can be high cardinality; prefer containername alone",cw=ECS/ContainerInsights/ContainerMemoryUtilization,https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-enhanced-observability-metrics-ECS.html -Availability/Health,UnHealthyContainerHealthStatus,true,UNEXPECTED — Container health check status. 1=UNHEALTHY 0=HEALTHY. Enhanced Observability only. Requires health check in task definition.,1,summary,ecs_containerinsights_un_healthy_container_health_status,query-value|timeseries,max,none,context.containername:10|context.servicename:25|context.clustername:10|context.taskdefinitionfamily:25,context.env|context.team,context.cloud.region|context.taskid,Enhanced Observability not enabled or health check not configured,Safe,cw=ECS/ContainerInsights/UnHealthyContainerHealthStatus,https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-enhanced-observability-metrics-ECS.html -Capacity/Saturation,TaskCpuUtilization,true,UNEXPECTED — Task-level CPU utilization percentage. Enhanced Observability only.,percent,summary,ecs_containerinsights_task_cpu_utilization,timeseries|query-value,avg,none,context.servicename:25|context.clustername:10|context.taskdefinitionfamily:25,context.env|context.team,context.cloud.region|context.taskid,Enhanced Observability not enabled,Safe,cw=ECS/ContainerInsights/TaskCpuUtilization,https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-enhanced-observability-metrics-ECS.html -Capacity/Saturation,TaskMemoryUtilization,true,UNEXPECTED — Task-level memory utilization percentage. Enhanced Observability only.,percent,summary,ecs_containerinsights_task_memory_utilization,timeseries|query-value,avg,none,context.servicename:25|context.clustername:10|context.taskdefinitionfamily:25,context.env|context.team,context.cloud.region|context.taskid,Enhanced Observability not enabled,Safe,cw=ECS/ContainerInsights/TaskMemoryUtilization,https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-enhanced-observability-metrics-ECS.html -Capacity/Saturation,TaskEphemeralStorageUtilization,true,UNEXPECTED — Task-level ephemeral storage utilization percentage. Enhanced Observability only. Fargate only.,percent,summary,ecs_containerinsights_task_ephemeral_storage_utilization,query-value|timeseries,avg,none,context.servicename:25|context.clustername:10|context.taskdefinitionfamily:25,context.env|context.team,context.cloud.region|context.taskid,Enhanced Observability not enabled or not Fargate,Safe,cw=ECS/ContainerInsights/TaskEphemeralStorageUtilization,https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-enhanced-observability-metrics-ECS.html -Cost,NOT_APPLICABLE,false,"ECS does not have dedicated cost metrics. Cost is driven by Fargate vCPU/memory-seconds and EC2 instance hours. Proxy signals: CpuUtilized vs CpuReserved (over-provisioning) and RunningTaskCount (scale). Use AWS Cost Explorer for actual cost data.",N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A diff --git a/plugins/tsuga/skills/knowledge-technology/references/aws-ecs/queries.md b/plugins/tsuga/skills/knowledge-technology/references/aws-ecs/queries.md deleted file mode 100644 index febdd72..0000000 --- a/plugins/tsuga/skills/knowledge-technology/references/aws-ecs/queries.md +++ /dev/null @@ -1,57 +0,0 @@ -# AWS ECS - -Managed container orchestrator (EC2 or Fargate capacity). Healthy: running tasks = desired, no restart loops, CPU/memory within reservation. - -## Incident shapes - -- **Task restart loop** — `ecs_containerinsights_restart_count` climbs → bad deploy, missing secret, failed health check -- **Scheduler starvation** — `ecs_containerinsights_desired_task_count > ecs_containerinsights_running_task_count` → cluster lacks capacity -- **Memory pressure / OOM** — `ecs_containerinsights_memory_utilized / ecs_containerinsights_memory_reserved → 1.0` → OOM-killed -- **CPU throttling** — `ecs_containerinsights_cpu_utilized / ecs_containerinsights_cpu_reserved → 1.0` sustained → requests queue -- **Ephemeral-storage exhaustion** — Fargate ephemeral disk full → ENOSPC -- **Deployment stuck** — `ecs_containerinsights_deployment_count > 1` long → new revision never goes healthy - -## Key metrics - -| Metric | Unit | Signal | -|---|---|---| -| `ecs_containerinsights_running_task_count` | count | Tasks in RUNNING | -| `ecs_containerinsights_pending_task_count` | count | Waiting to be placed | -| `ecs_containerinsights_desired_task_count` | count | Target from service | -| `ecs_containerinsights_deployment_count` | count | > 1 = in-progress rollout | -| `ecs_containerinsights_restart_count` | count | Delta-over-window = restart count | -| `aws_ecs_cpu_utilization` | % | Cluster-level | -| `aws_ecs_memory_utilization` | % | Cluster-level | -| `ecs_containerinsights_cpu_utilized` / `ecs_containerinsights_cpu_reserved` | cpu-units | Per-task / service | -| `ecs_containerinsights_memory_utilized` / `ecs_containerinsights_memory_reserved` | bytes | Per-task / service | -| `aws_ecs_cpu_reservation` / `aws_ecs_memory_reservation` | % | Cluster reservation pressure | -| `ecs_containerinsights_ephemeral_storage_utilized` / `Reserved` | bytes | Fargate ephemeral disk | -| `aws_ecs_ebs_filesystem_utilization` | % | EBS-backed tasks | -| `ecs_containerinsights_network_rx_bytes` / `TxBytes` | bytes | Per-task traffic | - -## Derived signals - -- `ecs_containerinsights_running_task_count / ecs_containerinsights_desired_task_count` — readiness. < 1.0 after rollout = stuck. -- `ecs_containerinsights_memory_utilized / ecs_containerinsights_memory_reserved` — saturation. > 0.85 sustained = OOM risk. -- `ecs_containerinsights_cpu_utilized / ecs_containerinsights_cpu_reserved` — CPU saturation. > 0.8 + throttling = under-provisioned. -- `aws_ecs_cpu_reservation + aws_ecs_memory_reservation` — cluster reservation pressure. Near 100% = no placement headroom. -- `Δ ecs_containerinsights_restart_count` over 5m — any steady positive = unhealthy service. - -## Log patterns - -- `Essential container in task exited` — essential container died; task stops -- `Unable to place a task because no container instance met all of its requirements` — scheduler can't fit -- `OutOfMemory: Container killed by the kernel` — OOM -- `CannotPullContainerError` — image pull failed -- `ResourceInitializationError: unable to pull secrets or registry auth` — secrets/ECR auth -- `Task failed ELB health checks` — health-check failure -- `exited with code 137` — SIGKILL (usually OOM) -- `exited with code 139` — SIGSEGV - -## Gotchas - -- `ecs_containerinsights_running_task_count == ecs_containerinsights_desired_task_count` doesn't prove health; tasks can flip RUNNING between crashes. Check `ecs_containerinsights_restart_count` delta too. -- Fargate vs EC2 expose different metric subsets. Fargate has no instance-level metrics. -- Cluster-level `aws_ecs_cpu_utilization` averages across instances; one hot instance can hide. -- Soft vs hard memory limits: soft is unlimited when cluster has headroom. Under-provisioned soft limits are a common silent failure. -- During scale-down, `ecs_containerinsights_running_task_count > ecs_containerinsights_desired_task_count` is normal, not an incident. diff --git a/plugins/tsuga/skills/knowledge-technology/references/aws-efs/README.md b/plugins/tsuga/skills/knowledge-technology/references/aws-efs/README.md deleted file mode 100644 index 559eb39..0000000 --- a/plugins/tsuga/skills/knowledge-technology/references/aws-efs/README.md +++ /dev/null @@ -1,265 +0,0 @@ -# AWS EFS Integration Context Bundle - -## Metadata -**Technology:** AWS EFS -**Deployment:** managed -**Environment:** prod -**Persona:** SRE Dev and ops -**Telemetry preference:** mixed -**Integration scope:** core service only -**Primary use-case:** reliability and performance - -## How to use this bundle -- Use `01_aws-efs_metrics.csv` as the source of truth for metric names, units, temporality assumptions, and safe query patterns. -- Use `02_aws-efs_dashboard_plan.yaml` as the implementation blueprint for sections, widgets, derived signals, explanation notes, triage chains, and playbooks. -- Use `03_aws-efs_state.yaml` as the machine-readable state file for stage status, inferred namespace mappings, log intelligence status, and unresolved unknowns. -- Use `04_aws-efs_memory.md` for the human-readable Stage 1 handoff narrative and Stage 2 priority checks. -- Stage 2 will create `05_aws-efs_metric_catalog.csv` as the discovered inventory and reconciliation memory for actual Tsuga metric and attribute coverage. -- Stage 4 should read `00` under `Log intelligence (Stage 4 handoff)` and `03.log_intel` before attempting log route creation. - -## What it is and what "good" looks like - -### Confirmed by sources -- Amazon EFS is a managed NFS file system for EC2, containers, and other AWS compute that exposes health primarily through CloudWatch metrics around throughput, I/O composition, storage size, client connections, throughput headroom, and optional replication lag. [S1][S2][S3][S6] -- The core operational posture is shaped by file system type (Regional vs One Zone), performance mode (General Purpose vs Max I/O legacy), and throughput mode (Elastic, Provisioned, or Bursting). Those choices directly change what "good" throughput headroom looks like and whether burst credits matter. [S3][S4] -- Good for EFS means client connections are stable, metered throughput stays comfortably below permitted throughput, `PercentIOLimit` does not pin near the General Purpose limit, storage growth is expected across Standard and cold tiers, and replication lag remains near the documented 15-minute RPO envelope when replication is enabled. [S2][S3][S6] -- The fastest dashboard split for incidents is: headroom pressure (`fleet-health`), workload shape change (`throughput-mix`), unexpected storage growth or tiering drift (`storage-lifecycle`), and DR freshness risk (`replication-resilience`). - -### Best-practice inference -- Incident shape 1: **Throughput ceiling or credit exhaustion.** Start in `fleet-health`; check `I/O Limit Utilization`, `Throughput Utilization`, `Burst Credit Balance`, then `Permitted Throughput`. -- Incident shape 2: **Workload pattern changed.** Start in `throughput-mix`; compare `Metadata Throughput`, `Metadata Read vs Write`, and `Metered vs Total Throughput` to tell metadata-heavy churn from broader file throughput pressure. -- Incident shape 3: **Storage bill or cold-data movement surprise.** Start in `storage-lifecycle`; determine whether total bytes are growing in Standard or whether IA and Archive shares changed after lifecycle policy edits. -- Incident shape 4: **Disaster recovery freshness risk.** Start in `replication-resilience`; `TimeSinceLastSync` is the only high-signal service metric if replication exists. -- A useful EFS dashboard is narrower than a database or queue dashboard: EFS exposes file-system service counters, not application latency, per-directory hot spots, or rich server-side error codes. Logs and client-side telemetry are needed for mount failures and NFS semantics problems. - -## Key concepts - -### Glossary - -| Term | Definition | Operational meaning | Dashboard section | -|---|---|---|---| -| EFS file system | Managed NFS storage service in AWS | Main blast-radius boundary for all service metrics | all | -| FileSystemId | Unique identifier for one EFS file system | Primary safe group-by for fleet views | all | -| DestinationFileSystemId | Replica destination file system id | Required to interpret replication lag by pair | replication-resilience | -| Regional file system | EFS file system replicated across multiple AZs in one Region | Default durability posture for production | fleet-health | -| One Zone file system | EFS file system stored in one AZ | Lower resilience; storage and DR sections matter more | fleet-health | -| General Purpose mode | Default EFS performance mode with lower latency | Only mode where `PercentIOLimit` meaningfully indicates limit pressure | fleet-health | -| Max I/O mode | Older high-parallelism mode with higher latency | `PercentIOLimit` can be absent or less relevant; higher latency may be expected | fleet-health | -| Elastic throughput | Throughput mode that scales automatically with workload | Burst credits do not accrue or drain here | fleet-health | -| Provisioned throughput | Fixed configured throughput independent of file system size | `PermittedThroughput` becomes a contractual capacity signal | fleet-health | -| Bursting throughput | Throughput mode tied to Standard storage size plus credits | `BurstCreditBalance` becomes a runway signal | fleet-health | -| Burst credits | Accumulated capacity that lets bursting-mode EFS exceed baseline throughput | Falling credits mean future throttling risk before current outage | fleet-health | -| PermittedThroughput | Maximum throughput the file system can currently drive | Best denominator for utilization math | fleet-health | -| MeteredIOBytes | Throughput-accounted bytes after EFS read discount rules | Best numerator for throughput headroom | throughput-mix | -| TotalIOBytes | Actual bytes processed without read discount | Separates true load from billing-weighted load | throughput-mix | -| DataReadIOBytes | Bytes read from the file system | Read workload volume | throughput-mix | -| DataWriteIOBytes | Bytes written to the file system | Write workload volume and write pressure | throughput-mix | -| MetadataIOBytes | Bytes used by metadata operations | High values often mean directory traversal or metadata-heavy workloads | throughput-mix | -| MetadataReadIOBytes | Bytes from metadata reads | Explains read-heavy namespace scans | throughput-mix | -| MetadataWriteIOBytes | Bytes from metadata writes | Explains create/unlink/chmod style metadata churn | throughput-mix | -| PercentIOLimit | Percent of the General Purpose I/O limit consumed | Early warning that latency pain is limit-driven | fleet-health | -| ClientConnections | Number of connected clients | Useful blast-radius and mount-fanout indicator | fleet-health | -| StorageBytes | File system size across storage classes | Capacity and cost proxy metric | storage-lifecycle | -| EFS Standard | Hot SSD-backed storage class | Growth here affects bursting baseline and cost | storage-lifecycle | -| EFS IA | Infrequent Access storage class | Rising share suggests lifecycle policy is moving cold data | storage-lifecycle | -| EFS Archive | Coldest EFS storage class | Useful for cost posture, but not for latency-sensitive data | storage-lifecycle | -| IASizeOverhead | Metering overhead from 128 KiB minimum billing in IA | Large overhead means many tiny cold files | storage-lifecycle | -| ArchiveSizeOverhead | Metering overhead from 128 KiB minimum billing in Archive | Large overhead means Archive is inefficient for small files | storage-lifecycle | -| TimeSinceLastSync | Seconds since the last successful replication sync | Direct DR freshness signal | replication-resilience | -| Close-to-open consistency | EFS/NFS consistency model | Explains app complaints that do not show as service-level EFS faults | throughput-mix | -| Mount helper | `amazon-efs-utils` client tool | Stage 4 log source for mount failures and TLS/watchdog issues | replication-resilience | - -[S2][S3][S4][S5][S6][S7][S8] - -### Concept Map -Client workload -> mounts -> EFS file system (why: all service metrics aggregate at file-system scope) -EC2 instance or pod -> creates -> NFS client connection (why: `ClientConnections` is often the first fanout signal) -Application read path -> drives -> DataReadIOBytes (why: read throughput is a core demand shape) -Application write path -> drives -> DataWriteIOBytes (why: write throughput is the clearest mutation signal) -Directory traversal and stat-heavy code -> drives -> MetadataReadIOBytes (why: namespace scans can hurt without large data throughput) -Create/delete/chmod-heavy code -> drives -> MetadataWriteIOBytes (why: metadata churn is operationally distinct from data writes) -DataReadIOBytes + DataWriteIOBytes + MetadataIOBytes -> contribute to -> TotalIOBytes (why: total workload shape matters) -Read discount rules -> transform -> MeteredIOBytes (why: metered throughput is what counts against throughput limits) -MeteredIOBytes -> compared against -> PermittedThroughput (why: this reveals throughput utilization) -Bursting throughput mode -> consumes -> BurstCreditBalance (why: credit drain predicts future throughput pressure) -Standard storage growth -> increases -> bursting baseline throughput (why: larger hot footprint raises baseline) -Elastic throughput mode -> removes dependence on -> BurstCreditBalance (why: credit-based reasoning becomes misleading) -General Purpose mode -> constrained by -> I/O limit (why: `PercentIOLimit` signals limit proximity) -Max I/O mode -> trades -> higher latency for more parallelism tolerance (why: low-level latency expectations change) -Lifecycle management -> moves data from -> Standard to IA and Archive (why: storage mix shifts cost and latency posture) -Small cold files -> inflate -> IA and Archive overhead dimensions (why: `StorageBytes` meter can rise faster than actual bytes) -Regional file system -> replicates durability across -> multiple AZs (why: higher resilience baseline) -One Zone file system -> concentrates risk in -> single AZ (why: replication and backup posture matter more) -Replication configuration -> synchronizes source -> destination file system (why: DR depends on successful syncs) -TimeSinceLastSync -> indicates -> replica freshness debt (why: failover usefulness depends on this lag) -Mount helper -> emits -> client-side logs (why: service metrics do not explain mount failures) -CloudWatch Logs mount-status integration -> captures -> mount success or failure events (why: Stage 4 can route mount attempts) -context.env and context.team -> scope -> ownership and blast radius (why: filters should match oncall routing) -context.cloud.region -> separates -> regional fleet posture (why: EFS is regional, and replication may span regions) -context.filesystemid -> anchors -> per-file-system triage (why: safest entity key for widgets) -context.destinationfilesystemid -> anchors -> replication pair attribution (why: one source can have one or more replication relationships) - -### Entities and dimensions - -| Entity or dimension | Why useful | Cardinality risk | Safe top-N suggestion | Do NOT group-by guidance | -|---|---|---|---|---| -| `FileSystemId` | Primary EFS identity and safest drilldown axis | Low | 10 | Default group-by for almost every fleet widget | -| `DestinationFileSystemId` | Distinguishes replication pairs | Low | 10 | Use only on replication widgets | -| `context.env` | Separates prod and non-prod posture | Low | 6 | Must exist as a global filter | -| `context.team` | Maps ownership and routing | Low | 12 | Must exist as a global filter | -| `context.cloud.region` | Distinguishes regional fleets and DR topology | Low | 10 | Useful global filter and fleet split | -| `context.cloud.account.id` | Separates accounts or tenants | Low | 10 | Good optional global filter | -| `StorageClass` or equivalent | Required for `StorageBytes` Standard/IA/Archive analysis | Low | 6 | Do not assume exact key name until Stage 2 confirms it | -| Throughput mode | Changes whether burst credits matter | Low | 4 | Treat as optional metadata; do not make it a primary chart split unless confirmed | -| Performance mode | Explains `PercentIOLimit` relevance | Low | 3 | Optional metadata, not a first-line filter | -| File system type | Regional vs One Zone resilience posture | Low | 3 | Optional metadata, not a high-frequency split | -| Availability Zone | Useful only for One Zone inventory context | Medium | 6 | Avoid in core metric widgets unless emitted | -| Mount target id | Helpful for networking or mount-target diagnostics | Medium | 10 | Do not assume it exists on service metrics | -| Client host or pod | Useful for mount-failure logs, not service metrics | High | 10 | Do not use as a metric group-by by default | -| Access point id | Can isolate application tenancy patterns | Medium | 10 | Useful only if actually emitted; otherwise avoid | -| NFS operation or path | Would be powerful for app diagnosis | Very High | 0 | Do not expect on EFS service metrics | - -### Tsuga field mapping - -| Vendor/exporter dimension | Recommended context.* key | Must-exist vs optional | -|---|---|---| -| `FileSystemId` | `context.filesystemid` | Confirm in Stage 2 | -| `DestinationFileSystemId` | `context.destinationfilesystemid` | Unknown in this workspace; verify in Stage 3 only if replication metrics appear | -| Storage class dimension (`Total`, `Standard`, `IA`, `Archive`, overhead variants) | `context.storageclass` | Confirmed in Stage 2 | -| AWS Region | `context.cloud.region` | Confirmed in Stage 2 | -| AWS Account ID | `context.cloud.account.id` | Confirmed in Stage 2 | -| Performance mode | `context.performance_mode` | Optional | -| Throughput mode | `context.throughput_mode` | Optional | -| File system type (`Regional`, `One Zone`) | `context.file_system_type` | Optional | -| Org environment enrichment | `context.env` | Confirmed in Stage 2 | -| Org team enrichment | `context.team` | Confirmed in Stage 2 | - -### Confirmed by sources -- AWS documents `FileSystemId` as the core dimension for all EFS metrics and `DestinationFileSystemId` as the additional dimension for `TimeSinceLastSync`. [S2][S6] -- `StorageBytes` is emitted with storage-class dimensions that distinguish `Total`, `Standard`, `IA`, `IASizeOverhead`, `Archive`, and `ArchiveSizeOverhead`. [S2] - -### Best-practice inference -- Stage 2 confirmed flat lowercase keys in this workspace, including `context.filesystemid` and `context.storageclass`. -- Throughput mode and performance mode metadata were not discovered on live metrics, so those concepts stay in note text rather than dashboard filters. - -## Golden signals - -### Confirmed by sources -| Signal | What it means for AWS EFS | Typical degradation causes | Best telemetry sources | What people page on | Section questions | -|---|---|---|---|---|---| -| Traffic | Actual file-system throughput plus metadata intensity | App demand spike, backup/scan jobs, deploy-induced metadata storms | `TotalIOBytes`, `MeteredIOBytes`, `MetadataIOBytes`, `MetadataReadIOBytes`, `MetadataWriteIOBytes` [S2] | Throughput jumps or metadata share changes unexpectedly | Is traffic higher, and is it metadata-heavy? | -| Errors | EFS service metrics do not expose rich error counts; failures are usually seen as mount issues or app-side NFS errors | mount helper failures, network path issues, security groups, TLS setup, stale NFS clients | mount helper logs, mount attempt CloudWatch Logs, support logs [S7][S8][S9] | Mount attempts failing or clients unable to connect | Is this a service headroom issue or a client-side mount issue? | -| Latency | Best inferred from headroom and mode because EFS service metrics expose throughput and I/O limit more directly than per-op latency | hitting General Purpose I/O limit, credit depletion, Max I/O mode expectations, cold-storage access expectations | `PercentIOLimit`, `PermittedThroughput`, `BurstCreditBalance`, performance mode context [S2][S3] | `PercentIOLimit` pinned high with user-visible slowness | Is slowness caused by I/O limit pressure or workload shape? | -| Saturation | Headroom exhaustion in throughput or burst-credit runway | bursting credits draining, metered throughput near permitted ceiling, unexpected client fanout | `MeteredIOBytes`, `PermittedThroughput`, `BurstCreditBalance`, `ClientConnections` [S2][S3] | Throughput utilization sustained high or credit runway collapsing | Which limit is closest to failing right now? | - -### Best-practice inference -- `MeteredIOBytes` compared with `PermittedThroughput` is more actionable than raw byte totals alone because it encodes EFS read discounts and actual limit pressure. -- `MetadataIOBytes` is often the overlooked signal that explains "EFS is slow" reports when the app is doing directory-walks, file-stat storms, or lots of tiny-file operations. -- For EFS, logs and client telemetry are more important for "errors" than service metrics; an empty error widget would be misleading, so the dashboard should treat mount logs as Stage 4 scope instead. - -## Telemetry sources - -### Confirmed by sources -| Source type | How collected | What it provides | Pros/cons | Common pitfalls | -|---|---|---|---|---| -| EFS CloudWatch metrics | Native AWS/EFS namespace in CloudWatch | File-system throughput, I/O mix, storage bytes, client connections, headroom, replication lag | Best service-level baseline and always-on for core metrics | Does not expose application latency or server-side error counts [S1][S2] | -| CloudWatch storage-class dimensions | StorageBytes dimensions on AWS/EFS metrics | Standard, IA, Archive, and overhead storage mix | Best cost and lifecycle posture signal | Exporter may normalize storage-class dimension names unexpectedly [S2] | -| Performance/throughput configuration docs | AWS EFS config and performance metadata | Explains whether credits matter and whether `PercentIOLimit` is relevant | Essential for interpreting metrics correctly | These are inventory facts, not always emitted as metric labels [S3][S4] | -| Replication metrics | CloudWatch plus replication configuration | `TimeSinceLastSync` per source/destination pair | Clear DR freshness signal | Metric absent when replication is not configured [S2][S6] | -| Mount helper support logs | `amazon-efs-utils` logs on clients | TLS, watchdog, mount helper failures | Best source for client mount failures | Lives on clients, not the EFS service; requires log collection [S8][S9] | -| Mount attempt CloudWatch Logs | Optional `amazon-efs-utils` CloudWatch logging | Remote success/failure of mount attempts and mount status events | Good Stage 4 route candidate | Must be explicitly enabled in `efs-utils.conf` [S7] | - -### Best-practice inference -- In this workspace, `TimeSinceLastSync` is absent entirely, so replication freshness remains gated rather than shown as healthy-zero. -- "No data" on `BurstCreditBalance` can be legitimate if the file system uses Elastic or Provisioned throughput rather than Bursting. -- "No data" on storage-class splits may mean the exporter kept only the total storage series or the file system has not tiered data yet. - -## Log intelligence (Stage 4 handoff) - -### Confirmed by sources -1. **Log sources matrix** - -| Source | Access path | Typical format | Structured vs unstructured | Evidence | -|---|---|---|---|---| -| Mount helper support logs | `/var/log/amazon/efs` on client hosts | Plain-text helper, watchdog, and optional stunnel logs | Unstructured/semi-structured | [S8] | -| Mount attempt CloudWatch Logs | Configured in `efs-utils.conf`, default pattern `/aws/efs/utils/{fs_id}` when customized | Line-oriented mount success/failure events | Semi-structured | [S7] | -| EFS mount helper runtime | `mount.efs` via `amazon-efs-utils` | Client-side command and TLS/watchdog behavior | Unstructured | [S8][S9] | - -2. **Known log formats** -- **Mount helper support log**: line-oriented text in `/var/log/amazon/efs`; includes mount helper, watchdog, and optional stunnel log content. Good for TLS bootstrap, watchdog restart, and mount negotiation failures. [S8] -- **Mount attempt CloudWatch log**: success/failure notification stream created when CloudWatch logging is enabled in `efs-utils.conf`; useful for remote mount-attempt auditing without logging into instances. [S7] - -3. **Candidate query filters for Stage 4** -- Precise: `context.log_group:/aws/efs/utils/ AND context.filesystemid:*` - Rationale: targets opt-in mount-attempt log groups after normalization. - Risk: requires CloudWatch logging to be enabled and the log-group field to be preserved. -- Fallback: `(message:*amazon-efs-mount-watchdog* OR message:*mount.efs* OR message:*efs-utils*) AND context.team:*` - Rationale: catches client-side EFS helper logs even if log-group metadata is inconsistent. - Risk: may mix helper logs from multiple hosts and file systems. - -4. **Attribute mapping hints** - -| Raw field | Suggested Tsuga key | Confidence | Notes | -|---|---|---|---| -| file system id (`fs-...`) | `context.filesystemid` | High | Best join key between logs and metrics | -| destination file system id | `context.destinationfilesystemid` | Medium | Useful only if replication tooling logs it | -| mount target or DNS name | `context.mount_target` | Medium | Good for network-path failures | -| client hostname | `context.host.name` | Medium | Useful in logs, avoid in metric widgets | -| availability zone | `context.cloud.availability_zone` | Medium | Useful for One Zone mount troubleshooting | -| helper component (`mount helper`, `watchdog`, `stunnel`) | `context.component` | High | Supports split parsers and log filtering | -| mount status | `context.mount.status` | High | Useful derived status field for mount-attempt logs | -| error reason | `context.error.reason` | Medium | Best free-text classification target | - -5. **Parsing risks** -- Support logs are client-side and line-oriented; they are not guaranteed to include a stable JSON structure. -- Optional stunnel logging is disabled by default, so TLS details may be absent even when mounts fail. [S8] -- CloudWatch mount-attempt logs are opt-in and may not exist in the workspace at all. [S7] -- Filesystem id may appear in different forms across mount paths, helper arguments, or log group names. -- Multi-host aggregation can create noisy mixed streams unless file-system id and host identity are extracted. - -### Best-practice inference -- Stage 4 should prefer parsing helper and watchdog component names plus filesystem id first; these are lower-cardinality and more reusable than raw mount commands. -- If no EFS-specific logs are present in Tsuga, a route may still be useful for generic NFS mount-failure logs from EC2 or Kubernetes node agents. - -## Caveats and footguns -- **[fleet-health]** `PercentIOLimit` applies to the General Purpose performance mode; treat absence or low relevance carefully on Max I/O or unsupported surfaces. (S3) -- **[fleet-health]** `BurstCreditBalance` is meaningful only for Bursting throughput; Elastic throughput does not accrue or consume credits. (S3) -- **[fleet-health]** `PermittedThroughput` can exceed provisioned throughput when Standard storage size would allow more; do not assume it equals a configured provisioned number. (S2)(S3) -- **[fleet-health, throughput-mix]** Reads are discounted in EFS throughput accounting, so `MeteredIOBytes` and `TotalIOBytes` diverging is normal for read-heavy workloads. (S2)(S3) -- **[throughput-mix]** `MetadataIOBytes` can rise sharply during namespace scans, deploys that touch many files, or small-file storms; this does not necessarily mean user data throughput increased. (Inference) -- **[throughput-mix]** `SampleCount` on EFS byte metrics represents operation count, not bytes. If Tsuga preserves it, divide by seconds for ops/s rather than treating it as throughput. (S2) -- **[fleet-health]** `ClientConnections` only supports the `Sum` statistic in CloudWatch; averages across long windows need period-aware interpretation. (S2) -- **[fleet-health]** One standard client usually means one connection per mounted EC2 instance; high client count can reflect fanout, not necessarily unhealthy load. (S2) -- **[storage-lifecycle]** `StorageBytes` is emitted every 15 minutes, not every minute like most EFS metrics. Short time windows can make storage widgets look stale. (S2) -- **[storage-lifecycle]** IA and Archive metered size include 128 KiB minimum billing effects on small files; overhead growth can be a billing artifact, not actual data expansion. (S2)(S4) -- **[storage-lifecycle]** Archive is supported only with Elastic throughput; if Archive data exists, do not assume throughput mode can be switched freely. (S4) -- **[storage-lifecycle]** Cold-storage tiers have tens-of-milliseconds first-byte latency; rising IA or Archive share is not automatically bad, but it changes user latency expectations. (S3)(S4) -- **[replication-resilience]** `TimeSinceLastSync` measures elapsed time since the last successful sync, not a point-in-time-consistent replication lag guarantee. (S6) -- **[replication-resilience]** AWS states a 15-minute RPO for most file systems after initial sync, but very large or frequently changing file systems can exceed that. (S6) -- **[replication-resilience]** Absence of replication metrics usually means replication is not configured, not that lag is zero. (S2)(S6) -- **[fleet-health, replication-resilience]** One Zone file systems are not resilient to loss of the AZ; treat DR freshness and backup posture more seriously there. (S4) -- **[throughput-mix]** EFS does not expose per-directory, per-path, or per-user service metrics, so hot-subtree diagnosis requires client or application telemetry. (Inference) -- **[throughput-mix, fleet-health]** Application-visible slowness may come from NFS semantics, client-side caching, or mount/network issues that are invisible in EFS service metrics. (S5)(S8)(S9) -- **[replication-resilience]** Initial sync time depends on file count and file size; newly enabled replication can look unhealthy for a while without indicating ongoing replication failure. (S6) -- **[fleet-health]** Max I/O has inherently higher per-operation latency than General Purpose; do not compare those modes as if they share the same normal latency baseline. (S3) -- **[storage-lifecycle]** Standard storage growth can increase bursting baseline throughput, so rising storage is not always purely a cost negative. (S3) -- **[throughput-mix]** Read-heavy workloads can look healthier in metered throughput than in actual throughput because reads count at one-third rate. (S2)(S3) - -## Confirmed Tsuga prefixes -- `aws_efs_*` — **CONFIRMED** (10 metrics found in Tsuga via `tsuga_search_metrics.py '^aws_efs.*'` and validated with MCP `get_metric`) -- `aws.efs.*` — **UNKNOWN / RULED OUT** (no dot-normalized EFS family found in Tsuga) - -## Discovery status -- Discovery complete in Stage 2: 10 metrics confirmed, 3 expected AWS-documented metrics missing (`DataReadIOBytes`, `DataWriteIOBytes`, `TimeSinceLastSync`), 0 unexpected metrics found. - -## Top sources -- [S1] https://docs.aws.amazon.com/efs/latest/ug/monitoring-cloudwatch.html - Canonical entry point for EFS CloudWatch monitoring behavior and 1-minute metric cadence. -- [S2] https://docs.aws.amazon.com/efs/latest/ug/efs-metrics.html - Primary source for exact EFS metric names, dimensions, statistics, storage-class splits, and replication metric semantics. -- [S3] https://docs.aws.amazon.com/efs/latest/ug/performance.html - Explains throughput modes, performance modes, burst credits, and why `PercentIOLimit` and `PermittedThroughput` matter. -- [S4] https://docs.aws.amazon.com/efs/latest/ug/features.html - Defines Regional vs One Zone durability and storage-class behavior that shape dashboard interpretation. -- [S5] https://docs.aws.amazon.com/efs/latest/ug/efs-mount-helper.html - Grounds the client-side mount helper as a key operational surface for non-service-side failures. -- [S6] https://docs.aws.amazon.com/efs/latest/ug/efs-replication.html - Primary source for replication behavior, RPO language, and `TimeSinceLastSync` usage. -- [S7] https://docs.aws.amazon.com/efs/latest/ug/how-to-monitor-mount-status.html - Best source for mount-attempt CloudWatch Logs and opt-in log-group behavior. -- [S8] https://docs.aws.amazon.com/efs/latest/ug/mount-helper-logs.html - Primary source for support log location, helper/watchdog logging, and log retention caveats. -- [S9] https://docs.aws.amazon.com/efs/latest/ug/lifecycle-management-efs.html - Lifecycle policy source for Standard, IA, and Archive movement and cost-oriented storage posture. -- [S10] https://aws.amazon.com/efs/pricing/ - Pricing source for why `StorageBytes` and throughput/operation volume are the practical cost proxies. diff --git a/plugins/tsuga/skills/knowledge-technology/references/aws-efs/metrics.csv b/plugins/tsuga/skills/knowledge-technology/references/aws-efs/metrics.csv deleted file mode 100644 index 9388948..0000000 --- a/plugins/tsuga/skills/knowledge-technology/references/aws-efs/metrics.csv +++ /dev/null @@ -1,15 +0,0 @@ -theme,name,confirmed,definition,unit,type,tsuga_metric_name,widget_suggestions,aggregation,post_function,group_by,required_context_fields,optional_context_fields,absence_means,cardinality_footguns,variants,citations -Availability/Health,PercentIOLimit,true,Shows how close a General Purpose EFS file system is to reaching its I/O limit. Primary headroom signal when available.,%,summary,aws_efs_percent_io_limit,query-value|timeseries,max,none,context.filesystemid:10,context.env|context.team,context.cloud.region|context.cloud.account.id,File system is not General Purpose or exporter omitted the metric,Safe; group by file system only,cloudwatch=PercentIOLimit,https://docs.aws.amazon.com/efs/latest/ug/efs-metrics.html|https://docs.aws.amazon.com/efs/latest/ug/performance.html -Capacity/Saturation,BurstCreditBalance,true,Burst credits available to a bursting-mode file system. Falling balance means future throughput pressure even if current traffic still looks acceptable.,By,summary,aws_efs_burst_credit_balance,query-value|timeseries,min,none,context.filesystemid:10,context.env|context.team,context.cloud.region|context.cloud.account.id,File system is not using Bursting throughput or exporter omitted the metric,Safe,cloudwatch=BurstCreditBalance,https://docs.aws.amazon.com/efs/latest/ug/efs-metrics.html|https://docs.aws.amazon.com/efs/latest/ug/performance.html -Capacity/Saturation,PermittedThroughput,true,Maximum throughput the file system can currently drive. Use as the denominator for throughput utilization and to distinguish healthy load from throughput ceilings.,By/s,summary,aws_efs_permitted_throughput,query-value|timeseries|top-list,average,none,context.filesystemid:10,context.env|context.team,context.cloud.region|context.cloud.account.id,Exporter gap or file system missing from inventory,Safe,cloudwatch=PermittedThroughput,https://docs.aws.amazon.com/efs/latest/ug/efs-metrics.html|https://docs.aws.amazon.com/efs/latest/ug/performance.html -Throughput/Usage,MeteredIOBytes,true,Throughput-accounted bytes for all operations after EFS read-discount rules. Best numerator for throughput utilization math.,By,summary,aws_efs_metered_io_bytes,timeseries|query-value|top-list,sum,per-second,context.filesystemid:10,context.env|context.team,context.cloud.region|context.cloud.account.id,File system is idle or exporter omitted the metric,Safe,cloudwatch=MeteredIOBytes,https://docs.aws.amazon.com/efs/latest/ug/efs-metrics.html|https://docs.aws.amazon.com/efs/latest/ug/performance.html -Throughput/Usage,TotalIOBytes,true,Actual bytes processed for all EFS operations without read discounts. Useful for true workload volume and discount-ratio math.,By,summary,aws_efs_total_io_bytes,timeseries|query-value|top-list,sum,per-second,context.filesystemid:10,context.env|context.team,context.cloud.region|context.cloud.account.id,File system is idle or exporter omitted the metric,Safe,cloudwatch=TotalIOBytes,https://docs.aws.amazon.com/efs/latest/ug/efs-metrics.html -Throughput/Usage,MetadataIOBytes,true,Actual bytes consumed by metadata operations. High values often indicate namespace or tiny-file pressure.,By,summary,aws_efs_metadata_io_bytes,timeseries|query-value|top-list,sum,per-second,context.filesystemid:10,context.env|context.team,context.cloud.region|context.cloud.account.id,No metadata-heavy operations during the period,Safe,cloudwatch=MetadataIOBytes,https://docs.aws.amazon.com/efs/latest/ug/efs-metrics.html -Throughput/Usage,MetadataReadIOBytes,true,Actual bytes consumed by metadata read operations such as namespace scans and stats.,By,summary,aws_efs_metadata_read_io_bytes,timeseries|top-list,sum,per-second,context.filesystemid:10,context.env|context.team,context.cloud.region|context.cloud.account.id,No metadata reads during the period,Safe,cloudwatch=MetadataReadIOBytes,https://docs.aws.amazon.com/efs/latest/ug/efs-metrics.html -Throughput/Usage,MetadataWriteIOBytes,true,Actual bytes consumed by metadata write operations such as creates, deletes, or permission changes.,By,summary,aws_efs_metadata_write_io_bytes,timeseries|top-list,sum,per-second,context.filesystemid:10,context.env|context.team,context.cloud.region|context.cloud.account.id,No metadata writes during the period,Safe,cloudwatch=MetadataWriteIOBytes,https://docs.aws.amazon.com/efs/latest/ug/efs-metrics.html -Availability/Health,ClientConnections,true,Number of client connections to a file system. Good blast-radius signal for fleet fanout and mount churn.,{connection},summary,aws_efs_client_connections,query-value|timeseries|top-list,sum,none,context.filesystemid:10,context.env|context.team,context.cloud.region|context.cloud.account.id,No connected clients or exporter omitted the metric,Safe for file system; do not infer per-host health from this alone,cloudwatch=ClientConnections,https://docs.aws.amazon.com/efs/latest/ug/efs-metrics.html -Capacity/Saturation,StorageBytes,true,File system size across storage classes. Primary storage and cost proxy. Emitted every 15 minutes rather than every minute.,By,summary,aws_efs_storage_bytes,query-value|timeseries|bar,max,none,context.filesystemid:10|context.storageclass:6,context.env|context.team,context.cloud.region|context.cloud.account.id|context.storageclass,Storage-class dimension missing or file system missing from inventory,Safe on file system id and storageclass,cloudwatch=StorageBytes,https://docs.aws.amazon.com/efs/latest/ug/efs-metrics.html|https://docs.aws.amazon.com/efs/latest/ug/features.html -Throughput/Usage,DataReadIOBytes,false,MISSING — documented by AWS but not present in Tsuga for this workspace. Read/write demand must be inferred from TotalIOBytes, MeteredIOBytes, and metadata split instead.,By,summary,MISSING,N/A,N/A,N/A,context.env|context.team,context.cloud.region|context.cloud.account.id,Metric not exported by collector or omitted from workspace,N/A,cloudwatch=DataReadIOBytes,https://docs.aws.amazon.com/efs/latest/ug/efs-metrics.html -Throughput/Usage,DataWriteIOBytes,false,MISSING — documented by AWS but not present in Tsuga for this workspace. Write-heavy behavior must be inferred indirectly because the direct write series is absent.,By,summary,MISSING,N/A,N/A,N/A,context.env|context.team,context.cloud.region|context.cloud.account.id,Metric not exported by collector or omitted from workspace,N/A,cloudwatch=DataWriteIOBytes,https://docs.aws.amazon.com/efs/latest/ug/efs-metrics.html -Availability/Health,TimeSinceLastSync,false,MISSING — documented by AWS but not present in Tsuga for this workspace. Replication freshness must remain gated until the metric appears.,s,summary,MISSING,N/A,N/A,N/A,context.env|context.team,context.cloud.region|context.cloud.account.id,Replication not configured or metric not exported,N/A,cloudwatch=TimeSinceLastSync,https://docs.aws.amazon.com/efs/latest/ug/efs-metrics.html|https://docs.aws.amazon.com/efs/latest/ug/efs-replication.html -Cost,NOT_APPLICABLE,false,EFS does not expose a direct spend metric. Use StorageBytes by storage class as the primary cost proxy and MeteredIOBytes as a throughput-activity proxy. Detailed billing still belongs in AWS Cost Explorer and pricing views.,N/A,N/A,N/A,N/A,N/A,N/A,N/A,context.env|context.team,context.cloud.region|context.cloud.account.id,N/A,N/A,N/A,https://aws.amazon.com/efs/pricing/|https://docs.aws.amazon.com/efs/latest/ug/efs-metrics.html diff --git a/plugins/tsuga/skills/knowledge-technology/references/aws-efs/queries.md b/plugins/tsuga/skills/knowledge-technology/references/aws-efs/queries.md deleted file mode 100644 index 85f12b8..0000000 --- a/plugins/tsuga/skills/knowledge-technology/references/aws-efs/queries.md +++ /dev/null @@ -1,50 +0,0 @@ -# AWS EFS - -Shared NFS filesystem. Mounted by EC2 / ECS / EKS / Lambda. Healthy: throughput within limits, metadata / data IO balanced, burst credits positive, connection count reasonable. - -## Incident shapes - -- **Burst credit exhaustion** — `aws_efs_burst_credit_balance → 0` → IOPS throttled to baseline → latency cliff -- **IO limit hit** — `aws_efs_percent_io_limit = 100%` sustained (General Purpose mode) → consider Max I/O or provisioned throughput -- **Throughput ceiling** — metered throughput at `aws_efs_permitted_throughput` → reads/writes slow -- **Connection storm** — `aws_efs_client_connections` spikes (often Lambda cold-start storms) -- **Metadata-heavy workload** — `aws_efs_metadata_io_bytes` dominates → small-file or listdir pattern - -## Key metrics - -| Metric | Unit | Signal | -|---|---|---| -| `aws_efs_percent_io_limit` | % | GP-mode saturation (N/A in Max I/O mode) | -| `aws_efs_burst_credit_balance` | bytes | 0 = throttled to baseline | -| `aws_efs_permitted_throughput` | bytes/s | Current allowed throughput | -| `aws_efs_metered_io_bytes` | bytes | Total IO this interval | -| `aws_efs_total_io_bytes` | bytes | Incl. metadata ops | -| `By` / `MISSING` | bytes | User-data IO | -| `aws_efs_metadata_io_bytes` | bytes | Metadata ops (list, stat, mkdir) | -| `aws_efs_client_connections` | count | Open mounts | -| `aws_efs_storage_bytes` | bytes | Filesystem size (by class) | -| `MISSING` | seconds | Cross-region replication lag | - -## Derived signals - -- `aws_efs_permitted_throughput - aws_efs_metered_io_bytes` — throughput headroom. Zero = bound. -- `aws_efs_metadata_io_bytes / aws_efs_total_io_bytes` — metadata dominance. > 0.3 = small-files-heavy workload. -- `aws_efs_burst_credit_balance / aws_efs_metered_io_bytes` — seconds of runway at current rate; predict the cliff. - -## Log patterns - -EFS has no direct logs. Use NFS-mount + app + kernel logs: - -- `NFS server not responding` / `still trying` — mount hiccup -- `server X.X.X.X OK` — recovered -- `stale NFS file handle` — mount reset after failure -- `rpc.statd` issues — lock daemon -- `Permission denied` — IAM / POSIX - -## Gotchas - -- Burst throughput is based on filesystem size; a 1 GB filesystem has minimal baseline. Need provisioned throughput or a bigger dataset. -- `aws_efs_permitted_throughput` drop is often the visible part of burst-credit exhaustion, not a separate incident. -- Lambda + EFS spawns many short-lived connections; `aws_efs_client_connections` spikes during cold-start storms are usually benign. -- Metadata-heavy ops (`ls -R` on many files) exhaust burst credits fast even when total bytes are small. -- Cross-AZ NFS latency is 2-3 ms minimum; different-AZ consumers always see higher latency than same-AZ. diff --git a/plugins/tsuga/skills/knowledge-technology/references/aws-elasticache/README.md b/plugins/tsuga/skills/knowledge-technology/references/aws-elasticache/README.md deleted file mode 100644 index 4e148c8..0000000 --- a/plugins/tsuga/skills/knowledge-technology/references/aws-elasticache/README.md +++ /dev/null @@ -1,299 +0,0 @@ -# AWS ElastiCache Integration Context Bundle - -## Metadata -**Technology:** AWS ElastiCache -**Deployment:** managed -**Environment:** prod -**Persona:** SRE Dev and ops -**Telemetry preference:** mixed -**Integration scope:** core service only -**Primary use-case:** reliability and performance - -## How to use this bundle -- Use `01_aws-elasticache_metrics.csv` as the source of truth for dashboard-safe metric names, units, temporality assumptions, and safe aggregation patterns. -- Use `02_aws-elasticache_dashboard_plan.yaml` for the dashboard blueprint: sections, widgets, derived signals, explanation notes, triage chains, playbooks, and coverage intent. -- Use `03_aws-elasticache_state.yaml` for machine-readable status, assumptions, unknowns, and Stage 4 log intelligence handoff. -- Use `04_aws-elasticache_memory.md` for the readable Stage 1 handoff narrative and the Stage 2 verification order. -- Stage 2 will create `05_aws-elasticache_metric_catalog.csv` as the discovered Tsuga catalog for reconciliation, attribute validation, and coverage checks. -- Stage 4 should read this file's `Log intelligence (Stage 4 handoff)` section and `03_aws-elasticache_state.yaml` `log_intel` block first before designing any log route. - -## What it is and what "good" looks like -### Confirmed by sources -AWS ElastiCache is a managed in-memory caching service that supports Valkey, Redis OSS, and Memcached, with two deployment models: serverless and self-designed node-based clusters. Serverless gives a single endpoint and automatically scales memory, compute, and network capacity, while self-designed clusters expose explicit nodes, shards, replicas, and placement choices. [How ElastiCache works](https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/WhatIs.corecomponents.html), [ElastiCache components and features](https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/WhatIs.Components.html) - -For Valkey and Redis OSS, "good" means the cache is absorbing demand with stable read and write latency, high cache efficiency, enough headroom in engine CPU and memory, and low replica lag when replicas exist. AWS explicitly recommends watching CPU, engine CPU, swap, evictions, connections, memory, latency, replication, and traffic-management signals because these are the first indicators of impending saturation or instability. [Which Metrics Should I Monitor?](https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/CacheMetrics.WhichShouldIMonitor.html), [Metrics for Valkey and Redis OSS](https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/CacheMetrics.Redis.html), [Host-Level Metrics](https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/CacheMetrics.HostLevel.html) - -For Memcached, "good" means request and connection volume are stable, hit efficiency remains high enough for the workload, evictions are rare and explainable, and host CPU and memory are not being consumed by undersized nodes or hot keys. AWS documents Memcached-specific request, item, connection, and memory metrics at node level. [Metrics for Memcached](https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/CacheMetrics.Memcached.html), [Host-Level Metrics](https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/CacheMetrics.HostLevel.html) - -For serverless ElastiCache, "good" means data storage and ElastiCache Processing Units stay within configured limits, pre-scaling is used ahead of sharp demand spikes, and throttling or out-of-memory behaviors never appear in production because operators have enough warning from `BytesUsedForCache` and `ElastiCacheProcessingUnits`. [How ElastiCache works](https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/WhatIs.corecomponents.html), [Scaling ElastiCache](https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/Scaling.html) - -Top incident shape 1: cache inefficiency or memory pressure. Start in `ec-redis-efficiency` or `ec-memcached-efficiency` first to determine whether misses, evictions, or memory headroom changed before latency. Top incident shape 2: write interruption or replica instability in Valkey/Redis. Start in `ec-redis-resilience`. Top incident shape 3: serverless throttling or scaling-limit pressure. Start in `ec-serverless-capacity`. - -### Best-practice inference -The technology name is broader than a single metric family. A useful ElastiCache dashboard should treat Valkey/Redis OSS, Memcached, and serverless as explicit sub-surfaces rather than collapsing them into one undifferentiated cache view, because replication, log support, scaling behavior, and even the meaning of CPU pressure differ materially across them. - -For first-response dashboards, cache efficiency matters more than raw request volume. A traffic spike with stable hit rate and stable latency is often a healthy growth event; a flat request rate with falling hit rate or rising evictions is usually more urgent because it signals a working set or sizing problem. - -The dashboard should preserve ownership boundaries: host-level metrics answer whether the managed node is under OS pressure, engine metrics answer whether the cache engine is the bottleneck, and serverless ECPU metrics answer whether AWS-managed compute elasticity is nearing configured limits. - -## Key concepts -### Glossary -| Term | Definition | Operational meaning | Dashboard section | -|---|---|---|---| -| Serverless cache | ElastiCache deployment with automatic capacity management and a single endpoint | Headroom is tracked through storage and ECPU, not node counts | ec-serverless-capacity | -| Self-designed cluster | Node-based ElastiCache deployment where operators choose node size and count | Capacity planning and blast radius are explicit operator concerns | ec-fleet-health | -| Cache node | Smallest ElastiCache compute and memory unit | Main host-level failure and saturation boundary | ec-fleet-health | -| Shard | Valkey/Redis partition of data across node groups | Write scaling and hotspot analysis depend on shard distribution | ec-redis-resilience | -| Replication group | Valkey/Redis grouping of primaries and replicas | High availability and replica lag analysis anchor here | ec-redis-resilience | -| Primary node | Read/write Redis or Valkey node | Write latency and failover ownership start here | ec-redis-resilience | -| Read replica | Read-only replica following a primary asynchronously | Replica lag is the early warning for stale reads and failover risk | ec-redis-resilience | -| Configuration endpoint | Entry point clients use to discover topology | Connection failures here can affect the whole cache surface | ec-fleet-health | -| Reader endpoint | Redis endpoint that spreads reads across replicas | Useful when demand is read-heavy and replica lag matters | ec-redis-resilience | -| Auto Discovery | Memcached mechanism that lets clients learn current node endpoints | Client behavior during scale events depends on this being used correctly | ec-memcached-efficiency | -| Cache hit rate | Share of lookups served from cache instead of missing | Best top-line efficiency signal for caching value | ec-redis-efficiency | -| Eviction | Removal of non-expired items due to memory pressure | Strong sign that the working set no longer fits or TTL policy is wrong | ec-redis-efficiency | -| Freeable memory | OS memory that the host can reclaim | Low headroom often precedes instability before the engine reports it directly | ec-fleet-health | -| Engine CPU | CPU used by the Valkey/Redis engine thread | Better saturation indicator than host CPU on larger Redis nodes | ec-redis-efficiency | -| Host CPU | CPU consumed by the whole cache host | Important when background managed processes compete with the engine | ec-fleet-health | -| Successful read latency | Latency of successful read commands | Measures the user-facing cost of cache reads | ec-redis-efficiency | -| Successful write latency | Latency of successful write commands | Measures mutation path cost and persistence/replication stress | ec-redis-resilience | -| Replication lag | Delay between primary and replica applying updates | Direct indicator of stale-read and failover exposure | ec-redis-resilience | -| Traffic management active | Redis signal showing ElastiCache is shedding or shaping workload for stability | Indicates undersized nodes or bandwidth constraints | ec-redis-resilience | -| ECPU | ElastiCache Processing Unit for serverless compute billing and scaling | Main compute-intensity signal for serverless caches | ec-serverless-capacity | -| Data tiering | Redis/Valkey feature that stores colder data on SSD | Changes memory semantics and adds disk read/write signals | ec-redis-efficiency | -| Slow log | Redis/Valkey log stream of slow commands | Best source for query-pattern root cause after metrics show latency | ec-redis-resilience | -| Engine log | Redis/Valkey operational log stream | Best source for warnings, notices, and engine-side anomalies | ec-redis-resilience | -| SNS event notification | ElastiCache control-plane event stream | Best source for topology changes, replacements, and scaling events | ec-redis-resilience | - -### Concept Map -Application -> connects to -> cache endpoint (why: every cache incident starts with whether clients can still reach the cache) -Cache endpoint -> resolves to -> serverless proxy layer or node endpoints (why: connection behavior differs by deployment model) -Serverless proxy layer -> routes requests to -> backend cache nodes (why: client topology is hidden but scaling pressure still exists underneath) -Self-designed cluster -> contains -> cache nodes (why: host saturation and blast radius are node scoped) -Valkey/Redis cluster -> partitions data into -> shards (why: write scaling and hotspots are shard dependent) -Shard -> contains -> primary node (why: writes and failover ownership land on the primary) -Shard -> contains -> read replicas (why: read scale and replica lag depend on replica health) -Primary node -> asynchronously replicates to -> read replica (why: stale reads and failover safety depend on lag) -Memcached cluster -> partitions data across -> nodes (why: node loss or hot key imbalance can create partial misses) -Client auto discovery -> refreshes -> Memcached node map (why: scaling events are only safe if clients learn new nodes) -Working set size -> drives -> memory consumption (why: memory pressure is the root cause of many cache incidents) -Memory consumption -> triggers -> evictions when max memory is reached (why: eviction spikes usually mean cache value is falling) -Cache misses -> increase -> backend dependency load (why: cache degradation propagates into downstream databases) -Engine CPU -> constrains -> Redis command throughput (why: Redis is effectively single-threaded for engine work) -Host CPU -> includes -> engine work plus managed background processes (why: host overload can exist even when engine CPU is moderate) -TrafficManagementActive -> indicates -> node cannot process current incoming load safely (why: this is an undersizing symptom, not a healthy steady state) -Successful request latency -> reflects -> command execution cost (why: rising latency is the user-visible outcome of saturation) -Replication lag -> grows when -> primary or network cannot keep replicas caught up (why: read consistency and failover posture degrade) -Serverless ECPU -> grows with -> transferred data and compute cost per command (why: this is the serverless compute saturation and cost proxy) -Serverless storage limits -> cap -> retained data size (why: nearing limits can cause eviction or OOM behavior) -Slow log entries -> reveal -> expensive commands and hot access patterns (why: metrics show symptoms; logs show command-level cause) -Engine logs -> reveal -> warnings and operational events (why: useful for correlation when behavior changes abruptly) -SNS events -> reveal -> topology, scaling, and replacement changes (why: control-plane events explain sudden metric shifts) -Availability Zone placement -> affects -> resilience and failover blast radius (why: replicas in one AZ do not provide real HA) -Context filters -> map to -> environment, team, region, engine, cluster, replica group (why: triage needs safe ownership and topology slicing) - -### Entities and dimensions -#### Confirmed by sources -| Entity / dimension | Why useful | Cardinality risk | Safe top-N suggestion | -|---|---|---|---| -| CacheClusterId | Primary identity for node-based cluster telemetry and logs | Low to medium | Top 20 | -| CacheNodeId | Needed for host-level and log-level node troubleshooting | Medium | Top 20 | -| ReplicationGroupId | Primary Redis HA boundary | Low | Top 20 | -| NodeGroupId / shard | Needed for Redis hotspot and replication topology analysis | Medium | Top 20 | -| Engine | Separates Valkey, Redis OSS, and Memcached behavior | Very low | Top 5 | -| Tier | Distinguishes memory vs SSD for data-tiered Redis metrics | Very low | Top 2 | -| Availability Zone | Main blast-radius split for HA and placement | Low | Top 6 | -| Region | Required for multi-region or global datastore interpretation | Low | Top 10 | -| Serverless cache name | Primary identity for serverless telemetry | Low | Top 20 | -| Log type | Distinguishes slow log vs engine log | Very low | Top 5 | -| Command family | Useful for latency drilldowns if exported | Medium to high | Top 15 | -| Event name | Best control-plane change grouping | Medium | Top 20 | - -#### Best-practice inference -| Entity / dimension | Why useful | Cardinality risk | Safe top-N suggestion | -|---|---|---|---| -| context.team | Ownership routing | Low | Top 20 | -| context.env | Production vs staging split | Low | Top 10 | -| context.cloud.account.id | Multi-account blast-radius separation | Low | Top 20 | -| context.client.address | Useful for debugging abusive clients, but often too high cardinality | High | Do NOT group-by by default | -| key / command text | Useful in logs only, not metrics | Very high | Do NOT group-by | -| security group | Helpful for access incidents if present | Medium | Top 20 | - -### Tsuga field mapping -#### Confirmed by sources -| Vendor / exporter dimension | Recommended context.* key | Must-exist vs optional | -|---|---|---| -| CacheClusterId | `context.cacheclusterid` | Optional | -| CacheNodeId | `context.cachenodeid` | Optional | -| ReplicationGroupId | `context.replicationgroupid` | Optional | -| NodeGroupId / shard | `context.nodegroupid` | Optional | -| Availability Zone | `context.cloud.availability_zone` | Optional | -| Region | `context.cloud.region` | Optional | -| Engine | `context.engine` | Optional | -| Tier | `context.tier` | Optional | -| Log type | `context.log.type` | Optional | - -#### Best-practice inference -| Vendor / exporter dimension | Recommended context.* key | Must-exist vs optional | -|---|---|---| -| Serverless cache name | `context.serverless_cache.name` | Optional | -| Cluster / cache display name | `context.cache.name` | Optional | -| Account ID | `context.cloud.account.id` | Optional | -| Team | `context.team` | Must-exist | -| Environment | `context.env` | Must-exist | -| Service name | `context.service.name` | Optional | - -### Stage 2 discovery update -- The live Tsuga catalog confirms `context.env`, `context.team`, `context.cloud.region`, `context.cloud.account.id`, `context.cacheclusterid`, and `context.cachenodeid` across the discovered `aws_elasticache_*` family. -- `context.replicationgroupid`, `context.nodegroupid`, and `context.role` are present only on a small Redis-family subset, notably `aws_elasticache_engine_cpu_utilization` and the counted-for-evict memory/capacity metrics. -- `context.engine` and `context.serverless_cache.name` were not observed in the live metric catalog, so they should not be used as global filters in Stage 3. - -## Golden signals -### Confirmed by sources -| Signal | What it means for ElastiCache | Typical causes when degraded | Best telemetry sources | What people page on | Section questions | -|---|---|---|---|---|---| -| Traffic | Connections, commands, bytes, and serverless ECPU showing load shape | Release spike, hot key, backend fallback, uneven key distribution | `CurrConnections`, Memcached `NewConnections`, host network, `ElastiCacheProcessingUnits` | Sudden demand increase, throttling, traffic shape change | "Did demand change?" "Which engine or cache is hottest?" | -| Errors | Miss inflation, auth failures, error count, traffic management, OOM-like symptoms | Working set overflow, auth/config drift, undersized nodes, limit exhaustion | `CacheMisses`, `ErrorCount`, `AuthenticationFailures`, `TrafficManagementActive`, SNS events | Miss storm, throttling, write failures, auth failures | "Is the cache still returning value?" "Is the engine rejecting or shaping work?" | -| Latency | Read and write command cost as seen in successful command latency | CPU saturation, slow commands, replication pressure, hot keys | `SuccessfulReadRequestLatency`, `SuccessfulWriteRequestLatency`, slow log | Sustained latency increase on successful commands | "Are users paying more for cache operations?" | -| Saturation | CPU, memory, storage, replica lag, and ECPU/storage-limit pressure | Node undersizing, poor TTL policy, replication backlog, serverless max limits | `EngineCPUUtilization`, `CPUUtilization`, `FreeableMemory`, `DatabaseMemoryUsagePercentage`, `ReplicationLag`, `BytesUsedForCache`, `ElastiCacheProcessingUnits` | Headroom collapse, evictions, replica lag, throttling risk | "What resource is binding first?" "Is this recoverable by scale-out or only by design changes?" | - -### Best-practice inference -- For cache systems, traffic is only useful when paired with efficiency. High traffic with high hit rate is often healthy; moderate traffic with collapsing hit rate is usually the more operationally important signal. -- Replication lag belongs under saturation for Redis/Valkey, not just availability, because lag typically grows when primaries or networks are already overloaded. -- For serverless caches, ECPU is both a cost and performance signal. It should be treated as a leading indicator for throttling risk when explicit usage limits are configured. - -## Telemetry sources -### Confirmed by sources -| Source type | How collected | What it provides | Pros / cons | Common pitfalls | -|---|---|---|---|---| -| ElastiCache engine metrics for Valkey / Redis OSS | Native AWS/ElastiCache CloudWatch metrics, mostly derived from `INFO` | Cache efficiency, latency, CPU, memory, replication, traffic management | Richest engine-specific signal set / only valid for Redis-family caches | Some metrics are replica-only, data-tiering-only, or engine-version dependent [Metrics for Valkey and Redis OSS](https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/CacheMetrics.Redis.html) | -| ElastiCache engine metrics for Memcached | Native AWS/ElastiCache CloudWatch metrics from Memcached stats | Connections, request volume, hit/miss, evictions, item and memory behavior | Clean Memcached-specific request semantics / no replication semantics | `UnusedMemory` is not true free memory and evictions can still happen [Metrics for Memcached](https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/CacheMetrics.Memcached.html) | -| Host-level metrics | AWS/ElastiCache CloudWatch host metrics every 60s | CPU, freeable memory, network IO, CPU credit behavior | Cross-engine baseline / less specific than engine metrics | Host CPU and engine CPU mean different things on Redis-family caches [Host-Level Metrics](https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/CacheMetrics.HostLevel.html) | -| Serverless scaling metrics | CloudWatch serverless usage metrics | `BytesUsedForCache`, `ElastiCacheProcessingUnits`, limit proximity | Best serverless capacity picture / less topology detail than node-based clusters | Usage limits can create throttling or OOM-like behavior before infra auto-scales enough [Scaling ElastiCache](https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/Scaling.html), [How ElastiCache works](https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/WhatIs.corecomponents.html) | -| Log delivery | Redis/Valkey slow log or engine log to CloudWatch Logs or Firehose in JSON or TEXT | Slow commands, engine warnings, client origin, command details | Strong root-cause detail / supported only for Redis-family engines and certain versions | Slow log has finite retention and delivery may miss older entries if `slowlog-max-len` is too small [Log delivery](https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/Log_Delivery.html) | -| SNS event notifications | ElastiCache control-plane notifications to SNS | Node replacement, provisioning, scaling, security-group and parameter changes | Explains topology or config change causality / not continuous performance telemetry | Topic and region/account requirements can silently prevent expected notifications [Event Notifications and Amazon SNS](https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/ElastiCacheSNS.html) | - -### Best-practice inference -- "No data" can mean different things by surface: Redis replica lag may be absent because there are no replicas; slow logs may be absent because log delivery is disabled; serverless ECPU may be absent if the environment does not use serverless caches at all. -- Stage 2 should verify whether Tsuga exposes one generic `aws_elasticache_*` namespace or splits serverless and node-based surfaces more explicitly. - -## Log intelligence (Stage 4 handoff) -### Confirmed by sources -1. **Log sources matrix** - -| Source | Access path | Typical format | Structured vs unstructured | Evidence | -|---|---|---|---|---| -| Redis / Valkey slow log | CloudWatch Logs or Firehose via log delivery | JSON or TEXT slow-log entries | Structured or semi-structured | [Log delivery](https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/Log_Delivery.html) | -| Redis / Valkey engine log | CloudWatch Logs or Firehose via log delivery | JSON or TEXT engine log entries | Structured or semi-structured | [Log delivery](https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/Log_Delivery.html) | -| ElastiCache SNS events | SNS delivery from ElastiCache control plane | Event notification messages | Structured enough for event routing | [Event Notifications and Amazon SNS](https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/ElastiCacheSNS.html) | -| Memcached engine logs | Unknown in AWS-managed delivery docs | Unknown | Unknown | No equivalent Memcached log-delivery page found in primary docs reviewed | - -2. **Known log formats** -- `Redis/Valkey slow log` - - Sample shape: cluster id, node id, log entry id, unix timestamp, execution duration in microseconds, command, client address, and optional client name. - - Delimiter notes: can be emitted as JSON or TEXT. - - Timestamp pattern: Unix timestamp for slow-log event time. - - Quoting behavior: command text is sanitized so real key/value arguments are replaced rather than emitted verbatim. - - Optional fields: client name is only present if set. [Log delivery](https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/Log_Delivery.html) -- `Redis/Valkey engine log` - - Sample shape: cluster id, node id, log level, UTC timestamp, role, and message body. - - Timestamp pattern: `DD MMM YYYY hh:mm:ss.ms UTC`. - - Quoting behavior: plain text or JSON depending on delivery configuration. - - Optional fields: role and message content vary with engine state. [Log delivery](https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/Log_Delivery.html) - -3. **Candidate query filters for Stage 4** -- Precise: `context.service.name:"aws-elasticache" AND context.cacheclusterid:* AND (context.log.type:"slow-log" OR context.log.type:"engine-log")` - - Rationale: targets Redis/Valkey log delivery with both identity and log-type boundaries. - - Risk: assumes Tsuga already normalizes service name and log-type fields. -- Fallback: `"CacheClusterId" AND ("slowlog" OR "LogLevel" OR "ClientAddress")` - - Rationale: broad text search that can catch raw JSON or text delivery payloads. - - Risk: may mix different AWS log streams and misses Memcached entirely. - -4. **Attribute mapping hints** - -| Raw field | Suggested Tsuga key | Confidence | Notes | -|---|---|---|---| -| CacheClusterId | `context.cacheclusterid` | High | Explicitly documented in slow and engine log contents | -| CacheNodeId | `context.cachenodeid` | High | Explicitly documented in slow and engine log contents | -| Duration | `context.db.query.duration_us` | High | Slow log duration is in microseconds | -| Command | `context.db.operation` | Medium | Prefer operation family rather than full command text for low cardinality | -| ClientAddress | `context.client.address` | Medium | May need splitting into host and port | -| ClientName | `context.client.name` | Medium | Optional field | -| Log level | `context.level` | High | Engine log has documented levels | -| Role | `context.cache.role` | Medium | Useful for primary vs replica interpretation if present | -| Event name | `context.event.name` | Medium | For SNS event routes | - -5. **Parsing risks** -- Slow log delivery is only supported for Valkey 7.x+ and Redis OSS 6.0+; engine log delivery is only supported for Valkey 7.x+ and Redis OSS 6.2+. [Log delivery](https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/Log_Delivery.html) -- Slow log retrieval is bounded by `slowlog-max-len`, so a busy system can drop older slow-log entries before delivery. [Log delivery](https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/Log_Delivery.html) -- JSON and TEXT are both supported, so Stage 4 should confirm the actual sink format before committing to a parser. [Log delivery](https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/Log_Delivery.html) -- Memcached log delivery is not confirmed in the primary docs reviewed, so Stage 4 should not fabricate a Memcached parser path. -- SNS event messages and engine logs are operationally useful but semantically different; they should not share one generic parser without a split condition. - -### Best-practice inference -- Prefer routing Redis/Valkey slow logs separately from engine logs because their target attributes and cardinality budgets differ. -- If only SNS events are present, Stage 4 should build an event route rather than pretending command-level logs exist. -- Full command text should stay as raw log content or an optional low-priority attribute, not a default group-by key. - -## Caveats and footguns -- **[ec-fleet-health]** `CPUUtilization` and `EngineCPUUtilization` are not interchangeable for Valkey/Redis OSS; host CPU includes background managed processes while engine CPU tracks the cache engine thread. ([Host-Level Metrics](https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/CacheMetrics.HostLevel.html), [Metrics for Valkey and Redis OSS](https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/CacheMetrics.Redis.html)) -- **[ec-fleet-health]** On small Redis-family nodes, watching only engine CPU can hide host overload from background processes. ([Which Metrics Should I Monitor?](https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/CacheMetrics.WhichShouldIMonitor.html)) -- **[ec-fleet-health]** Host-level metrics are emitted per cache node every 60 seconds, so brief spikes can disappear in coarse windows. ([Host-Level Metrics](https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/CacheMetrics.HostLevel.html)) -- **[ec-fleet-health]** Multi-AZ for Valkey/Redis OSS only helps when replicas exist in different Availability Zones. ([Minimizing downtime in ElastiCache by using Multi-AZ with Valkey and Redis OSS](https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/AutoFailover.html)) -- **[ec-redis-efficiency]** `CacheHitRate` can fall because of misses, evictions, or expired keys; it does not by itself distinguish which one changed. ([Metrics for Valkey and Redis OSS](https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/CacheMetrics.Redis.html)) -- **[ec-redis-efficiency]** Data-tiered Redis metrics can carry a `Tier` dimension, so collapsing memory and SSD signals together can hide where pressure really lives. ([Metrics for Valkey and Redis OSS](https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/CacheMetrics.Redis.html)) -- **[ec-redis-efficiency]** `DatabaseMemoryUsagePercentage` and `DatabaseCapacityUsagePercentage` are not equivalent once data tiering is enabled. ([Metrics for Valkey and Redis OSS](https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/CacheMetrics.Redis.html)) -- **[ec-redis-efficiency]** A healthy cache can still show high `CurrConnections` because ElastiCache itself uses some monitoring connections. ([Metrics for Valkey and Redis OSS](https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/CacheMetrics.Redis.html)) -- **[ec-redis-efficiency]** `BytesUsedForCache` includes dataset, buffers, and other memory purposes, so it is broader than "live items only." ([Metrics for Valkey and Redis OSS](https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/CacheMetrics.Redis.html)) -- **[ec-redis-resilience]** `ReplicationLag` is replica-only; absence may mean there is no replica rather than that replication is healthy. ([Metrics for Valkey and Redis OSS](https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/CacheMetrics.Redis.html)) -- **[ec-redis-resilience]** `TrafficManagementActive` is a boolean symptom of undersizing or bandwidth stress, not a normal operating mode to ignore. ([Metrics for Valkey and Redis OSS](https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/CacheMetrics.Redis.html)) -- **[ec-redis-resilience]** Successful read and write latency track only successful commands; failing or throttled paths can be underrepresented. ([Metrics for Valkey and Redis OSS](https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/CacheMetrics.Redis.html)) -- **[ec-redis-resilience]** Background save activity can degrade performance and can be misread as generic latency regression if `SaveInProgress` is not checked. ([Metrics for Valkey and Redis OSS](https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/CacheMetrics.Redis.html)) -- **[ec-memcached-efficiency]** Memcached is multi-threaded, so CPU interpretation differs from Redis-family guidance. ([Which Metrics Should I Monitor?](https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/CacheMetrics.WhichShouldIMonitor.html)) -- **[ec-memcached-efficiency]** `UnusedMemory` is not true free memory; evictions can still occur while `UnusedMemory` is nonzero. ([Metrics for Memcached](https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/CacheMetrics.Memcached.html)) -- **[ec-memcached-efficiency]** Memcached has no Redis-style replica lag or failover metrics, so using Redis HA playbooks on Memcached is wrong. ([ElastiCache components and features](https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/WhatIs.Components.html)) -- **[ec-memcached-efficiency]** Memcached clusters repartition data when nodes are added or removed, so hit-rate changes around scale events may be expected. ([ElastiCache components and features](https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/WhatIs.Components.html)) -- **[ec-serverless-capacity]** Hitting a serverless data-storage maximum can trigger eviction or OOM behavior; hitting an ECPU maximum can throttle requests. ([Scaling ElastiCache](https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/Scaling.html)) -- **[ec-serverless-capacity]** Serverless pre-scaling changes may take up to 60 minutes to become available, so they are not instantaneous incident mitigations. ([Scaling ElastiCache](https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/Scaling.html)) -- **[ec-serverless-capacity]** Minimum serverless limits incur cost even if actual usage stays below them. ([Scaling ElastiCache](https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/Scaling.html)) -- **[ec-serverless-capacity]** Aggregate serverless scale can still be uneven per slot; skewed key distribution can create hot-slot symptoms before cache-wide metrics look extreme. ([Scaling ElastiCache](https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/Scaling.html)) -- **[ec-redis-resilience]** Slow and engine log delivery are Redis-family features with engine-version prerequisites, so a missing log stream may be a feature gap rather than an ingestion problem. ([Log delivery](https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/Log_Delivery.html)) -- **[ec-redis-resilience]** SNS notifications require same-account, same-region topics and do not work effectively with encrypted topics in the described configuration. ([Event Notifications and Amazon SNS](https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/ElastiCacheSNS.html)) - -## Confirmed Tsuga prefixes -- `aws_elasticache_*` — **CONFIRMED** (77 live metrics found in the last 24 hours via `tools/tsuga_search_metrics.py '^aws_elasticache_.*'` and catalog bootstrap) - -## Discovery status -Discovery complete for Stage 2 bootstrap and reconciliation: -- 77 live `aws_elasticache_*` metrics found in Tsuga over the last 24 hours. -- 32 metrics are now represented in `01_aws-elasticache_metrics.csv` and confirmed live. -- 10 Stage 1 metrics are missing from the live environment, all in the Memcached-specific or serverless-specific surface. -- The discovered environment is effectively Redis-family plus host-level ElastiCache telemetry; no Memcached get/hit metrics and no serverless ECPU/storage metrics were found. -- Spot-check aggregation against the raw scalar endpoint returned HTTP 400, so point-in-time scalar validation was inconclusive; the live catalog itself still proves recent presence because discovery used a 24-hour window. - -Remaining unknowns after discovery: -- Whether any Memcached or serverless metrics exist outside the last 24-hour window or in a different account / environment slice. -- Whether Redis/Valkey slow logs, engine logs, or only SNS events are available for Stage 4. -- Whether command-family latency summaries are operationally useful enough to promote into Stage 3 enrichment widgets. - -## Top sources -- https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/WhatIs.corecomponents.html - Why: best high-level explanation of deployment models, serverless behavior, and ECPU semantics. -- https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/WhatIs.Components.html - Why: grounds nodes, shards, replication groups, endpoints, and Memcached cluster behavior. -- https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/CacheMetrics.Redis.html - Why: source of truth for Valkey/Redis OSS metric names and meanings. -- https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/CacheMetrics.Memcached.html - Why: source of truth for Memcached metric names and meanings. -- https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/CacheMetrics.HostLevel.html - Why: defines cross-engine host CPU, memory, and network metrics and sampling cadence. -- https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/CacheMetrics.WhichShouldIMonitor.html - Why: AWS's operational prioritization of which metrics matter most. -- https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/Log_Delivery.html - Why: authoritative for Redis/Valkey slow log and engine log support, fields, and formats. -- https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/AutoFailover.html - Why: confirms Multi-AZ and replica prerequisites for Redis/Valkey HA. -- https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/ElastiCacheSNS.html - Why: authoritative event-notification source for operational changes and route ideas. -- https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/Scaling.html - Why: defines serverless scaling, ECPU and storage limits, throttling risk, and pre-scaling behavior. diff --git a/plugins/tsuga/skills/knowledge-technology/references/aws-elasticache/metrics.csv b/plugins/tsuga/skills/knowledge-technology/references/aws-elasticache/metrics.csv deleted file mode 100644 index 9def428..0000000 --- a/plugins/tsuga/skills/knowledge-technology/references/aws-elasticache/metrics.csv +++ /dev/null @@ -1,43 +0,0 @@ -theme,name,confirmed,definition,unit,type,tsuga_metric_name,widget_suggestions,aggregation,post_function,group_by,required_context_fields,optional_context_fields,absence_means,cardinality_footguns,variants,citations -Fleet/Health,CPUUtilization,true,"Host CPU usage for the cache node. Use for all engines, and especially for Memcached or smaller Redis-family nodes. [Inferred from Std]",percent,gauge,aws_elasticache_cpu_utilization,query-value|timeseries|top-list,max,none,context.cacheclusterid:20|context.cachenodeid:20,context.env|context.team,context.cloud.region,"Metric missing, wrong namespace, or the selected engine surface is not present in Tsuga","Prefer engine, cluster, or node grouping; do not assume client-level fields exist",cloudwatch=CPUUtilization|tsuga_guess=aws_elasticache_cpu_utilization,https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/CacheMetrics.HostLevel.html|https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/CacheMetrics.WhichShouldIMonitor.html -Fleet/Health,FreeableMemory,true,Reclaimable host memory and a leading headroom signal before cache instability. [Inferred from Std],By,gauge,aws_elasticache_freeable_memory,query-value|timeseries|top-list,min,none,context.cacheclusterid:20|context.cachenodeid:20,context.env|context.team,context.cloud.region,"Metric missing, wrong namespace, or unsupported surface in Tsuga",Node grouping is safe; avoid grouping by any raw client identity,cloudwatch=FreeableMemory|tsuga_guess=aws_elasticache_freeable_memory,https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/CacheMetrics.HostLevel.html -Fleet/Health,NetworkBytesIn,true,"Host-level inbound network bytes, useful when demand shape changes before cache-efficiency metrics move. [Inferred from Std]",By,counter,aws_elasticache_network_bytes_in,timeseries|top-list,sum,per-second,context.cacheclusterid:20|context.cachenodeid:20,context.env|context.team,context.cloud.region,"Metric missing, wrong namespace, or unsupported surface in Tsuga",Use node or cluster grouping only,cloudwatch=NetworkBytesIn|tsuga_guess=aws_elasticache_network_bytes_in,https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/CacheMetrics.HostLevel.html -Fleet/Health,NetworkBytesOut,true,"Host-level outbound network bytes, useful for egress-heavy cache patterns and replica or client fan-out. [Inferred from Std]",By,counter,aws_elasticache_network_bytes_out,timeseries|top-list,sum,per-second,context.cacheclusterid:20|context.cachenodeid:20,context.env|context.team,context.cloud.region,"Metric missing, wrong namespace, or unsupported surface in Tsuga",Use node or cluster grouping only,cloudwatch=NetworkBytesOut|tsuga_guess=aws_elasticache_network_bytes_out,https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/CacheMetrics.HostLevel.html -Redis/Valkey Efficiency,EngineCPUUtilization,true,CPU usage of the Valkey or Redis OSS engine thread and the most precise Redis-family saturation signal on larger nodes. [Inferred from Std],percent,gauge,aws_elasticache_engine_cpu_utilization,query-value|timeseries,max,none,context.replicationgroupid:20|context.cacheclusterid:20|context.cachenodeid:20,context.env|context.team,context.cloud.region|context.nodegroupid,,Do not compare blindly across engines; this is Redis-family only,cloudwatch=EngineCPUUtilization|tsuga_guess=aws_elasticache_engine_cpu_utilization,https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/CacheMetrics.Redis.html|https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/CacheMetrics.WhichShouldIMonitor.html -Redis/Valkey Efficiency,CacheHitRate,true,"Cache efficiency ratio for Valkey or Redis OSS. Lower values often indicate evictions, expirations, or missing keys. [Inferred from Std]",percent,gauge,aws_elasticache_cache_hit_rate,query-value|timeseries,max,none,context.cacheclusterid:20,context.env|context.team,context.cloud.region,,Use cluster or shard grouping; avoid over-grouping by node when the question is cache value,cloudwatch=CacheHitRate|tsuga_guess=aws_elasticache_cache_hit_rate,https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/CacheMetrics.Redis.html -Redis/Valkey Efficiency,CacheHits,true,Successful read-only key lookups in the main dictionary. [Inferred from Std],count,counter,aws_elasticache_cache_hits,timeseries|query-value|top-list,sum,per-second,context.cacheclusterid:20,context.env|context.team,context.cloud.region,"No Redis-family traffic, wrong namespace, or no Redis-family telemetry present",Useful as numerator in hit-ratio and cache-value formulas,cloudwatch=CacheHits|tsuga_guess=aws_elasticache_cache_hits,https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/CacheMetrics.Redis.html -Redis/Valkey Efficiency,CacheMisses,true,Unsuccessful read-only key lookups in the main dictionary. [Inferred from Std],count,counter,aws_elasticache_cache_misses,timeseries|query-value|top-list,sum,per-second,context.cacheclusterid:20,context.env|context.team,context.cloud.region,"No Redis-family traffic, wrong namespace, or no Redis-family telemetry present",Misses are high-signal when shown as a ratio or paired with backend pressure,cloudwatch=CacheMisses|tsuga_guess=aws_elasticache_cache_misses,https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/CacheMetrics.Redis.html -Redis/Valkey Efficiency,BytesUsedForCache_Redis,true,"Total bytes allocated by Valkey or Redis OSS for dataset, buffers, and other cache purposes. [Inferred from Std]",By,gauge,aws_elasticache_bytes_used_for_cache,query-value|timeseries,max,none,context.cacheclusterid:20,context.env|context.team,context.cloud.region,,Tier splits are useful when data tiering is enabled,cloudwatch=BytesUsedForCache|tsuga_guess=aws_elasticache_bytes_used_for_cache,https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/CacheMetrics.Redis.html -Redis/Valkey Efficiency,DatabaseMemoryUsagePercentage,true,Percent of memory used by the Redis-family cluster. [Inferred from Std],percent,gauge,aws_elasticache_database_memory_usage_percentage,query-value|timeseries,max,none,context.cacheclusterid:20,context.env|context.team,context.cloud.region,,Do not confuse with total capacity percentage on data-tiered clusters,cloudwatch=DatabaseMemoryUsagePercentage|tsuga_guess=aws_elasticache_database_memory_usage_percentage,https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/CacheMetrics.Redis.html -Redis/Valkey Efficiency,DatabaseCapacityUsagePercentage,true,"Percent of total Redis-family data capacity in use, including SSD on tiered clusters. [Inferred from Std]",percent,gauge,aws_elasticache_database_capacity_usage_percentage,query-value|timeseries,max,none,context.cacheclusterid:20,context.env|context.team,context.cloud.region,,Most useful where data tiering exists; otherwise can overlap with memory-usage percentage,cloudwatch=DatabaseCapacityUsagePercentage|tsuga_guess=aws_elasticache_database_capacity_usage_percentage,https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/CacheMetrics.Redis.html -Redis/Valkey Efficiency,Evictions_Redis,true,Number of keys evicted because the Redis-family cache hit maxmemory. [Inferred from Std],count,counter,aws_elasticache_evictions,timeseries|query-value|top-list,sum,per-second,context.cacheclusterid:20,context.env|context.team,context.cloud.region,"No evictions, wrong namespace, or no Redis-family telemetry present",Show as rate or grouped hotspot chart; do not average away,cloudwatch=Evictions|tsuga_guess=aws_elasticache_evictions,https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/CacheMetrics.Redis.html|https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/CacheMetrics.WhichShouldIMonitor.html -Redis/Valkey Efficiency,CurrConnections_Redis,true,Number of client connections excluding replica connections for Valkey or Redis OSS. [Inferred from Std],count,gauge,aws_elasticache_curr_connections,query-value|timeseries,max,none,context.cacheclusterid:20|context.cachenodeid:20,context.env|context.team,context.cloud.region,,"Some monitoring connections are expected; interpret trends, not absolute purity",cloudwatch=CurrConnections|tsuga_guess=aws_elasticache_curr_connections,https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/CacheMetrics.Redis.html -Redis/Valkey Resilience,SuccessfulReadRequestLatency,true,Latency of successful read requests in Redis-family caches. [Inferred from Std],us,gauge,aws_elasticache_successful_read_request_latency,query-value|timeseries|top-list,average,none,context.cacheclusterid:20|context.cachenodeid:20,context.env|context.team,context.cloud.region,"Metric missing, wrong namespace, or no Redis-family telemetry present",Prefer percentiles if supported live; do not combine reads and writes into one chart,cloudwatch=SuccessfulReadRequestLatency|tsuga_guess=aws_elasticache_successful_read_request_latency,https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/CacheMetrics.Redis.html -Redis/Valkey Resilience,SuccessfulWriteRequestLatency,true,Latency of successful write requests in Redis-family caches. [Inferred from Std],us,gauge,aws_elasticache_successful_write_request_latency,query-value|timeseries|top-list,average,none,context.cacheclusterid:20|context.cachenodeid:20,context.env|context.team,context.cloud.region,"Metric missing, wrong namespace, or no Redis-family telemetry present",Prefer percentiles if supported live; keep separate from read latency,cloudwatch=SuccessfulWriteRequestLatency|tsuga_guess=aws_elasticache_successful_write_request_latency,https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/CacheMetrics.Redis.html -Redis/Valkey Resilience,ReplicationLag,true,Replica lag behind the primary node. Replica-only metric and key failover-readiness signal. [Inferred from Std],s,gauge,aws_elasticache_replication_lag,query-value|timeseries|top-list,max,none,context.cachenodeid:20,context.env|context.team,context.cloud.region,"No replicas, wrong namespace, or no Redis-family telemetry present",Replica-only metric; absence is ambiguous until topology is confirmed,cloudwatch=ReplicationLag|tsuga_guess=aws_elasticache_replication_lag,https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/CacheMetrics.Redis.html -Redis/Valkey Resilience,TrafficManagementActive,true,Boolean signal that ElastiCache is actively managing traffic because the node cannot process all incoming work safely. [Inferred from Std],bool,gauge,aws_elasticache_traffic_management_active,query-value|timeseries,max,none,context.cacheclusterid:20|context.cachenodeid:20,context.env|context.team,context.cloud.region,,Keep as boolean/state visual; do not normalize as a rate,cloudwatch=TrafficManagementActive|tsuga_guess=aws_elasticache_traffic_management_active,https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/CacheMetrics.Redis.html -Redis/Valkey Resilience,ErrorCount,true,Total number of failed Redis-family commands during the period. [Inferred from Std],count,counter,aws_elasticache_error_count,timeseries|query-value|top-list,sum,per-second,context.cacheclusterid:20|context.cachenodeid:20,context.env|context.team,context.cloud.region,"No command failures, wrong namespace, or no Redis-family telemetry present",Pair with latency or traffic-management state to avoid overreacting to tiny counts,cloudwatch=ErrorCount|tsuga_guess=aws_elasticache_error_count,https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/CacheMetrics.Redis.html -Redis/Valkey Resilience,AuthenticationFailures,true,Failed AUTH attempts for Valkey or Redis OSS and a strong security signal. [Inferred from Std],count,counter,aws_elasticache_authentication_failures,timeseries|query-value|top-list,sum,per-second,context.cacheclusterid:20|context.cachenodeid:20,context.env|context.team,context.cloud.region,"No auth failures, wrong namespace, or no Redis-family telemetry present",Likely low-volume but high-value; avoid noisy groupings,cloudwatch=AuthenticationFailures|tsuga_guess=aws_elasticache_authentication_failures,https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/CacheMetrics.Redis.html -Memcached Efficiency,CurrConnections_Memcached,false,"Current Memcached connections, including expected ElastiCache and internal connections. [Inferred from Std]",count,gauge,aws_elasticache_curr_connections_memcached,query-value|timeseries,max,none,context.cacheclusterid:20|context.cachenodeid:20,context.env|context.team,context.cloud.region|context.cloud.availability_zone,Metric not present in the live Tsuga catalog for this environment,Interpret trend rather than assuming all connections are client traffic,cloudwatch=CurrConnections|tsuga_guess=aws_elasticache_curr_connections_memcached,https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/CacheMetrics.Memcached.html -Memcached Efficiency,GetHits,false,Memcached get requests where the key was found. [Inferred from Std],count,counter,aws_elasticache_get_hits,timeseries|query-value|top-list,sum,per-second,context.cacheclusterid:20|context.cachenodeid:20,context.env|context.team,context.cloud.region,Metric not present in the live Tsuga catalog for this environment,Best used in a hit-rate formula with GetMisses,cloudwatch=GetHits|tsuga_guess=aws_elasticache_get_hits,https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/CacheMetrics.Memcached.html -Memcached Efficiency,GetMisses,false,Memcached get requests where the key was not found. [Inferred from Std],count,counter,aws_elasticache_get_misses,timeseries|query-value|top-list,sum,per-second,context.cacheclusterid:20|context.cachenodeid:20,context.env|context.team,context.cloud.region,Metric not present in the live Tsuga catalog for this environment,Best used in a hit-rate formula with GetHits,cloudwatch=GetMisses|tsuga_guess=aws_elasticache_get_misses,https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/CacheMetrics.Memcached.html -Memcached Efficiency,Evictions_Memcached,false,Non-expired Memcached items evicted to make space for new writes. [Inferred from Std],count,counter,aws_elasticache_evictions_memcached,timeseries|query-value|top-list,sum,per-second,context.cacheclusterid:20|context.cachenodeid:20,context.env|context.team,context.cloud.region,Metric not present in the live Tsuga catalog for this environment,Do not average away; evictions are a direct memory-fit symptom,cloudwatch=Evictions|tsuga_guess=aws_elasticache_evictions_memcached,https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/CacheMetrics.Memcached.html -Memcached Efficiency,NewConnections,true,New Memcached connections received during the period. [Inferred from Std],count,counter,aws_elasticache_new_connections,timeseries|query-value|top-list,sum,per-second,context.cacheclusterid:20|context.cachenodeid:20,context.env|context.team,context.cloud.region,"No Memcached traffic, wrong namespace, or no Memcached telemetry present",Good denominator for connection churn or instability analysis,cloudwatch=NewConnections|tsuga_guess=aws_elasticache_new_connections,https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/CacheMetrics.Memcached.html -Memcached Efficiency,BytesReadIntoMemcached,false,Bytes read from the network by the Memcached cache node. [Inferred from Std],By,counter,aws_elasticache_bytes_read_into_memcached,timeseries|top-list,sum,per-second,context.cacheclusterid:20|context.cachenodeid:20,context.env|context.team,context.cloud.region,Metric not present in the live Tsuga catalog for this environment,Node or cluster grouping only,cloudwatch=BytesReadIntoMemcached|tsuga_guess=aws_elasticache_bytes_read_into_memcached,https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/CacheMetrics.Memcached.html -Memcached Efficiency,BytesWrittenOutFromMemcached,false,Bytes written to the network by the Memcached cache node. [Inferred from Std],By,counter,aws_elasticache_bytes_written_out_from_memcached,timeseries|top-list,sum,per-second,context.cacheclusterid:20|context.cachenodeid:20,context.env|context.team,context.cloud.region,Metric not present in the live Tsuga catalog for this environment,Node or cluster grouping only,cloudwatch=BytesWrittenOutFromMemcached|tsuga_guess=aws_elasticache_bytes_written_out_from_memcached,https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/CacheMetrics.Memcached.html -Memcached Efficiency,UnusedMemory,false,"Memcached memory not used by data, but not necessarily available for more data because overhead still consumes memory. [Inferred from Std]",By,gauge,aws_elasticache_unused_memory,query-value|timeseries,min,none,context.cacheclusterid:20|context.cachenodeid:20,context.env|context.team,context.cloud.region,Metric not present in the live Tsuga catalog for this environment,Do not interpret as true allocatable free space,cloudwatch=UnusedMemory|tsuga_guess=aws_elasticache_unused_memory,https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/CacheMetrics.Memcached.html -Memcached Efficiency,CurrItems,false,Current number of Memcached items stored in cache. [Inferred from Std],count,gauge,aws_elasticache_curr_items_memcached,query-value|timeseries,max,none,context.cacheclusterid:20|context.cachenodeid:20,context.env|context.team,context.cloud.region,Metric not present in the live Tsuga catalog for this environment,Use cluster or node grouping only,cloudwatch=CurrItems|tsuga_guess=aws_elasticache_curr_items_memcached,https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/CacheMetrics.Memcached.html -Serverless Capacity,ElastiCacheProcessingUnits,false,Serverless compute-consumption metric tracking ECPU usage driven by CPU time and data transferred. [Inferred from Std],ecpu/s,gauge,aws_elasticache_processing_units,query-value|timeseries|top-list,max,none,context.engine:5|context.serverless_cache.name:20,context.env|context.team,context.cloud.region,Metric not present in the live Tsuga catalog for this environment,Most useful per serverless cache or engine; avoid mixing with node-based capacity charts,cloudwatch=ElastiCacheProcessingUnits|tsuga_guess=aws_elasticache_processing_units,https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/WhatIs.corecomponents.html|https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/Scaling.html -Serverless Capacity,BytesUsedForCache_Serverless,false,Serverless data-storage metric and main proxy for stored-data growth. [Inferred from Std],By,gauge,aws_elasticache_bytes_used_for_cache_serverless,query-value|timeseries,max,none,context.engine:5|context.serverless_cache.name:20,context.env|context.team,context.cloud.region,Metric not present in the live Tsuga catalog for this environment,May overlap in name with Redis-family metric; Stage 3 should keep it gated unless live serverless telemetry appears,cloudwatch=BytesUsedForCache|tsuga_guess=aws_elasticache_bytes_used_for_cache_serverless,https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/Scaling.html|https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/WhatIs.corecomponents.html -Ops/Security,SaveInProgress,true,Binary metric showing a background save is currently running on Redis-family nodes. [Inferred from Std],bool,gauge,aws_elasticache_save_in_progress,query-value|timeseries,max,none,context.cacheclusterid:20|context.cachenodeid:20,context.env|context.team,context.cloud.region,,"Use as an annotation-like operational clue, not as a throughput KPI",cloudwatch=SaveInProgress|tsuga_guess=aws_elasticache_save_in_progress,https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/CacheMetrics.Redis.html -Fleet/Health,CPUCreditBalance,true,Remaining CPU credits on burstable ElastiCache nodes and a key signal for T-class headroom collapse.,credits,gauge,aws_elasticache_cpu_credit_balance,query-value|timeseries|top-list,min,none,context.cacheclusterid:20|context.cachenodeid:20,context.env|context.team|context.cacheclusterid,context.cachenodeid|context.cloud.region|context.cloud.account.id,Node is not burstable or metric not present in the live Tsuga catalog,Useful only on burstable node families; do not generalize across all clusters,cloudwatch=CPUCreditBalance|tsuga=aws_elasticache_cpu_credit_balance,https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/CacheMetrics.HostLevel.html -Fleet/Health,CPUCreditUsage,true,"CPU credits consumed by burstable ElastiCache nodes, which rise when sustained CPU demand exceeds the baseline allowance.",credits,counter,aws_elasticache_cpu_credit_usage,query-value|timeseries|top-list,sum,per-second,context.cacheclusterid:20|context.cachenodeid:20,context.env|context.team|context.cacheclusterid,context.cachenodeid|context.cloud.region|context.cloud.account.id,Node is not burstable or metric not present in the live Tsuga catalog,"Useful only on burstable node families; treat as a rate, not a level",cloudwatch=CPUCreditUsage|tsuga=aws_elasticache_cpu_credit_usage,https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/CacheMetrics.HostLevel.html -Redis/Valkey Efficiency,ActiveDefragHits,true,Active defragmentation reallocations on Redis-family nodes and a signal that the engine is reclaiming fragmented memory.,count,counter,aws_elasticache_active_defrag_hits,timeseries|query-value|top-list,sum,per-second,context.cacheclusterid:20|context.cachenodeid:20,context.env|context.team|context.cacheclusterid,context.cachenodeid|context.cloud.region|context.cloud.account.id,"Feature disabled, unsupported engine version, or metric not present in the live Tsuga catalog",Interpret with memory fragmentation; raw totals alone are not a problem statement,cloudwatch=ActiveDefragHits|tsuga=aws_elasticache_active_defrag_hits,https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/CacheMetrics.Redis.html -Redis/Valkey Efficiency,DatabaseMemoryUsageCountedForEvictPercentage,true,Percent of Redis-family memory counted toward eviction and therefore closer to practical maxmemory pressure than broad memory usage alone.,percent,gauge,aws_elasticache_database_memory_usage_counted_for_evict_percentage,query-value|timeseries|top-list,max,none,context.replicationgroupid:20|context.nodegroupid:20,context.env|context.team,context.replicationgroupid|context.nodegroupid|context.cloud.region|context.cloud.account.id,Metric not present in the live Tsuga catalog for this environment,"Best interpreted with memory usage and evictions, not by itself",cloudwatch=DatabaseMemoryUsageCountedForEvictPercentage|tsuga=aws_elasticache_database_memory_usage_counted_for_evict_percentage,https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/CacheMetrics.Redis.html -Redis/Valkey Efficiency,MemoryFragmentationRatio,true,Redis-family allocator fragmentation ratio showing how much extra memory overhead exists beyond the active dataset.,ratio,gauge,aws_elasticache_memory_fragmentation_ratio,query-value|timeseries|top-list,max,none,context.cacheclusterid:20|context.cachenodeid:20,context.env|context.team|context.cacheclusterid,context.cachenodeid|context.cloud.region|context.cloud.account.id,Metric not present in the live Tsuga catalog for this environment,High values need memory context; fragmentation alone does not prove user-visible impact,cloudwatch=MemoryFragmentationRatio|tsuga=aws_elasticache_memory_fragmentation_ratio,https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/CacheMetrics.Redis.html -Redis/Valkey Resilience,MasterLinkHealthStatus,true,Replica link health status for Redis-family topology and a direct indicator of whether a follower is cleanly connected to its primary.,status,gauge,aws_elasticache_master_link_health_status,query-value|timeseries|top-list,min,none,context.cacheclusterid:20|context.cachenodeid:20,context.env|context.team|context.cacheclusterid,context.cachenodeid|context.cloud.region|context.cloud.account.id,"No replicas, unsupported topology, or metric not present in the live Tsuga catalog",Replica-only metric; absence is ambiguous without topology context,cloudwatch=MasterLinkHealthStatus|tsuga=aws_elasticache_master_link_health_status,https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/CacheMetrics.Redis.html -Redis/Valkey Resilience,ReplicationBytes,true,Bytes replicated from Redis-family primaries to replicas and a direct measure of replication write volume.,By,counter,aws_elasticache_replication_bytes,timeseries|query-value|top-list,sum,per-second,context.cacheclusterid:20|context.cachenodeid:20,context.env|context.team|context.cacheclusterid,context.cachenodeid|context.cloud.region|context.cloud.account.id,"No replicas, unsupported topology, or metric not present in the live Tsuga catalog",Best read with replication lag so throughput is not mistaken for distress,cloudwatch=ReplicationBytes|tsuga=aws_elasticache_replication_bytes,https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/CacheMetrics.Redis.html -Fleet/Health,NetworkBandwidthInAllowanceExceeded,true,"Events where inbound network bandwidth allowance is exceeded on an ElastiCache node, indicating host-network saturation on ingress.",count,counter,aws_elasticache_network_bandwidth_in_allowance_exceeded,timeseries|query-value|top-list,sum,per-second,context.cacheclusterid:20|context.cachenodeid:20,context.env|context.team|context.cacheclusterid,context.cachenodeid|context.cloud.region|context.cloud.account.id,No allowance breaches or metric not present in the live Tsuga catalog,Use with host and traffic context; sparse counters should not dominate the overview alone,cloudwatch=NetworkBandwidthInAllowanceExceeded|tsuga=aws_elasticache_network_bandwidth_in_allowance_exceeded,https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/CacheMetrics.HostLevel.html -Fleet/Health,NetworkBandwidthOutAllowanceExceeded,true,"Events where outbound network bandwidth allowance is exceeded on an ElastiCache node, indicating host-network saturation on egress.",count,counter,aws_elasticache_network_bandwidth_out_allowance_exceeded,timeseries|query-value|top-list,sum,per-second,context.cacheclusterid:20|context.cachenodeid:20,context.env|context.team|context.cacheclusterid,context.cachenodeid|context.cloud.region|context.cloud.account.id,No allowance breaches or metric not present in the live Tsuga catalog,Use with host and traffic context; sparse counters should not dominate the overview alone,cloudwatch=NetworkBandwidthOutAllowanceExceeded|tsuga=aws_elasticache_network_bandwidth_out_allowance_exceeded,https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/CacheMetrics.HostLevel.html -Fleet/Health,NetworkConntrackAllowanceExceeded,true,"Events where an ElastiCache node exceeds connection-tracking allowance, indicating too much concurrent connection state for the host network budget.",count,counter,aws_elasticache_network_conntrack_allowance_exceeded,timeseries|query-value|top-list,sum,per-second,context.cacheclusterid:20|context.cachenodeid:20,context.env|context.team|context.cacheclusterid,context.cachenodeid|context.cloud.region|context.cloud.account.id,No conntrack breaches or metric not present in the live Tsuga catalog,Pair with connection counts; this is a saturation symptom rather than normal traffic accounting,cloudwatch=NetworkConntrackAllowanceExceeded|tsuga=aws_elasticache_network_conntrack_allowance_exceeded,https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/CacheMetrics.HostLevel.html -Fleet/Health,NetworkPacketsPerSecondAllowanceExceeded,true,"Events where an ElastiCache node exceeds packet-rate allowance, indicating packet-heavy traffic is outgrowing host network capacity.",count,counter,aws_elasticache_network_packets_per_second_allowance_exceeded,timeseries|query-value|top-list,sum,per-second,context.cacheclusterid:20|context.cachenodeid:20,context.env|context.team|context.cacheclusterid,context.cachenodeid|context.cloud.region|context.cloud.account.id,No packet-rate breaches or metric not present in the live Tsuga catalog,Most useful when packet counts or connection churn indicate many small requests,cloudwatch=NetworkPacketsPerSecondAllowanceExceeded|tsuga=aws_elasticache_network_packets_per_second_allowance_exceeded,https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/CacheMetrics.HostLevel.html diff --git a/plugins/tsuga/skills/knowledge-technology/references/aws-elasticache/queries.md b/plugins/tsuga/skills/knowledge-technology/references/aws-elasticache/queries.md deleted file mode 100644 index 5d9e473..0000000 --- a/plugins/tsuga/skills/knowledge-technology/references/aws-elasticache/queries.md +++ /dev/null @@ -1,54 +0,0 @@ -# AWS ElastiCache - -Managed Redis or Memcached. Engine-layer = `redis.md`. This file covers AWS-managed-service signals. - -## Incident shapes - -- **Engine CPU saturation** — `aws_elasticache_engine_cpu_utilization` (not `aws_elasticache_cpu_utilization`) > 90% sustained → command-shape issue -- **Memory pressure / eviction** — `aws_elasticache_database_memory_usage_percentage` near 100%, `aws_elasticache_evictions` climbs -- **Connection saturation** — `aws_elasticache_curr_connections` / `aws_elasticache_curr_connections_memcached` climb, `aws_elasticache_new_connections` spikes, admission fails -- **Replication lag** — `aws_elasticache_replication_lag` (Redis) climbs under primary write burst -- **Traffic management active** — `aws_elasticache_traffic_management_active = 1` → cluster-level throttling engaged - -## Key metrics - -| Metric | Unit | Signal | -|---|---|---| -| `aws_elasticache_cpu_utilization` | % | Host CPU (includes backups, replication) | -| `aws_elasticache_engine_cpu_utilization` | % | Redis main-thread CPU — THE blocking-command signal | -| `aws_elasticache_freeable_memory` | bytes | Host memory available | -| `aws_elasticache_bytes_used_for_cache` | bytes | Cache footprint | -| `aws_elasticache_database_memory_usage_percentage` | % | Near 100% = eviction imminent | -| `aws_elasticache_cache_hit_rate` | % | Direct cache efficiency | -| `aws_elasticache_cache_hits` / `aws_elasticache_cache_misses` | count | Hit/miss components | -| `aws_elasticache_evictions` | count | Sustained rate = memory pressure | -| `aws_elasticache_curr_connections` | count | Client count | -| `aws_elasticache_new_connections` | count | New-connection rate | -| `aws_elasticache_successful_read_request_latency` / `WriteRequestLatency` | μs | p99 latency | -| `aws_elasticache_replication_lag` | seconds | Primary-replica drift | -| `aws_elasticache_network_bytes_in` / `Out` | bytes/s | Traffic volume | -| `aws_elasticache_error_count` | count | Engine-returned errors | -| `aws_elasticache_authentication_failures` | count | AUTH failures — client misconfig or rotation | - -## Derived signals - -- `aws_elasticache_database_memory_usage_percentage` — direct memory utilization. -- `aws_elasticache_cache_hit_rate` — cache effectiveness. Cache workloads < 0.95 sustained = concern. -- Per-second derivative of `aws_elasticache_evictions` — sustained + `noeviction` policy = writes failing. -- `maxclients - aws_elasticache_curr_connections` — headroom. `maxclients` is node config, not a metric. - -## Log patterns - -- `OOM command not allowed when used memory > 'maxmemory'` — memory cap hit -- `MISCONF Redis is configured to save RDB snapshots` — RDB failure; writes blocked -- `Max clients reached` — connection ceiling -- `NOAUTH Authentication required` — client auth misconfig -- `WRONGTYPE Operation against a key holding the wrong kind of value` — app bug -- `LOADING Redis is loading the dataset` — failover / startup state - -## Gotchas - -- `aws_elasticache_cpu_utilization` vs `aws_elasticache_engine_cpu_utilization` confusion is the #1 misdiagnosis. Always use engine CPU for command-workload analysis. -- `aws_elasticache_replication_lag` only applies to Redis cluster-mode-disabled replicas. Cluster-mode-enabled has partition-level lag harder to surface. -- `aws_elasticache_traffic_management_active = 1` means AWS is throttling the cluster due to overload — secondary signal, not root cause. -- Memory metrics come from `INFO memory`; if engine is blocked by a slow command, scrapes stall and return stale values. diff --git a/plugins/tsuga/skills/knowledge-technology/references/aws-elb/README.md b/plugins/tsuga/skills/knowledge-technology/references/aws-elb/README.md deleted file mode 100644 index 3322aed..0000000 --- a/plugins/tsuga/skills/knowledge-technology/references/aws-elb/README.md +++ /dev/null @@ -1,259 +0,0 @@ -# AWS ELB Integration Context Bundle - -## Metadata -**Technology:** AWS ELB - Network ELB (NLB) surface -**Deployment:** managed -**Environment:** prod -**Persona:** SRE Dev and ops -**Telemetry preference:** mixed -**Integration scope:** core service only -**Primary use-case:** reliability and performance - -## How to use this bundle -Read `01_aws-elb_metrics.csv` first for the metric source of truth and the current Stage 1 assumptions about NLB metric names, units, and safe math. Read `02_aws-elb_dashboard_plan.yaml` next for the dashboard sections, widgets, derived signals, explanation notes, triage chains, and playbooks. - -Use `03_aws-elb_state.yaml` for machine-readable unknowns, assumptions, and log intelligence handoff. Use `04_aws-elb_memory.md` for the narrative summary of what Stage 2 should verify first. Stage 2 will create `05_aws-elb_metric_catalog.csv` as the discovered Tsuga metric catalog for reconciliation and coverage checks. Stage 4 should read this file's `Log intelligence (Stage 4 handoff)` section and `03_aws-elb_state.yaml` `log_intel` block before attempting route creation. - -## What it is and what "good" looks like -### Confirmed by sources -AWS Network Load Balancer is a layer-4 load balancer that forwards TCP, TLS, UDP, TCP_UDP, QUIC, and TCP_QUIC traffic to target groups while preserving source IP and supporting zonal load-balancer nodes. It is designed for very high connection rates, long-lived connections, and low-latency forwarding across enabled Availability Zones. Good NLB posture means new-flow demand is absorbed without rising rejected flows, reset rates stay near baseline, healthy targets remain available in every enabled zone, and capacity-unit consumption grows proportionally with traffic rather than spiking from inefficient connection patterns. AWS documents that NLB metrics are emitted in 60-second CloudWatch intervals when traffic exists, and that some metrics are absent when there is no traffic rather than explicitly reporting zero. [CloudWatch metrics](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-cloudwatch-metrics.html), [NLB overview](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/introduction.html), [How ELB works](https://docs.aws.amazon.com/elasticloadbalancing/latest/userguide/how-elastic-load-balancing-works.html) - -At incident time, the first question is whether the NLB is accepting and forwarding connections. The next question is whether failures originate at the load balancer edge, the target fleet, or a zonal health problem. The third question is whether capacity pressure is coming from connection churn, bytes transferred, active flows, or port allocation. Incident shape 1 is flow rejection or connection establishment failure during burst load; start in `nlb-failures-resets`. Incident shape 2 is rising resets with healthy-host erosion; start in `nlb-target-health`. Incident shape 3 is cost or headroom drift without an outright outage; start in `nlb-capacity-efficiency`. - -### Best-practice inference -For NLB-heavy systems, resets and rejected flows usually tell operators more than absolute traffic counts because they represent direct connection failure instead of normal demand. A healthy dashboard should let an on-call engineer distinguish three patterns quickly: traffic surge with sufficient headroom, traffic surge with capacity rejection, and target instability masked as a network symptom. - -If the environment uses mixed protocols, protocol-specific breakdowns become valuable only after the main health view is stable. For first-response dashboards, it is better to lead with all-flow metrics and keep protocol-split metrics as deep-dive diagnostics because protocol cardinality is lower than per-target or per-client dimensions but still increases visual noise. - -## Key concepts -### Glossary -| Term | Definition | Operational meaning | Dashboard section | -|---|---|---|---| -| Network Load Balancer | AWS managed layer-4 load balancer | Primary ingress or east-west TCP/TLS/UDP traffic control plane | nlb-demand-throughput | -| Listener | Protocol and port definition that accepts client connections | Traffic shape depends on listener protocol mix | nlb-demand-throughput | -| Target group | Set of registered backend targets | Failures often localize by target group before the whole NLB degrades | nlb-target-health | -| Load balancer node | Per-AZ node created for an enabled subnet/AZ | Zonal problems often appear before whole-balancer issues | nlb-target-health | -| Availability Zone | Isolated AWS zone hosting NLB nodes and targets | Blast radius and failover analysis depend on AZ splits | nlb-target-health | -| NewFlowCount | Number of new client-to-target flows | Best traffic denominator for rejection and reset ratios | nlb-demand-throughput | -| ActiveFlowCount | Concurrent active flows | Tracks occupancy and long-lived connection pressure | nlb-capacity-efficiency | -| ProcessedBytes | Total bytes processed by the NLB | Throughput and bytes-per-flow context | nlb-demand-throughput | -| PeakBytesPerSecond | Period burst throughput gauge | Detects short spikes hidden by average byte rates | nlb-capacity-efficiency | -| PeakPacketsPerSecond | Period burst packet-rate gauge | Surfaces packet-heavy workloads and small-packet storms | nlb-capacity-efficiency | -| RejectedFlowCount | Connections rejected by the load balancer | Direct customer-impact signal | nlb-failures-resets | -| PortAllocationErrorCount | Backend port allocation failures | Strong indicator of saturation or ephemeral-port pressure | nlb-failures-resets | -| TCP_Target_Reset_Count | TCP resets from targets | Backend instability, health, or protocol mismatch symptom | nlb-failures-resets | -| TCP_ELB_Reset_Count | TCP resets initiated by the load balancer | Edge-side timeout or lifecycle symptom | nlb-failures-resets | -| TCP_Client_Reset_Count | TCP resets initiated by clients | Client churn or upstream timeout symptom | nlb-failures-resets | -| HealthyHostCount | Healthy registered targets | Available serving capacity denominator | nlb-target-health | -| UnHealthyHostCount | Unhealthy registered targets | Capacity erosion and failover risk | nlb-target-health | -| ZonalHealthStatus | Zone-level health signal | AZ degradation and DNS failover context | nlb-target-health | -| ClientTLSNegotiationErrorCount | Failed TLS negotiations at the client edge | TLS-specific ingress failures that do not always show as target issues | nlb-failures-resets | -| SecurityGroupBlockedFlowCount | TLS flows blocked by NLB security groups | Security policy rejection that does not equal target unhealthiness | nlb-failures-resets | -| Cross-zone load balancing | NLB feature that lets nodes send traffic to targets in all enabled AZs | Changes how zonal imbalance should be interpreted | nlb-target-health | -| Passive health checks | Unconfigurable NLB behavior that observes target response quality | Backend instability can surface before active health checks mark targets unhealthy | nlb-target-health | -| Fail open | NLB behavior when all targets fail health checks or a target group is empty | Healthy-host metrics alone can mislead during total-failure scenarios | nlb-target-health | -| NLCU | Network Load Balancer capacity unit | Capacity and cost pressure roll up to whichever dimension dominates the hour | nlb-capacity-efficiency | - -### Concept Map -Client -> connects to -> NLB listener (why: every incident begins with whether the listener accepted work) -Listener -> forwards to -> target group (why: target-group health determines forwarding safety) -Target group -> contains -> targets (why: host health erosion reduces serving capacity) -Enabled Availability Zone -> hosts -> NLB node (why: zonal failure can remove an IP from DNS) -NLB node -> forwards to -> healthy targets in its zone by default (why: zonal imbalance matters even without a full outage) -Cross-zone load balancing -> widens routing set to -> healthy targets in all enabled zones (why: cross-zone changes how AZ-local symptoms appear) -NewFlowCount -> measures -> demand churn (why: best denominator for rejection and reset ratios) -ActiveFlowCount -> measures -> connection occupancy (why: long-lived flows consume headroom differently than burst traffic) -ProcessedBytes -> captures -> total transferred bytes (why: byte-heavy traffic can dominate NLCU cost) -PeakBytesPerSecond -> highlights -> burst throughput (why: short spikes can be hidden in totals) -PeakPacketsPerSecond -> highlights -> packet intensity (why: small-packet floods stress networking differently than large transfers) -RejectedFlowCount -> indicates -> direct connection denial (why: user impact starts before target-side investigation) -PortAllocationErrorCount -> indicates -> backend port exhaustion pressure (why: strong clue for capacity bottlenecks) -TCP_Target_Reset_Count -> points to -> target-side connection termination (why: target instability and protocol mismatch show here) -TCP_ELB_Reset_Count -> points to -> load-balancer-side connection termination (why: edge lifecycle and timeout behavior show here) -TCP_Client_Reset_Count -> points to -> client-side aborts (why: not all resets indicate backend failure) -HealthyHostCount -> bounds -> available serving pool (why: low healthy capacity magnifies rejection and reset risk) -UnHealthyHostCount -> amplifies -> routing instability (why: degraded fleets often precede customer-visible failures) -ZonalHealthStatus -> summarizes -> per-AZ health posture (why: useful for DNS failover and zonal isolation) -Passive health checks -> detect -> target problems earlier than active checks (why: resets can rise before unhealthy counts do) -Fail open -> routes to -> all targets when all are unhealthy (why: traffic can continue while health counters look alarming) -SecurityGroupBlockedFlowCount -> separates -> policy blocks from target failures (why: remediation differs from health-based outages) -ClientTLSNegotiationErrorCount -> separates -> TLS edge failures from target health issues (why: cert/policy problems look different from app outages) -NLCU -> is dominated by -> highest of new flows, active flows, or processed bytes (why: capacity and cost diagnostics need all three dimensions) -Load balancer identity -> maps to -> context.loadbalancer (why: ownership and blast radius require a stable primary grouping key) -Target group identity -> maps to -> context.targetgroup (why: backend isolation needs a second-level grouping key) -Availability Zone -> maps to -> context.availabilityzone (why: zonal impairment must be surfaced explicitly) -Environment and team tags -> map to -> context.env and context.team (why: dashboards need multi-tenant filtering and routing context) - -### Entities and dimensions -| Entity/Dimension | Why useful | Cardinality risk | Safe top-N suggestion | Do NOT group-by guidance | -|---|---|---|---|---| -| `context.loadbalancer` | Primary NLB ownership and blast-radius key | Low | 20 | Use as first grouping before any target details | -| `context.targetgroup` | Localizes unhealthy hosts and reset-heavy backends | Medium | 20 | Avoid combining with ephemeral or client identifiers | -| `context.availabilityzone` | Detects zonal impairment and failover asymmetry | Low | 6 | Do not interpret alone without load balancer context | -| `context.cloud.region` | Separates regional fleets | Low | 10 | Prefer as a global filter in single-region dashboards | -| `context.cloud.account.id` | Splits multi-account NLB estates | Low | 10 | Better as a global filter than a chart series | -| `context.env` | Distinguishes prod, staging, and test | Low | 5 | Use as dashboard filter, not chart group-by | -| `context.team` | Ownership and routing | Low | 10 | Keep as filter, not a timeseries split | -| `context.listener` | Protocol-port isolation for listener-specific incidents | Medium | 10 | Do not put on overview charts by default | -| `context.protocol` | Distinguishes TCP/TLS/UDP style behavior | Medium | 10 | Use only when the NLB actually mixes protocols | -| `context.target.ip` | Helpful for deep host isolation | High | 15 | Never use on overview; only use in deep-dive top lists if verified | -| `context.target.port` | Distinguishes shared-target services | High | 10 | Avoid unless troubleshooting a specific port mapping problem | -| `context.security_group` | Useful for policy-block diagnosis | Medium | 10 | Only use when security-group-blocked metrics are confirmed | -| `context.scope.name` | Fallback generic workload identity | Medium | 20 | Use only if `context.loadbalancer` is absent in Stage 2 discovery | - -### Tsuga field mapping -#### Confirmed by sources -| Vendor/exporter dimension | Recommended context.* key | Must-exist vs optional | -|---|---|---| -| LoadBalancer | `context.loadbalancer` | Must-exist | -| TargetGroup | `context.targetgroup` | Strongly preferred | -| AvailabilityZone | `context.availabilityzone` | Strongly preferred | -| AWS Region | `context.cloud.region` | Optional | -| AWS Account ID | `context.cloud.account.id` | Optional | -| Environment tag | `context.env` | Must-exist | -| Team tag | `context.team` | Must-exist | -| Listener ARN or port/protocol metadata | `context.listener` | Optional | -| Protocol | `context.protocol` | Optional | - -AWS documents `LoadBalancer`, `AvailabilityZone`, and `TargetGroup` as dimensions on multiple NLB metrics, so these are the strongest expected Tsuga mappings for grouping and filters. [CloudWatch metrics](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-cloudwatch-metrics.html) - -#### Best-practice inference -If a CloudWatch-to-Tsuga pipeline normalizes dimension names differently, `context.scope.name` is the safest fallback for the primary identity field until Stage 2 confirms the real attribute set. Listener, target IP, target port, and security-group fields are useful only if the ingestion pipeline preserves them consistently enough for bounded groupings. - -## Golden signals -### Confirmed by sources -| Signal | What it means for NLB | Typical causes when it degrades | Best telemetry sources | What people page on | Section questions | -|---|---|---|---|---|---| -| Traffic | Connection and byte demand through the NLB | burst connection churn, sudden demand drops, packet floods | `NewFlowCount`, `ActiveFlowCount`, `ProcessedBytes`, `PeakBytesPerSecond`, `PeakPacketsPerSecond` | demand collapse, unexpected surge, unplanned traffic shape shift | Is the NLB receiving expected flow demand? Are bursts changing throughput shape? | -| Errors | Connection acceptance and reset failures | target instability, port exhaustion, TLS negotiation failures, policy blocks | `RejectedFlowCount`, reset counters, `PortAllocationErrorCount`, TLS/security-group metrics | rejected flow spikes, sustained reset growth | Are failures edge-side, target-side, or client-side? | -| Latency | NLB itself is low-latency, so latency proxies come from resets, churn, and target health rather than an app-layer response-time metric | overloaded backends, SYN/backlog pressure, TLS negotiation issues | reset counters, `ActiveFlowCount`, `HealthyHostCount`, logs | connection establishment delay, connection timeouts, rising resets during stable demand | Is there connection instability even if raw traffic looks normal? | -| Saturation | Headroom consumed by flows, bytes, and available healthy targets | port allocation pressure, active-flow buildup, uneven zonal capacity, byte-heavy workloads | `ConsumedLCUs`, `PortAllocationErrorCount`, `ActiveFlowCount`, healthy/unhealthy host counts | rejection with capacity pressure, cost/headroom drift, zonal exhaustion | Do we still have safe serving headroom? Which capacity dimension is binding? | - -### Best-practice inference -For NLBs, latency is rarely a first-class single metric the way it is for ALBs. Operators usually infer user-visible latency trouble from a combination of connection resets, host health, and rising connection occupancy. A practical dashboard should therefore emphasize failure ratios and capacity efficiency over a synthetic latency number that the platform does not emit directly. - -## Telemetry sources -### Confirmed by sources -| Source type | How collected | What it provides | Pros/cons | Common pitfalls | -|---|---|---|---|---| -| CloudWatch NLB metrics | AWS publishes 60-second metrics when traffic exists | Canonical NLB flow, reset, host-health, and capacity metrics | Best source for baseline health; no app-layer payload context | No traffic often means no datapoints, not explicit zero; security-group rejections are not captured for all paths | -| NLB access logs to S3 | Optional access logging, legacy path | TLS connection details and log-line level triage data | High forensic value for TLS listeners | Disabled by default; TLS-only; eventual consistency; best-effort delivery | -| NLB logs to CloudWatch Logs/Data Firehose/S3 | Enhanced logging pipeline | Centralized log delivery with more flexible sinks | Better operational routing than legacy S3-only flow | Still optional; format and field choices must be verified in environment | -| Target health APIs and console state | AWS target group health state and reason codes | Registration, unhealthy, draining, unavailable reasons | Explains host-count drops | Separate from metric stream; not a metric source by itself | -| Pricing and capacity references | AWS pricing and capacity reservation docs | Meaning of LCUs and throughput conversions | Grounds capacity-efficiency widgets | Not operational telemetry; use for interpretation only | - -NLB metrics are reported only while requests are flowing, access logs are optional and TLS-listener scoped, and target health APIs expose reason codes such as failed health checks or deregistration progress. [CloudWatch metrics](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-cloudwatch-metrics.html), [Access logs](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-access-logs.html), [CloudWatch logs](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-cloudwatch-logs.html), [Check target health](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/check-target-health.html) - -### Best-practice inference -If there is no data for `HealthyHostCount` or `UnHealthyHostCount`, the meaning is ambiguous until Stage 2 confirms the ingestion model: it can mean no registered targets in scope, no traffic-driven metric emission in the selected window, or an ingestion gap. Dashboards should therefore use missing-note language instead of treating null as healthy. - -## Log intelligence (Stage 4 handoff) -### Confirmed by sources -1. Log sources matrix - -| Source | Access path | Typical format | Structured vs unstructured | Evidence | -|---|---|---|---|---| -| Legacy NLB access logs | S3 bucket objects under `AWSLogs/.../elasticloadbalancing/.../net....log.gz` | Space-delimited log entries | Semi-structured text | [Access logs](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-access-logs.html) | -| Enhanced NLB logs | CloudWatch Logs, Data Firehose, or S3 | AWS-managed delivered log records, optionally Parquet in some sinks | Structured delivery path, exact payload to verify | [CloudWatch logs](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-cloudwatch-logs.html) | - -2. Known log formats - -- `NLB legacy TLS access log` - - Sample line: `tls 2.0 2024-01-01T00:00:00.000000Z net/my-nlb/1234567890abcdef 192.0.2.10:44321 198.51.100.20:443 0.000 0.002 0.000 200 200 0 57 "TLSv1.2" "ECDHE-RSA-AES128-GCM-SHA256" ...` - - Delimiter and shape notes: space-delimited fields with quoted string fields later in the record; field count can evolve over time. - - Timestamp pattern: ISO-8601 UTC with microseconds. - - Quoting behavior: some string fields are double-quoted. - - Optional fields: AWS notes that parsers should stop at the last documented field and tolerate trailing additions. [Access logs](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-access-logs.html) - -3. Candidate query filters for Stage 4 - -- Precise: `context.service.name:"aws-networkelb" AND (context.loadbalancer:* OR context.targetgroup:*)` - - Rationale: narrows to logs already normalized as NLB traffic while keeping the two most useful identity keys. - - Risk: depends on Stage 4 confirming the service-name mapping in the current pipeline. -- Fallback: `"elasticloadbalancing" AND "net/" AND ("TLS" OR "tcp")` - - Rationale: catches raw AWS-delivered NLB log lines when structured enrichment is weak. - - Risk: can overmatch adjacent ELB log sources and may miss non-TLS paths. - -4. Attribute mapping hints - -| Raw field | Suggested Tsuga key | Confidence | Notes | -|---|---|---|---| -| load balancer id | `context.loadbalancer` | High | Should map from `net//` path segment | -| client ip:port | `context.client.address` / `context.client.port` | Medium | Split required | -| target ip:port | `context.target.address` / `context.target.port` | Medium | Split required | -| tls protocol | `context.tls.version` | Medium | Legacy logs are TLS-listener scoped | -| tls cipher | `context.tls.cipher` | Medium | Useful for negotiation-error triage | -| elb status / connection outcome | `context.status` | Medium | Exact field naming depends on sink format | -| bytes received/sent | `context.network.bytes_in` / `context.network.bytes_out` | Low | Verify field names in actual environment | - -5. Parsing risks - -- Legacy access logs are TLS-listener only, so absence of logs can mean feature not enabled rather than no traffic. -- Delivery is eventually consistent and best effort, so log counts should not be used as an authoritative traffic ledger. -- Sample-field schemas may evolve; AWS recommends parsers tolerate appended fields. -- Mixed raw and enriched delivery formats are possible if S3 legacy logging and CloudWatch Logs are both enabled. -- Target and client endpoint fields require splitting host and port safely. - -### Best-practice inference -If the environment uses enhanced CloudWatch Logs delivery, Stage 4 should prefer the actual delivered schema over the legacy text format even when both are enabled. If only metric telemetry is present and no logs are configured, Stage 4 should route through target health reason codes and infrastructure events rather than fabricating an NLB log route. - -## Caveats and footguns -- **[nlb-demand-throughput]** NLB metrics are emitted only when requests are flowing, so missing points can mean idle traffic rather than telemetry failure. ([CloudWatch metrics](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-cloudwatch-metrics.html)) -- **[nlb-demand-throughput]** For Network Load Balancers with security groups, traffic rejected by the security groups is not captured in the standard CloudWatch metrics discussed on the metrics page. ([CloudWatch metrics](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-cloudwatch-metrics.html)) -- **[nlb-demand-throughput]** `ProcessedBytes` and `PeakBytesPerSecond` can rise because of payload growth even when connection counts remain flat. (Inference) -- **[nlb-demand-throughput]** Packet-heavy small-payload traffic can make `PeakPacketsPerSecond` more informative than bytes. (Inference) -- **[nlb-failures-resets]** `RejectedFlowCount` is direct customer impact; do not average it away as a background signal. ([CloudWatch metrics](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-cloudwatch-metrics.html)) -- **[nlb-failures-resets]** `TCP_Target_Reset_Count` and `TCP_ELB_Reset_Count` indicate different ownership domains and should not be collapsed into one root-cause KPI. ([CloudWatch metrics](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-cloudwatch-metrics.html)) -- **[nlb-failures-resets]** `TCP_Client_Reset_Count` can spike during client-side timeout or retry storms; it does not prove the NLB or target is unhealthy. ([CloudWatch metrics](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-cloudwatch-metrics.html)) -- **[nlb-failures-resets]** Port allocation errors usually indicate urgent backend flow-capacity problems and should be triaged alongside rejected flows. ([Troubleshooting](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-troubleshooting.html)) -- **[nlb-failures-resets]** TLS negotiation failures belong with edge failure analysis, not target-health analysis. ([CloudWatch metrics](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-cloudwatch-metrics.html)) -- **[nlb-failures-resets]** Security-group-blocked flow metrics are TLS/security-group specific and may be absent entirely in otherwise healthy environments. (Inference) -- **[nlb-target-health]** Healthy-host counts should be interpreted by target group and AZ, not only as a global sum. ([CloudWatch metrics](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-cloudwatch-metrics.html)) -- **[nlb-target-health]** Passive health checks cannot be directly monitored, so reset spikes can lead host-count changes. ([Health checks](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/target-group-health-checks.html)) -- **[nlb-target-health]** If all targets fail health checks at once, NLB can fail open, so traffic may still flow while health looks bad. ([Health checks](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/target-group-health-checks.html)) -- **[nlb-target-health]** If a zone has no healthy targets, AWS can remove that zone's NLB IP from DNS, which changes client routing patterns quickly. ([Health checks](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/target-group-health-checks.html)) -- **[nlb-target-health]** HTTP or HTTPS health checks use the load balancer node IP and listener port in the host header, which can surprise backends that expect a virtual host. ([Health checks](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/target-group-health-checks.html)) -- **[nlb-target-health]** UDP and QUIC services often rely on non-UDP health checks, so health semantics can diverge from the served protocol. ([Health checks](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/target-group-health-checks.html)) -- **[nlb-capacity-efficiency]** NLCU cost and pressure are driven by the highest of new flows, active flows, or processed bytes for the hour, not the sum of those dimensions. ([ELB FAQ](https://aws.amazon.com/th/elasticloadbalancing/faqs/)) -- **[nlb-capacity-efficiency]** `ConsumedLCUs` is capacity context, not a direct outage indicator; it becomes actionable when paired with rejection, reset, or host-health deterioration. (Inference) -- **[nlb-capacity-efficiency]** Active-flow growth can be healthy for long-lived connections; interpret it with new-flow churn and resets before calling it saturation. (Inference) -- **[nlb-capacity-efficiency]** Protocol-specific flow metrics (`ActiveFlowCount_TCP`, `_TLS`, `_UDP`) are diagnostic enrichments and can clutter overview panels if used without a protocol filter. (Inference) -- **[nlb-capacity-efficiency]** Target quotas and listener quotas can become design constraints before metrics obviously fail, so persistent headroom widgets need an accompanying note rather than a hard threshold. ([NLB quotas](https://docs.aws.amazon.com/en_us/elasticloadbalancing/latest/network/load-balancer-limits.html)) - -## Confirmed Tsuga prefixes -- `aws_networkelb_*` - **CONFIRMED** (21 live metrics in the current Tsuga catalog; verified by exact-prefix search and 24-hour/30-day CLI enumeration) - -## Discovery status -Discovery: completed in Stage 2 for the current Tsuga environment. -- Prefix preflight with exact matching confirmed 21 `aws_networkelb_*` metrics. -- Live catalog reconciliation result: - - 21 metrics present in Tsuga - - 5 Stage 1 candidate metrics missing in this environment: `aws_networkelb_active_flow_count_tls`, `aws_networkelb_active_flow_count_udp`, `aws_networkelb_client_tls_negotiation_error_count`, `aws_networkelb_security_group_blocked_flow_count_inbound_tls`, `aws_networkelb_security_group_blocked_flow_count_outbound_tls` - - 6 live-only metrics added to the bundle: `aws_networkelb_consumed_lc_us_tcp`, `aws_networkelb_new_flow_count_tcp`, `aws_networkelb_processed_bytes_tcp`, `aws_networkelb_processed_packets`, `aws_networkelb_security_group_blocked_flow_count_inbound_tcp`, `aws_networkelb_unhealthy_routing_flow_count` -- Actual context fields confirmed broadly across the live catalog: `context.env`, `context.team`, `context.loadbalancer`, `context.availabilityzone`, `context.cloud.region`, and `context.cloud.account.id` -- `context.targetgroup` is present on host-health metrics, but not on reset/rejection/flow metrics, so target-group grouping is limited to the target-health section. -- Count-like NLB metrics are exposed in Tsuga as `summary` with `cumulative` temporality, and this bundle now uses `per-second` on count-like widgets by explicit user request. - -## Top sources -1. https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-cloudwatch-metrics.html - Why: canonical metric list, dimensions, and "no traffic means no datapoints" behavior. -2. https://docs.aws.amazon.com/elasticloadbalancing/latest/network/introduction.html - Why: NLB architecture, protocol support, target groups, and zonal node model. -3. https://docs.aws.amazon.com/elasticloadbalancing/latest/userguide/how-elastic-load-balancing-works.html - Why: shared ELB routing, DNS, and zonal behavior context. -4. https://docs.aws.amazon.com/elasticloadbalancing/latest/network/target-group-health-checks.html - Why: active/passive health checks, fail-open behavior, and zonal removal semantics. -5. https://docs.aws.amazon.com/elasticloadbalancing/latest/network/check-target-health.html - Why: target states and reason codes for health-related triage. -6. https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-target-groups.html - Why: target-group thresholds, DNS failover, and health-threshold behavior. -7. https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-access-logs.html - Why: legacy TLS access-log format, path, and parser caveats. -8. https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-cloudwatch-logs.html - Why: enhanced log-delivery options and Stage 4 route implications. -9. https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-troubleshooting.html - Why: operational failure modes including reset growth and port allocation errors. -10. https://aws.amazon.com/th/elasticloadbalancing/faqs/ - Why: LCU interpretation for NLB flow, active connection, and processed-byte dimensions. -11. https://docs.aws.amazon.com/en_us/elasticloadbalancing/latest/network/load-balancer-limits.html - Why: quotas that affect capacity and saturation interpretation. diff --git a/plugins/tsuga/skills/knowledge-technology/references/aws-elb/metrics.csv b/plugins/tsuga/skills/knowledge-technology/references/aws-elb/metrics.csv deleted file mode 100644 index 09fae10..0000000 --- a/plugins/tsuga/skills/knowledge-technology/references/aws-elb/metrics.csv +++ /dev/null @@ -1,28 +0,0 @@ -theme,name,confirmed,definition,unit,type,tsuga_metric_name,widget_suggestions,aggregation,post_function,group_by,required_context_fields,optional_context_fields,absence_means,cardinality_footguns,variants,citations -Throughput/Usage,NewFlowCount,true,New client-to-target flows and the primary demand denominator for rejection and reset ratios.,{flow},counter,aws_networkelb_new_flow_count,timeseries|query-value|top-list,sum,per-second,context.loadbalancer:20|context.availabilityzone:6,context.env|context.team|context.loadbalancer,context.cloud.region|context.cloud.account.id,"No recent traffic in the selected window or metric not ingested",Use load balancer or AZ splits only; target group is not present on the live metric,cloudwatch=NewFlowCount|tsuga=aws_networkelb_new_flow_count,https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-cloudwatch-metrics.html -Throughput/Usage,NewFlowCount_TCP,true,TCP-only new client-to-target flows for TCP listeners.,{flow},counter,aws_networkelb_new_flow_count_tcp,timeseries|query-value|top-list,sum,per-second,context.loadbalancer:20|context.availabilityzone:6,context.env|context.team|context.loadbalancer,context.cloud.region|context.cloud.account.id,"TCP listener not enabled, no TCP traffic, or metric not ingested",Protocol-specific metric; keep out of overview unless TCP detail is needed,live_catalog=aws_networkelb_new_flow_count_tcp,https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-cloudwatch-metrics.html -Throughput/Usage,ActiveFlowCount,true,Concurrent open client-to-target flows showing connection occupancy and long-lived load.,{flow},gauge,aws_networkelb_active_flow_count,timeseries|query-value|top-list,max,none,context.loadbalancer:20|context.availabilityzone:6,context.env|context.team|context.loadbalancer,context.cloud.region|context.cloud.account.id,"No matching traffic, no active flows, or ingestion gap",Do not sum widely across unrelated NLBs when the operational question is hotspot detection,cloudwatch=ActiveFlowCount|tsuga=aws_networkelb_active_flow_count,https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-cloudwatch-metrics.html -Throughput/Usage,ActiveFlowCount_TCP,true,Concurrent TCP flows for TCP listeners only.,{flow},gauge,aws_networkelb_active_flow_count_tcp,timeseries|query-value|max,none,context.loadbalancer:20|context.availabilityzone:6,context.env|context.team|context.loadbalancer,context.cloud.region|context.cloud.account.id,"TCP listener not enabled, no TCP traffic, or metric not ingested",Protocol-specific metric; exclude unless TCP-specific diagnostics are needed,cloudwatch=ActiveFlowCount_TCP|tsuga=aws_networkelb_active_flow_count_tcp,https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-cloudwatch-metrics.html -Throughput/Usage,ActiveFlowCount_TLS,false,Concurrent TLS flows for TLS listeners only.,{flow},gauge,aws_networkelb_active_flow_count_tls,timeseries|query-value|max,none,context.loadbalancer:20|context.availabilityzone:6,context.env|context.team|context.loadbalancer,context.cloud.region|context.cloud.account.id,"TLS listener not enabled in this environment or metric not ingested",Do not assume TLS listener telemetry exists from docs alone,cloudwatch=ActiveFlowCount_TLS|tsuga=aws_networkelb_active_flow_count_tls,https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-cloudwatch-metrics.html -Throughput/Usage,ActiveFlowCount_UDP,false,Concurrent UDP flows for UDP listeners only.,{flow},gauge,aws_networkelb_active_flow_count_udp,timeseries|query-value|max,none,context.loadbalancer:20|context.availabilityzone:6,context.env|context.team|context.loadbalancer,context.cloud.region|context.cloud.account.id,"UDP listener not enabled in this environment or metric not ingested",Do not assume UDP listener telemetry exists from docs alone,cloudwatch=ActiveFlowCount_UDP|tsuga=aws_networkelb_active_flow_count_udp,https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-cloudwatch-metrics.html -Throughput/Usage,ProcessedBytes,true,Total bytes processed by the NLB and the main throughput numerator for bytes-per-flow analysis.,By,counter,aws_networkelb_processed_bytes,timeseries|query-value|top-list,sum,per-second,context.loadbalancer:20|context.availabilityzone:6,context.env|context.team|context.loadbalancer,context.cloud.region|context.cloud.account.id,"No recent traffic or metric not ingested",Prefer load-balancer or AZ grouping; raw target-level byte groupings are unavailable in the live catalog,cloudwatch=ProcessedBytes|tsuga=aws_networkelb_processed_bytes,https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-cloudwatch-metrics.html -Throughput/Usage,ProcessedBytes_TCP,true,TCP-only processed bytes for TCP listeners.,By,counter,aws_networkelb_processed_bytes_tcp,timeseries|query-value|top-list,sum,per-second,context.loadbalancer:20|context.availabilityzone:6,context.env|context.team|context.loadbalancer,context.cloud.region|context.cloud.account.id,"TCP listener not enabled, no TCP traffic, or metric not ingested",Correlated with all-flow bytes in TCP-only environments; usually a protocol-specific diagnostic,live_catalog=aws_networkelb_processed_bytes_tcp,https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-cloudwatch-metrics.html -Throughput/Usage,ProcessedPackets,true,Total packets processed by the NLB and a packet-volume companion to byte throughput.,{packet},counter,aws_networkelb_processed_packets,timeseries|query-value|top-list,sum,per-second,context.loadbalancer:20|context.availabilityzone:6,context.env|context.team|context.loadbalancer,context.cloud.region|context.cloud.account.id,"No recent traffic or metric not ingested",Packet totals can be redundant when peak packet gauges already tell the story,live_catalog=aws_networkelb_processed_packets,https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-cloudwatch-metrics.html -Throughput/Usage,PeakBytesPerSecond,true,Burst bytes-per-second gauge for the period and a strong signal for short throughput spikes.,By/s,gauge,aws_networkelb_peak_bytes_per_second,timeseries|query-value|max,none,context.loadbalancer:20|context.availabilityzone:6,context.env|context.team|context.loadbalancer,context.cloud.region|context.cloud.account.id,"No traffic during the period or metric not ingested",Peak gauges are burst-sensitive and should be trended before alarming,cloudwatch=PeakBytesPerSecond|tsuga=aws_networkelb_peak_bytes_per_second,https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-cloudwatch-metrics.html -Throughput/Usage,PeakPacketsPerSecond,true,Burst packet-rate gauge for detecting packet-heavy or small-payload traffic spikes.,{packet}/s,gauge,aws_networkelb_peak_packets_per_second,timeseries|query-value|max,none,context.loadbalancer:20|context.availabilityzone:6,context.env|context.team|context.loadbalancer,context.cloud.region|context.cloud.account.id,"No traffic during the period or metric not ingested",Interpret with PeakBytesPerSecond to distinguish packet storms from large transfers,cloudwatch=PeakPacketsPerSecond|tsuga=aws_networkelb_peak_packets_per_second,https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-cloudwatch-metrics.html -Errors/Failures,RejectedFlowCount,true,Connections rejected by the NLB and the clearest direct customer-impact metric for acceptance failures.,{flow},counter,aws_networkelb_rejected_flow_count,timeseries|query-value|top-list,sum,per-second,context.loadbalancer:20|context.availabilityzone:6,context.env|context.team|context.loadbalancer,context.cloud.region|context.cloud.account.id,"Healthy baseline or missing traffic; use ratio context before alerting",Never aggregate away with averages; use as rate or ratio,cloudwatch=RejectedFlowCount|tsuga=aws_networkelb_rejected_flow_count,https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-cloudwatch-metrics.html -Errors/Failures,PortAllocationErrorCount,true,Backend port allocation failures that indicate immediate flow-capacity pressure.,{error},counter,aws_networkelb_port_allocation_error_count,timeseries|query-value|top-list,sum,per-second,context.loadbalancer:20|context.availabilityzone:6,context.env|context.team|context.loadbalancer,context.cloud.region|context.cloud.account.id,"No errors, no traffic, or metric not ingested",Treat sustained nonzero values as urgent; keep series counts small for readability,cloudwatch=PortAllocationErrorCount|tsuga=aws_networkelb_port_allocation_error_count,https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-cloudwatch-metrics.html|https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-troubleshooting.html -Errors/Failures,TCP_Target_Reset_Count,true,TCP resets sent by targets and the primary target-side connection instability signal.,{reset},counter,aws_networkelb_tcp_target_reset_count,timeseries|query-value|top-list,sum,per-second,context.loadbalancer:20|context.availabilityzone:6,context.env|context.team|context.loadbalancer,context.cloud.region|context.cloud.account.id,"No target resets, no traffic, or metric not ingested",Target group is not available on the live metric; use load balancer or AZ splits instead,cloudwatch=TCP_Target_Reset_Count|tsuga=aws_networkelb_tcp_target_reset_count,https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-cloudwatch-metrics.html -Errors/Failures,TCP_ELB_Reset_Count,true,TCP resets sent by the load balancer and the main edge-side reset signal.,{reset},counter,aws_networkelb_tcp_elb_reset_count,timeseries|query-value|top-list,sum,per-second,context.loadbalancer:20|context.availabilityzone:6,context.env|context.team|context.loadbalancer,context.cloud.region|context.cloud.account.id,"No ELB resets, no traffic, or metric not ingested",Keep separate from target resets because remediation paths differ,cloudwatch=TCP_ELB_Reset_Count|tsuga=aws_networkelb_tcp_elb_reset_count,https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-cloudwatch-metrics.html|https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-troubleshooting.html -Errors/Failures,TCP_Client_Reset_Count,true,TCP resets initiated by clients and useful for separating client aborts from edge or target failures.,{reset},counter,aws_networkelb_tcp_client_reset_count,timeseries|query-value|top-list,sum,per-second,context.loadbalancer:20|context.availabilityzone:6,context.env|context.team|context.loadbalancer,context.cloud.region|context.cloud.account.id,"No client resets, no traffic, or metric not ingested",Do not treat as backend failure without corroborating target or health signals,cloudwatch=TCP_Client_Reset_Count|tsuga=aws_networkelb_tcp_client_reset_count,https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-cloudwatch-metrics.html -Errors/Failures,ClientTLSNegotiationErrorCount,false,Failed client-side TLS negotiations on TLS listeners and an edge-specific handshake failure signal.,{error},counter,aws_networkelb_client_tls_negotiation_error_count,timeseries|query-value|top-list,sum,per-second,context.loadbalancer:20|context.availabilityzone:6,context.env|context.team|context.loadbalancer,context.listener|context.cloud.region,"TLS listener metric not present in the live Tsuga catalog for this environment",TLS-specific metric; keep documented but gated out of dashboards,cloudwatch=ClientTLSNegotiationErrorCount|tsuga=aws_networkelb_client_tls_negotiation_error_count,https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-cloudwatch-metrics.html -Errors/Failures,SecurityGroupBlockedFlowCount_Inbound_TCP,true,Inbound TCP flows blocked by NLB security groups and a policy-specific rejection diagnostic.,{flow},counter,aws_networkelb_security_group_blocked_flow_count_inbound_tcp,timeseries|query-value|top-list,sum,per-second,context.loadbalancer:20|context.availabilityzone:6,context.env|context.team|context.loadbalancer,context.cloud.region|context.cloud.account.id,"Security groups may not be attached or no blocked traffic occurred",Feature-specific metric; useful when policy blocks are suspected,live_catalog=aws_networkelb_security_group_blocked_flow_count_inbound_tcp,https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-cloudwatch-metrics.html -Errors/Failures,SecurityGroupBlockedFlowCount_Inbound_TLS,false,TLS flows blocked inbound by NLB security groups.,{flow},counter,aws_networkelb_security_group_blocked_flow_count_inbound_tls,timeseries|query-value|top-list,sum,per-second,context.loadbalancer:20|context.availabilityzone:6,context.env|context.team|context.loadbalancer,context.security_group|context.listener|context.cloud.region,"TLS-specific security-group metric not present in the live Tsuga catalog",Feature-specific metric absent in this environment,cloudwatch=SecurityGroupBlockedFlowCount_Inbound_TLS|tsuga=aws_networkelb_security_group_blocked_flow_count_inbound_tls,https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-cloudwatch-metrics.html -Errors/Failures,SecurityGroupBlockedFlowCount_Outbound_TLS,false,TLS flows blocked outbound by NLB security groups.,{flow},counter,aws_networkelb_security_group_blocked_flow_count_outbound_tls,timeseries|query-value|top-list,sum,per-second,context.loadbalancer:20|context.availabilityzone:6,context.env|context.team|context.loadbalancer,context.security_group|context.listener|context.cloud.region,"TLS-specific security-group metric not present in the live Tsuga catalog",Feature-specific metric absent in this environment,cloudwatch=SecurityGroupBlockedFlowCount_Outbound_TLS|tsuga=aws_networkelb_security_group_blocked_flow_count_outbound_tls,https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-cloudwatch-metrics.html -Errors/Failures,UnhealthyRoutingFlowCount,true,Flows routed to unhealthy targets and a strong direct-impact signal when fail-open or unhealthy-routing behavior occurs.,{flow},counter,aws_networkelb_unhealthy_routing_flow_count,timeseries|query-value|top-list,sum,per-second,context.loadbalancer:20|context.availabilityzone:6,context.env|context.team|context.loadbalancer,context.cloud.region|context.cloud.account.id,"No unhealthy routing events, no traffic, or metric not ingested",Interpret with host health because nonzero unhealthy routing is a severe symptom,live_catalog=aws_networkelb_unhealthy_routing_flow_count,https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-cloudwatch-metrics.html -Availability/Health,HealthyHostCount,true,Healthy targets registered behind the NLB and the primary serving-capacity denominator.,{host},gauge,aws_networkelb_healthy_host_count,timeseries|query-value|top-list,average|min,none,context.loadbalancer:20|context.targetgroup:20|context.availabilityzone:6,context.env|context.team|context.loadbalancer,context.targetgroup|context.cloud.region|context.cloud.account.id,"No registered targets in scope, no data due to traffic pattern, or metric missing",Always pair with unhealthy hosts and zonal health before concluding capacity is fine,cloudwatch=HealthyHostCount|tsuga=aws_networkelb_healthy_host_count,https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-cloudwatch-metrics.html -Availability/Health,UnHealthyHostCount,true,Unhealthy targets behind the NLB and the main capacity-erosion gauge.,{host},gauge,aws_networkelb_un_healthy_host_count,timeseries|query-value|top-list,average|max,none,context.loadbalancer:20|context.targetgroup:20|context.availabilityzone:6,context.env|context.team|context.loadbalancer,context.targetgroup|context.cloud.region|context.cloud.account.id,"No unhealthy targets, no registered targets, or metric not ingested",Global sums can hide a single failing target group; keep groupings bounded,cloudwatch=UnHealthyHostCount|tsuga=aws_networkelb_un_healthy_host_count,https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-cloudwatch-metrics.html -Availability/Health,ZonalHealthStatus,true,Zone-level health status for identifying AZ impairment and failover behavior.,1,gauge,aws_networkelb_zonal_health_status,timeseries|query-value|top-list,max,none,context.loadbalancer:20|context.availabilityzone:6,context.env|context.team|context.loadbalancer,context.cloud.region|context.cloud.account.id,"Feature not present in ingestion, no data, or wrong prefix assumption",Interpret with host counts because zonal status alone is too coarse,cloudwatch=ZonalHealthStatus|tsuga=aws_networkelb_zonal_health_status,https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-cloudwatch-metrics.html -Capacity/Saturation,ConsumedLCUs,true,Network Load Balancer capacity-unit consumption used as a headroom and cost-pressure proxy.,{lcu},gauge,aws_networkelb_consumed_lc_us,query-value|timeseries|max,none,context.loadbalancer:20|context.availabilityzone:6,context.env|context.team|context.loadbalancer,context.cloud.region|context.cloud.account.id,"No traffic, metric not ingested, or the selected window misses load",Use only with traffic and failure context because LCU rise alone is not an incident,cloudwatch=ConsumedLCUs|tsuga=aws_networkelb_consumed_lc_us,https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-cloudwatch-metrics.html|https://aws.amazon.com/th/elasticloadbalancing/faqs/ -Capacity/Saturation,ConsumedLCUs_TCP,true,TCP-specific consumed load-balancer capacity units for TCP listeners.,{lcu},gauge,aws_networkelb_consumed_lc_us_tcp,query-value|timeseries|max,none,context.loadbalancer:20|context.availabilityzone:6,context.env|context.team|context.loadbalancer,context.cloud.region|context.cloud.account.id,"TCP listener not enabled, no TCP traffic, or metric not ingested",Protocol-specific capacity metric; useful only when TCP detail matters,live_catalog=aws_networkelb_consumed_lc_us_tcp,https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-cloudwatch-metrics.html -Cost,NOT_APPLICABLE,false,"AWS does not emit direct dollar-cost telemetry in the NLB metric namespace. Use `ConsumedLCUs`, `ConsumedLCUs_TCP`, `NewFlowCount`, `ActiveFlowCount`, `ProcessedBytes`, `PeakBytesPerSecond`, and `PeakPacketsPerSecond` as cost-pressure proxies.",Unknown,Unknown,Unknown,query-value|timeseries,Unknown,none,context.loadbalancer:20,context.env|context.team|context.loadbalancer,context.cloud.region|context.cloud.account.id,"Not applicable because cost is derived from pricing rules rather than a direct metric",Keep cost separate from reliability severity in dashboard framing,pricing=NLB_LCU_rules,https://aws.amazon.com/th/elasticloadbalancing/faqs/ diff --git a/plugins/tsuga/skills/knowledge-technology/references/aws-elb/queries.md b/plugins/tsuga/skills/knowledge-technology/references/aws-elb/queries.md deleted file mode 100644 index 21922ae..0000000 --- a/plugins/tsuga/skills/knowledge-technology/references/aws-elb/queries.md +++ /dev/null @@ -1,57 +0,0 @@ -# AWS ELB (NLB / ALB / GLB / CLB) - -NLB = L4, ALB = L7, GLB = GENEVE, CLB = legacy. Metric names below skew NLB. ALB has its own (`HTTPCode_Target_2XX_Count`, `TargetResponseTime`, etc.) but reasoning is the same. - -## Incident shapes - -- **Target fleet unhealthy** — `aws_networkelb_un_healthy_host_count > 0` → backend problem, not LB -- **Flow rejections** — `aws_networkelb_rejected_flow_count` spikes → per-LB connection limits or AZ capacity -- **TLS handshake failures** — `aws_networkelb_client_tls_negotiation_error_count` climbs → cert / SNI / TLS-version mismatch -- **TCP resets** — target / ELB / client reset counts spike → abrupt close -- **Zonal imbalance** — `aws_networkelb_zonal_health_status = 0` on one AZ → traffic shifts, overloads peers -- **LCU ceiling** — `aws_networkelb_consumed_lc_us` climbs → cost spike, potential throttling - -## Key metrics - -| Metric | Unit | Signal | -|---|---|---| -| `aws_networkelb_active_flow_count` | flows | Concurrent connections | -| `aws_networkelb_new_flow_count` | flows/min | New-connection rate | -| `aws_networkelb_processed_bytes` | bytes | Traffic volume | -| `aws_networkelb_healthy_host_count` | count | Per-target-group healthy | -| `aws_networkelb_un_healthy_host_count` | count | Per-target-group unhealthy | -| `aws_networkelb_rejected_flow_count` | count | Connection rejections | -| `aws_networkelb_unhealthy_routing_flow_count` | count | Flows to unhealthy target (AZ failover) | -| `aws_networkelb_tcp_elb_reset_count` | count | LB-initiated resets | -| `aws_networkelb_tcp_target_reset_count` | count | Target-initiated resets | -| `aws_networkelb_tcp_client_reset_count` | count | Client-initiated resets | -| `aws_networkelb_client_tls_negotiation_error_count` | count | TLS handshake failures | -| `SecurityGroupBlockedFlowCount_*` | count | SG denies | -| `aws_networkelb_port_allocation_error_count` | count | NLB source-NAT port exhaustion | -| `aws_networkelb_zonal_health_status` | 0/1 | Per-AZ health | -| `aws_networkelb_consumed_lc_us` | units | Cost / sizing | - -## Derived signals - -- `aws_networkelb_healthy_host_count / (aws_networkelb_healthy_host_count + aws_networkelb_un_healthy_host_count)` — fleet health. < 1 = partial outage. -- `(ELB + Target + Client resets) / aws_networkelb_new_flow_count` — reset ratio. Spike = instability. -- `aws_networkelb_rejected_flow_count / (aws_networkelb_new_flow_count + aws_networkelb_rejected_flow_count)` — rejection rate. Any sustained = capacity or SG problem. -- Per-AZ `aws_networkelb_new_flow_count` skew > 2× with other AZs `aws_networkelb_zonal_health_status=1` = client routing issue. - -## Log patterns - -ALB access logs (S3-delivered): - -- `request_processing_time=-1` — connection failure to target -- `target_processing_time=-1` — target didn't respond -- `elb_status_code=504` — gateway timeout -- `elb_status_code=502` — bad gateway (malformed backend response) -- `elb_status_code=503` — service unavailable (often health-check failure) -- `target_status_code="-"` — no target response received - -## Gotchas - -- `aws_networkelb_un_healthy_host_count` lags real health by one or two health-check intervals (~30s). Brief outages may not show. -- `aws_networkelb_rejected_flow_count` can be zero during overload if rejections happen at target (target returns 503). -- NLB `aws_networkelb_port_allocation_error_count` = source-NAT exhaustion on ENI; add subnets or switch to client-IP preservation. -- ALB 5xx with no target-metric correlation = LB couldn't reach target (DNS, routing, SG). Check `aws_networkelb_healthy_host_count` at that timestamp. diff --git a/plugins/tsuga/skills/knowledge-technology/references/aws-eventbridge/README.md b/plugins/tsuga/skills/knowledge-technology/references/aws-eventbridge/README.md deleted file mode 100644 index 73e5306..0000000 --- a/plugins/tsuga/skills/knowledge-technology/references/aws-eventbridge/README.md +++ /dev/null @@ -1,283 +0,0 @@ -# AWS EventBridge Integration Context Bundle - -## Metadata -**Technology:** AWS EventBridge -**Deployment:** managed -**Environment:** prod -**Persona:** SRE Dev and ops -**Telemetry preference:** mixed -**Integration scope:** core service only -**Primary use-case:** reliability and performance - -## How to use this bundle -- Use `01_aws-eventbridge_metrics.csv` as the source of truth for metric names, units, safe aggregations, and expected Tsuga mappings. -- Use `02_aws-eventbridge_dashboard_plan.yaml` as the implementation blueprint for sections, widgets, derived signals, note copy, triage chains, and playbooks. -- Use `03_aws-eventbridge_state.yaml` for machine-readable state, assumptions, log intelligence status, and unresolved unknowns. -- Use `04_aws-eventbridge_memory.md` for the readable Stage 1 handoff narrative and Stage 2 priority checks. -- Stage 2 will create `05_aws-eventbridge_metric_catalog.csv` for discovered metric inventory, attribute-key reconciliation, and description curation. -- Stage 4 should read this file's `Log intelligence (Stage 4 handoff)` section and `03_aws-eventbridge_state.yaml` `log_intel` block before authoring any log route payloads. - -## What it is and what "good" looks like - -### Confirmed by sources -- EventBridge is AWS's serverless event router. It receives events from AWS services, SaaS partners, and custom applications, matches them against rules on an event bus, and invokes one or more targets. [S1][S2] -- AWS exposes `AWS/Events` CloudWatch metrics for the rule-matching and target-invocation path, including `MatchedEvents`, `TriggeredRules`, `InvocationsCreated`, `InvocationAttempts`, `SuccessfulInvocationAttempts`, `Invocations`, `FailedInvocations`, and ingestion-to-invocation latency metrics. [S1] -- AWS guidance for EventBridge operations is to watch invocation failures, retry behavior, dead-letter-queue flow, and throttle limits because EventBridge can retry for up to 24 hours and up to 185 times before giving up. [S2][S7][S8] -- "Good" for EventBridge is not host health. Good means expected events are being matched, the intended number of targets are being invoked, failures are rare, retries are low, and end-to-end delivery latency stays close to baseline. [S1][S2] -- The current Tsuga namespace spot check found 10 live `aws_events_*` metrics, which is a compact surface. That makes a single dashboard more appropriate than separate overview and deep-dive dashboards for this integration. [Tsuga preflight] - -### Best-practice inference -- Incident shape 1: **routing looks normal but delivery fails**. `Matched Events` and `Triggered Rules` stay flat while `Permanent Failure Rate (%)` rises. Start in `delivery-reliability`. -- Incident shape 2: **fanout explosion**. `Average Invocations Per Matched Event` jumps while `Successful Attempt Rate (%)` falls or latencies climb. Start in `fanout-capacity`. -- Incident shape 3: **managed service delay without obvious hard failure**. `Matched Events` and `Successful Invocation Attempts` keep moving, but ingestion-to-success latency widens. Start in `latency-path`. -- Dashboard success criteria: one surface should show whether events are getting matched, whether targets are keeping up, whether failures are durable or retryable, and whether routing fanout changed enough to threaten quotas or downstream systems. - -## Key concepts - -### Glossary -| Term | Definition | Operational meaning | Dashboard section | -|---|---|---|---| -| Event bus | The logical router that receives events and evaluates rules. | Primary blast-radius boundary for EventBridge incidents. | routing-volume | -| Rule | A pattern plus target set that defines when EventBridge should act. | Most useful ownership and fanout dimension after bus name. | routing-volume | -| Target | The downstream AWS service or endpoint invoked by a matched rule. | Determines latency, retry, and failure behavior. | delivery-reliability | -| Matched event | An event that matched at least one rule. | Best traffic proxy for routed work entering the rule engine. | routing-volume | -| Triggered rule | A rule that matched and fired for an event. | Shows how much rule evaluation is producing actionable work. | routing-volume | -| Invocation created | A target invocation that EventBridge scheduled from a rule match. | Primary fanout signal because one event can create multiple invocations. | fanout-capacity | -| Invocation attempt | An attempt to deliver work to a target, including retries. | Higher than invocations indicates retries or repeated delivery pressure. | delivery-reliability | -| Successful invocation attempt | A target delivery attempt that succeeded. | Best success numerator for delivery quality. | delivery-reliability | -| Invocation | A completed target invocation accounting line item in AWS metrics. | Useful denominator for permanent-failure ratio. | delivery-reliability | -| Failed invocation | A target invocation that exhausted retries or otherwise failed permanently. | Non-zero is the strongest failure signal in the current namespace. | delivery-reliability | -| Ingestion-to-invocation start latency | Time from event ingestion until EventBridge begins target delivery. | Captures routing and queueing delay before the first attempt. | latency-path | -| Ingestion-to-invocation complete latency | Time from ingestion until an invocation attempt completes. | Helps separate queue/start delay from attempt completion time. | latency-path | -| Ingestion-to-invocation success latency | Time from ingestion until EventBridge records a successful delivery. | Best end-to-end latency signal for healthy successful paths. | latency-path | -| Retry policy | EventBridge's retry window and attempt policy for failed deliveries. | Explains why attempts can rise without immediate permanent failures. | delivery-reliability | -| Dead-letter queue (DLQ) | Optional SQS queue for undelivered target events. | Critical for post-failure capture, but DLQ metrics are absent from the current namespace. | delivery-reliability | -| Invocation throttle limit | Service quota limiting target invocations per Region. | Exceeding it produces delay and throttling symptoms before total failure. | fanout-capacity | -| Event bus logs | Structured execution logs emitted by EventBridge to CloudWatch Logs, Firehose, or S3. | Best Stage 4 source for tracing individual event routing steps. | all | -| CloudTrail management event | API audit record for EventBridge control-plane actions. | Best source for config-change correlation when routing behavior changes suddenly. | routing-volume | -| Schema discovery | EventBridge feature that infers event schemas into a registry. | Useful for producer/event-shape debugging, but not a delivery SLI by itself. | routing-volume | -| Archive and replay | EventBridge feature for storing and replaying past events. | Important remediation tool when rules were wrong or targets were unavailable. | delivery-reliability | -| Custom event | Application-published event typically sent via `PutEvents`. | If publish-side signals are missing, CloudTrail or producer metrics may be needed. | routing-volume | -| Managed rule fanout | The ratio of rules and targets generated from each matched event. | Sudden increases can create cost, latency, and quota pressure. | fanout-capacity | - -### Concept Map -Event producer -> sends -> Event bus (why: EventBridge starts all routing from bus ingestion) -AWS service event -> lands on -> Event bus (why: managed AWS sources bypass custom producer code) -Custom application -> calls -> PutEvents (why: producer-side event publishing path) -Event bus -> evaluates -> Rule (why: rules decide whether downstream work is created) -Rule -> matches -> Event (why: only matching events generate delivery work) -Matched event -> can trigger -> multiple rules (why: one event can fan out across several rule patterns) -Triggered rule -> creates -> one or more target invocations (why: each rule may have multiple targets) -Invocations created -> become -> invocation attempts (why: EventBridge must actually deliver to targets) -Invocation attempt -> succeeds -> successful invocation attempt (why: downstream target accepted delivery) -Invocation attempt -> fails transiently -> retry policy (why: EventBridge retries before marking durable failure) -Retry policy -> increases -> invocation attempts (why: retries consume attempt budget and time) -Retry exhaustion -> produces -> failed invocation (why: EventBridge gives up after retry policy completes) -Target unavailability -> increases -> ingestion-to-success latency (why: retries extend end-to-end completion time) -Rule explosion -> increases -> invocations created (why: more rules or more targets per rule multiply delivery work) -Higher fanout -> consumes -> invocation throttle quota (why: each created invocation counts against regional capacity) -Quota pressure -> delays -> invocation start (why: throttling shows up before end-to-end success) -Event bus logs -> record -> execution steps (why: they show where the pipeline stalled) -CloudTrail -> records -> configuration changes (why: rule edits or target edits can explain sudden behavior shifts) -DLQ -> captures -> undelivered events (why: preserves payloads for remediation after delivery failure) -Archive -> stores -> historical events (why: replay can recover from bad rules or temporary outages) -Replay -> re-enters -> event bus (why: EventBridge reruns matching and invocation logic) -Schema discovery -> describes -> event structure (why: helps validate producers and event patterns) -Account and Region -> scope -> quotas and metrics (why: EventBridge limits and CloudWatch metrics are regional) -Team ownership -> maps to -> rule and bus boundaries (why: routing incidents are usually owned at those layers) -Managed service health -> is observed through -> routing, attempts, failures, and latency (why: there is no host-level EventBridge dashboard) - -### Entities and dimensions -| Entity or dimension | Why useful | Cardinality risk | Safe top-N suggestion | Do NOT group-by guidance | -|---|---|---|---|---| -| `context.env` | Environment boundary for prod and non-prod routing differences. | Low | 5 | Keep as dashboard filter, not a chart legend default. | -| `context.team` | Ownership split for event buses or rule sets. | Low | 10 | Use for filtering, not primary triage. | -| `context.cloud.region` | Quotas and EventBridge behavior are regional. | Low | 10 | Avoid mixing Region with per-rule groupings in the same chart. | -| `context.cloud.account.id` | Multi-account organizations often isolate buses by account. | Low | 10 | Use for filtering before deeper breakdowns. | -| `context.rulename` | Best confirmed dimension for rule explosion or broken patterns. | Medium | 20 | Prefer top-lists before dense multi-line charts. | -| `context.source` | Best confirmed source-family dimension across all 10 metrics. | Medium | 12 | Do not pair with too many secondary labels by default. | -| `context.cloud.region` | Primary fallback for `aws_events_invocations_created`, which lacks `context.rulename`. | Low | 10 | Use after rule and source views if the issue is regional. | -| Event bus name | Operationally important concept, but not present as a confirmed metric attribute in the current namespace. | N/A | N/A | Use logs or config inventory, not metric group-by. | -| Target id | Operationally important concept, but not present as a confirmed metric attribute in the current namespace. | N/A | N/A | Use logs or downstream service telemetry instead. | -| `context.eventsourcename` | Distinguishes AWS source, partner source, or custom app source when available outside the current metric set. | Medium | 12 | Not confirmed in the current namespace. | -| `context.detailtype` | Helps spot one noisy event family. | Medium-High | 12 | Avoid as first-line dashboard split if producers emit many types. | -| `context.replayname` | Important only when archives or replays are active. | Low | 10 | Gate or hide when replay is unused. | -| `context.endpointname` | Relevant if API destinations are used. | Low-Medium | 10 | Keep out of default widgets unless API destinations are confirmed. | -| `context.schemaregistry` | Useful only for schema-discovery troubleshooting. | Low | 10 | Not a routing SLI dimension. | - -### Tsuga field mapping -| Vendor or exporter dimension | Recommended context.* key | Must-exist vs optional | -|---|---|---| -| RuleName | `context.rulename` | Confirmed optional field on 9 of 10 metrics | -| Source | `context.source` | Confirmed optional field on all 10 metrics | -| EventBusName | Unknown in current metric catalog | Not present in confirmed metric attributes | -| TargetId | Unknown in current metric catalog | Not present in confirmed metric attributes | -| TargetArn | Unknown in current metric catalog | Not present in confirmed metric attributes | -| EventSourceName | `context.eventsourcename` | Optional | -| DetailType | `context.detailtype` | Optional | -| ReplayName | `context.replayname` | Optional | -| API destination or connection identifier | `context.endpointname` | Optional | -| Region | `context.cloud.region` | Confirmed optional field | -| Account | `context.cloud.account.id` | Confirmed optional field | -| Environment | `context.env` | Must-exist | -| Team | `context.team` | Must-exist | - -### Confirmed by sources -- AWS documents EventBridge metrics with dimensions around event buses, rules, and the invocation pipeline. [S1] -- Tsuga metadata confirms `context.rulename`, `context.source`, `context.cloud.account.id`, `context.cloud.region`, `context.env`, and `context.team` on the live `aws_events_*` namespace. [Tsuga Stage 2] -- Event bus logs include execution-step metadata such as event bus, rules, targets, timestamps, and details blocks, which supports mapping several routing dimensions into Tsuga log fields even when the metric catalog does not expose them. [S5][S6] - -### Best-practice inference -- Stage 2 confirmed that the current metric catalog does not carry event-bus or target identifiers, so those pivots must come from logs or other telemetry. -- The namespace likely comes from a CloudWatch-style exporter because keys are lowercased and normalized in the same style as other AWS integrations in this repo. - -## Golden signals - -### Confirmed by sources -| Signal | What it means for EventBridge | Typical causes when it degrades | Best telemetry sources | What people page on | Section questions | -|---|---|---|---|---|---| -| Traffic | How many events are matching rules and how much delivery work gets created. | Producer outage, rule edits, event pattern drift, event storms. | `MatchedEvents`, `TriggeredRules`, `InvocationsCreated` [S1] | Expected event volume disappears or fanout spikes unexpectedly. | Are events still matching? Did rule or target fanout change? | -| Errors | Whether delivery attempts are failing durably instead of succeeding after retries. | Broken target IAM, deleted target resources, downstream endpoint failures, exhausted retries. | `FailedInvocations`, `InvocationAttempts`, `SuccessfulInvocationAttempts` [S1][S2][S7][S8] | Persistent non-zero failure rate or falling success rate. | Are failures transient, retryable, or permanent? | -| Latency | Time from event ingestion to target delivery start, completion, and success. | Invocation throttling, target latency, retry backoff, service-side queueing. | `IngestiontoInvocationStartLatency`, `IngestiontoInvocationCompleteLatency`, `IngestiontoInvocationSuccessLatency` [S1] | Success latency grows materially above start latency baseline. | Where is delay entering: before attempts, during attempts, or across retries? | -| Saturation | Pressure from fanout growth or regional invocation limits. | Too many rules per event, too many targets per rule, downstream slowdown, quota exhaustion. | `InvocationsCreated`, `InvocationAttempts`, quotas and throttle guidance [S1][S2][S7] | Attempts rise much faster than invocations created or latency rises with steady traffic. | Are we creating more work than the service or targets can drain? | - -### Best-practice inference -- For EventBridge, `consumer lag`-style thinking maps to end-to-end invocation latency rather than host CPU. Managed service incidents usually surface first as routing delay or retry amplification, not infrastructure gauges. -- `FailedInvocations` matters more than raw `Invocations` because it reflects events that never made it to the target even after EventBridge exhausted retries. - -## Telemetry sources - -### Confirmed by sources -| Source type | How collected | What it provides | Pros and cons | Common pitfalls | -|---|---|---|---|---| -| `AWS/Events` CloudWatch metrics | Native AWS service metrics, typically exported into Tsuga from CloudWatch | Counts for matches, rules, invocations, attempts, failures, and latency | Best coarse operational SLI surface; low implementation effort. Limited per-payload context. | Missing publisher-side `PutEvents*` or DLQ metrics means the namespace can look healthier than the full pipeline. [S1] | -| Event bus logs | EventBridge emits structured logs to CloudWatch Logs, Firehose, or S3 | Per-event execution steps, rule matches, target attempts, and outcomes | Best source for Stage 4 parsing and root-cause drill-down. Requires logging to be enabled and costs extra. | Log volume can be high on busy buses; logs can contain sensitive event data and need careful scope selection. [S5][S6] | -| CloudTrail management events | AWS CloudTrail captures EventBridge API activity | Configuration changes such as rule edits, bus policy changes, target changes | Excellent for correlating behavior shifts to config drift. Does not explain per-event delivery outcome. | Easy to overuse for runtime debugging even though it is control-plane only. [S4] | -| SQS dead-letter queues | Optional EventBridge rule target DLQ | Original undelivered event payloads plus failure context | Best evidence for post-failure triage and replay workflows. | DLQ metrics are absent from the current `aws_events_*` namespace, so dashboards must call out this blind spot. [S8] | -| Schema discovery and registry | Optional EventBridge schemas feature | Event structure inventory for producers and consumers | Useful for understanding event shape drift. Not itself a health signal. | Customer managed keys are not supported for discovered schema sources. [S9] | - -### Best-practice inference -- "No data" in `aws_events_*` often means the EventBridge path is idle, logging is disabled, or an exporter omitted certain AWS metrics. It does not automatically mean healthy zero. -- Pair EventBridge metrics with one log source and one control-plane source. Metrics alone cannot tell whether a failure came from IAM, target deletion, payload shape drift, or quota pressure. - -## Log intelligence (Stage 4 handoff) - -### Confirmed by sources -1. **Log sources matrix** - -| Source | Access path | Typical format | Structured vs unstructured | Evidence | -|---|---|---|---|---| -| Event bus logs | CloudWatch Logs log group, Kinesis Data Firehose, or Amazon S3 destination configured on an event bus | JSON record with metadata, execution step, and details block | Structured | [S5][S6] | -| CloudTrail management events | CloudTrail event history, S3 trail, or CloudWatch Logs trail integration | JSON audit event | Structured | [S4] | -| SQS dead-letter queue payloads | SQS queue configured as EventBridge DLQ | JSON event payload plus queue envelope fields | Structured | [S8] | - -2. **Known log formats** -- **Event bus log record** - - Sample line: `{"resource_arn":"arn:aws:events:REGION:ACCOUNT:event-bus/default","message_type":"EVENT_RECEIPT","log_level":"INFO","details":{"event_bus_name":"default","source":"my.app","detail_type":"order.created"}}` - - Shape notes: single JSON object per event-processing step; top-level metadata plus nested `details`. - - Timestamp pattern: epoch-millis style timestamps plus step-specific timestamps inside `details`. [S6] - - Quoting behavior: standard JSON quoting. - - Optional fields: rule, target, error, replay, trace headers vary by execution step. -- **CloudTrail management event** - - Sample line: `{"eventSource":"events.amazonaws.com","eventName":"PutRule","awsRegion":"eu-west-1","requestParameters":{"name":"my-rule"}}` - - Shape notes: standard CloudTrail JSON envelope with nested request and response blocks. [S4] - -3. **Candidate query filters for Stage 4** -- Precise: `resource_arn:*event-bus/* AND message_type:*INVOCATION*` - - Rationale: targets the EventBridge delivery path and filters out unrelated control-plane noise. - - Risk: depends on Stage 4 confirming exact field names in emitted logs. -- Fallback: `"events.amazonaws.com" OR "event-bus" OR "FailedInvocations"` - - Rationale: broader catch-all when the exact structured fields are unknown. - - Risk: will pull CloudTrail and operational noise together and need post-filtering. - -4. **Attribute mapping hints** - -| Raw field | Suggested Tsuga key | Confidence | Notes | -|---|---|---|---| -| `resource_arn` | `context.resource.arn` | Medium | Stable top-level field in event bus logs. | -| `details.event_bus_name` | `context.eventbusname` | Medium | Preferred bus dimension if present. | -| `details.rule_name` | `context.rulename` | Medium | Key field for rule explosion or misconfiguration. | -| `details.target_id` | `context.targetid` | Medium | Useful for target-specific failure attribution. | -| `details.source` | `context.source` | Medium | Maps event producer namespace. | -| `details.detail_type` | `context.detailtype` | Medium | Good for high-level event family filtering. | -| `log_level` | `context.log.level` | High | Straightforward log attribute. | -| `message_type` | `context.message.type` | Medium | Distinguishes execution phases. | -| `eventName` | `context.aws.api.action` | High | CloudTrail control-plane action. | -| `awsRegion` | `context.cloud.region` | High | Shared between CloudTrail and AWS resource context. | - -5. **Parsing risks** -- Different event bus execution steps emit different `details` shapes. [S6] -- Log volume can be very high if all steps are logged for a busy bus. [S5] -- Event payload fields may be large or sensitive; avoid exploding them into top-level attributes without need. [S5] -- CloudTrail records config changes, not runtime delivery outcomes, so mixing both streams can blur meaning. [S4] -- DLQ messages wrap the original event with SQS delivery metadata, which can create dual-format parsing requirements. [S8] - -### Best-practice inference -- Start Stage 4 with event bus logs if they are enabled. They have the best chance of linking a failed metric spike back to a specific rule or target. -- If event bus logs are disabled, CloudTrail plus DLQ ingestion is the next-best combination, but it will not cover transient retry behavior as clearly. - -## Caveats and footguns -- **[routing-volume]** `MatchedEvents` means events that matched at least one rule, not events published to the bus. A producer outage and a rule-pattern mistake can look similar if you only watch this metric. (S1) -- **[routing-volume]** `TriggeredRules` can exceed `MatchedEvents` because one matched event can trigger multiple rules. Treat the ratio as a fanout signal, not a health regression by itself. (S1) -- **[fanout-capacity]** `InvocationsCreated` can exceed both `MatchedEvents` and `TriggeredRules` if triggered rules have multiple targets. A sudden jump often reflects configuration drift. (S1) -- **[delivery-reliability]** `InvocationAttempts` includes retries, so rising attempts do not automatically mean more unique business events. (S1, S7) -- **[delivery-reliability]** `SuccessfulInvocationAttempts` can stay high while `FailedInvocations` rises if EventBridge is succeeding eventually for some targets but exhausting retries for others. (Inference) -- **[delivery-reliability]** `FailedInvocations` is a durable-failure signal after retry behavior, not an immediate first-attempt error counter. (S1, S2, S7) -- **[delivery-reliability]** The current namespace lacks explicit retry counters and DLQ counters, so retry amplification must be inferred from ratios between attempts and successes. (Tsuga preflight) -- **[latency-path]** `IngestiontoInvocationStartLatency` widening with stable success latency delta usually indicates service-side queueing or throttling before delivery even begins. (Inference) -- **[latency-path]** `IngestiontoInvocationSuccessLatency` includes both queueing and retry delay, so it is not directly comparable to downstream target runtime alone. (S1) -- **[latency-path]** Latency metrics in the current Tsuga namespace are typed as summaries; Stage 2 must verify whether `average`, `max`, or percentile aggregations are safe in Tsuga. (Tsuga preflight) -- **[fanout-capacity]** Regional invocation throttle limits can create delay before you see obvious permanent failures. Watch latency and attempts together. (S2, S7) -- **[fanout-capacity]** A new rule with broad event patterns can multiply target work without any producer-side traffic increase. (S2) -- **[routing-volume, fanout-capacity]** Archive and replay operations can legitimately spike matched events and invocations. Treat replay windows separately from live traffic. (S5) -- **[routing-volume]** Schema discovery is useful for payload understanding but is not a routing success metric. Do not place schema counts in the main health surface. (S9) -- **[delivery-reliability]** DLQ absence is not proof of health. If DLQs are not configured, failures may still happen but only surface as `FailedInvocations` or logs. (S8) -- **[delivery-reliability]** Target IAM failures, deleted targets, and target throttling can all collapse into delivery-failure symptoms. Metrics alone will not disambiguate them. (S8) -- **[routing-volume]** Event bus logs can contain full event data. Turning them on broadly may create privacy or cost issues and should be scoped deliberately. (S5) -- **[routing-volume]** CloudTrail is control-plane only. A clean CloudTrail timeline does not prove runtime delivery is healthy. (S4) -- **[fanout-capacity]** Grouping by raw target ARN in charts can become noisy if rules target many unique resources. Prefer top-lists and stage gated views. (Inference) -- **[routing-volume, delivery-reliability, latency-path, fanout-capacity]** The current `aws_events_*` namespace contains only 10 metrics, so dashboards must call out blind spots instead of inventing absent publisher, retry, or DLQ signals. (Tsuga preflight) - -## Confirmed Tsuga prefixes -- `aws_events*` — **CONFIRMED** (10/10 metrics present after Stage 2 catalog bootstrap and MCP metadata checks on 2026-03-18; exact names: `aws_events_failed_invocations`, `aws_events_ingestion_to_invocation_complete_latency`, `aws_events_ingestion_to_invocation_start_latency`, `aws_events_ingestion_to_invocation_success_latency`, `aws_events_invocation_attempts`, `aws_events_invocations`, `aws_events_invocations_created`, `aws_events_matched_events`, `aws_events_successful_invocation_attempts`, `aws_events_triggered_rules`). - -## Discovery status -- Discovery: Stage 2 reconciliation completed. -- `METRICS_FOUND`: 10 `aws_events_*` metrics in Tsuga, all of which match the operational metrics already documented in `01_aws-eventbridge_metrics.csv`. -- Confirmed context field registry: `context.rulename` on 9 metrics, `context.source` on all 10 metrics, plus `context.cloud.account.id`, `context.cloud.region`, `context.env`, and `context.team`. -- Confirmed metric metadata: all 10 metrics are `summary` with `cumulative` temporality and support `avg|count|max|min|sum`. -- Notable gap: a broader catalog search did not find AWS-documented `PutEvents*`, retry, or DLQ metric families for EventBridge in this environment. -- Spot-check note: MCP scalar aggregation returned an internal server error, so recent-data verification was inconclusive even though the catalog and metadata reconciliation succeeded. - -## Top sources -1. https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-monitoring.html - Why: canonical CloudWatch metric names, latency metrics, and operational dimensions for `AWS/Events`. -2. https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-monitoring-events-best-practices.html - Why: AWS's own guidance on what to watch for failures, DLQs, retries, and throttling. -3. https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-event-bus-logs.html - Why: authoritative description of event bus logging destinations, controls, and tradeoffs. -4. https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-event-logs-schema.html - Why: schema for EventBridge event bus log records used in Stage 4 parsing. -5. https://docs.aws.amazon.com/eventbridge/latest/userguide/logging-using-cloudtrail.html - Why: confirms EventBridge control-plane coverage in CloudTrail and the audit-event model. -6. https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-quota.html - Why: quota context for invocation throttle behavior and capacity interpretation. -7. https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-rule-retry-policy.html - Why: retry window and attempt semantics that explain attempt amplification. -8. https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-rule-dlq.html - Why: DLQ behavior, failure scenarios, and why DLQ metrics or messages matter for triage. -9. https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-schemas-infer.html - Why: schema discovery behavior and constraints for event-shape debugging. -10. https://aws.amazon.com/eventbridge/pricing/ - Why: cost model reference when using event volume and invocation volume as cost proxies. - -**Citation key** -- [S1] https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-monitoring.html -- [S2] https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-monitoring-events-best-practices.html -- [S3] https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-event-bus-logs.html -- [S4] https://docs.aws.amazon.com/eventbridge/latest/userguide/logging-using-cloudtrail.html -- [S5] https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-event-bus-logs.html -- [S6] https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-event-logs-schema.html -- [S7] https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-quota.html -- [S8] https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-rule-dlq.html -- [S9] https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-schemas-infer.html diff --git a/plugins/tsuga/skills/knowledge-technology/references/aws-eventbridge/metrics.csv b/plugins/tsuga/skills/knowledge-technology/references/aws-eventbridge/metrics.csv deleted file mode 100644 index d3933b5..0000000 --- a/plugins/tsuga/skills/knowledge-technology/references/aws-eventbridge/metrics.csv +++ /dev/null @@ -1,12 +0,0 @@ -theme,name,confirmed,definition,unit,type,tsuga_metric_name,widget_suggestions,aggregation,post_function,group_by,required_context_fields,optional_context_fields,absence_means,cardinality_footguns,variants,citations -Throughput/Usage,MatchedEvents,true,"Count of events that matched at least one EventBridge rule. Best traffic proxy for routing work entering the rule engine; it does not measure all published events.",{event},summary,aws_events_matched_events,timeseries|query-value|top-list,sum,none,context.rulename:20|context.source:12,context.env|context.team,context.cloud.region|context.cloud.account.id|context.rulename|context.source,Bus is idle, producer traffic is absent, or rules no longer match incoming events,Prefer rule or source groupings; `context.rulename` is bounded but still better in top-lists than dense multi-line charts,cw_metric=MatchedEvents|tsuga=aws_events_matched_events,https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-monitoring.html -Throughput/Usage,TriggeredRules,true,"Count of rules that triggered from matched events. Higher than MatchedEvents usually means multi-rule fanout rather than more producer traffic.",{rule},summary,aws_events_triggered_rules,timeseries|query-value|top-list,sum,none,context.rulename:20|context.source:12,context.env|context.team,context.cloud.region|context.cloud.account.id|context.rulename|context.source,No rules are triggering because the namespace is idle or rule patterns no longer match,Rule names can grow if teams create many narrow rules so default to top-lists before dense timeseries,cw_metric=TriggeredRules|tsuga=aws_events_triggered_rules,https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-monitoring.html -Capacity/Saturation,InvocationsCreated,true,"Count of target invocations created by EventBridge. Primary fanout metric because one matched event can create multiple target deliveries.",{invocation},summary,aws_events_invocations_created,timeseries|query-value|top-list,sum,none,context.source:12|context.cloud.region:10,context.env|context.team,context.cloud.region|context.cloud.account.id|context.source,No target work is being created because no rules are firing or the namespace is idle,This metric has no `context.rulename` in Tsuga so keep grouping at source or Region level,cw_metric=InvocationsCreated|tsuga=aws_events_invocations_created,https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-monitoring.html -Errors/Failures,InvocationAttempts,true,"Count of target invocation attempts including retries. Rising faster than InvocationsCreated usually indicates downstream retry pressure or repeated delivery failures.",{attempt},summary,aws_events_invocation_attempts,timeseries|query-value|top-list,sum,none,context.rulename:20|context.source:12,context.env|context.team,context.cloud.region|context.cloud.account.id|context.rulename|context.source,No delivery attempts are happening because no target work is being created or the namespace is idle,Attempts are retry-inflated so do not compare directly to business event counts,cw_metric=InvocationAttempts|tsuga=aws_events_invocation_attempts,https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-monitoring.html -Availability/Health,SuccessfulInvocationAttempts,true,"Count of successful target delivery attempts. Best success numerator for delivery quality in the current namespace.",{attempt},summary,aws_events_successful_invocation_attempts,timeseries|query-value|top-list,sum,none,context.rulename:20|context.source:12,context.env|context.team,context.cloud.region|context.cloud.account.id|context.rulename|context.source,No successful deliveries are occurring because traffic is idle or delivery attempts are failing,Use alongside InvocationAttempts and FailedInvocations to separate idle from broken,cw_metric=SuccessfulInvocationAttempts|tsuga=aws_events_successful_invocation_attempts,https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-monitoring.html -Availability/Health,Invocations,true,"Count of target invocations recorded by EventBridge. Useful denominator for durable failure share and overall target work completed.",{invocation},summary,aws_events_invocations,timeseries|query-value,sum,none,context.rulename:20|context.source:12,context.env|context.team,context.cloud.region|context.cloud.account.id|context.rulename|context.source,No completed invocation work is being recorded because the namespace is idle or deliveries are stalling,Interpret with care because retries live in InvocationAttempts not Invocations,cw_metric=Invocations|tsuga=aws_events_invocations,https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-monitoring.html -Errors/Failures,FailedInvocations,true,"Count of failed target invocations after EventBridge could not deliver successfully. Strongest durable failure metric in the current namespace.",{invocation},summary,aws_events_failed_invocations,timeseries|query-value|top-list,sum,none,context.rulename:20|context.source:12,context.env|context.team,context.cloud.region|context.cloud.account.id|context.rulename|context.source,No durable target failures were recorded or the namespace is idle,Use top-lists instead of many-line timeseries when many rules exist,cw_metric=FailedInvocations|tsuga=aws_events_failed_invocations,https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-monitoring.html -Performance/Latency,IngestiontoInvocationStartLatency,true,"Latency from event ingestion until EventBridge starts target invocation. Best signal for queueing or throttle delay before the first delivery attempt.",ms,summary,aws_events_ingestion_to_invocation_start_latency,timeseries|query-value,average|max,none,context.rulename:20|context.source:12,context.env|context.team,context.cloud.region|context.cloud.account.id|context.rulename|context.source,No successful routing work reached the start point or the metric is stale,Tsuga reports summary+cumulative with avg|max|min|sum capabilities; avoid percentiles unless Stage 3 explicitly re-validates them,cw_metric=IngestionToInvocationStartLatency|tsuga=aws_events_ingestion_to_invocation_start_latency,https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-monitoring.html -Performance/Latency,IngestiontoInvocationCompleteLatency,true,"Latency from event ingestion until an invocation attempt completes. Useful midpoint to separate start delay from end-to-end success delay.",ms,summary,aws_events_ingestion_to_invocation_complete_latency,timeseries|query-value,average|max,none,context.rulename:20|context.source:12,context.env|context.team,context.cloud.region|context.cloud.account.id|context.rulename|context.source,No completed attempts were recorded or the metric is stale,Use with start and success latency rather than alone,cw_metric=IngestionToInvocationCompleteLatency|tsuga=aws_events_ingestion_to_invocation_complete_latency,https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-monitoring.html -Performance/Latency,IngestiontoInvocationSuccessLatency,true,"Latency from event ingestion until EventBridge records a successful delivery. Best end-to-end latency signal for user-visible delivery health.",ms,summary,aws_events_ingestion_to_invocation_success_latency,timeseries|query-value,average|max,none,context.rulename:20|context.source:12,context.env|context.team,context.cloud.region|context.cloud.account.id|context.rulename|context.source,No successful deliveries completed or the metric is stale,Use rule-level top lists rather than noisy target splits because no target id field is present,cw_metric=IngestionToInvocationSuccessLatency|tsuga=aws_events_ingestion_to_invocation_success_latency,https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-monitoring.html -Cost,NOT_APPLICABLE,false,"EventBridge cost is driven mainly by published custom events, matched rules, and target invocations, but the current `aws_events_*` namespace does not expose a direct billing metric. Use `aws_events_matched_events` and `aws_events_invocations_created` as operational cost proxies and use AWS Cost Explorer for actual spend.",N/A,N/A,N/A,N/A,N/A,N/A,context.env|context.team,context.cloud.account.id,No direct cost metric in this namespace,N/A,N/A,https://aws.amazon.com/eventbridge/pricing/ diff --git a/plugins/tsuga/skills/knowledge-technology/references/aws-eventbridge/queries.md b/plugins/tsuga/skills/knowledge-technology/references/aws-eventbridge/queries.md deleted file mode 100644 index 3609cca..0000000 --- a/plugins/tsuga/skills/knowledge-technology/references/aws-eventbridge/queries.md +++ /dev/null @@ -1,46 +0,0 @@ -# AWS EventBridge - -Event bus. Events → matched rules → invoked targets. Healthy: events matched and invoked with low failure, bounded latency. - -## Incident shapes - -- **Failed invocations** — target sick, IAM wrong, or target-side throttling -- **Invocation latency** — `aws_events_ingestion_to_invocation_complete_latency` p95 climbs -- **No matches** — `aws_events_matched_events` drops to zero on a previously busy rule → upstream stopped OR rule drifted -- **Rule-config drift** — `aws_events_triggered_rules` changes without a deploy → check audit trail -- **Account throttling** — PutEvents throttled; producers retry, effective rate drops - -## Key metrics - -| Metric | Unit | Signal | -|---|---|---| -| `aws_events_matched_events` | count | Events matching at least one rule | -| `aws_events_triggered_rules` | count | Rule fires | -| `aws_events_invocations_created` | count | Target invocations queued | -| `aws_events_invocation_attempts` | count | Tries including retries | -| `aws_events_successful_invocation_attempts` | count | Successes | -| `aws_events_failed_invocations` | count | Terminal failures | -| `aws_events_ingestion_to_invocation_start_latency` | ms | Ingest → first attempt | -| `aws_events_ingestion_to_invocation_complete_latency` | ms | Ingest → success | - -## Derived signals - -- `aws_events_successful_invocation_attempts / aws_events_invocation_attempts` — success rate. Baseline > 0.99. -- `(aws_events_invocation_attempts - aws_events_invocations_created) / aws_events_invocations_created` — retry rate. High = target rejecting. -- `aws_events_triggered_rules / aws_events_matched_events` — fan-out ratio. Sudden shift = rule config changed. - -## Log patterns - -- CloudTrail `PutRule` / `PutTargets` / `DeleteRule` / `DisableRule` — config changes -- Target Lambda: `Task timed out` — invocation timed out target-side -- Target SQS DLQ depth growing = terminal failures -- Schema registry errors on PutEvents — schema enforcement (if enabled) -- `ValidationException` on PutEvents — malformed event JSON - -## Gotchas - -- Not exactly-once: targets can be invoked multiple times. Targets must be idempotent. -- `aws_events_failed_invocations` increments only after retries exhausted. Flaky targets show as elevated `aws_events_invocation_attempts` before failures appear. -- Rule patterns: AND within a field, OR within `anything_but`. Subtle changes can silently match zero events. -- Archive-and-replay can inflate `aws_events_matched_events` without real traffic change. -- Cross-region / cross-account (global endpoints) has separate throttling. diff --git a/plugins/tsuga/skills/knowledge-technology/references/aws-firehose/README.md b/plugins/tsuga/skills/knowledge-technology/references/aws-firehose/README.md deleted file mode 100644 index a10e1f4..0000000 --- a/plugins/tsuga/skills/knowledge-technology/references/aws-firehose/README.md +++ /dev/null @@ -1,402 +0,0 @@ -# AWS Firehose Integration Context Bundle - -## Metadata - -**Technology:** Amazon Data Firehose (formerly Amazon Kinesis Data Firehose) -**Deployment:** managed -**Environment:** prod -**Persona:** SRE Dev and ops -**Telemetry preference:** Prometheus CloudWatch Exporter (CONFIRMED — Stage 2 discovery) -**Integration scope:** core service only -**Primary use-case:** reliability and performance -**Destination (this org):** HTTP endpoint (CONFIRMED — DeliveryToS3.* metrics absent; delivery_to_http_endpoint_* metrics present) -**Stage 2 discovery:** COMPLETED 2026-03-16 - ---- - -## How to use this bundle - -- `01_aws-firehose_metrics.csv` — source-of-truth for all metric names, units, temporality, safe aggregations, and group-bys. Use this as the widget data layer. -- `02_aws-firehose_dashboard_plan.yaml` — dashboard blueprint: sections, widgets, derived signals, explanation notes, triage chains, and playbooks. -- `03_aws-firehose_state.yaml` — machine-readable stage status, confirmed/inferred namespace prefixes, assumptions, and the unknowns list that Stage 2 must resolve. -- `04_aws-firehose_memory.md` — human-readable Stage 1 summary, key tradeoffs, and Stage 2 priority checks. -- Stage 2 will create `05_aws-firehose_metric_catalog.csv` — discovered metric inventory with attribute keys and AI-curated descriptions for reconciliation and coverage checks. -- Stage 4: read "Log intelligence (Stage 4 handoff)" section below and `03_aws-firehose_state.yaml` → `log_intel` before designing log routes. - ---- - -## What it is and what "good" looks like - -Amazon Data Firehose is a fully managed streaming ETL service that reliably ingests and delivers streaming data to storage and analytics destinations (Amazon S3, Redshift, OpenSearch Service, OpenSearch Serverless, Splunk, Snowflake, HTTP endpoints, and custom destinations). It handles buffering, compression, encryption, format conversion, and optional Lambda transformation. Producers write to a Delivery Stream via PutRecord/PutRecordBatch API calls or connect Kinesis Data Streams / MSK (Kafka) as a managed source. - -**What "good" looks like (this org — HTTP endpoint destination, Prometheus CloudWatch Exporter):** -- `aws_firehose_delivery_to_http_endpoint_data_freshness` (Max) stays flat and low (well below buffer interval × 2 — typically below 120s for a 60s buffer) -- `aws_firehose_delivery_to_http_endpoint_processed_bytes` ≈ `aws_firehose_delivery_to_http_endpoint_bytes` (HTTP Endpoint Retry Rate near 0%) -- `aws_firehose_throttled_records` = 0 (no records rejected at ingestion — critical: throttled records are silently dropped) -- `aws_firehose_incoming_records` tracks expected application throughput (flat or predictable growth) -- `aws_firehose_failed_validation_records` = 0 (no MSK/Kafka schema validation failures) -- No KMS error metrics non-zero (all four kms_key_* metrics stay at 0) -- `aws_firehose_put_record_batch_latency` stays low (< 500ms average under normal load) - -**Paging intent (symptom-level only):** -- Page if `aws_firehose_delivery_to_http_endpoint_data_freshness` (Max) climbs significantly above the buffer interval — records are not reaching the HTTP endpoint -- Page if `aws_firehose_throttled_records` is sustained non-zero — data is being permanently lost (not retried) -- Page if any KMS error metric is non-zero — all delivery is blocked until KMS is fixed -- Page if `aws_firehose_failed_validation_records` spikes — MSK/Kafka records are being silently discarded - -**Top 3 incident shapes (updated for HTTP endpoint destination):** - -1. **Delivery stall** (`aws_firehose_delivery_to_http_endpoint_data_freshness` climbing, HTTP Endpoint Retry Rate rising) → Start at "Delivery Freshness" section; check HTTP endpoint availability and response codes in CloudWatch error logs (`/aws/kinesisfirehose/{stream-name}`). KMS errors take priority — check Total KMS Errors first. -2. **Ingestion throttling** (`aws_firehose_throttled_records` non-zero) → Start at "Ingestion" section; notify producers immediately (data loss in progress); file AWS Service Quotas increase. Note: BytesPerSecondLimit/RecordsPerSecondLimit not exported by org's collector — use Throttle Rate as the primary saturation signal. -3. **MSK/Kafka schema validation failure** (`aws_firehose_failed_validation_records` non-zero) → Check `context.sourcepartitionid` in the Failed Validation widget to identify the failing Kafka partition; coordinate with producer team on schema alignment. Lambda transformation failures are not applicable (Lambda not enabled on this org's streams). - -**Confirmed by sources:** -- DataFreshness as primary operational signal: https://docs.aws.amazon.com/firehose/latest/dev/monitoring-with-cloudwatch-metrics.html -- KMS errors as immediate delivery-blocker: https://docs.aws.amazon.com/firehose/latest/dev/encryption.html -- ThrottledRecords as silent data-loss indicator: AWS Firehose Developer Guide, Monitoring section - -**Best-practice inference:** -- "Good DataFreshness" threshold = buffer interval × retry buffer (inferred from standard Firehose buffer configuration patterns) -- Lambda 5-minute hard timeout as the critical alert threshold (inferred from AWS Lambda limits documentation) - ---- - -## Key concepts - -### Glossary - -| Term | Definition | Operational meaning | Dashboard section | -|---|---|---|---| -| Delivery Stream | The named Firehose resource — a configured pipeline from source to destination with buffering, transformation, and compression settings. | Primary group-by dimension in all widgets. One stream = one pipeline. | All sections | -| DataFreshness | The age in seconds of the oldest record currently held by Firehose that has not yet been delivered. | THE primary SRE signal. Rising DataFreshness = data backlog. Use Maximum stat for worst-case. | delivery-freshness | -| Buffer Interval | Configurable time limit (60s–900s) after which Firehose flushes buffered data to the destination, even if the buffer size is not reached. | Baseline for DataFreshness. Healthy DataFreshness ≤ buffer interval × 2. | delivery-freshness | -| Buffer Size | Configurable data size limit (varies by destination) that triggers a flush when reached before the buffer interval. | Affects how frequently S3 objects are created. Smaller = more frequent small objects. | delivery-throughput | -| Direct PUT | Data ingestion method where producers call PutRecord/PutRecordBatch API directly on the Delivery Stream. | Source for IncomingRecords/IncomingBytes. Throttling occurs here if limits exceeded. | ingestion | -| Kinesis Data Streams (source) | Optional: use a Kinesis Data Stream as the Firehose source instead of Direct PUT. Firehose reads from the stream. | Adds KinesisMillisBehindLatest as a separate freshness signal. Throttling at GetRecords level. | ingestion | -| MSK (Kafka) source | Optional: use an Amazon MSK Kafka cluster as the Firehose source. | Adds KafkaOffsetLag as freshness signal. Similar pattern to Kinesis source. | ingestion | -| ThrottledRecords | Records rejected by Firehose because ingestion rate exceeded service limits. These records are NOT retried — they are silently dropped by the service. | Non-zero = data loss. The producer must retry dropped records itself. Critical to alert on. | ingestion | -| BytesPerSecondLimit | Current service-side throughput limit for the delivery stream (bytes/s before throttling occurs). | Compare against IncomingBytes rate to see headroom. Approaching limit = imminent throttling. | ingestion | -| RecordsPerSecondLimit | Current service-side records/sec limit for the delivery stream. | Similar to BytesPerSecondLimit but for record count. | ingestion | -| Lambda Transformation | Optional feature: Firehose invokes an AWS Lambda function to transform each record batch before delivery. | Adds ExecuteProcessing.* and SucceedProcessing.* metrics. Duration limit = 5 minutes hard. | transformation | -| ExecuteProcessing.Success | Ratio (0–1) of successful Lambda invocations to total invocations over the period. Average stat gives the success rate. | Drop below 1.0 = Lambda transformation errors. Key signal for transformation health. | transformation | -| Format Conversion | Optional feature: Firehose converts record format from JSON to Apache Parquet or ORC before S3 delivery. Requires a Glue Data Catalog schema. | Adds FailedConversion.* and SucceedConversion.* metrics. Failures = records routed to error S3 prefix. | transformation | -| Dynamic Partitioning | Optional feature: Firehose routes records to different S3 prefixes based on data content (using JQ or inline parsing). | Adds PartitionCount and PartitionCountExceeded metrics. Hard limit: 500 active partitions. | backup-limits | -| DeliveryToS3.Success | Count (Sum) or ratio (Average 0–1) of successful S3 PUT commands. Use Average stat to get success rate. | Average near 1.0 = healthy. Dropping = delivery failures. Use Sum to count absolute PUT successes. | delivery-errors | -| Backup to S3 | When Lambda transformation or Redshift delivery is enabled, Firehose can write source records to a backup S3 bucket. BackupToS3.* metrics track this. | BackupToS3.DataFreshness is a secondary freshness signal for transformation pipelines. | backup-limits | -| KMS Encryption | Optional: Firehose can encrypt data at rest using AWS KMS. KMS errors immediately block all delivery. | KMS* metrics: any non-zero value = all delivery blocked. Highest severity failure. | delivery-errors | -| Error Output S3 Prefix | When delivery fails after all retries (or format conversion fails), records are written to a special error prefix in S3. These records need manual reprocessing. | Not directly metriced — you discover this via CloudWatch error logs and DataFreshness. | delivery-errors | -| Retry Duration | Configurable window (0–7200s) during which Firehose retries failed deliveries before routing to the error prefix. | Affects how long DataFreshness will climb before records are abandoned to the error prefix. | delivery-freshness | -| PutRecordBatch | Bulk PUT API call: up to 500 records or 4 MiB per call. More efficient than PutRecord for high-throughput producers. | Source for IncomingPutRequests metric. Throttled at same limits as PutRecord. | ingestion | -| Delivery Stream ARN | Globally unique AWS identifier for the stream. | Used as resource identifier in IAM policies and CloudWatch dimensions. | — | -| KinesisMillisBehindLatest | When Kinesis Data Streams is the source: age (ms) of the last-read Firehose record relative to the newest record in the Kinesis stream. | Critical freshness signal for Kinesis-sourced streams. Rising = Firehose falling behind. | ingestion | -| JQ Processing | When dynamic partitioning is enabled, Firehose uses JQ expressions to extract partition keys from records. JQProcessing.Duration tracks execution time. | Slow JQ = slower partitioning, potential DataFreshness impact. | backup-limits | -| Redshift COPY | When Redshift is the destination, Firehose buffers to an intermediate S3 location then issues a Redshift COPY command. DeliveryToRedshift.Success tracks success of these COPY commands. | COPY failures → DataFreshness climbs + BackupToS3.DataFreshness climbs. | delivery-errors | -| ObjectCount | S3 objects delivered per period (dynamic partitioning only). Higher partition count = more objects per buffer flush. | Rising ObjectCount with flat DataFreshness = healthy high-partition throughput. | backup-limits | - ---- - -### Concept Map - -``` -Producer (app/service) -> PutRecord/PutRecordBatch API -> Delivery Stream (ingestion layer) -Producer -> Kinesis Data Stream (source) -> Delivery Stream reads via GetRecords (Kinesis source mode) -Producer -> MSK/Kafka (source) -> Delivery Stream reads via Kafka consumer (MSK source mode) - -Delivery Stream -> buffer (size/interval threshold) -> triggers flush -Delivery Stream ingestion -> ThrottledRecords if IncomingBytes > BytesPerSecondLimit (why: service limits; throttled records are silently dropped) -Delivery Stream -> optionally invokes Lambda (transformation layer) -> ExecuteProcessing.* metrics -Lambda transformation -> ExecuteProcessing.Duration approaches 5-min limit -> timeout failure path -Lambda failure -> records can route to error prefix or delivery blocked (why: transformation required) - -Delivery Stream buffer -> DeliveryToS3 (primary S3 destination) -DeliveryToS3 failure -> retry window (configurable) -> retry exhausted -> error prefix in S3 -DeliveryToS3 delivery blocked -> DataFreshness climbs -> oldest record age grows -DataFreshness climbing -> leading indicator of all delivery problems (why: any blocker increases record age) - -Delivery Stream -> optionally BackupToS3 (when Lambda transform enabled or Redshift destination) -BackupToS3 failure -> BackupToS3.DataFreshness climbs (why: backup independently buffered) - -Delivery Stream -> optionally format conversion (JSON→Parquet/ORC via Glue schema) -Format conversion failure -> FailedConversion.Records -> records to error prefix (why: schema mismatch) - -Delivery Stream -> optionally Dynamic Partitioning -> PartitionCount (hard limit 500) -PartitionCount approaching 500 -> PartitionCountExceeded=1 -> overflow records to default prefix (why: partition limit breached) - -KMS encryption enabled -> KMS key must be accessible for all delivery operations -KMS key problem -> KMSKey* metrics non-zero -> ALL delivery blocked immediately (why: encryption required for every write) - -Delivery Stream -> CloudWatch error logs -> /aws/kinesisfirehose/{stream-name} -CloudWatch error logs -> DestinationDelivery stream (destination errors) -CloudWatch error logs -> BackupDelivery stream (backup S3 errors, if backup enabled) -``` - ---- - -### Entities and dimensions - -| Entity | Why useful | Cardinality risk | Safe top-N | Notes | -|---|---|---|---|---| -| `DeliveryStreamName` | Primary triage dimension — every metric belongs to one stream. Breakdown by stream reveals which pipeline is affected. | Medium (10s–100s of streams per org) | 10 | Safe for group-by in all sections | -| `context.cloud.region` | Delivery to S3/Redshift is region-scoped; cross-region delivery adds latency. Useful for multi-region setups. | Low (AWS regions in use) | 10 | Safe for group-by | -| `context.cloud.account.id` | Multi-account orgs need to separate streams by account. Critical for cost attribution. | Low (few accounts per org) | 10 | Safe for group-by | -| `context.env` | Separate prod/staging/dev streams — baseline freshness and throughput differ. | Low | 5 | Always include as global filter | -| `context.team` | Different teams own different pipelines. Team-level aggregation for SLO tracking. | Low | 10 | Always include as global filter | -| Destination type | S3 / Redshift / OpenSearch / Splunk / Snowflake / HTTP — different SLA characteristics. | Very low (fixed per stream) | N/A | Not a CloudWatch dimension; encode in stream naming convention or use separate dashboard filters | -| Source type | Direct PUT / Kinesis / MSK — changes which metrics are available. | Very low | N/A | Not a CloudWatch dimension; changes available metrics | -| Lambda function name | If Lambda transformation enabled, function ARN is a Lambda-level dimension, not a Firehose dimension. | Low–medium | 10 | Not in Firehose CloudWatch namespace — look in AWS/Lambda namespace | -| S3 bucket | Delivery destination bucket — not a CloudWatch dimension for Firehose metrics. | N/A | N/A | Do NOT group by S3 bucket in Firehose metrics; track via S3 metrics separately | -| AWS account + region | Full stream identity is account + region + stream name | Low | N/A | Compound key for multi-org setups | -| Partition key (dynamic partitioning) | Partition routing key — not a CloudWatch dimension | N/A | N/A | Track via PartitionCount, not by key value | -| Error code | Delivery error codes from CloudWatch Logs | Low–medium | N/A | Only available in log data, not metrics namespace | -| Retry attempt | Retry attempt number — not exposed as a CloudWatch dimension | N/A | N/A | Not available as a metric dimension | - ---- - -### Tsuga field mapping - -| Vendor/exporter dimension | Confirmed context.* key | Must-exist vs optional | Stage 2 status | -|---|---|---|---| -| `DeliveryStreamName` (CloudWatch dimension) | `context.deliverystreamname` | Must-exist (primary group-by) | **CONFIRMED** via get-metric attribute inspection | -| MSK/Kafka source partition | `context.sourcepartitionid` | Optional (only on failed_validation_records) | **CONFIRMED** present on aws_firehose_failed_validation_records | -| AWS Account ID (implicit in CW data) | `context.cloud.account.id` | Must-exist for multi-account | Confirmed in catalog | -| AWS Region (implicit in CW data) | `context.cloud.region` | Must-exist for multi-region | Confirmed in catalog | -| — (from `.env`) | `context.env` | Must-exist (global filter) | Confirmed in catalog | -| — (from `.env`) | `context.team` | Must-exist (global filter) | Confirmed in catalog | - -**Stage 2 correction:** The assumed attribute key `context.aws.firehose.delivery_stream_name` was incorrect. The actual Tsuga attribute key is **`context.deliverystreamname`** (snake_case, no OTel aws prefix — consistent with Prometheus CloudWatch Exporter label conventions). All widget group_by fields updated in `02_aws-firehose_dashboard_plan.yaml`. - ---- - -## Golden signals - -### Traffic — How much data is flowing? - -**For Firehose:** Traffic = ingestion rate (records and bytes entering the delivery stream). -- **Why it matters:** Unexpected traffic drops or spikes are the first sign of producer failure or runaway event generation. -- **Typical causes of degradation:** Producer outage (traffic drops to zero), retry storm (traffic spikes), application deployment (traffic pattern change). -- **Best telemetry (confirmed):** `aws_firehose_incoming_records` (sum + rate), `aws_firehose_put_record_batch_bytes` (sum + rate, proxy for IncomingBytes — `IncomingBytes` not exported by this org's collector). -- **What people page on:** Traffic drops to zero while expected producers are running (producer failure), or sudden spike combined with throttling. -- **Section questions:** "Is data flowing in at the expected rate?" / "Are producers hitting ingestion limits?" - -**Confirmed by sources:** https://docs.aws.amazon.com/firehose/latest/dev/monitoring-with-cloudwatch-metrics.html -**Best-practice inference:** Traffic → zero combined with DataFreshness staying flat (not rising) indicates no new data, not a delivery block. - ---- - -### Errors — Are delivery failures occurring? - -**For Firehose:** Errors = delivery failures (S3 PUT failures, Lambda invocation failures, format conversion failures, KMS errors, throttled records). -- **Why it matters:** Failed deliveries result in data loss (records routed to error S3 prefix after retry exhaustion). ThrottledRecords at ingestion are silently lost. -- **Typical causes:** IAM permission issues (S3 bucket policy, Lambda execution role), destination unavailability, Lambda function bugs, schema mismatches, KMS key revocation. -- **Best telemetry (confirmed):** `aws_firehose_throttled_records` (sum + rate), KMS error metrics (`aws_firehose_kms_key_*`), `aws_firehose_failed_validation_records` (sum + rate). Note: `DeliveryToS3.Success` absent (S3 not destination); use HTTP Endpoint Retry Rate signal (processed_bytes vs delivery_bytes) for endpoint-level error detection. Lambda and format conversion metrics absent (features not enabled). -- **What people page on:** Success rate dropping, ThrottledRecords non-zero (data loss), any KMS error. -- **Section questions:** "Are delivery failures occurring?" / "Are records being lost to the error prefix?" - -**Confirmed by sources:** ThrottledRecords data loss behavior documented at https://docs.aws.amazon.com/firehose/latest/dev/monitoring-with-cloudwatch-metrics.html -**Best-practice inference:** KMS errors blocking all delivery; IAM failures as leading cause of permission errors. - ---- - -### Latency / Freshness — Is data being delivered on time? - -**For Firehose:** Latency = DataFreshness — the age of the oldest undelivered record. -- **Why it matters for Firehose (not generic latency):** DataFreshness is the canonical end-to-end delivery lag signal. A rising freshness value means data is piling up and not reaching its destination. Unlike traditional request latency, this is the "queue depth clock" — it tells you how stale the destination is. -- **Why `DataFreshness` beats other signals:** It's a leading indicator of ALL delivery problems (permission errors, destination issues, Lambda failures, retry exhaustion) before records are lost. -- **Typical causes of degradation:** Destination unreachable (IAM, network, destination outage), Lambda transformation failures or timeouts, retry loops from transient errors, KMS key issues. -- **Best telemetry (confirmed):** `aws_firehose_delivery_to_http_endpoint_data_freshness` (max aggregation) — always use Maximum stat for worst-case SLO tracking. This org uses HTTP endpoint; the S3 variant (`DeliveryToS3.DataFreshness`) is absent. -- **What people page on:** DataFreshness exceeds configured retry duration (data is about to be abandoned to error prefix). -- **Section questions:** "Is data reaching the destination on time?" / "What is the age of the oldest undelivered record?" - -**Confirmed by sources:** DataFreshness as primary Firehose operational signal: https://docs.aws.amazon.com/firehose/latest/dev/monitoring-with-cloudwatch-metrics.html -**Best-practice inference:** Alert threshold = retry duration (default 300s); severe = DataFreshness > 900s. - ---- - -### Saturation — Are we approaching service limits? - -**For Firehose:** Saturation = proximity to ingestion limits + delivery stream resource utilization. -- **Why it matters:** ThrottledRecords = data is already being lost. `BytesPerSecondLimit` and `RecordsPerSecondLimit` show headroom before throttling starts. -- **Typical causes:** Traffic growth without limit increase requests, uncontrolled retry storm from a broken producer, poorly configured dynamic partitioning hitting the 500-partition limit. -- **Best telemetry (confirmed):** `aws_firehose_throttled_records` (sum + rate) — the only available saturation signal (Throttle Rate % derived signal). `BytesPerSecondLimit`, `RecordsPerSecondLimit`, and `PartitionCount` are not exported by this org's collector and are absent from Tsuga. -- **What people page on:** ThrottledRecords sustained non-zero, PartitionCountExceeded = 1. -- **Section questions:** "Are service limits being hit?" / "How much headroom remains before throttling?" - -**Confirmed by sources:** Throttling behavior at https://docs.aws.amazon.com/firehose/latest/dev/limits.html -**Best-practice inference:** Partition limit of 500 from service limits documentation. - ---- - -## Telemetry sources - -| Source type | How collected | What it provides | Pros/Cons | Common pitfalls | -|---|---|---|---|---| -| **CloudWatch Metric Streams (recommended)** | CloudWatch Metric Streams → OpenTelemetry/Firehose → Tsuga (OTel OTLP ingestion). Metrics are pushed in near real-time (sub-1-minute). | All AWS/Firehose metrics with full dimension metadata. OTel format: metric names as `amazonaws.com/AWS/Firehose/{MetricName}`, type=SUMMARY. | Pro: near real-time, low latency, no polling. Con: Metric Stream itself requires setup; metrics arrive as OTel SUMMARY type (sum, count, min, max quantiles available). | SUMMARY type requires careful aggregation selection in Tsuga. The `sum` stat quantile=0 gives min, quantile=1 gives max. CloudWatch rollup period is 1 minute. | -| **Prometheus CloudWatch Exporter** | Prometheus scrape of `cloudwatch_exporter` → Tsuga Prometheus receiver | Same CloudWatch metrics as above, but in Prometheus format: `aws_firehose_{metric_snake_case}_{statistic}` (e.g., `aws_firehose_incoming_bytes_sum`). Type=gauge. | Pro: familiar Prometheus semantics, per-statistic metrics. Con: polling introduces 1-minute+ lag; adds Prometheus infrastructure overhead. | CloudWatch API polling incurs costs. Metrics are pre-aggregated gauges, not raw events — do NOT use `rate()` on the `_sum` suffix metrics as if they were Prometheus counters. | -| **AWS CloudWatch Console** | Manual dashboard (not Tsuga) | All metrics with visual charts | Pro: zero setup. Con: not for programmatic alerting or cross-service correlation. | Monitoring gaps between CloudWatch console and Tsuga — different sampling/rendering. | - -**Optional features that change available metrics:** -- Lambda transformation enabled → adds `ExecuteProcessing.*`, `SucceedProcessing.*` -- Format conversion enabled → adds `SucceedConversion.*`, `FailedConversion.*` -- Dynamic partitioning enabled → adds `PartitionCount`, `PartitionCountExceeded`, `JQProcessing.Duration`, `ActivePartitionsLimit` -- Kinesis Data Streams source → adds `DataReadFromKinesisStream.*`, `KinesisMillisBehindLatest`, `ThrottledGetRecords`, `ThrottledGetShardIterator` -- MSK (Kafka) source → adds `DataReadFromSource.*`, `KafkaOffsetLag`, `SourceThrottled.Delay` -- KMS encryption enabled → adds `KMSKeyAccessDenied`, `KMSKeyDisabled`, `KMSKeyInvalidState`, `KMSKeyNotFound` -- S3 Backup enabled → adds `BackupToS3.*` -- CloudWatch Logs source with decompression → adds `OutputDecompressedBytes.*`, `OutputDecompressedRecords.*` -- Snowflake destination → adds `DeliveryToSnowflake.*` - -**What "no data" usually means per source:** -- `IncomingBytes/Records` = 0: no producers are writing to the stream (producer outage, stream not being used, or wrong stream name selected) -- `DeliveryToS3.*` missing entirely: stream may not have S3 as a destination, or metric collection is not enabled -- `ExecuteProcessing.*` missing: Lambda transformation not enabled for this stream -- `FailedConversion.*` missing: format conversion not enabled for this stream -- `KMS*` metrics missing: SSE with KMS not enabled - -**Confirmed by sources:** Metric Streams encoding as SUMMARY: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-metric-streams-formats-opentelemetry-translation.html -**Best-practice inference:** Prometheus CloudWatch Exporter gauge behavior described as industry convention. - ---- - -## Log intelligence (Stage 4 handoff) - -### Confirmed by sources - -**Log sources matrix:** - -| Source | Access path | Typical format | Structured vs unstructured | Evidence | -|---|---|---|---|---| -| Firehose error logs | CloudWatch Logs group: `/aws/kinesisfirehose/{delivery-stream-name}` | JSON-like error objects (error code + message + action) | Semi-structured: error code dot-notation, free-text message | https://docs.aws.amazon.com/firehose/latest/dev/troubleshooting.html | -| Log stream: DestinationDelivery | Within above log group — destination delivery errors | Error code + message + timestamp | Semi-structured | AWS Firehose Developer Guide | -| Log stream: BackupDelivery | Within above log group — S3 backup errors (if backup enabled) | Same as DestinationDelivery | Semi-structured | AWS Firehose Developer Guide | - -**Known log formats (confirmed):** - -Firehose error logs in CloudWatch Logs are NOT standard structured JSON. Each log event is a text record containing: -``` -{ - "deliveryStreamARN": "arn:aws:firehose:us-east-1:123456789012:deliverystream/my-stream", - "destination": "arn:aws:s3:::my-bucket", - "deliveryStreamVersionId": 1, - "message": "The S3 bucket is not accessible. Ensure that the bucket exists and the role has access to the bucket.", - "errorCode": "S3.AccessDenied", - "processor": "s3-destination" -} -``` -- Timestamp: provided by CloudWatch Logs (not embedded in the message) -- Each log event = one error occurrence -- Error code format: `Service.ErrorType` (e.g., `S3.AccessDenied`, `Lambda.InvokeLimitExceeded`) -- Messages are English prose (variable length, no fixed delimiter) - -**Candidate query filters for Stage 4:** - -- **Precise:** `context.aws.log_group:/aws/kinesisfirehose/` — targets Firehose error log groups only. Risk: low (all Firehose error logs in the org, filtered by stream name if needed). -- **Broader fallback:** `context.aws.log_group:kinesisfirehose` — catches variations in log group naming. Risk: may match other Kinesis-related log groups. - -**Attribute mapping hints:** - -| Raw field | Suggested Tsuga key | Confidence | Notes | -|---|---|---|---| -| `errorCode` | `error.type` | High | Standard OTel attribute for error category | -| `deliveryStreamARN` | `aws.firehose.delivery_stream_arn` | Medium | Follows OTel AWS semconv pattern | -| `destination` | `aws.firehose.destination` | Medium | ARN of the destination resource | -| `message` | `message` (body) | High | Log message field | -| `processor` | `aws.firehose.processor` | Low | Internal processor name; low operational value | -| CloudWatch log group name | `context.aws.log_group` | High | Standard Tsuga/OTel CloudWatch Log attribute | -| CloudWatch stream name | `context.aws.log_stream` | High | Standard Tsuga/OTel CloudWatch Log attribute | - -**Parsing risks:** -- Firehose samples error logs at high failure rates — you will NOT see every error in CloudWatch Logs during large-scale failures. -- The log format is AWS-internal and can change between regions or service versions without notice. -- `deliveryStreamARN` may be split across log entries (truncation possible for very long ARNs in some edge cases). -- CloudWatch Logs logging is opt-in (must be configured on the stream) — logs may simply not exist for streams that didn't enable it. -- No per-record delivery log — only error events. Normal successful deliveries produce no log output. - -### Best-practice inference - -- Error code (`errorCode`) is the highest-value extracted field for alerting on log routes — it disambiguates the failure type. -- Parsing `deliveryStreamARN` to extract stream name (last segment after `/`) gives a higher-cardinality but more useful attribute than the full ARN. -- Lambda transformation errors will also appear in `/aws/lambda/{function-name}` log groups — NOT in the Firehose log group. A separate log route for Lambda may be needed to capture transformation errors. - ---- - -## Caveats and footguns - -- **[ingestion]** `ThrottledRecords` non-zero = data is already lost. Firehose does NOT retry throttled records — the producer must retry on its own. This is unlike most retry semantics. (Confirmed: https://docs.aws.amazon.com/firehose/latest/dev/monitoring-with-cloudwatch-metrics.html) -- **[ingestion]** `IncomingBytes` and `IncomingRecords` exclude throttled records — they count only successfully accepted records. So a drop in `IncomingRecords` could mean either producers slowed down OR throttling is happening. Always check `ThrottledRecords` alongside. (Confirmed: AWS documentation) -- **[delivery-freshness]** Always use **Maximum** statistic for `DataFreshness` alerting, not Average. Average hides worst-case records; a single stuck record can cause data loss without triggering an Average-based alert. (Best-practice inference) -- **[delivery-freshness]** DataFreshness will naturally be near the buffer interval under normal conditions — it's not zero even when healthy. "Alarming" starts when freshness exceeds buffer interval × (1 + retry count factor). (Confirmed: AWS documentation behavior description) -- **[delivery-freshness]** When DataFreshness drops sharply to zero, it may mean the stream is idle (no records in flight), not that delivery improved. Always correlate with `IncomingRecords`. (Best-practice inference) -- **[delivery-throughput, delivery-freshness]** `DeliveryToS3.Records` is the number of records in each S3 PUT, NOT the number of successful PUTs. With batching, a single PUT delivers many records. Do not compare `IncomingRecords` to `DeliveryToS3.Records` within short windows — they will not match due to buffering. (Confirmed: AWS documentation) -- **[delivery-errors]** `DeliveryToS3.Success` has two valid interpretations depending on the CloudWatch statistic used: **Sum** = count of successful S3 PUT operations; **Average** = success rate (0–1). Always clarify which stat you're using. Use Average for ratio-based signals; Sum for absolute operation counts. (Confirmed: AWS CloudWatch documentation) -- **[transformation]** Lambda transformation has a **5-minute hard timeout** per batch. If `ExecuteProcessing.Duration` (Max) approaches 300,000 ms, delivery will fail. There is no graceful partial failure — the entire batch fails. (Confirmed: AWS Lambda limits documentation) -- **[transformation]** Lambda transformation failures can cause `DataFreshness` to climb even when the destination is healthy — the bottleneck is in the transformation layer, not delivery. Check `ExecuteProcessing.Success` when DataFreshness rises unexpectedly. (Best-practice inference) -- **[transformation]** `SucceedProcessing.Records` and `SucceedProcessing.Bytes` only count records that successfully passed through Lambda. Records that Lambda returned with `Dropped` status are NOT counted here and are silently excluded from delivery. (Confirmed: AWS documentation) -- **[delivery-errors]** KMS key errors (`KMSKeyAccessDenied`, etc.) block ALL delivery immediately — not just for new records, but for any buffered data that requires encryption before writing. These must be treated as P1 incidents. (Confirmed: https://docs.aws.amazon.com/firehose/latest/dev/encryption.html) -- **[transformation]** Format conversion (JSON → Parquet/ORC) requires a Glue Data Catalog table schema. If the schema evolves and records don't match, `FailedConversion.Records` will rise. Failed records go to a `processing-failed/` error prefix in S3 — they are NOT retried. (Confirmed: AWS documentation) -- **[backup-limits]** `PartitionCount` approaching 500 means `PartitionCountExceeded` = 1, causing records to overflow to the default S3 prefix. This is a soft data routing failure — data is not lost but is incorrectly partitioned, which breaks downstream assumptions. (Confirmed: service limits documentation) -- **[ingestion]** `BytesPerSecondLimit` is a live metric representing the current service quota, not a static value. If your account has requested quota increases, this value will reflect the new limit. Do not hardcode thresholds — compare dynamically. (Best-practice inference) -- **[delivery-freshness, delivery-errors]** Firehose retries delivery for a configurable retry duration (default varies by destination; S3 default is 0s but backup can be configured up to 7200s). During retry, DataFreshness climbs. After retry exhaustion, records go to error prefix and DataFreshness resets. DataFreshness resetting doesn't mean the problem is fixed — it may mean data was abandoned. (Confirmed: AWS documentation) -- **[ingestion]** CloudWatch Metric Streams deliver metrics as SUMMARY type (not counter or gauge). This affects aggregation options in Tsuga — `sum` and `count` from SUMMARY are available, as well as min (quantile=0) and max (quantile=1). Treat Sum-based metrics (IncomingBytes, IncomingRecords) as pre-aggregated delta counters: `average` or `sum` aggregation, with `per-second` post-function if delta. (Confirmed: CloudWatch Metric Streams OTel format documentation) -- **[delivery-throughput]** `DeliveryToS3.ObjectCount` is only available when dynamic partitioning is enabled. Do not gate the entire delivery-throughput section on this metric — it's supplemental, not core. (Confirmed: AWS documentation) -- **[delivery-errors, ingestion]** CloudWatch error logs are sampled at high failure rates. During a catastrophic delivery failure affecting many streams simultaneously, not all errors will be logged. Metrics (`DataFreshness`, `DeliveryToS3.Success`) are the reliable signal; logs are for root cause analysis after the alert fires. (Confirmed: AWS documentation) -- **[ingestion]** `IncomingPutRequests` counts successful API calls — a single PutRecordBatch call counts as 1 request regardless of the number of records it contains. Use `IncomingRecords` for data volume, `IncomingPutRequests` for API throughput. (Confirmed: AWS documentation) -- **[delivery-freshness]** When Kinesis Data Streams is the source, `KinesisMillisBehindLatest` is a separate freshness signal that measures how far behind Firehose is in reading from the source stream. This can grow independently of `DataFreshness` (destination-side freshness). Both can be elevated simultaneously or independently. (Confirmed: AWS documentation) -- **[delivery-errors]** `DeliveryToRedshift.Success` (Average) approaching 0 while `BackupToS3.DataFreshness` is low indicates the Redshift COPY commands are failing but records are being safely staged in S3. Records can be manually re-COPYed. This is recoverable, unlike S3-destination failures where records go to the error prefix. (Best-practice inference) -- **[ingestion]** Direct PUT and Kinesis-source streams have different throttle mechanisms. For Kinesis-source streams, throttling appears as `ThrottledGetRecords`/`ThrottledGetShardIterator`, NOT as `ThrottledRecords` (which is for Direct PUT). These are in separate metric rows. (Confirmed: AWS documentation) - ---- - -## Confirmed Tsuga prefixes - -- `aws_firehose_*` — **CONFIRMED** (Stage 2 discovery via `tsuga_search_metrics.py "firehose"` — 15 matching metrics found with this prefix) - -**Prefix format:** Prometheus CloudWatch Exporter snake_case. No statistic suffixes — each metric is a SUMMARY type with sum/count/min/max quantiles available. Not the OTel dot-format (`aws.firehose.*`) and not the CW Metric Streams format. - -**Ruled out:** `aws.firehose.*` (OTel dots — not present), `amazonaws.com/AWS/Firehose/*` (CW Metric Streams format — not present). - -**15 confirmed metrics in Tsuga:** -1. `aws_firehose_incoming_records` -2. `aws_firehose_throttled_records` -3. `aws_firehose_put_record_batch_bytes` -4. `aws_firehose_put_record_batch_records` -5. `aws_firehose_put_record_batch_requests` -6. `aws_firehose_put_record_batch_latency` -7. `aws_firehose_delivery_to_http_endpoint_data_freshness` -8. `aws_firehose_delivery_to_http_endpoint_bytes` -9. `aws_firehose_delivery_to_http_endpoint_records` -10. `aws_firehose_delivery_to_http_endpoint_processed_bytes` -11. `aws_firehose_failed_validation_records` -12. `aws_firehose_kms_key_access_denied` -13. `aws_firehose_kms_key_disabled` -14. `aws_firehose_kms_key_invalid_state` -15. `aws_firehose_kms_key_not_found` - -**Key discovery findings:** -- Destination is **HTTP endpoint** (not S3) — all `DeliveryToS3.*` metrics absent -- **MSK/Kafka is the source** (inferred from presence of `aws_firehose_failed_validation_records`) -- Lambda transformation **not enabled** — `ExecuteProcessing.*` absent -- Format conversion **not enabled** — `FailedConversion.*` absent -- `IncomingBytes` **not exported** — use `aws_firehose_put_record_batch_bytes` as proxy - ---- - -## Discovery status - -**Stage 2 discovery: COMPLETED** (2026-03-16) - -Discovered via `tsuga_search_metrics.py "firehose"` scanning 3433 total metrics. All unknowns from Stage 1 (u_01 through u_07) resolved. Metric catalog bootstrapped: `05_aws-firehose_metric_catalog.csv` (15 metrics, 0 quality gate errors). - ---- - -## Top sources - -1. https://docs.aws.amazon.com/firehose/latest/dev/monitoring-with-cloudwatch-metrics.html — Authoritative AWS Firehose CloudWatch metrics reference: complete list of all metrics with statistics, units, dimensions, and feature requirements. -2. https://docs.aws.amazon.com/firehose/latest/dev/what-is-this-service.html — Amazon Data Firehose product overview: architecture, delivery stream concepts, source/destination types. -3. https://docs.aws.amazon.com/firehose/latest/dev/limits.html — Service limits and quotas: BytesPerSecondLimit values, partition limits, Lambda timeout, retry duration bounds. -4. https://docs.aws.amazon.com/firehose/latest/dev/data-transformation.html — Lambda transformation feature: record format, failure handling, timeout behavior, backup S3 bucket semantics. -5. https://docs.aws.amazon.com/firehose/latest/dev/record-format-conversion.html — Format conversion (JSON→Parquet/ORC): Glue schema requirements, FailedConversion behavior, error prefix routing. -6. https://docs.aws.amazon.com/firehose/latest/dev/dynamic-partitioning.html — Dynamic partitioning: JQ expressions, partition count limits, PartitionCountExceeded behavior. -7. https://docs.aws.amazon.com/firehose/latest/dev/encryption.html — KMS encryption: how KMS errors block all delivery, key access requirements. -8. https://docs.aws.amazon.com/firehose/latest/dev/troubleshooting.html — Troubleshooting guide: error log format, error code taxonomy, CloudWatch Logs setup for Firehose. -9. https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-metric-streams-formats-opentelemetry-translation.html — CloudWatch Metric Streams → OTel format: SUMMARY type encoding, metric name format, quantile encoding. -10. https://docs.aws.amazon.com/firehose/latest/dev/basic-deliver.html — S3 delivery configuration: buffer size, buffer interval, retry duration, S3 prefix patterns, backup bucket behavior. diff --git a/plugins/tsuga/skills/knowledge-technology/references/aws-firehose/metrics.csv b/plugins/tsuga/skills/knowledge-technology/references/aws-firehose/metrics.csv deleted file mode 100644 index 717c825..0000000 --- a/plugins/tsuga/skills/knowledge-technology/references/aws-firehose/metrics.csv +++ /dev/null @@ -1,33 +0,0 @@ -theme,name,confirmed,definition,unit,type,tsuga_metric_name,widget_suggestions,aggregation,post_function,group_by,required_context_fields,optional_context_fields,absence_means,cardinality_footguns,variants,citations -Throughput/Usage,aws_firehose_incoming_records,true,Records successfully accepted by the delivery stream per CloudWatch period. Excludes throttled records. Use sum+rate for per-second ingestion rate. Primary producer throughput signal.,{record},summary,aws_firehose_incoming_records,timeseries|query-value,sum,per-second,context.deliverystreamname:10,context.env|context.team,context.cloud.region|context.cloud.account.id,No data ingested — stream is idle or producers are not writing,Safe for group-by; bounded by delivery stream count,prometheus=aws_firehose_incoming_records,https://docs.aws.amazon.com/firehose/latest/dev/monitoring-with-cloudwatch-metrics.html -Errors/Failures,aws_firehose_throttled_records,true,Records rejected at ingestion because the delivery stream exceeded its service throughput or records-per-second limit. CRITICAL: throttled records are NOT retried by Firehose — they are silently dropped. The producer must retry.,{record},summary,aws_firehose_throttled_records,timeseries|query-value,sum,per-second,context.deliverystreamname:10,context.env|context.team,context.cloud.region,No throttling — ingestion is within service limits (healthy),Safe,prometheus=aws_firehose_throttled_records,https://docs.aws.amazon.com/firehose/latest/dev/monitoring-with-cloudwatch-metrics.html -Throughput/Usage,aws_firehose_put_record_batch_bytes,true,Total bytes submitted to the delivery stream via PutRecordBatch API calls per period. Includes both accepted and throttled records. Measures producer-side volume before Firehose filtering.,By,summary,aws_firehose_put_record_batch_bytes,timeseries|query-value,sum,per-second,context.deliverystreamname:10,context.env|context.team,context.cloud.region,No API calls — producers not writing,Safe,prometheus=aws_firehose_put_record_batch_bytes,https://docs.aws.amazon.com/firehose/latest/dev/monitoring-with-cloudwatch-metrics.html -Throughput/Usage,aws_firehose_put_record_batch_records,true,Records submitted via PutRecordBatch API calls per period. A single call carries up to 500 records or 4 MiB. Includes both accepted and any subsequently throttled records.,{record},summary,aws_firehose_put_record_batch_records,timeseries,sum,per-second,context.deliverystreamname:10,context.env|context.team,context.cloud.region,No API calls — producers not writing,Safe,prometheus=aws_firehose_put_record_batch_records,https://docs.aws.amazon.com/firehose/latest/dev/monitoring-with-cloudwatch-metrics.html -Throughput/Usage,aws_firehose_put_record_batch_requests,true,Number of PutRecordBatch API calls per period. One call = 1 request regardless of record count. Use with put_record_batch_records to compute average batch size efficiency.,{request},summary,aws_firehose_put_record_batch_requests,timeseries,sum,per-second,context.deliverystreamname:10,context.env|context.team,context.cloud.region,No API calls — producers not writing,Safe,prometheus=aws_firehose_put_record_batch_requests,https://docs.aws.amazon.com/firehose/latest/dev/monitoring-with-cloudwatch-metrics.html -Performance/Latency,aws_firehose_put_record_batch_latency,true,"Per-call latency in milliseconds for PutRecordBatch API operations. Use average aggregation. Increases with record size, Firehose service pressure, or throttling. Producer-observed API response time.",ms,summary,aws_firehose_put_record_batch_latency,timeseries|query-value,average,none,context.deliverystreamname:10,context.env|context.team,context.cloud.region,No API calls — producers not writing,Safe,prometheus=aws_firehose_put_record_batch_latency,https://docs.aws.amazon.com/firehose/latest/dev/monitoring-with-cloudwatch-metrics.html -Availability/Health,aws_firehose_delivery_to_http_endpoint_data_freshness,true,Age in seconds of the oldest record currently buffered in the delivery stream that has not yet been delivered to the HTTP endpoint. THE primary Firehose operational signal. Use max aggregation for worst-case. Rising value = delivery is blocked or retrying.,s,summary,aws_firehose_delivery_to_http_endpoint_data_freshness,timeseries|query-value,max,none,context.deliverystreamname:10,context.env|context.team,context.cloud.region,Stream is idle (no records in flight) — confirm with incoming_records,Safe; max aggregation is correct for SLO tracking,prometheus=aws_firehose_delivery_to_http_endpoint_data_freshness,https://docs.aws.amazon.com/firehose/latest/dev/monitoring-with-cloudwatch-metrics.html -Throughput/Usage,aws_firehose_delivery_to_http_endpoint_bytes,true,Bytes delivered to and acknowledged by the HTTP endpoint destination per CloudWatch period. Use sum+rate for per-second delivery throughput. Should approach incoming bytes over longer windows accounting for buffer delay.,By,summary,aws_firehose_delivery_to_http_endpoint_bytes,timeseries|query-value,sum,per-second,context.deliverystreamname:10,context.env|context.team,context.cloud.region,HTTP endpoint delivery not configured or stream is idle,Safe,prometheus=aws_firehose_delivery_to_http_endpoint_bytes,https://docs.aws.amazon.com/firehose/latest/dev/monitoring-with-cloudwatch-metrics.html -Throughput/Usage,aws_firehose_delivery_to_http_endpoint_records,true,Records delivered to and acknowledged by the HTTP endpoint destination per CloudWatch period. Over longer windows should approach incoming_records accounting for buffer interval delay.,{record},summary,aws_firehose_delivery_to_http_endpoint_records,timeseries,sum,per-second,context.deliverystreamname:10,context.env|context.team,context.cloud.region,HTTP endpoint delivery not configured or stream is idle,Safe; do not compare to incoming_records in short windows due to buffering,prometheus=aws_firehose_delivery_to_http_endpoint_records,https://docs.aws.amazon.com/firehose/latest/dev/monitoring-with-cloudwatch-metrics.html -Throughput/Usage,aws_firehose_delivery_to_http_endpoint_processed_bytes,true,Total bytes sent to the HTTP endpoint per period including all delivery attempts and retries. Exceeds delivery_to_http_endpoint_bytes when the endpoint is rejecting or timing out requests. Use sum+rate.,By,summary,aws_firehose_delivery_to_http_endpoint_processed_bytes,timeseries,sum,per-second,context.deliverystreamname:10,context.env|context.team,context.cloud.region,HTTP endpoint delivery not configured,Safe; compare to delivery_bytes to measure retry overhead,prometheus=aws_firehose_delivery_to_http_endpoint_processed_bytes,https://docs.aws.amazon.com/firehose/latest/dev/monitoring-with-cloudwatch-metrics.html -Errors/Failures,aws_firehose_failed_validation_records,true,Records rejected during schema validation when an MSK (Kafka) source is configured. Non-zero value indicates records from the Kafka source stream are failing the schema check at ingestion. Use sum+rate.,{record},summary,aws_firehose_failed_validation_records,timeseries|query-value,sum,per-second,context.deliverystreamname:10,context.env|context.team,context.cloud.region|context.sourcepartitionid,MSK source not configured or all records passing validation (healthy),context.sourcepartitionid can be used to identify which Kafka partition is producing invalid records,prometheus=aws_firehose_failed_validation_records,https://docs.aws.amazon.com/firehose/latest/dev/monitoring-with-cloudwatch-metrics.html -Errors/Failures,aws_firehose_kms_key_access_denied,true,Number of KMSAccessDeniedException errors during delivery. Any non-zero value means the delivery stream IAM role lacks permission to use the KMS key — all delivery is blocked immediately. Requires KMS-based SSE enabled.,{error},summary,aws_firehose_kms_key_access_denied,timeseries|query-value,sum,per-second,context.deliverystreamname:10,context.env,context.cloud.region,KMS encryption not enabled — metric absent (healthy),Safe; should always be 0,prometheus=aws_firehose_kms_key_access_denied,https://docs.aws.amazon.com/firehose/latest/dev/encryption.html -Errors/Failures,aws_firehose_kms_key_disabled,true,Number of KMSDisabledExceptions during delivery. The KMS key has been disabled — all delivery is blocked until re-enabled. Requires KMS-based SSE enabled.,{error},summary,aws_firehose_kms_key_disabled,timeseries|query-value,sum,per-second,context.deliverystreamname:10,context.env,context.cloud.region,KMS encryption not enabled — metric absent,Safe; should always be 0,prometheus=aws_firehose_kms_key_disabled,https://docs.aws.amazon.com/firehose/latest/dev/encryption.html -Errors/Failures,aws_firehose_kms_key_invalid_state,true,"Number of KMSInvalidStateExceptions during delivery. The KMS key is in an invalid state (e.g., pending deletion). All delivery blocked. Requires KMS-based SSE enabled.",{error},summary,aws_firehose_kms_key_invalid_state,timeseries|query-value,sum,per-second,context.deliverystreamname:10,context.env,context.cloud.region,KMS encryption not enabled — metric absent,Safe; should always be 0,prometheus=aws_firehose_kms_key_invalid_state,https://docs.aws.amazon.com/firehose/latest/dev/encryption.html -Errors/Failures,aws_firehose_kms_key_not_found,true,Number of KMSNotFoundExceptions during delivery. The KMS key ARN no longer exists. All delivery blocked until stream configuration is updated. Requires KMS-based SSE enabled.,{error},summary,aws_firehose_kms_key_not_found,timeseries|query-value,sum,per-second,context.deliverystreamname:10,context.env,context.cloud.region,KMS encryption not enabled — metric absent,Safe; should always be 0,prometheus=aws_firehose_kms_key_not_found,https://docs.aws.amazon.com/firehose/latest/dev/encryption.html -Throughput/Usage,IncomingBytes,false,MISSING — not present in Tsuga. Bytes successfully ingested by the delivery stream. Absent because the Prometheus CloudWatch Exporter is not exporting this metric for the org's streams.,By,summary,MISSING,N/A,N/A,N/A,N/A,N/A,N/A,Metric not exported by collector,N/A,cw_metric=IncomingBytes,https://docs.aws.amazon.com/firehose/latest/dev/monitoring-with-cloudwatch-metrics.html -Capacity/Saturation,BytesPerSecondLimit,false,MISSING — not present in Tsuga. Current throughput limit for the delivery stream in bytes/s. Absent from the Prometheus CloudWatch Exporter's metric set for the org's streams.,By/s,summary,MISSING,N/A,N/A,N/A,N/A,N/A,N/A,Metric not exported by collector,N/A,cw_metric=BytesPerSecondLimit,https://docs.aws.amazon.com/firehose/latest/dev/limits.html -Capacity/Saturation,RecordsPerSecondLimit,false,MISSING — not present in Tsuga. Current records/s limit for the delivery stream. Absent from the Prometheus CloudWatch Exporter's metric set.,{record}/s,summary,MISSING,N/A,N/A,N/A,N/A,N/A,N/A,Metric not exported by collector,N/A,cw_metric=RecordsPerSecondLimit,https://docs.aws.amazon.com/firehose/latest/dev/limits.html -Availability/Health,DeliveryToS3.DataFreshness,false,MISSING — org uses HTTP endpoint destination not S3. All DeliveryToS3.* metrics are absent. Use aws_firehose_delivery_to_http_endpoint_data_freshness instead.,s,summary,MISSING,N/A,N/A,N/A,N/A,N/A,N/A,S3 not the configured destination,N/A,cw_metric=DeliveryToS3.DataFreshness,https://docs.aws.amazon.com/firehose/latest/dev/monitoring-with-cloudwatch-metrics.html -Availability/Health,DeliveryToS3.Success,false,MISSING — org uses HTTP endpoint destination not S3.,1,summary,MISSING,N/A,N/A,N/A,N/A,N/A,N/A,S3 not the configured destination,N/A,cw_metric=DeliveryToS3.Success,https://docs.aws.amazon.com/firehose/latest/dev/monitoring-with-cloudwatch-metrics.html -Throughput/Usage,DeliveryToS3.Bytes,false,MISSING — org uses HTTP endpoint destination not S3.,By,summary,MISSING,N/A,N/A,N/A,N/A,N/A,N/A,S3 not the configured destination,N/A,cw_metric=DeliveryToS3.Bytes,https://docs.aws.amazon.com/firehose/latest/dev/monitoring-with-cloudwatch-metrics.html -Throughput/Usage,DeliveryToS3.Records,false,MISSING — org uses HTTP endpoint destination not S3.,{record},summary,MISSING,N/A,N/A,N/A,N/A,N/A,N/A,S3 not the configured destination,N/A,cw_metric=DeliveryToS3.Records,https://docs.aws.amazon.com/firehose/latest/dev/monitoring-with-cloudwatch-metrics.html -Performance/Latency,ExecuteProcessing.Duration,false,MISSING — Lambda transformation not enabled on the org's delivery streams.,ms,summary,MISSING,N/A,N/A,N/A,N/A,N/A,N/A,Lambda transformation not enabled,N/A,cw_metric=ExecuteProcessing.Duration,https://docs.aws.amazon.com/firehose/latest/dev/data-transformation.html -Availability/Health,ExecuteProcessing.Success,false,MISSING — Lambda transformation not enabled on the org's delivery streams.,1,summary,MISSING,N/A,N/A,N/A,N/A,N/A,N/A,Lambda transformation not enabled,N/A,cw_metric=ExecuteProcessing.Success,https://docs.aws.amazon.com/firehose/latest/dev/data-transformation.html -Throughput/Usage,SucceedProcessing.Records,false,MISSING — Lambda transformation not enabled.,{record},summary,MISSING,N/A,N/A,N/A,N/A,N/A,N/A,Lambda transformation not enabled,N/A,cw_metric=SucceedProcessing.Records,https://docs.aws.amazon.com/firehose/latest/dev/data-transformation.html -Errors/Failures,FailedConversion.Records,false,MISSING — format conversion not enabled on the org's delivery streams.,{record},summary,MISSING,N/A,N/A,N/A,N/A,N/A,N/A,Format conversion not enabled,N/A,cw_metric=FailedConversion.Records,https://docs.aws.amazon.com/firehose/latest/dev/record-format-conversion.html -Throughput/Usage,SucceedConversion.Records,false,MISSING — format conversion not enabled.,{record},summary,MISSING,N/A,N/A,N/A,N/A,N/A,N/A,Format conversion not enabled,N/A,cw_metric=SucceedConversion.Records,https://docs.aws.amazon.com/firehose/latest/dev/record-format-conversion.html -Availability/Health,BackupToS3.DataFreshness,false,MISSING — S3 backup not enabled (Lambda transformation not active).,s,summary,MISSING,N/A,N/A,N/A,N/A,N/A,N/A,Backup not enabled,N/A,cw_metric=BackupToS3.DataFreshness,https://docs.aws.amazon.com/firehose/latest/dev/data-transformation.html -Availability/Health,BackupToS3.Success,false,MISSING — S3 backup not enabled.,1,summary,MISSING,N/A,N/A,N/A,N/A,N/A,N/A,Backup not enabled,N/A,cw_metric=BackupToS3.Success,https://docs.aws.amazon.com/firehose/latest/dev/data-transformation.html -Capacity/Saturation,PartitionCount,false,MISSING — dynamic partitioning not enabled on the org's delivery streams.,{partition},summary,MISSING,N/A,N/A,N/A,N/A,N/A,N/A,Dynamic partitioning not enabled,N/A,cw_metric=PartitionCount,https://docs.aws.amazon.com/firehose/latest/dev/dynamic-partitioning.html -Performance/Latency,KinesisMillisBehindLatest,false,MISSING — Kinesis Data Streams is not the source for the org's delivery streams.,ms,summary,MISSING,N/A,N/A,N/A,N/A,N/A,N/A,Kinesis source not configured,N/A,cw_metric=KinesisMillisBehindLatest,https://docs.aws.amazon.com/firehose/latest/dev/monitoring-with-cloudwatch-metrics.html -Cost,NOT_APPLICABLE,false,"Firehose does not expose a direct cost metric. Cost is driven by ingested data volume. Use aws_firehose_put_record_batch_bytes (sum+rate) as cost proxy — Firehose pricing is per-GB ingested. For detailed cost analysis use AWS Cost Explorer with service filter 'AmazonKinesisFirehose'.",N/A,N/A,N/A,N/A,N/A,N/A,N/A,context.env|context.team,context.cloud.account.id,N/A,N/A,N/A,https://aws.amazon.com/firehose/pricing/ diff --git a/plugins/tsuga/skills/knowledge-technology/references/aws-firehose/queries.md b/plugins/tsuga/skills/knowledge-technology/references/aws-firehose/queries.md deleted file mode 100644 index 81c5271..0000000 --- a/plugins/tsuga/skills/knowledge-technology/references/aws-firehose/queries.md +++ /dev/null @@ -1,56 +0,0 @@ -# AWS Kinesis Data Firehose - -Managed streaming delivery: producers → buffer → transform → destination (S3 / Redshift / OpenSearch / HTTP / Splunk). Healthy: incoming within limits, no throttling, bounded freshness, no KMS / destination errors. - -## Incident shapes - -- **Throttling** — `ThrottledRecords` nonzero → `MISSING` / `IncomingRecords` quota hit -- **Delivery lag** — `DeliveryTo*.DataFreshness` climbs → data in Firehose but not delivered -- **Destination errors** — `MISSING < 1.0` → buffer grows, eventually backup / DLQ -- **Transform failures** — `MISSING < 1.0` → Lambda transform sick -- **KMS errors** — access_denied / disabled / invalid_state / not_found → key rotation -- **Validation failures** — `aws_firehose_failed_validation_records` → schema rejects - -## Key metrics - -| Metric | Unit | Signal | -|---|---|---| -| `aws_firehose_incoming_records` | count | Inbound rate | -| `MISSING` | bytes | Volume vs `MISSING` | -| `aws_firehose_throttled_records` | count | Throttling | -| `MISSING` / `MISSING` | limit | Current ceilings | -| `MISSING` | seconds | Oldest undelivered age | -| `MISSING` | ratio | Successful batch fraction | -| `MISSING` / `Records` | count | Delivered volume | -| `MISSING` | ms | Transform Lambda duration | -| `MISSING` | ratio | Transform success | -| `aws_firehose_kms_key_*` | count | KMS error counters | -| `aws_firehose_failed_validation_records` | count | Schema rejects | -| `aws_firehose_put_record_batch_latency` | ms | Producer API latency | - -## Derived signals - -- `throttled_records / (incoming_records + throttled_records)` — throttle rate. Any sustained positive = capacity issue. -- `DataFreshness` — linear growth = delivery stuck; sawtooth = normal buffer-flush cycles. -- `MISSING` — drop = transform broken. -- `MISSING / MISSING` — quota headroom. > 0.8 sustained = request increase. - -## Log patterns - -Firehose error CloudWatch log group: - -- `KMS key ... is in invalid state` — rotation issue -- `AccessDenied` — role lacks destination permissions -- `Invalid data format` — doesn't match destination format -- `Connection refused` / `reset` — HTTP endpoint unreachable -- `HTTP 4xx` / `5xx` on HTTP endpoint -- `ProcessingFailed` — transform Lambda errored -- Backup-to-S3 entries — delivery failed, records in backup bucket - -## Gotchas - -- Firehose buffers before delivery (size or time). Low freshness = "would be delivered at next flush," not "delivered now." -- Destination failures don't produce producer errors immediately; internal retry loop (default 24h) before backup. -- Transform Lambda can return `Dropped` status intentionally — looks like loss but is by design. -- Dynamic partition keys can create millions of small S3 objects; healthy in Firehose metrics but blows up downstream. -- Quotas are region + account level; multiple streams share. diff --git a/plugins/tsuga/skills/knowledge-technology/references/aws-lambda/README.md b/plugins/tsuga/skills/knowledge-technology/references/aws-lambda/README.md deleted file mode 100644 index d95be9a..0000000 --- a/plugins/tsuga/skills/knowledge-technology/references/aws-lambda/README.md +++ /dev/null @@ -1,249 +0,0 @@ -# AWS Lambda Integration Context Bundle - -## Metadata -**Technology:** AWS Lambda -**Deployment:** managed -**Environment:** prod -**Persona:** SRE Dev and ops -**Telemetry preference:** mixed -**Integration scope:** core service only -**Primary use-case:** reliability and performance - -## How to use this bundle -- Use `01_aws-lambda_metrics.csv` as the source of truth for Lambda metrics, safe aggregations, and grouping advice. -- Use `02_aws-lambda_dashboard_plan.yaml` for the section model, widgets, derived signals, triage chains, and playbooks. -- Use `03_aws-lambda_state.yaml` for machine-readable unknowns, field-mapping risks, and Stage 2 verification priorities. -- Use `04_aws-lambda_memory.md` for the human-readable handoff narrative and tradeoffs from Stage 1. -- Stage 2 will create `05_aws-lambda_metric_catalog.csv` as the discovered Tsuga inventory used for reconciliation and coverage checks. -- Stage 4 should read this file's `Log intelligence (Stage 4 handoff)` section and `03_aws-lambda_state.yaml` `log_intel` first before proposing any log route. - -## What it is and what "good" looks like -### Confirmed by sources -AWS Lambda is AWS's managed function runtime: work arrives through synchronous requests, asynchronous invokes, or event source mappings, then Lambda schedules execution environments, enforces concurrency limits, and emits CloudWatch metrics for invocations, failures, latency, async delivery, and stream lag. Good operational posture means successful invocations dominate total demand, throttles stay rare, execution duration stays inside the workload's expected envelope, concurrency remains below account or reserved limits, and async or stream-backed sources do not accumulate age or lag. AWS documents the primary service metrics in the Lambda monitoring guide and separates concurrency, async, event source mapping, and performance signals explicitly. -Sources: https://docs.aws.amazon.com/lambda/latest/dg/monitoring-metrics-types.html, https://docs.aws.amazon.com/lambda/latest/dg/configuration-concurrency.html, https://docs.aws.amazon.com/lambda/latest/dg/provisioned-concurrency.html - -### Best-practice inference -- Incident shape 1: customer-facing invocation failures or throttles. Start with `invoke-health`, then move to `concurrency-throttling`. -- Incident shape 2: slow executions or runtime-tail growth without outright errors. Start with `latency-runtime`, then compare with `concurrency-throttling`. -- Incident shape 3: async or stream-trigger backlog growth. Start with `async-delivery` for async invokes or `stream-consumers` for queue/stream sources. -- Paging intent: use dashboards to split demand, code/runtime failure, concurrency exhaustion, and backlog amplification before touching service-specific downstream dashboards. - -## Key concepts -### Glossary -| Term | Definition | Operational meaning | Dashboard section | -|---|---|---|---| -| Invocation | A single request accepted by Lambda for execution | Baseline demand and denominator for failure ratios | invoke-health | -| Error | A function invocation that returns an error to Lambda | Direct customer-impact or workload-failure symptom | invoke-health | -| Throttle | An invocation rejected because concurrency capacity was unavailable | Capacity exhaustion symptom, often account or reserved-limit related | concurrency-throttling | -| Duration | Time spent running handler code, excluding some init semantics | Main execution latency signal | latency-runtime | -| Concurrent execution | Number of in-flight function executions | Primary saturation indicator | concurrency-throttling | -| Reserved concurrency | Capacity reserved for one function and removed from the shared pool | Isolation and blast-radius control | concurrency-throttling | -| Unreserved pool | Shared account concurrency left after reservations | Hidden coupling across functions | concurrency-throttling | -| Provisioned concurrency | Pre-initialized execution environments kept warm | Cold-start mitigation and latency control | provisioned-concurrency | -| Spillover | Invocations that exceeded provisioned capacity and fell back to on-demand | Loss of provisioned-latency guarantee | provisioned-concurrency | -| Async invocation | Invocation mode where Lambda queues work before executing it | Backlog and delayed-delivery risk | async-delivery | -| Async event age | Age of the oldest event in Lambda's async queue | Queueing pressure and delayed processing symptom | async-delivery | -| Destination failure | Failure to write async outcome to the configured destination | Post-processing reliability issue | async-delivery | -| Dead-letter error | Failure when sending a failed async event to DLQ | Loss of failure evidence and replay path | async-delivery | -| Recursive invocation drop | Invocations Lambda drops to break recursive loops | Severe logic or event wiring bug | invoke-health | -| Event source mapping | Lambda poller binding between a stream/queue and a function | Ownership point for iterator age and lag | stream-consumers | -| Iterator age | Age of the last processed record for stream-based sources | Consumer lag symptom for Kinesis/DynamoDB Streams | stream-consumers | -| Offset lag | Kafka offset lag observed by the mapping | Backlog depth for Kafka/MSK consumers | stream-consumers | -| Oversized record | Kafka record that exceeded Lambda event size constraints | Silent throughput loss if ignored | stream-consumers | -| Post-runtime extensions duration | Time spent after handler completion while extensions finish | Tail-latency drag from agents/extensions | latency-runtime | -| Execution version | Version or alias actually executed | Needed to separate rollout issues from global failure | latency-runtime | -| Function resource | Qualified function ARN or alias form exported with metrics | Stable identity for alias/version drilldown | concurrency-throttling | -| Log group | CloudWatch Logs stream `/aws/lambda/` for Lambda execution logs | Main Stage 4 log route source | async-delivery | - -### Concept Map -```text -Caller -> invokes -> Lambda function (why: demand enters execution path) -Async producer -> enqueues -> Lambda async queue (why: delivery can succeed before execution starts) -Stream or queue source -> feeds -> event source mapping (why: poller health determines backlog) -Event source mapping -> dispatches -> Lambda execution environment (why: iterator age rises when dispatch cannot keep up) -Lambda scheduler -> allocates -> concurrent executions (why: saturation happens before code runs) -Reserved concurrency -> carves out -> function-specific capacity (why: one hot function can be isolated) -Provisioned concurrency -> pre-warms -> execution environments (why: latency variability drops when capacity is ready) -Provisioned concurrency spillover -> indicates -> on-demand fallback (why: latency protection is being exceeded) -Invocation -> results in -> success or error (why: top-level health split) -Invocation demand -> competes for -> account concurrency pool (why: throttles can be cross-function, not local) -Function code -> emits -> duration distribution (why: runtime or dependency slowness surfaces here) -Extensions -> add tail time to -> post-runtime extensions duration (why: agents can slow completion without code change) -Async queue backlog -> increases -> async event age (why: delayed processing hurts timeliness) -Async failures -> attempt delivery to -> destination or DLQ (why: failure evidence must not be lost) -Kinesis or DynamoDB stream backlog -> increases -> iterator age (why: consumer is behind real time) -Kafka backlog -> increases -> offset lag (why: partition progress is behind head) -FunctionName + Resource + ExecutedVersion -> map to -> ownership and rollout slices (why: isolate alias/version regressions) -context.env + context.team -> map to -> operational ownership (why: global filtering and incident routing) -context.cloud.region + context.cloud.account.id -> map to -> blast radius (why: regional/account isolation) -CloudWatch Logs -> contain -> START/END/REPORT and application logs (why: Stage 4 can correlate metrics to failures) -``` - -### Entities and dimensions -| Entity/Dimension | Why useful | Cardinality risk | Safe top-N | Do NOT group-by guidance | -|---|---|---|---|---| -| `context.functionname` | Primary Lambda ownership key in Tsuga discovery | Low | 20 | Use as the first grouping level before any resource split | -| `context.resource` | Distinguishes alias-qualified or qualified resources | Medium | 15 | Avoid mixing with request IDs or log stream names | -| `context.cloud.account.id` | Multi-account blast-radius split | Low | 10 | Use for account-pool widgets, not per-function charts | -| `context.cloud.region` | Regional blast-radius isolation | Low | 12 | Prefer as global filter in single-region accounts | -| `context.env` | Environment segmentation | Low | 5 | Global filter only | -| `context.team` | Team ownership routing | Low | 10 | Global filter only | -| `context.cloud.provider` | Confirms provider family | Low | 3 | Rarely worth chart space | -| `context.resource` | Alias/resource drilldown for rollouts | Medium | 15 | Prefer after function grouping | -| `context.source` | Exporter provenance | Low | 5 | Debug-only; not an operational group-by | -| `context.unit` | Metric unit tag | Low | 3 | Never group by | -| Provisioned-concurrency dimensions | Not discovered in Tsuga | Unknown | Unknown | Keep the section gated instead of guessing | -| Stream mapping dimensions | Not discovered in Tsuga | Unknown | Unknown | Keep stream-consumer section gated instead of guessing | - -### Tsuga field mapping -| Vendor/exporter dimension | Recommended context.* key | Must-exist vs optional | -|---|---|---| -| FunctionName | `context.functionname` | Must-exist | -| Resource | `context.resource` | Optional but preferred | -| ExecutedVersion | `Unknown` | Optional and not discovered in Tsuga | -| EventSourceMappingUUID | `Unknown` | Optional and not discovered in Tsuga | -| AWS Region | `context.cloud.region` | Optional but preferred | -| AWS Account ID | `context.cloud.account.id` | Optional | -| Environment tag | `context.env` | Must-exist | -| Team tag | `context.team` | Must-exist | - -#### Confirmed by sources -AWS documents FunctionName, Resource, ExecutedVersion, and EventSourceMappingUUID as Lambda metric dimensions, and the service semantics cleanly map them to function, version, and event-source slices that a dashboard needs. -Sources: https://docs.aws.amazon.com/lambda/latest/dg/monitoring-metrics-view.html, https://docs.aws.amazon.com/lambda/latest/dg/monitoring-metrics-types.html - -#### Best-practice inference -Stage 2 discovery confirmed `context.functionname` and `context.resource`, but did not discover version or event-source-mapping keys. Provisioned-concurrency and stream-consumer sections should stay explicitly gated until those richer dimensions appear. - -## Golden signals -### Confirmed by sources -| Signal | Lambda meaning | Typical degradations | Best telemetry sources | What people page on | Section questions | -|---|---|---|---|---|---| -| Traffic | Accepted Lambda demand across sync, async, and event-source flows | traffic drop, demand spike, missing async intake | `Invocations`, `AsyncEventsReceived` | sudden invocation collapse or unexpected surge | Are functions receiving work? Is demand shape normal? | -| Errors | Code failures, delivery failures, throttles, recursive drops | bad deploy, dependency outage, concurrency exhaustion | `Errors`, `Throttles`, `DestinationDeliveryFailures`, `DeadLetterErrors`, `RecursiveInvocationsDropped` | errors or throttles rise against stable demand | Are failures from code, capacity, or delivery plumbing? | -| Latency | Execution time and backlog wait before processing | slow dependencies, heavy payloads, backlog accumulation | `Duration`, `PostRuntimeExtensionsDuration`, `AsyncEventAge`, `IteratorAge`, `OffsetLag` | latency tails or consumer lag rising | Is work slow to execute or just waiting to start? | -| Saturation | Capacity headroom for on-demand or provisioned concurrency | account limit pressure, insufficient provisioned warm pool | `ConcurrentExecutions`, `ProvisionedConcurrentExecutions`, `ProvisionedConcurrencyUtilization`, `ProvisionedConcurrencySpilloverInvocations` | concurrency and spillover rise before user-visible failure | Are we running out of execution capacity? | - -### Best-practice inference -For Lambda, backlog age is often more actionable than raw invocation count during incidents because delayed work can hide behind normal request volume. - -## Telemetry sources -### Confirmed by sources -| Source type | How collected | What it provides | Pros/cons | Common pitfalls | -|---|---|---|---|---| -| CloudWatch Lambda service metrics | Native AWS namespace `AWS/Lambda` | Canonical invocation, error, latency, async, and concurrency signals | Best source for service health; limited request context | Some metrics are feature-gated and absent unless the invocation mode is used | -| CloudWatch Logs for function log groups | `/aws/lambda/` | START/END/REPORT lines plus app logs | Best source for request-level failure details | Log formats vary by runtime and advanced logging mode | -| Telemetry API to extensions | Runtime/extension API stream | Platform events and lifecycle timing | Strong for extension-aware diagnostics | Only available if extensions are present and configured | -| Lambda Insights | Optional enhanced monitoring extension | Memory, CPU, network, and system telemetry | Useful for deep runtime analysis | Not a guaranteed baseline; do not assume it exists | - -### Best-practice inference -Treat service metrics as the baseline dashboard contract and keep Lambda Insights out of the core dashboard unless Stage 2 discovers the metrics. -Sources: https://docs.aws.amazon.com/lambda/latest/dg/monitoring-metrics-types.html, https://docs.aws.amazon.com/lambda/latest/dg/monitoring-cloudwatchlogs-logformat.html, https://docs.aws.amazon.com/lambda/latest/dg/runtimes-telemetry-api.html, https://docs.aws.amazon.com/lambda/latest/dg/monitoring-insights.html - -## Log intelligence (Stage 4 handoff) -### Confirmed by sources -**Log sources matrix** - -| Source | Access path | Typical format | Structured vs unstructured | Evidence | -|---|---|---|---|---| -| Function execution logs | CloudWatch Logs group `/aws/lambda/` | START / END / REPORT platform lines + application output | Mixed | https://docs.aws.amazon.com/lambda/latest/dg/monitoring-cloudwatchlogs-logformat.html | -| Advanced logging controls | Same CloudWatch Logs destination | JSON platform and application logs when enabled for supported runtimes | Structured | https://docs.aws.amazon.com/lambda/latest/dg/monitoring-cloudwatchlogs-logformat.html | -| Telemetry API stream to extension | Local runtime API subscriber | JSON events such as platform start/report/runtime statuses | Structured | https://docs.aws.amazon.com/lambda/latest/dg/runtimes-telemetry-api.html | -| Lambda Insights logs | `/aws/lambda-insights` extension output | Embedded metrics and diagnostics records | Structured-ish | https://docs.aws.amazon.com/lambda/latest/dg/monitoring-insights.html | - -**Known log formats** - -1. Standard text platform lines - Sample line: `REPORT RequestId: 11111111-1111-1111-1111-111111111111 Duration: 933.59 ms Billed Duration: 934 ms Memory Size: 128 MB Max Memory Used: 94 MB Init Duration: 203.12 ms` - Shape notes: space-delimited key/value fragments, fixed uppercase prefix, optional `Init Duration`, request ID repeats across START/END/REPORT. Timestamp comes from CloudWatch event envelope, not the line body. Quoting is minimal. - Evidence: https://docs.aws.amazon.com/lambda/latest/dg/monitoring-cloudwatchlogs-logformat.html - -2. JSON platform report lines with advanced logging controls - Sample shape: `{"time":"2024-03-13T18:56:24.046Z","type":"platform.report","record":{"requestId":"...","metrics":{"durationMs":..., "billedDurationMs":..., "memorySizeMB":..., "maxMemoryUsedMB":...}}}` - Shape notes: nested JSON with explicit event type and metrics object. Timestamp is embedded. Optional fields depend on runtime and logging settings. - Evidence: https://docs.aws.amazon.com/lambda/latest/dg/monitoring-cloudwatchlogs-logformat.html - -**Candidate query filters for Stage 4** -- Precise: `context.log.group:/aws/lambda/* AND context.functionname:*` - Rationale: narrow to Lambda execution log groups while requiring function identity. - Risk: depends on Tsuga preserving the CloudWatch log group field. -- Broader fallback: `(message:*RequestId* OR message:*timed* OR message:*exited*)` - Rationale: catches standard Lambda platform lines even if log-group metadata is missing. - Risk: can match unrelated copied log text. - -**Attribute mapping hints** - -| Raw field | Suggested Tsuga key | Confidence | Notes | -|---|---|---|---| -| CloudWatch log group `/aws/lambda/` | `context.functionname` | High | Derive `` from suffix if explicit function field is absent | -| `RequestId` | `context.aws.lambda.request_id` | High | Shared across START/END/REPORT lines | -| `Duration` / `durationMs` | `context.aws.lambda.duration_ms` | High | Useful for log-to-metric spot checks | -| `Billed Duration` / `billedDurationMs` | `context.aws.lambda.billed_duration_ms` | Medium | Cost-adjacent but not a Stage 1 core metric | -| `Memory Size` | `context.aws.lambda.memory_size_mb` | Medium | Good for tuning and outlier analysis | -| `Max Memory Used` | `context.aws.lambda.max_memory_used_mb` | Medium | Useful for memory headroom correlation | -| JSON `type` | `context.aws.lambda.platform_event_type` | Medium | Important when parsing Telemetry API or JSON platform logs | - -**Parsing risks** -- Standard text and JSON platform logs can coexist during rollout or across runtimes. -- REPORT lines have optional fields such as `Init Duration`, so rigid token counts will break. -- Application logs may include embedded JSON strings or multiline stack traces. -- Telemetry API event schemas differ from CloudWatch platform text lines; do not reuse one parser blindly. -- Timezone handling should use CloudWatch envelope timestamps when parsing classic text lines. - -### Best-practice inference -If Tsuga already extracts function identity from CloudWatch metadata, prefer the discovered `context.functionname` field over parsing it from the log group string. If not, derive function name from `/aws/lambda/` and keep request ID extraction separate from app-log parsing. - -## Caveats and footguns -- **[invoke-health]** `Invocations` count accepted requests, not completed successes; pair with `Errors` and `Throttles` before declaring health. (https://docs.aws.amazon.com/lambda/latest/dg/monitoring-metrics-types.html) -- **[invoke-health]** `Errors` does not cover throttled invokes because throttles are tracked separately. (https://docs.aws.amazon.com/lambda/latest/dg/monitoring-metrics-types.html) -- **[invoke-health, concurrency-throttling]** `Throttles` can come from account concurrency exhaustion rather than one unhealthy function. (https://docs.aws.amazon.com/lambda/latest/dg/configuration-concurrency.html) -- **[latency-runtime]** `Duration` excludes cold-start init time in older interpretations but user experience can still include init-related delay; keep deployment and init context in mind. (Inference) -- **[latency-runtime]** `PostRuntimeExtensionsDuration` can rise because of observability agents, not business logic regressions. (https://docs.aws.amazon.com/lambda/latest/dg/monitoring-metrics-types.html) -- **[provisioned-concurrency]** `ProvisionedConcurrencyUtilization` is meaningless when provisioned concurrency is not configured. (https://docs.aws.amazon.com/lambda/latest/dg/provisioned-concurrency.html) -- **[provisioned-concurrency]** `ProvisionedConcurrencySpilloverInvocations` indicates loss of warm-capacity guarantees, not necessarily outright failure. (https://docs.aws.amazon.com/lambda/latest/dg/monitoring-metrics-types.html) -- **[async-delivery]** Async metrics only appear for asynchronous invocation flows; zero data may mean the mode is unused, not broken. (https://docs.aws.amazon.com/lambda/latest/dg/monitoring-metrics-types.html) -- **[async-delivery]** `AsyncEventAge` can rise while `Invocations` still look normal because the queue delays work before execution starts. (https://docs.aws.amazon.com/lambda/latest/dg/monitoring-metrics-types.html) -- **[async-delivery]** `DestinationDeliveryFailures` and `DeadLetterErrors` are delivery-pipeline problems after execution, not function-code latency metrics. (https://docs.aws.amazon.com/lambda/latest/dg/monitoring-metrics-types.html) -- **[invoke-health]** `RecursiveInvocationsDropped` is rare but severe; even a small non-zero value is usually a wiring defect, not normal noise. (https://docs.aws.amazon.com/lambda/latest/dg/monitoring-metrics-types.html) -- **[stream-consumers]** `IteratorAge` is only relevant for Kinesis and DynamoDB Streams event source mappings. (https://docs.aws.amazon.com/lambda/latest/dg/monitoring-metrics-types.html) -- **[stream-consumers]** `OffsetLag` and `OversizedRecordCount` are Kafka-specific; gate the section rather than showing zeros for non-Kafka consumers. (https://docs.aws.amazon.com/lambda/latest/dg/monitoring-metrics-types.html) -- **[stream-consumers]** Grouping by partition, shard, or request ID would explode cardinality; prefer event source mapping UUID or function name. (Inference) -- **[concurrency-throttling]** `ConcurrentExecutions` is a level, not a rate; never apply `rate` to it. (https://docs.aws.amazon.com/lambda/latest/dg/monitoring-metrics-types.html) -- **[concurrency-throttling]** Reserved concurrency can protect one function while starving the unreserved pool for others, so global account symptoms can hide behind healthy local charts. (https://docs.aws.amazon.com/lambda/latest/dg/configuration-concurrency.html) -- **[latency-runtime]** Average duration can hide tail spikes; deep dive should prefer max or percentile-like views where the Tsuga metric family supports it. (Inference) -- **[invoke-health, latency-runtime]** ExecutedVersion matters during rollouts; aggregating all aliases together can dilute a bad canary or version-specific regression. (https://docs.aws.amazon.com/lambda/latest/dg/monitoring-metrics-view.html) -- **[async-delivery, stream-consumers]** Backlog age is often the customer-visible symptom before error counts move. (Inference) -- **[invoke-health, async-delivery]** Standard log lines and JSON logs can coexist, so Stage 4 parsers must branch by shape instead of assuming one format. (https://docs.aws.amazon.com/lambda/latest/dg/monitoring-cloudwatchlogs-logformat.html) - -## Confirmed Tsuga prefixes -- `aws_lambda*` — **CONFIRMED** (11 metrics present in Tsuga from `tools/tsuga_search_metrics.py '^aws_lambda.*'`) - -## Discovery status -Discovery: completed in Stage 2. -- Metrics found: 11 total -- Confirmed against Stage 1 inventory: 11 -- Missing from Stage 1 vendor expectations: 10 optional or disabled metrics -- Unexpected but useful metrics added: `aws_lambda_claimed_account_concurrency`, `aws_lambda_unreserved_concurrent_executions` -- Confirmed Tsuga function key: `context.functionname` -- Confirmed Tsuga resource key: `context.resource` - -## Top sources -1. https://docs.aws.amazon.com/lambda/latest/dg/monitoring-metrics-types.html - Why: canonical Lambda metric names, dimensions, and feature-gated metric families. -2. https://docs.aws.amazon.com/lambda/latest/dg/monitoring-metrics-view.html - Why: metric dimension behavior and console viewing model for FunctionName, Resource, and ExecutedVersion. -3. https://docs.aws.amazon.com/lambda/latest/dg/configuration-concurrency.html - Why: concurrency semantics, reserved concurrency behavior, and account-pool interactions. -4. https://docs.aws.amazon.com/lambda/latest/dg/provisioned-concurrency.html - Why: provisioned concurrency semantics and operational tradeoffs. -5. https://docs.aws.amazon.com/lambda/latest/dg/monitoring-cloudwatchlogs-logformat.html - Why: official Lambda log formats for standard and JSON modes. -6. https://docs.aws.amazon.com/lambda/latest/dg/runtimes-telemetry-api.html - Why: platform telemetry event model for extension-aware logging. -7. https://docs.aws.amazon.com/lambda/latest/dg/monitoring-insights.html - Why: optional Lambda Insights telemetry and why it should be gated. -8. https://docs.aws.amazon.com/lambda/latest/dg/invocation-async.html - Why: async invocation semantics and why age/delivery metrics matter operationally. -9. https://docs.aws.amazon.com/lambda/latest/dg/invocation-recursion.html - Why: recursive loop detection and drop semantics. -10. https://docs.aws.amazon.com/lambda/latest/dg/with-msk-process.html - Why: Kafka-backed Lambda consumption context for offset lag and consumer backlog interpretation. diff --git a/plugins/tsuga/skills/knowledge-technology/references/aws-lambda/metrics.csv b/plugins/tsuga/skills/knowledge-technology/references/aws-lambda/metrics.csv deleted file mode 100644 index 129a5c1..0000000 --- a/plugins/tsuga/skills/knowledge-technology/references/aws-lambda/metrics.csv +++ /dev/null @@ -1,23 +0,0 @@ -theme,name,confirmed,definition,unit,type,tsuga_metric_name,widget_suggestions,aggregation,post_function,group_by,required_context_fields,optional_context_fields,absence_means,cardinality_footguns,variants,citations -Throughput/Usage,Invocations,true,"Accepted Lambda invocation count; primary demand baseline and denominator for health ratios.",{invocation},counter,aws_lambda_invocations,"timeseries|query-value|top-list",sum,rate,"context.functionname:20|context.cloud.region:10",context.env|context.team|context.functionname,context.cloud.region|context.cloud.account.id|context.resource,"No matching traffic, wrong namespace mapping, or collector gap.","Stay at function or region level; do not group by request-level fields.",cloudwatch=Invocations,https://docs.aws.amazon.com/lambda/latest/dg/monitoring-metrics-types.html -Errors/Failures,Errors,true,"Function invocation errors returned to Lambda; code or dependency failure symptom.",{error},counter,aws_lambda_errors,"timeseries|query-value|top-list",sum,rate,"context.functionname:20|context.cloud.region:10",context.env|context.team|context.functionname,context.cloud.region|context.cloud.account.id|context.resource,"No recent execution failures, no traffic, or missing data.","Keep separate from throttles because they represent different failure domains.",cloudwatch=Errors,https://docs.aws.amazon.com/lambda/latest/dg/monitoring-metrics-types.html -Errors/Failures,Throttles,true,"Invocations rejected because concurrency capacity was unavailable.",{throttle},counter,aws_lambda_throttles,"timeseries|query-value|top-list",sum,rate,"context.functionname:20|context.cloud.region:10",context.env|context.team|context.functionname,context.cloud.region|context.cloud.account.id|context.resource,"No capacity rejection, no traffic, or missing data.","Interpret against account concurrency signals before blaming one function.",cloudwatch=Throttles,https://docs.aws.amazon.com/lambda/latest/dg/monitoring-metrics-types.html|https://docs.aws.amazon.com/lambda/latest/dg/configuration-concurrency.html -Performance/Latency,Duration,true,"Execution duration for Lambda handler runtime; primary processing-latency signal.",ms,summary,aws_lambda_duration,"timeseries|query-value|top-list",average,none,"context.functionname:20|context.resource:15",context.env|context.team|context.functionname,context.cloud.region|context.cloud.account.id|context.resource,"No executions in window, unsupported statistic selection, or ingest gap.","Average can hide tails; prefer max or top-list views for triage.",cloudwatch=Duration,https://docs.aws.amazon.com/lambda/latest/dg/monitoring-metrics-types.html -Capacity/Saturation,ConcurrentExecutions,true,"Current number of in-flight Lambda executions; primary function-level saturation signal.",{execution},summary,aws_lambda_concurrent_executions,"timeseries|query-value|top-list",max,none,"context.functionname:20|context.cloud.region:10",context.env|context.team|context.functionname,context.cloud.region|context.cloud.account.id|context.resource,"Function idle, not instrumented, or namespace mismatch.","Treat as level-like despite Tsuga catalog temporality being cumulative/summary.",cloudwatch=ConcurrentExecutions,https://docs.aws.amazon.com/lambda/latest/dg/monitoring-metrics-types.html -Throughput/Usage,AsyncEventsReceived,true,"Async events accepted into Lambda's internal queue.",{event},counter,aws_lambda_async_events_received,"timeseries|query-value|top-list",sum,rate,"context.functionname:20|context.cloud.region:10",context.env|context.team|context.functionname,context.cloud.region|context.cloud.account.id|context.resource,"No async traffic, sync-only function, or missing data.","Interpret only for async-capable workflows.",cloudwatch=AsyncEventsReceived,https://docs.aws.amazon.com/lambda/latest/dg/monitoring-metrics-types.html|https://docs.aws.amazon.com/lambda/latest/dg/invocation-async.html -Errors/Failures,AsyncEventsDropped,true,"Async events dropped before successful processing.",{event},counter,aws_lambda_async_events_dropped,"timeseries|query-value|top-list",sum,rate,"context.functionname:20|context.cloud.region:10",context.env|context.team|context.functionname,context.cloud.region|context.cloud.account.id|context.resource,"No async drops, async path unused, or missing data.","Small counts still represent lost work.",cloudwatch=AsyncEventsDropped,https://docs.aws.amazon.com/lambda/latest/dg/monitoring-metrics-types.html|https://docs.aws.amazon.com/lambda/latest/dg/invocation-async.html -Performance/Latency,AsyncEventAge,true,"Age of the oldest event in Lambda's async queue.",ms,summary,aws_lambda_async_event_age,"timeseries|query-value|top-list",max,none,"context.functionname:20|context.cloud.region:10",context.env|context.team|context.functionname,context.cloud.region|context.cloud.account.id|context.resource,"No async backlog, async path unused, or missing data.","Age matters more than count when consumers are delayed.",cloudwatch=AsyncEventAge,https://docs.aws.amazon.com/lambda/latest/dg/monitoring-metrics-types.html|https://docs.aws.amazon.com/lambda/latest/dg/invocation-async.html -Performance/Latency,PostRuntimeExtensionsDuration,true,"Tail time spent after the handler returns while extensions continue processing.",ms,summary,aws_lambda_post_runtime_extensions_duration,"timeseries|query-value|top-list",max,none,"context.functionname:20|context.resource:15",context.env|context.team|context.functionname,context.cloud.region|context.cloud.account.id|context.resource,"No extensions present, no executions, or feature not emitted.","Useful only when compared to Duration; do not treat as the primary latency signal.",cloudwatch=PostRuntimeExtensionsDuration,https://docs.aws.amazon.com/lambda/latest/dg/monitoring-metrics-types.html|https://docs.aws.amazon.com/lambda/latest/dg/runtimes-telemetry-api.html -Capacity/Saturation,ClaimedAccountConcurrency,true,"Amount of account concurrency currently claimed by Lambda workloads; account-level saturation context.",{execution},summary,aws_lambda_claimed_account_concurrency,"timeseries|query-value",max,none,"context.cloud.account.id:10|context.cloud.region:10",context.env|context.team,context.cloud.region|context.cloud.account.id,"Metric absent if account-level concurrency export is disabled.","Account-level only; no function grouping available.",cloudwatch=ClaimedAccountConcurrency,https://docs.aws.amazon.com/lambda/latest/dg/monitoring-metrics-types.html|https://docs.aws.amazon.com/lambda/latest/dg/configuration-concurrency.html -Capacity/Saturation,UnreservedConcurrentExecutions,true,"Concurrent executions consuming the shared unreserved account pool.",{execution},summary,aws_lambda_unreserved_concurrent_executions,"timeseries|query-value",max,none,"context.cloud.account.id:10|context.cloud.region:10",context.env|context.team,context.cloud.region|context.cloud.account.id,"Metric absent if no unreserved concurrency activity is exported.","Account-level only; use with throttles to spot shared-pool pressure.",cloudwatch=UnreservedConcurrentExecutions,https://docs.aws.amazon.com/lambda/latest/dg/monitoring-metrics-types.html|https://docs.aws.amazon.com/lambda/latest/dg/configuration-concurrency.html -Capacity/Saturation,ProvisionedConcurrentExecutions,false,"Number of provisioned execution environments actively serving work. Not found in Tsuga during Stage 2 and should remain a gated optional metric.",{execution},Unknown,aws_lambda_provisioned_concurrent_executions,"timeseries|query-value|top-list",max,none,"context.functionname:20|context.cloud.region:10",context.env|context.team|context.functionname,context.cloud.region|context.cloud.account.id|context.resource,"Provisioned concurrency not configured, not ingested, or metric family absent.","Keep gated until Tsuga discovery confirms availability.",cloudwatch=ProvisionedConcurrentExecutions,https://docs.aws.amazon.com/lambda/latest/dg/monitoring-metrics-types.html|https://docs.aws.amazon.com/lambda/latest/dg/provisioned-concurrency.html -Capacity/Saturation,ProvisionedConcurrencyUtilization,false,"Share of provisioned concurrency in use. Not found in Tsuga during Stage 2 and should remain gated.",1,Unknown,aws_lambda_provisioned_concurrency_utilization,"timeseries|query-value|top-list",max,none,"context.functionname:20|context.cloud.region:10",context.env|context.team|context.functionname,context.cloud.region|context.cloud.account.id|context.resource,"Provisioned concurrency not enabled or not ingested.","Do not compare across functions unless the feature exists and is standardized.",cloudwatch=ProvisionedConcurrencyUtilization,https://docs.aws.amazon.com/lambda/latest/dg/monitoring-metrics-types.html|https://docs.aws.amazon.com/lambda/latest/dg/provisioned-concurrency.html -Throughput/Usage,ProvisionedConcurrencyInvocations,false,"Invocations served by provisioned concurrency environments. Not found in Tsuga during Stage 2.",{invocation},Unknown,aws_lambda_provisioned_concurrency_invocations,"timeseries|query-value|top-list",sum,rate,"context.functionname:20|context.cloud.region:10",context.env|context.team|context.functionname,context.cloud.region|context.cloud.account.id|context.resource,"Provisioned concurrency not enabled, not ingested, or no traffic.","Use with spillover only when Tsuga actually exports both metrics.",cloudwatch=ProvisionedConcurrencyInvocations,https://docs.aws.amazon.com/lambda/latest/dg/monitoring-metrics-types.html -Errors/Failures,ProvisionedConcurrencySpilloverInvocations,false,"Invocations that overflowed provisioned concurrency. Not found in Tsuga during Stage 2.",{invocation},Unknown,aws_lambda_provisioned_concurrency_spillover_invocations,"timeseries|query-value|top-list",sum,rate,"context.functionname:20|context.cloud.region:10",context.env|context.team|context.functionname,context.cloud.region|context.cloud.account.id|context.resource,"Provisioned concurrency not enabled, not ingested, or no spillover.","Keep gated until discovered.",cloudwatch=ProvisionedConcurrencySpilloverInvocations,https://docs.aws.amazon.com/lambda/latest/dg/monitoring-metrics-types.html|https://docs.aws.amazon.com/lambda/latest/dg/provisioned-concurrency.html -Errors/Failures,DeadLetterErrors,false,"Failures while writing failed async events to the configured dead-letter queue. Not found in Tsuga during Stage 2.",{error},Unknown,aws_lambda_dead_letter_errors,"timeseries|query-value|top-list",sum,rate,"context.functionname:20|context.cloud.region:10",context.env|context.team|context.functionname,context.cloud.region|context.cloud.account.id|context.resource,"No async dead-letter delivery failures, async path unused, or metric absent in Tsuga.","Keep out of synchronous sections and gate until discovery confirms presence.",cloudwatch=DeadLetterErrors,https://docs.aws.amazon.com/lambda/latest/dg/monitoring-metrics-types.html -Errors/Failures,DestinationDeliveryFailures,false,"Failures while writing async invocation records to a configured destination. Not found in Tsuga during Stage 2.",{error},Unknown,aws_lambda_destination_delivery_failures,"timeseries|query-value|top-list",sum,rate,"context.functionname:20|context.cloud.region:10",context.env|context.team|context.functionname,context.cloud.region|context.cloud.account.id|context.resource,"No destination failures, async destinations unused, or metric absent in Tsuga.","Keep gated until discovered.",cloudwatch=DestinationDeliveryFailures,https://docs.aws.amazon.com/lambda/latest/dg/monitoring-metrics-types.html -Errors/Failures,RecursiveInvocationsDropped,false,"Invocations Lambda dropped after detecting a recursive loop. Not found in Tsuga during Stage 2.",{drop},Unknown,aws_lambda_recursive_invocations_dropped,"timeseries|query-value|top-list",sum,rate,"context.functionname:20|context.cloud.region:10",context.env|context.team|context.functionname,context.cloud.region|context.cloud.account.id|context.resource,"No recursive loop detection events or metric absent in Tsuga.","Keep as documented but gated; even small counts would be high signal if the metric appears later.",cloudwatch=RecursiveInvocationsDropped,https://docs.aws.amazon.com/lambda/latest/dg/monitoring-metrics-types.html|https://docs.aws.amazon.com/lambda/latest/dg/invocation-recursion.html -Performance/Latency,IteratorAge,false,"Age of the last record for Kinesis and DynamoDB Streams event source mappings. Not found in Tsuga during Stage 2.",ms,Unknown,aws_lambda_iterator_age,"timeseries|query-value|top-list",max,none,"context.functionname:20|context.cloud.region:10",context.env|context.team|context.functionname,context.cloud.region|context.cloud.account.id,"No stream backlog, stream source unused, or metric absent in Tsuga.","Do not pretend mapping-level drilldown exists without discovered mapping attributes.",cloudwatch=IteratorAge,https://docs.aws.amazon.com/lambda/latest/dg/monitoring-metrics-types.html -Capacity/Saturation,OffsetLag,false,"Kafka offset lag observed by the event source mapping. Not found in Tsuga during Stage 2.",{record},Unknown,aws_lambda_offset_lag,"timeseries|query-value|top-list",max,none,"context.functionname:20|context.cloud.region:10",context.env|context.team|context.functionname,context.cloud.region|context.cloud.account.id,"No Kafka-backed mapping, no backlog, or metric absent in Tsuga.","Keep gated until Tsuga confirms both the metric and useful dimensions.",cloudwatch=OffsetLag,https://docs.aws.amazon.com/lambda/latest/dg/monitoring-metrics-types.html|https://docs.aws.amazon.com/lambda/latest/dg/with-msk-process.html -Errors/Failures,OversizedRecordCount,false,"Kafka records skipped because the event exceeded Lambda's size limits. Not found in Tsuga during Stage 2.",{record},Unknown,aws_lambda_oversized_record_count,"timeseries|query-value|top-list",sum,rate,"context.functionname:20|context.cloud.region:10",context.env|context.team|context.functionname,context.cloud.region|context.cloud.account.id,"No oversized Kafka records, Kafka unused, or metric absent in Tsuga.","A small count would still matter, but the metric is not currently exported in Tsuga.",cloudwatch=OversizedRecordCount,https://docs.aws.amazon.com/lambda/latest/dg/monitoring-metrics-types.html -Cost,NOT_APPLICABLE,true,"CloudWatch Lambda service metrics do not expose direct spend. Use invocation volume, duration, concurrency, and unreserved-pool pressure as cost-adjacent proxies until billing data is integrated.",Unknown,Unknown,Unknown,note,Unknown,none,Unknown,context.env|context.team,context.functionname|context.cloud.region,"Not applicable in the core metric family.","Do not invent cost widgets from runtime metrics alone.",proxy=Invocations|proxy=Duration|proxy=UnreservedConcurrentExecutions,https://docs.aws.amazon.com/lambda/latest/dg/monitoring-metrics-types.html|https://docs.aws.amazon.com/lambda/latest/dg/configuration-concurrency.html diff --git a/plugins/tsuga/skills/knowledge-technology/references/aws-lambda/queries.md b/plugins/tsuga/skills/knowledge-technology/references/aws-lambda/queries.md deleted file mode 100644 index e98f019..0000000 --- a/plugins/tsuga/skills/knowledge-technology/references/aws-lambda/queries.md +++ /dev/null @@ -1,58 +0,0 @@ -# AWS Lambda - -Serverless compute, billed per-invocation + duration. Healthy: low error rate, duration within SLO, low throttle rate, concurrency under limits. - -## Incident shapes - -- **Error spike** — `aws_lambda_errors / aws_lambda_invocations` rises → code defect, deploy regression, or dependency failure -- **Throttling** — `aws_lambda_throttles` nonzero → reserved concurrency too low or account ceiling hit -- **aws_lambda_duration regression** — p95 `aws_lambda_duration` doubles → dependency slow, cold-start surge, or algorithmic regression -- **Async queue growth** — `aws_lambda_async_event_age` climbs → async invocations backing up -- **Stream consumer lag** — `aws_lambda_iterator_age` / `aws_lambda_offset_lag` climb on Kinesis / DDB-streams / MSK triggers -- **Dead-letter drop** — `aws_lambda_dead_letter_errors > 0` → retries exhausted, messages lost - -## Key metrics - -| Metric | Unit | Signal | -|---|---|---| -| `aws_lambda_invocations` | count | Baseline rate | -| `aws_lambda_errors` | count | Handler errors | -| `aws_lambda_throttles` | count | Any nonzero = concurrency-ceiling hit | -| `aws_lambda_duration` | ms | Use p50/p95/p99, not avg | -| `aws_lambda_concurrent_executions` | count | Peak concurrency | -| `aws_lambda_claimed_account_concurrency` | count | Account-level ceiling usage | -| `aws_lambda_async_event_age` | ms | Oldest pending async event | -| `aws_lambda_async_events_dropped` | count | Nonzero = async retry exhaustion, lost | -| `aws_lambda_dead_letter_errors` | count | DLQ delivery failure | -| `aws_lambda_destination_delivery_failures` | count | Destination-config failure | -| `aws_lambda_iterator_age` | ms | Kinesis/DDB stream-trigger lag | -| `aws_lambda_offset_lag` | records | Kafka/MSK trigger lag | -| `aws_lambda_oversized_record_count` | count | Records exceeding batch-size limit (dropped) | -| `aws_lambda_provisioned_concurrency_utilization` | % | Pre-warmed pool usage | -| `aws_lambda_provisioned_concurrency_spillover_invocations` | count | Spilled to on-demand; cold-start hit | - -## Derived signals - -- `aws_lambda_errors / aws_lambda_invocations` — error rate. Baseline < 0.01. -- `aws_lambda_throttles / (aws_lambda_invocations + aws_lambda_throttles)` — throttle rate. Any sustained = capacity. -- `p95 aws_lambda_duration / function timeout` — saturation. > 0.8 = timeout regression risk. -- `ReservedConcurrency - max(aws_lambda_concurrent_executions)` — headroom. < 5 sustained = imminent throttle. - -## Log patterns - -- `Task timed out after N seconds` — timeout hit -- `Process exited before completing request` — unhandled crash / OOM -- `REPORT RequestId: ... aws_lambda_duration: X Max Memory Used: Y MB` — invocation report -- `Unable to import module` / `Runtime.ImportModuleError` — bad deploy -- `Rate Exceeded` — account throttle -- `connect ETIMEDOUT` — downstream timeout -- `Init aws_lambda_duration: X ms` — cold start marker - -## Gotchas - -- `aws_lambda_duration` is billable; use p95/p99. Average hides long-tail users feel. -- Metrics are 1-minute granular. 30-second outages may not record. -- `aws_lambda_concurrent_executions` is max-over-minute; sub-minute spikes that throttled can be invisible. -- `aws_lambda_errors` excludes throttles (counted separately). -- Stream-trigger functions that throttle accumulate `aws_lambda_iterator_age` / `aws_lambda_offset_lag` without raising `aws_lambda_errors`. -- `aws_lambda_async_events_dropped` = messages gone unless DLQ/destination configured. diff --git a/plugins/tsuga/skills/knowledge-technology/references/aws-nat-gateway/README.md b/plugins/tsuga/skills/knowledge-technology/references/aws-nat-gateway/README.md deleted file mode 100644 index e1db1d7..0000000 --- a/plugins/tsuga/skills/knowledge-technology/references/aws-nat-gateway/README.md +++ /dev/null @@ -1,266 +0,0 @@ -# AWS NAT Gateway Integration Context Bundle - -## Metadata -**Technology:** AWS NAT Gateway -**Deployment:** managed -**Environment:** prod -**Persona:** SRE Dev and ops -**Telemetry preference:** mixed -**Integration scope:** core service only -**Primary use-case:** reliability and performance - -## How to use this bundle -- Use `01_aws-nat-gateway_metrics.csv` as the metric source of truth for NAT Gateway traffic, translation health, packet drops, and port pressure. -- Use `02_aws-nat-gateway_dashboard_plan.yaml` for section structure, widget definitions, derived signals, explanation notes, triage chains, and playbooks. -- Use `03_aws-nat-gateway_state.yaml` for machine-readable unknowns, assumptions, log-route handoff context, and Stage 2 verification targets. -- Use `04_aws-nat-gateway_memory.md` for the short narrative handoff into Stage 2. -- Stage 2 will create `05_aws-nat-gateway_metric_catalog.csv` as the discovered Tsuga metric inventory and description substrate. -- Stage 4 should read this file's `Log intelligence (Stage 4 handoff)` section and `03_aws-nat-gateway_state.yaml` `log_intel` block before proposing any route payload. - -## What it is and what "good" looks like -### Confirmed by sources -AWS NAT Gateway is an AZ-scoped managed network address translation service that lets private subnets initiate outbound connections to the internet, other VPCs, or other networks through a translated source address. It exposes CloudWatch metrics for connection volume, translated byte and packet flow in each direction, active connections, failed port allocations, idle timeouts, dropped packets, and peak throughput. Good posture means outbound traffic is flowing in the expected direction, connection attempts are converting into established connections, `ErrorPortAllocation`, `IdleTimeoutCount`, and `PacketsDropCount` stay near zero, and peak throughput remains below the point where connection failures or drops start appearing. -Sources: https://docs.aws.amazon.com/vpc/latest/userguide/metrics-dimensions-nat-gateway.html, https://docs.aws.amazon.com/vpc/latest/userguide/nat-gateway-basics.html, https://docs.aws.amazon.com/vpc/latest/userguide/nat-gateway-troubleshooting.html - -### Best-practice inference -- Incident shape 1: outbound dependency traffic stalls while active connections remain high. Start in `translation-health` to check connection establishment success, then `drops-port-pressure`. -- Incident shape 2: burst traffic or fan-out change causes port exhaustion or packet drops. Start in `drops-port-pressure`. -- Incident shape 3: traffic volume looks healthy but response path collapses or return traffic skews. Start in `traffic-directionality`. - -## Key concepts -### Glossary -| Term | Definition | Operational meaning | Dashboard section | -|---|---|---|---| -| NAT Gateway | Managed AWS network address translator | Core egress path for private workloads | all sections | -| Private subnet | Subnet without direct internet ingress | Usually depends on NAT for outbound reachability | traffic-directionality | -| Public NAT gateway | NAT gateway backed by an Elastic IP | Common pattern for internet egress from private subnets | traffic-directionality | -| Private NAT gateway | NAT gateway without internet routing | Used for private network translation patterns | traffic-directionality | -| Source translation | Rewriting source IP/port on egress | Fundamental success path for outbound connections | translation-health | -| Connection attempt | A new outbound connection request seen by the gateway | Demand denominator for failure ratios | translation-health | -| Connection established | A connection successfully translated and opened | Health proxy for successful translation | translation-health | -| Active connection | Concurrent open translated connections | Saturation and concurrency pressure signal | connections-capacity | -| ErrorPortAllocation | CloudWatch metric for failed source-port allocation | Strong evidence of NAT capacity exhaustion per destination tuple | drops-port-pressure | -| Idle timeout | Connection closed after inactivity | Often signals long-lived idle flows or keepalive mismatch | translation-health | -| Packet drop | Packet not forwarded by the NAT gateway | Direct customer-impact symptom when sustained | drops-port-pressure | -| BytesInFromSource | Bytes entering NAT from private-side sources | Outbound request-side volume | traffic-directionality | -| BytesOutToDestination | Bytes exiting NAT toward destination | Forwarded outbound volume | traffic-directionality | -| BytesInFromDestination | Bytes returning from destination toward NAT | Response-side volume | traffic-directionality | -| BytesOutToSource | Bytes exiting NAT toward private-side source | Returned response volume | traffic-directionality | -| PacketsInFromSource | Packets entering NAT from private-side sources | Request packet-rate baseline | traffic-directionality | -| PacketsOutToDestination | Packets forwarded to destination | Outbound forwarding confirmation | traffic-directionality | -| PacketsInFromDestination | Packets returning from destination | Response path signal | traffic-directionality | -| PacketsOutToSource | Packets forwarded back to private source | End-to-end response delivery proxy | traffic-directionality | -| PeakBytesPerSecond | Highest byte throughput observed in the minute | Burstiness and headroom proxy | packet-shape-bursts | -| PeakPacketsPerSecond | Highest packet throughput observed in the minute | Packet-intensity burst proxy | packet-shape-bursts | -| Elastic IP | Public IP attached to a public NAT gateway | Important for internet egress identity and troubleshooting | traffic-directionality | -| Availability Zone | AZ containing the NAT gateway | Blast-radius boundary because NAT gateways are AZ-local | connections-capacity | -| VPC Flow Logs | VPC-level connection log records | Best Stage 4 fallback because NAT Gateway has no native service logs | translation-health | -| Traffic path | Flow Logs field indicating network path | Useful to distinguish internet/NAT path evidence when available | translation-health | - -### Concept Map -```text -Private workload -> sends -> outbound packet to private subnet route table (why: NAT is reached only when routing sends egress to it) -Private subnet route table -> targets -> NAT gateway (why: routing decides whether egress uses NAT) -NAT gateway -> performs -> source IP and port translation (why: private sources need translated identity) -NAT gateway -> consumes -> source ports per destination tuple (why: port pool limits create exhaustion risk) -ConnectionAttemptCount -> represents -> new outbound connection demand (why: best denominator for establishment success) -ConnectionEstablishedCount -> represents -> successful translated sessions (why: confirms the gateway opened flows) -ActiveConnectionCount -> reflects -> concurrent live translated sessions (why: shows concurrency pressure) -ErrorPortAllocation -> indicates -> no source port available for a new flow (why: strong saturation symptom) -IdleTimeoutCount -> indicates -> connection expired after inactivity (why: long-lived idle protocols can fail without keepalives) -PacketsDropCount -> indicates -> packets were discarded (why: direct customer impact if sustained) -BytesInFromSource -> should lead to -> BytesOutToDestination (why: outbound requests should be forwarded) -BytesInFromDestination -> should lead to -> BytesOutToSource (why: return path should be delivered back to clients) -PacketsInFromSource -> should lead to -> PacketsOutToDestination (why: request packets traverse the translator) -PacketsInFromDestination -> should lead to -> PacketsOutToSource (why: response packets traverse back to workloads) -PeakBytesPerSecond -> bounds -> burst throughput intensity (why: average traffic can hide short spikes) -PeakPacketsPerSecond -> bounds -> burst packet intensity (why: packet-heavy workloads can fail before byte-heavy workloads) -One NAT gateway -> lives in -> one Availability Zone (why: outages and scaling are AZ-scoped) -One subnet -> should prefer -> same-AZ NAT gateway (why: reduces cross-AZ dependency and blast radius) -Destination service -> can stall -> response path bytes and packets (why: request traffic alone is not enough for success) -Keepalive mismatch -> increases -> IdleTimeoutCount (why: NAT expires idle state after 350 seconds) -Fan-out to same destination -> increases -> ErrorPortAllocation risk (why: per-destination connection scaling is bounded) -PacketsDropCount -> often follows -> burst or pressure events (why: sustained overload degrades forwarding) -Traffic skew by NAT gateway id -> reveals -> hot gateway or routing imbalance (why: one gateway can saturate before peers) -context.env -> scopes -> safe global dashboard filter (why: mixed environments distort burst and error baselines) -context.team -> scopes -> owner routing (why: NAT issues often map to platform or network owners) -context.cloud.region -> scopes -> regional blast radius (why: metrics aggregate differently across regions) -context.natgatewayid -> scopes -> resource-level triage (why: most NAT issues are per gateway, not per account) -Flow logs -> validate -> connection-level evidence when metrics are ambiguous (why: no native NAT access logs exist) -``` - -### Entities and dimensions -| Entity/Dimension | Why useful | Cardinality risk | Safe top-N | Do NOT group-by guidance | -|---|---|---|---|---| -| `context.natgatewayid` | Primary resource identity for troubleshooting | Low | 20 | Prefer as first-level grouping | -| `context.cloud.region` | Regional blast-radius split | Low | 10 | Usually a global filter in single-region deployments | -| `context.cloud.account.id` | Multi-account isolation | Low | 10 | Optional if single-account estate | -| `context.cloud.provider` | Confirms AWS-only scope and mixed-cloud hygiene | Low | 4 | Best used as a filter sanity check, not a chart split | -| `context.subnet.id` | Identifies private-side route source | Medium | 15 | Avoid top-level KPI groupings unless strongly bounded | -| `context.env` | Environment filter | Low | 5 | Use as a global filter, not a chart split | -| `context.team` | Ownership routing | Low | 10 | Use as a global filter, not a chart split | -| `context.route_table.id` | Helps debug wrong routing | Medium | 10 | Debug-only, not default widget split | -| `context.destination.address_family` | Distinguishes IPv4-only NAT path use | Low | 4 | Avoid unless dual-stack behavior matters | -| `context.destination.service` | Helps isolate fan-out hot spots | Medium | 10 | Only if a bounded service tag exists in Tsuga | -| `context.interface.id` | Useful for flow-log joins | Medium | 10 | Never use as a top-level dashboard grouping unless verified and bounded | -| `context.source.workload` | Maps egress to calling workload | Medium | 12 | Avoid if derived from pod/task identity without aggregation | -| `context.traffic_path` | Useful Flow Logs path evidence | Low | 6 | Log-only or debug-only unless verified in metrics | - -### Tsuga field mapping -| Vendor/exporter dimension | Recommended context.* key | Must-exist vs optional | -|---|---|---| -| NAT gateway ID | `context.natgatewayid` | Must-exist | -| Availability Zone | `Unknown` | Optional | -| AWS Region | `context.cloud.region` | Must-exist | -| AWS Account ID | `context.cloud.account.id` | Must-exist | -| VPC ID | `Unknown` | Optional | -| Subnet ID | `context.subnet.id` | Optional | -| Route table ID | `context.route_table.id` | Optional | -| Team tag | `context.team` | Must-exist | -| Environment tag | `context.env` | Must-exist | -| Cloud provider | `context.cloud.provider` | Must-exist | -| Exporter ARN | `context.aws.exporter.arn` | Optional | -| ENI / interface id for Flow Logs correlation | `context.interface.id` | Optional | - -#### Confirmed by sources -AWS publishes NAT Gateway metrics per `NatGatewayId`, and NAT gateways themselves are AZ-scoped resources deployed in a VPC subnet. -Sources: https://docs.aws.amazon.com/vpc/latest/userguide/metrics-dimensions-nat-gateway.html, https://docs.aws.amazon.com/vpc/latest/userguide/nat-gateway-basics.html - -#### Best-practice inference -Stage 2 confirmed the resource key is `context.natgatewayid`, but VPC-, subnet-, route-table-, and AZ-level dimensions are not present on the discovered NAT metrics and must not be assumed in dashboard queries. - -## Golden signals -### Confirmed by sources -| Signal | NAT Gateway meaning | Typical degradations | Best telemetry sources | What people page on | Section questions | -|---|---|---|---|---|---| -| Traffic | Outbound and return byte/packet flow through translation | sudden traffic drop, directional skew, response path collapse | NAT Gateway CloudWatch bytes/packets metrics | "egress stopped", "return traffic vanished", "traffic burst exceeded normal" | Is traffic flowing both ways? Which gateway is hottest? | -| Errors | Failed translation or forwarding events | `ErrorPortAllocation`, `PacketsDropCount` spikes | NAT Gateway CloudWatch error metrics, VPC Flow Logs for evidence | "new connections fail", "packets dropped under load" | Are failures due to port exhaustion or packet drops? | -| Latency | No direct latency metric; use lifecycle proxies | `IdleTimeoutCount` growth, connection establishment gap, stalled return path | NAT Gateway metrics plus VPC Flow Logs for path evidence | "connections hang then fail", "idle sessions reset" | Are connections establishing and staying valid? | -| Saturation | Concurrency and burst pressure against a per-gateway translator | high `ActiveConnectionCount`, high peaks, rising port-allocation errors | Active connections, peak throughput, error metrics | "one gateway is hot", "burst traffic creates hard failures" | Are we approaching or exceeding safe NAT headroom? | - -### Best-practice inference -For NAT Gateway, connection-establishment success and packet-drop rate are usually more actionable than a generic "availability" KPI because the service is binary only at the connection path. There is no first-class latency metric, so dashboard design should present lifecycle proxies and explicitly say they are proxies, not direct latency. - -## Telemetry sources -### Confirmed by sources -| Source type | How collected | What it provides | Pros/cons | Common pitfalls | -|---|---|---|---|---| -| CloudWatch NAT Gateway metrics | AWS-native metrics | Canonical bytes, packets, connections, drops, port-allocation errors, idle timeouts, peaks | Best source for gateway health and load | No direct latency or per-destination breakdown | -| VPC Flow Logs | VPC/subnet/ENI flow logging | Connection-level records for src/dst/ports/action/log-status and optional path fields | Best evidence source when metrics show a gateway problem | Must be enabled; aggregation interval and field version affect usefulness | -| Route table / NAT configuration state | AWS control plane | Which private subnets use which NAT gateway and whether AZ-local routing is followed | Needed to understand blast radius and one-gateway hot spots | Not a metric source; configuration drift can explain metric skew | -| Exporter / collector normalization | Tsuga ingestion layer | Final `aws_natgateway_*` names and context field shapes | Determines dashboard query syntax | Naming/temporality may differ from CloudWatch expectations | - -### Best-practice inference -- "No data" on CloudWatch NAT metrics can mean the gateway is idle, the gateway is absent from the selected account/region, or collector coverage is missing. -- "No data" on Flow Logs usually means Flow Logs were never enabled or the selected field version does not include the desired attributes. -- Optional features matter: private vs public NAT changes operational context, but the base metric family remains the same. - -## Log intelligence (Stage 4 handoff) -### Confirmed by sources -1. **Log sources matrix** - -| Source | Access path | Typical format | Structured vs unstructured | Evidence | -|---|---|---|---|---| -| VPC Flow Logs to CloudWatch Logs or S3 | VPC / subnet / ENI flow log destination | Space-delimited record with versioned fields | Structured text | https://docs.aws.amazon.com/vpc/latest/userguide/flow-log-records.html | -| NAT Gateway metrics + troubleshooting context | CloudWatch metrics and AWS console | Metrics only, not raw logs | Structured metrics only | https://docs.aws.amazon.com/vpc/latest/userguide/metrics-dimensions-nat-gateway.html | -| NAT Gateway troubleshooting docs | AWS docs | Narrative examples and failure explanations | Unstructured documentation | https://docs.aws.amazon.com/vpc/latest/userguide/nat-gateway-troubleshooting.html | - -2. **Known log formats** -- **VPC Flow Logs default format** - - Sample line: `2 123456789012 eni-abc123 10.0.1.10 52.95.110.1 49152 443 6 10 840 1709980000 1709980060 ACCEPT OK` - - Delimiter/shape notes: space-delimited positional fields. - - Timestamp pattern: Unix epoch seconds for `start` and `end`. - - Quoting behavior: none in default format. - - Optional fields: custom formats can add fields such as `pkt-srcaddr`, `pkt-dstaddr`, `traffic-path`, `az-id`, or subnet metadata when supported. - -3. **Candidate query filters for Stage 4** -- Precise: `context.service.name:vpc-flow-logs AND (context.natgatewayid:* OR message:*nat-*gateway*)` - - Rationale: best if the ingestion pipeline already maps NAT gateway identity or source ENI metadata. - - Risk: likely misses data if NAT gateway identity is not normalized into Tsuga log attributes. -- Fallback: `context.service.name:vpc-flow-logs AND (dstport:443 OR dstport:80 OR action:REJECT OR log-status:SKIPDATA)` - - Rationale: broad enough to capture common egress flows and failure evidence for NAT-heavy paths. - - Risk: captures more non-NAT traffic and needs refinement by VPC/subnet/interface context. - -4. **Attribute mapping hints** - -| Raw field | Suggested Tsuga key | Confidence | Notes | -|---|---|---|---| -| `interface-id` | `context.interface.id` | medium | Useful join field if NAT ENI mapping is known | -| `srcaddr` | `network.client.ip` | medium | Private source workload IP for request-side flow | -| `dstaddr` | `network.destination.ip` | medium | Destination service IP | -| `srcport` | `network.client.port` | medium | Useful when debugging ephemeral-port churn | -| `dstport` | `network.destination.port` | high | Critical for egress protocol/service breakdown | -| `protocol` | `network.transport` | medium | Numeric in raw logs; requires mapping | -| `packets` | `network.packets` | high | Good for failure-volume evidence | -| `bytes` | `network.bytes` | high | Good for traffic evidence | -| `action` | `event.outcome` | high | Map ACCEPT/REJECT carefully | -| `log-status` | `aws.vpc.flow_log.status` | high | Important for missing-data interpretation | -| `traffic-path` | `context.traffic_path` | medium | Valuable when enabled; not guaranteed in every format | - -5. **Parsing risks** -- Flow Log field set is version-dependent; do not assume `traffic-path` or packet-level address fields exist. -- Records are positional, so the parser must match the exact configured format. -- NAT Gateway itself has no native access log, so route logic must not pretend there is a NAT-specific raw log stream. -- Interface-to-NAT-gateway resolution may need AWS inventory enrichment outside the log line itself. - -### Best-practice inference -Stage 4 should treat VPC Flow Logs as the primary evidence stream and keep NAT-specific parsing conservative. If NAT gateway id is not directly present in the logs, the first viable route may need to key off VPC, subnet, or ENI metadata rather than a clean NAT gateway attribute. - -## Caveats and footguns -- **[traffic-directionality]** `BytesInFromSource` and `BytesOutToDestination` are request-side path metrics; they do not prove the response path is healthy. (https://docs.aws.amazon.com/vpc/latest/userguide/metrics-dimensions-nat-gateway.html) -- **[traffic-directionality]** `BytesInFromDestination` and `BytesOutToSource` can fall even when outbound traffic is healthy if the upstream dependency is stalling or rejecting. (Inference) -- **[traffic-directionality]** Packet and byte asymmetry is normal for request/response protocols; alert only on material baseline shift, not perfect symmetry. (Inference) -- **[translation-health]** NAT Gateway exposes no direct latency metric; any "latency" widget is a proxy and must be labeled as such. (https://docs.aws.amazon.com/vpc/latest/userguide/metrics-dimensions-nat-gateway.html) -- **[translation-health]** `IdleTimeoutCount` reflects idle-flow expiry after 350 seconds and is often application keepalive mismatch, not gateway unavailability. (https://docs.aws.amazon.com/vpc/latest/userguide/nat-gateway-troubleshooting.html) -- **[translation-health]** `ConnectionAttemptCount` without `ConnectionEstablishedCount` gap growth can still hide app-level failure beyond the gateway. (Inference) -- **[translation-health]** A low establishment-success ratio can be caused by destination refusal or routing drift, not only NAT pressure. (Inference) -- **[translation-health]** Flow Logs `ACCEPT` does not prove the full application request succeeded; it only proves the network flow record outcome. (https://docs.aws.amazon.com/vpc/latest/userguide/flow-log-records.html) -- **[connections-capacity]** NAT gateways are AZ-scoped, so summing active connections across AZs can hide a single hot gateway. (https://docs.aws.amazon.com/vpc/latest/userguide/nat-gateway-basics.html) -- **[connections-capacity]** Cross-AZ private-subnet routing to a different AZ's NAT gateway increases blast radius and can distort one-gateway saturation. (https://docs.aws.amazon.com/vpc/latest/userguide/nat-gateway-basics.html) -- **[connections-capacity]** `ActiveConnectionCount` is a gauge; do not rate-normalize it. (Inference) -- **[connections-capacity]** Peak metrics are maxima over a minute and can look scary next to average throughput even when sustained load is moderate. (Inference) -- **[drops-port-pressure]** `ErrorPortAllocation` is the most direct hard-failure signal for port exhaustion and should not be averaged away into long windows. (https://docs.aws.amazon.com/vpc/latest/userguide/metrics-dimensions-nat-gateway.html) -- **[drops-port-pressure]** Port-allocation failures are destination-sensitive; a single hot destination can fail while total traffic still looks normal. (https://docs.aws.amazon.com/vpc/latest/userguide/nat-gateway-troubleshooting.html) -- **[drops-port-pressure]** `PacketsDropCount` proves forwarding loss but not root cause by itself; correlate with peaks and port-allocation errors. (https://docs.aws.amazon.com/vpc/latest/userguide/metrics-dimensions-nat-gateway.html) -- **[drops-port-pressure]** Low absolute error counts can still matter if the environment has few connection attempts; ratios need denominator context. (Inference) -- **[packet-shape-bursts]** `PeakBytesPerSecond` and `PeakPacketsPerSecond` can spike without any customer issue if the system is bursty but below failure thresholds. (Inference) -- **[packet-shape-bursts]** Byte-heavy and packet-heavy workloads stress different limits; do not assume one peak metric is sufficient. (Inference) -- **[packet-shape-bursts]** Average throughput alone hides burst headroom problems; always pair it with peak metrics. (https://docs.aws.amazon.com/vpc/latest/userguide/metrics-dimensions-nat-gateway.html) -- **[traffic-directionality, translation-health]** Resource-level triage depends on `context.natgatewayid`; if that key disappears from ingestion, the dashboard must fall back to region-only grouping. (Stage 2 discovery) -- **[translation-health, drops-port-pressure]** The discovered NAT counters are all `summary/cumulative` in Tsuga, so counter widgets must use `rate` instead of `per-second`. (Stage 2 discovery) -- **[packet-shape-bursts]** Peak-to-average ratios are diagnostic heuristics, not official AWS health indicators. (Inference) -- **[connections-capacity, drops-port-pressure]** No cost metric exists in the base NAT Gateway metric family; do not invent a cost dashboard section from throughput alone. (Inference) - -## Confirmed Tsuga prefixes -- `aws_natgateway_*` — **CONFIRMED** (16/16 operational NAT metrics present in Tsuga discovery; matched by `tools/tsuga_search_metrics.py '^aws_natgateway_.*'`) - -## Discovery status -Discovery: completed in Stage 2. -- Metrics found: 16 total -- Metrics confirmed from `01`: 16 -- Missing from `01`: 0 -- Unexpected additional metrics: 0 -- Confirmed reusable context keys: `context.natgatewayid`, `context.cloud.region`, `context.cloud.account.id`, `context.cloud.provider`, `context.env`, `context.team` -- All discovered NAT metrics report `type=summary` and `temporality=cumulative`; counters were reconciled to use `rate` - -## Top sources -1. https://docs.aws.amazon.com/vpc/latest/userguide/metrics-dimensions-nat-gateway.html - Why: canonical NAT Gateway metric names, meanings, and dimensions. -2. https://docs.aws.amazon.com/vpc/latest/userguide/nat-gateway-basics.html - Why: architecture, AZ scope, and operational behavior. -3. https://docs.aws.amazon.com/vpc/latest/userguide/nat-gateway-troubleshooting.html - Why: key failure modes such as idle timeout and port-allocation exhaustion. -4. https://docs.aws.amazon.com/vpc/latest/userguide/flow-log-records.html - Why: VPC Flow Logs schema and field options for Stage 4. -5. https://docs.aws.amazon.com/vpc/latest/userguide/nat-gateway-working-with.html - Why: lifecycle and operational management context for gateways. -6. https://docs.aws.amazon.com/vpc/latest/userguide/vpc-nat-gateway.html - Why: service overview and public/private NAT positioning. -7. https://docs.aws.amazon.com/vpc/latest/userguide/route-table-options.html#route-tables-nat-device - Why: route-table behavior that explains blast radius and skew. -8. https://docs.aws.amazon.com/vpc/latest/userguide/flow-logs-records-examples.html - Why: concrete Flow Logs examples to guide Stage 4 parsing. -9. https://docs.aws.amazon.com/vpc/latest/userguide/amazon-vpc-limits.html - Why: quota context for related VPC constructs and NAT planning. -10. https://docs.aws.amazon.com/whitepapers/latest/building-scalable-secure-multi-vpc-network-infrastructure/using-nat-gateway-for-centralized-egress.html - Why: architecture pattern context for hot-gateway and shared-egress scenarios. diff --git a/plugins/tsuga/skills/knowledge-technology/references/aws-nat-gateway/metrics.csv b/plugins/tsuga/skills/knowledge-technology/references/aws-nat-gateway/metrics.csv deleted file mode 100644 index 31ae011..0000000 --- a/plugins/tsuga/skills/knowledge-technology/references/aws-nat-gateway/metrics.csv +++ /dev/null @@ -1,18 +0,0 @@ -theme,name,confirmed,definition,unit,type,tsuga_metric_name,widget_suggestions,aggregation,post_function,group_by,required_context_fields,optional_context_fields,absence_means,cardinality_footguns,variants,citations -Throughput/Usage,BytesInFromSource,true,"Bytes entering the NAT gateway from private-side sources; request-side egress baseline.",By,counter,aws_natgateway_bytes_in_from_source,"timeseries|query-value|top-list",sum,"rate","context.natgatewayid:10|context.cloud.region:10",context.env|context.team,context.cloud.account.id|context.cloud.provider|context.cloud.region|context.natgatewayid,"No request-side traffic, missing collector coverage, or wrong namespace prefix.","Prefer nat gateway id over workload-level identity to avoid cardinality explosion.",cloudwatch=BytesInFromSource,https://docs.aws.amazon.com/vpc/latest/userguide/metrics-dimensions-nat-gateway.html -Throughput/Usage,BytesOutToDestination,true,"Bytes forwarded from the NAT gateway to external destinations; confirms outbound forwarding volume.",By,counter,aws_natgateway_bytes_out_to_destination,"timeseries|query-value|top-list",sum,"rate","context.natgatewayid:10|context.cloud.region:10",context.env|context.team,context.cloud.account.id|context.cloud.provider|context.cloud.region|context.natgatewayid,"No forwarded outbound traffic, idle gateway, or ingest gap.","Do not group by low-level source IP unless Flow Logs prove bounded normalization.",cloudwatch=BytesOutToDestination,https://docs.aws.amazon.com/vpc/latest/userguide/metrics-dimensions-nat-gateway.html -Throughput/Usage,BytesInFromDestination,true,"Bytes returning from external destinations into the NAT gateway; response-path volume proxy.",By,counter,aws_natgateway_bytes_in_from_destination,"timeseries|query-value|top-list",sum,"rate","context.natgatewayid:10|context.cloud.region:10",context.env|context.team,context.cloud.account.id|context.cloud.provider|context.cloud.region|context.natgatewayid,"No return traffic, stalled dependencies, or missing data.","Interpret with request-side metrics; response traffic is naturally asymmetric.",cloudwatch=BytesInFromDestination,https://docs.aws.amazon.com/vpc/latest/userguide/metrics-dimensions-nat-gateway.html -Throughput/Usage,BytesOutToSource,true,"Bytes forwarded from the NAT gateway back to private-side sources; confirms response delivery.",By,counter,aws_natgateway_bytes_out_to_source,"timeseries|query-value|top-list",sum,"rate","context.natgatewayid:10|context.cloud.region:10",context.env|context.team,context.cloud.account.id|context.cloud.provider|context.cloud.region|context.natgatewayid,"No responses returned to workloads, no traffic, or ingest gap.","Avoid grouping by ephemeral source address if collector exports raw IPs.",cloudwatch=BytesOutToSource,https://docs.aws.amazon.com/vpc/latest/userguide/metrics-dimensions-nat-gateway.html -Throughput/Usage,PacketsInFromSource,true,"Packets entering the NAT gateway from private-side sources; request packet-rate baseline.",{packet},counter,aws_natgateway_packets_in_from_source,"timeseries|query-value|top-list",sum,"rate","context.natgatewayid:10|context.cloud.region:10",context.env|context.team,context.cloud.account.id|context.cloud.provider|context.cloud.region|context.natgatewayid,"No request-side packets or missing ingestion.","Packet metrics can be noisier than bytes for chatty protocols.",cloudwatch=PacketsInFromSource,https://docs.aws.amazon.com/vpc/latest/userguide/metrics-dimensions-nat-gateway.html -Throughput/Usage,PacketsOutToDestination,true,"Packets forwarded from NAT gateway to external destinations; outbound forwarding confirmation.",{packet},counter,aws_natgateway_packets_out_to_destination,"timeseries|query-value|top-list",sum,"rate","context.natgatewayid:10|context.cloud.region:10",context.env|context.team,context.cloud.account.id|context.cloud.provider|context.cloud.region|context.natgatewayid,"No forwarded packets, idle gateway, or missing coverage.","Pair with request-side packet metrics to spot forwarding gaps.",cloudwatch=PacketsOutToDestination,https://docs.aws.amazon.com/vpc/latest/userguide/metrics-dimensions-nat-gateway.html -Throughput/Usage,PacketsInFromDestination,true,"Packets returning from external destinations into the NAT gateway; response-path packet baseline.",{packet},counter,aws_natgateway_packets_in_from_destination,"timeseries|query-value|top-list",sum,"rate","context.natgatewayid:10|context.cloud.region:10",context.env|context.team,context.cloud.account.id|context.cloud.provider|context.cloud.region|context.natgatewayid,"No return packets, dependency stall, or collector gap.","Do not mistake natural packet asymmetry for packet loss without other signals.",cloudwatch=PacketsInFromDestination,https://docs.aws.amazon.com/vpc/latest/userguide/metrics-dimensions-nat-gateway.html -Throughput/Usage,PacketsOutToSource,true,"Packets forwarded from NAT gateway back to private-side sources; response delivery proxy.",{packet},counter,aws_natgateway_packets_out_to_source,"timeseries|query-value|top-list",sum,"rate","context.natgatewayid:10|context.cloud.region:10",context.env|context.team,context.cloud.account.id|context.cloud.provider|context.cloud.region|context.natgatewayid,"No response packets delivered, no traffic, or ingest gap.","Prefer nat gateway id over subnet id for default groupings.",cloudwatch=PacketsOutToSource,https://docs.aws.amazon.com/vpc/latest/userguide/metrics-dimensions-nat-gateway.html -Availability/Health,ConnectionAttemptCount,true,"New outbound connection attempts observed by the NAT gateway; baseline for success and failure ratios.",{connection},counter,aws_natgateway_connection_attempt_count,"timeseries|query-value|top-list",sum,"rate","context.natgatewayid:10|context.cloud.region:10",context.env|context.team,context.cloud.account.id|context.cloud.provider|context.cloud.region|context.natgatewayid,"No new connection demand or missing metric coverage.","A single hot destination can create pressure even if total attempts remain modest.",cloudwatch=ConnectionAttemptCount,https://docs.aws.amazon.com/vpc/latest/userguide/metrics-dimensions-nat-gateway.html -Availability/Health,ConnectionEstablishedCount,true,"Connections successfully established by the NAT gateway; direct proxy for translation success.",{connection},counter,aws_natgateway_connection_established_count,"timeseries|query-value|top-list",sum,"rate","context.natgatewayid:10|context.cloud.region:10",context.env|context.team,context.cloud.account.id|context.cloud.provider|context.cloud.region|context.natgatewayid,"No successful connections, no demand, or wrong namespace mapping.","Interpret with attempts to distinguish idle from failing.",cloudwatch=ConnectionEstablishedCount,https://docs.aws.amazon.com/vpc/latest/userguide/metrics-dimensions-nat-gateway.html -Capacity/Saturation,ActiveConnectionCount,true,"Concurrent active translated connections; core concurrency and saturation gauge.",{connection},gauge,aws_natgateway_active_connection_count,"timeseries|query-value|top-list",max,none,"context.natgatewayid:10|context.cloud.region:10",context.env|context.team,context.cloud.account.id|context.cloud.provider|context.cloud.region|context.natgatewayid,"Gateway is idle, absent, or data is missing.","Do not normalize or rate this gauge.",cloudwatch=ActiveConnectionCount,https://docs.aws.amazon.com/vpc/latest/userguide/metrics-dimensions-nat-gateway.html -Errors/Failures,ErrorPortAllocation,true,"Failed port allocations for new connections; direct sign that the gateway could not find a source port.",{error},counter,aws_natgateway_error_port_allocation,"timeseries|query-value|top-list",sum,"rate","context.natgatewayid:10|context.cloud.region:10",context.env|context.team,context.cloud.account.id|context.cloud.provider|context.cloud.region|context.natgatewayid,"No port exhaustion observed or no connection demand.","Keep windows tight; averages can hide short failure bursts.",cloudwatch=ErrorPortAllocation,https://docs.aws.amazon.com/vpc/latest/userguide/metrics-dimensions-nat-gateway.html -Errors/Failures,IdleTimeoutCount,true,"Connections closed after 350 seconds of inactivity; lifecycle/keepalive risk proxy rather than direct availability signal.",{timeout},counter,aws_natgateway_idle_timeout_count,"timeseries|query-value|top-list",sum,"rate","context.natgatewayid:10|context.cloud.region:10",context.env|context.team,context.cloud.account.id|context.cloud.provider|context.cloud.region|context.natgatewayid,"No idle expiries, no long-lived idle flows, or missing metric coverage.","Can spike because of protocol behavior, not gateway failure.",cloudwatch=IdleTimeoutCount,https://docs.aws.amazon.com/vpc/latest/userguide/metrics-dimensions-nat-gateway.html|https://docs.aws.amazon.com/vpc/latest/userguide/nat-gateway-troubleshooting.html -Errors/Failures,PacketsDropCount,true,"Packets dropped by the NAT gateway; strong forwarding-failure symptom.",{packet},counter,aws_natgateway_packets_drop_count,"timeseries|query-value|top-list",sum,"rate","context.natgatewayid:10|context.cloud.region:10",context.env|context.team,context.cloud.account.id|context.cloud.provider|context.cloud.region|context.natgatewayid,"No packet drops, no traffic, or collector gap.","Use with packet baselines to avoid overreacting to tiny absolute counts.",cloudwatch=PacketsDropCount,https://docs.aws.amazon.com/vpc/latest/userguide/metrics-dimensions-nat-gateway.html -Capacity/Saturation,PeakBytesPerSecond,true,"Peak byte throughput observed during the minute; burst headroom proxy.",By/s,gauge,aws_natgateway_peak_bytes_per_second,"timeseries|query-value|top-list",max,none,"context.natgatewayid:10|context.cloud.region:10",context.env|context.team,context.cloud.account.id|context.cloud.provider|context.cloud.region|context.natgatewayid,"No burst recorded, idle gateway, or missing data.","Peak and average should be read together; peak alone can overstate steady-state risk.",cloudwatch=PeakBytesPerSecond,https://docs.aws.amazon.com/vpc/latest/userguide/metrics-dimensions-nat-gateway.html -Capacity/Saturation,PeakPacketsPerSecond,true,"Peak packet throughput observed during the minute; packet-intensity headroom proxy.",{packet}/s,gauge,aws_natgateway_peak_packets_per_second,"timeseries|query-value|top-list",max,none,"context.natgatewayid:10|context.cloud.region:10",context.env|context.team,context.cloud.account.id|context.cloud.provider|context.cloud.region|context.natgatewayid,"No burst recorded, idle gateway, or missing data.","Packet-heavy workloads can stress limits before byte-heavy ones.",cloudwatch=PeakPacketsPerSecond,https://docs.aws.amazon.com/vpc/latest/userguide/metrics-dimensions-nat-gateway.html -Cost,NOT_APPLICABLE,true,"The base NAT Gateway metric family does not expose direct cost metrics. Use throughput and active-connection efficiency as cost-adjacent proxies, but keep cost out of the core dashboard until a billing data source is available.",Unknown,Unknown,Unknown,"note",Unknown,none,Unknown,context.env|context.team,context.cloud.region|context.cloud.account.id,"Not applicable in the metric family.","Do not invent cost widgets from incomplete usage signals.",proxy=PeakBytesPerSecond|proxy=ActiveConnectionCount,https://docs.aws.amazon.com/vpc/latest/userguide/metrics-dimensions-nat-gateway.html diff --git a/plugins/tsuga/skills/knowledge-technology/references/aws-nat-gateway/queries.md b/plugins/tsuga/skills/knowledge-technology/references/aws-nat-gateway/queries.md deleted file mode 100644 index 93d39e1..0000000 --- a/plugins/tsuga/skills/knowledge-technology/references/aws-nat-gateway/queries.md +++ /dev/null @@ -1,50 +0,0 @@ -# AWS NAT Gateway - -Managed NAT for private-subnet egress. Healthy: traffic flows both ways, no drops, no port-allocation errors, active connections under 55k per (dest IP, dest port). - -## Incident shapes - -- **Port allocation exhaustion** — `aws_natgateway_error_port_allocation > 0` → many connections to same (dest IP, dest port) tuple exhaust ephemeral ports -- **Packet drops** — `aws_natgateway_packets_drop_count` climbs → malformed, SG denies, or NAT degraded -- **Connection-rate cap** — active connections near 55k per destination → new connections fail -- **Idle timeout storms** — `aws_natgateway_idle_timeout_count` spikes → long-lived connections without keepalives -- **Bandwidth ceiling** — peak-bandwidth saturates → throughput capped -- **Asymmetric traffic** — in/out byte mismatch → one-way connectivity failure - -## Key metrics - -| Metric | Unit | Signal | -|---|---|---| -| `aws_natgateway_active_connection_count` | count | Current connections | -| `aws_natgateway_connection_attempt_count` | count | New-connection attempts | -| `aws_natgateway_connection_established_count` | count | Established | -| `aws_natgateway_error_port_allocation` | count | Any > 0 = incident | -| `aws_natgateway_packets_drop_count` | count | Drops | -| `aws_natgateway_idle_timeout_count` | count | Idle-timed-out connections (default 350s) | -| `aws_natgateway_bytes_in_from_source` / `aws_natgateway_bytes_out_to_source` | bytes | VPC ↔ internet direction | -| `aws_natgateway_bytes_in_from_destination` / `aws_natgateway_bytes_out_to_destination` | bytes | Return direction | -| `aws_natgateway_packets_in_from_source` / `aws_natgateway_packets_out_to_source` | packets | Same, packets | -| `aws_natgateway_peak_bytes_per_second` / `aws_natgateway_peak_packets_per_second` | max | Sub-interval peaks | - -## Derived signals - -- `aws_natgateway_connection_established_count / aws_natgateway_connection_attempt_count` — success rate. < 0.99 sustained = outbound connectivity problem (DNS, SG, remote host). -- `aws_natgateway_packets_drop_count / (aws_natgateway_packets_in_from_source + aws_natgateway_packets_in_from_destination)` — drop rate. Any sustained positive = issue. -- `aws_natgateway_bytes_in_from_source / aws_natgateway_bytes_out_to_source` — asymmetry. Extremes either way = one-way failure. - -## Log patterns - -NAT GW has no logs. Use VPC flow logs + app logs: - -- Flow logs with `REJECT` action on NAT GW ENI -- App: `connect() failed: Cannot assign requested address` — ephemeral-port exhaustion (consumer side) -- App: `connection timed out` — egress failure -- App: `DNS resolution failed` — VPC DNS or upstream resolver issue - -## Gotchas - -- Port-allocation is per (dest IP, dest port). Many consumers to the same upstream on the same port hit 55k fast. Mitigate: more NAT GWs, VPC endpoints (skip NAT), or HTTP/2 reuse upstream. -- `aws_natgateway_idle_timeout_count` on long-lived connections (DB, gRPC) can cause silent failures if client doesn't send keepalives within 350s. -- NAT GW doesn't log drop causes. Distinguish SG / NACL / NAT-internal via flow logs. -- 100 Gbps peak per NAT GW; consistent near-peak = scale horizontally. -- Cross-AZ NAT traffic charges twice. Cost spikes can reveal misconfigured routing tables. diff --git a/plugins/tsuga/skills/knowledge-technology/references/aws-privatelink/README.md b/plugins/tsuga/skills/knowledge-technology/references/aws-privatelink/README.md deleted file mode 100644 index 6a513fc..0000000 --- a/plugins/tsuga/skills/knowledge-technology/references/aws-privatelink/README.md +++ /dev/null @@ -1,1158 +0,0 @@ -# AWS PrivateLink Integration Context Bundle - -**Technology**: AWS PrivateLink -**Deployment**: Managed cloud service -**Environment**: Production -**Persona Focus**: SRE, Dev, and Ops -**Telemetry Preference**: Mixed (AWS CloudWatch native + VPC Flow Logs + Route 53 query logs) -**Integration Scope**: Core service only (VPC endpoints and endpoint services; excludes gateway endpoints, VPC peering, Transit Gateway) -**Primary Use-Case**: Reliability and performance monitoring -**Bundle Created**: 2026-02-10 -**Bundle Version**: 1.0 - ---- - -## How to use this bundle - -This bundle provides everything needed to build production-ready Tsuga dashboards for AWS PrivateLink. **Start with 07** (`07_aws-privatelink_dashboard_plan.yaml`) for the complete dashboard structure and widget specifications. **Refer to 05** (`05_aws-privatelink_metric_inventory.csv`) for the source-of-truth metric definitions, including units, aggregations, and temporality. **Use 09** (`09_aws-privatelink_section_notes_and_playbooks.md`) for all dashboard note content (mission note, section notes, triage chains, playbooks). - -**For Stage 2 (metric discovery)**: The "Confirmed Tsuga prefixes" section below provides the starting point for API discovery. All counter metrics have temporality marked as **VERIFY** in files 05 and 07; Stage 2 will resolve delta vs cumulative via `/v1/metrics/metadata` endpoint. - -**For Stage 3 (dashboard creation)**: Use the dashboard plan (07), metric inventory (05), derived signals (06), and section notes (09) as implementation blueprints. All widgets reference exact metric names, aggregations, post-functions, and normalizers. Quality gates script (`tools/tsuga_quality_gates.py`) will validate payloads before submission. - ---- - -## Confirmed Tsuga prefixes - -- **`aws_privatelinkendpoints_*`** — **CONFIRMED** (PrivateLink consumer/provider CloudWatch metrics) - **Evidence**: 6 metrics found in Tsuga (5 expected + 1 unexpected). Naming is `aws_privatelinkendpoints_*` (NOT `aws_privatelink_*` as initially assumed). Metrics: `aws_privatelinkendpoints_active_connections`, `aws_privatelinkendpoints_new_connections`, `aws_privatelinkendpoints_bytes_processed`, `aws_privatelinkendpoints_packets_dropped`, `aws_privatelinkendpoints_rst_packets_received`, `aws_privatelinkservices_endpoints_count` (unexpected). All are type `summary` with `cumulative` temporality. - **Source**: [AWS PrivateLink CloudWatch metrics documentation](https://docs.aws.amazon.com/vpc/latest/privatelink/privatelink-cloudwatch-metrics.html) + Tsuga API discovery - -- **`aws_networkelb_*`** — **CONFIRMED** (NLB backend target health metrics for provider-side monitoring) - **Evidence**: 3/4 expected NLB metrics found in Tsuga. Naming is `aws_networkelb_*` (NOT `aws_nlb_*`). Metrics: `aws_networkelb_healthy_host_count`, `aws_networkelb_un_healthy_host_count` (note underscore in "un_healthy"), `aws_networkelb_processed_bytes`. MISSING: `target_connection_error_count` (not ingested). Alternative proxy metric available: `aws_networkelb_tcp_target_reset_count`. All are type `summary` with `cumulative` temporality. - **Source**: [AWS NLB CloudWatch metrics documentation](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-cloudwatch-metrics.html) + Tsuga API discovery - -**Discovery strategy for Stage 2**: -1. Query Tsuga API `/v1/metrics/names-and-types?namespace=aws_privatelink*&from=&to=` to list all PrivateLink metrics with dimensions -2. Query `/v1/metrics/metadata` for each metric to confirm `type` (gauge vs summary) and `temporality` (delta vs cumulative) for counters -3. Query `/v1/metrics/names-and-types?namespace=aws_nlb*` OR `namespace=aws_networkloadbalancer*` to locate NLB metrics -4. Validate context field names (context.endpointid vs context.endpoint_id; context.availabilityzone vs context.az) -5. Check for optional dimensions (context.region for cross-region deployments; context.connectionstate if state metrics exist) - ---- - -## Discovery status - -**Discovery**: ✅ COMPLETED (Stage 2 finished 2026-02-10) -**Metrics found**: 10 total (6 PrivateLink + 4 NLB) = **90% coverage** -- PrivateLink: 5/5 expected + 1 unexpected (`endpoints_count`) = 6 metrics -- NLB: 3/4 expected (missing `target_connection_error_count`) = 4 metrics (including `processed_bytes` excluded from dashboards) -**High-impact unknowns**: 0 (all resolved during Stage 2) -**Coverage**: Consumer-side 100% (all 5 PrivateLink metrics confirmed); provider-side 75% (3/4 NLB metrics, 1 missing) - -**CRITICAL DISCOVERY FINDINGS:** -1. **Metric naming**: `aws_privatelinkendpoints_*` and `aws_networkelb_*` (NOT `aws_privatelink_*` or `aws_nlb_*`) -2. **Temporality**: ALL counters are cumulative → use `rate` post-function (NOT `per-second`) -3. **Context fields**: Use SPACE-SEPARATED names: `context.vpc endpoint id` (NOT `context.endpointid`) -4. **NO Availability Zone dimension**: Per-AZ bandwidth analysis IMPOSSIBLE; major dashboard limitation -5. **Missing metric**: NLB `target_connection_error_count` not found; use `tcp_target_reset_count` as proxy - -**Files updated**: 05 (metric inventory corrected), 00 (this file - prefix status updated), 12 (reconciliation report created) - ---- - -## Bundle files - -| # | Filename | Purpose | -|---|---|---| -| 00 | `00_aws-privatelink_cover.md` | Bundle overview, metadata, Tsuga prefix handoff, top sources | -| 01 | `01_aws-privatelink_executive_overview.md` | What it is, where it runs, what "good" looks like, top 3 incident shapes, paging intent | -| 02 | `02_aws-privatelink_key_concepts.md` | Glossary (24 terms), concept map (28 relationships), entities/dimensions (12), Tsuga field mapping | -| 03 | `03_aws-privatelink_golden_signals.md` | Traffic/Errors/Latency/Saturation definitions, causes, telemetry sources, paging symptoms, section questions | -| 04 | `04_aws-privatelink_telemetry_sources.md` | Source matrix (8 sources), optional features, "no data" meanings, source coverage map | -| 05 | `05_aws-privatelink_metric_inventory.csv` | Source-of-truth metric definitions (17 metrics: 5 PrivateLink + 4 NLB + 8 derived/proxy), units, temporality, aggregations, group-by | -| 06 | `06_aws-privatelink_derived_signals.csv` | Formula widgets (12 derived signals: bandwidth utilization, reset rate, cost calculations, backend health ratio) | -| 07 | `07_aws-privatelink_dashboard_plan.yaml` | Dashboard structure (2 dashboards: overview + deep dive), 4 sections, 40+ widgets, coverage map | -| 09 | `09_aws-privatelink_section_notes_and_playbooks.md` | Mission note, 4 section explanation notes, 20 triage chains, 7 operational playbooks | -| 10 | `10_aws-privatelink_caveats_footguns.md` | 33 caveats across 6 categories (cardinality, misleading metrics, units, temporality, false alarms, optional features) | -| 11 | `11_aws-privatelink_unknowns_verify_next.yaml` | 10 unknowns (3 high-impact, 3 medium-impact, 4 low-impact) with verification steps and UI reflection strategies | - ---- - -## Top sources (Stage 3 "Learn more" links) - -1. **[CloudWatch metrics for AWS PrivateLink](https://docs.aws.amazon.com/vpc/latest/privatelink/privatelink-cloudwatch-metrics.html)** — Official AWS documentation defining all 5 PrivateLink CloudWatch metrics (ActiveConnections, NewConnections, BytesProcessed, PacketDropCount, ResetPacketsReceived), namespaces (AWS/PrivateLinkEndpoints, AWS/PrivateLinkServices), and 1-minute granularity. Primary source for metric definitions in 05. - -2. **[AWS PrivateLink concepts](https://docs.aws.amazon.com/vpc/latest/privatelink/concepts.html)** — Core architecture documentation explaining interface VPC endpoints, VPC endpoint services, service consumers, service providers, Network Load Balancer requirements, and connection lifecycle. Foundation for 01 (executive overview) and 02 (key concepts). - -3. **[Configure an endpoint service](https://docs.aws.amazon.com/vpc/latest/privatelink/configure-endpoint-service.html)** — Service provider configuration guide covering connection acceptance (manual vs automatic), allowed principals, endpoint service states, and NLB integration. Source for connection state definitions (Pending, Available, Rejected, Failed, Expired) used in 01 and 02. - -4. **[Fix network performance issues with AWS PrivateLink endpoints](https://repost.aws/knowledge-center/vpc-troubleshoot-network-performance-privatelink)** — AWS re:Post knowledge article documenting 10 Gbps baseline, 100 Gbps auto-scaling, 350-second idle timeout, PacketDropCount troubleshooting, and MTU mismatch issues. Critical source for 03 (golden signals) and 10 (caveats). - -5. **[Gain usage insights with Amazon CloudWatch metrics for AWS PrivateLink](https://aws.amazon.com/blogs/networking-and-content-delivery/gain-usage-insights-with-amazon-cloudwatch-metrics-for-aws-privatelink/)** — AWS blog announcing Contributor Insights integration for PrivateLink, including top-N consumer endpoint ranking by BytesProcessed, ActiveConnections, NewConnections, ResetPacketsReceived. Source for 04 (telemetry sources) Contributor Insights section. - -6. **[AWS PrivateLink pricing](https://aws.amazon.com/privatelink/pricing/)** — Official pricing page documenting hourly endpoint charges ($0.01/hour interface, $0.02/hour resource), data processing charges (tiered: $0.01/GB, $0.006/GB, $0.004/GB), and cross-region data transfer charges ($0.02/GB surcharge). Primary source for cost metrics in 05 and 06. - -7. **[Introducing cross-region connectivity for AWS PrivateLink](https://aws.amazon.com/blogs/networking-and-content-delivery/introducing-cross-region-connectivity-for-aws-privatelink/)** — AWS blog announcing November 2025 launch of cross-region PrivateLink, explaining consumer-provider region mismatch support, IAM permissions (vpce:AllowMultiRegion), and multi-region architecture patterns. Source for cross-region cost analysis in 05, 06, 07. - -8. **[CloudWatch metrics for Network Load Balancers](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-cloudwatch-metrics.html)** — Official AWS documentation for NLB metrics including HealthyHostCount, UnHealthyHostCount, TargetConnectionErrorCount, ProcessedBytes, health check semantics. Primary source for backend target health metrics in 05 (provider-side monitoring). - -9. **[Health checks for Network Load Balancer target groups](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/target-group-health-checks.html)** — NLB health check configuration guide covering health check intervals, thresholds, active vs passive health checks, and target state transitions. Source for backend health troubleshooting in 09 (playbook 4). - -10. **[Troubleshoot connectivity between VPC endpoint and endpoint service](https://repost.aws/knowledge-center/connect-endpoint-service-vpc)** — AWS re:Post troubleshooting guide for connection failures, security group misconfigurations, DNS resolution issues, and endpoint state diagnostics. Source for 09 (playbook 1: connection establishment failure) and 10 (connectivity footguns). - -11. **[Working with private hosted zones — Route 53](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/hosted-zones-private.html)** — Route 53 documentation on private hosted zones for PrivateLink private DNS, VPC associations, split-horizon DNS, and query logging. Source for 02 (key concepts) DNS sections and 09 (playbook 7: DNS resolution failure). - -12. **[VPC Flow Logs documentation](https://docs.aws.amazon.com/vpc/latest/userguide/flow-logs.html)** — Official AWS guide to VPC Flow Logs for ENI-level traffic visibility, REJECT action analysis (security group denials), and flow log aggregation windows (1-10 minutes). Source for 04 (telemetry sources) VPC Flow Logs section and 09 (triage chains using Flow Logs). - ---- - -## Stage 2 handoff checklist - -- [x] Tsuga metric namespace prefixes documented with status labels (INFERRED) -- [x] Discovery strategy documented (which API endpoints to query, what to validate) -- [x] All counter metrics marked with temporality VERIFY in 05 and 07 -- [x] Context field names include fallback options in 05 group_by column -- [x] Unknowns file (11) documents all high and medium impact verification needs -- [x] Top sources list includes >= 10 URLs with justifications -- [x] Bundle files table complete with all 11 files -- [x] Discovery status section explains what Stage 2 will resolve - ---- - -## Validation results (self-check before Stage 2) - -- ✅ Metric name claims cited or marked inferred (05: all metrics have citations or "Inference" notes) -- ✅ 07 coverage: all tsuga-mapped metrics from 05 included or excluded with reason (coverage_map complete) -- ✅ 07 → 09 consistency: all sections in 07 have matching note blocks in 09, all widget titles referenced in 09 exist in 07 -- ✅ 06 → 05 consistency: all `input_metrics` in 06 exist in 05 (verified: all derived signals reference valid base metrics) -- ✅ 06 → 07 consistency: all `section_id` in 06 exist in 07, all formula widgets in 07 have matching rows in 06 -- ✅ 10 → 07 consistency: all section tags in 10 match section ids in 07 (verified: all caveats tagged correctly) -- ✅ No jammed multi-metric widgets without single derived-signal purpose (all multi-query widgets in 07 have clear purpose) -- ✅ Counter temporality guidance exists (05: post_function column has per-second|rate; 07: widgets have # VERIFY comments) -- ✅ Optional features gated (cross-region, NLB provider-side, Contributor Insights sections include gating language) -- ✅ Stage 2 handoff ready: "Confirmed Tsuga prefixes" section complete with evidence and status labels -- ✅ Temporality markers: Counter metrics in 05 have `post_function` guidance; widget specs in 07 include `# VERIFY` comments - ---- - -## Summary - -This bundle provides **comprehensive monitoring coverage** for AWS PrivateLink across **consumer-side** (VPC endpoints) and **provider-side** (VPC endpoint services) perspectives. Two dashboards (overview KPI wall + deep dive operational investigation) cover **4 major sections**: Endpoint Connection Health, Data Transfer & Network Performance, Backend Target Health (provider-side), and Cost & Usage. - -**Strengths**: Grounded in official AWS documentation (12 primary sources), includes operational playbooks and triage chains (20 chains, 7 playbooks), comprehensive caveats (33 footguns documented), and clear unknowns tracking (10 items with verification strategies). - -**Limitations**: Counter temporality unconfirmed (affects all rate calculations; marked for Stage 2 verification), NLB metric namespace unknown (backend health section may be partial coverage), connection state metrics may not exist as CloudWatch metrics (endpoint state distribution may require API calls vs dashboard widgets). - -**Next step**: Proceed to Stage 2 (metric discovery & reconciliation) using Tsuga API to resolve all high and medium impact unknowns. - - ---- - -# AWS PrivateLink — Executive Overview - -## What it is -AWS PrivateLink is a managed networking service that enables private connectivity between VPCs, AWS services, and on-premises networks without exposing traffic to the public internet. Service providers host services behind Network Load Balancers in their VPCs and expose them as VPC endpoint services. Consumers create interface VPC endpoints in their VPCs that connect privately to these services using AWS's internal network backbone. - -## Where it runs -PrivateLink operates within AWS VPCs across single or multiple accounts and regions. Interface VPC endpoints are deployed as elastic network interfaces (ENIs) within consumer VPC subnets across one or more Availability Zones. The service provider side anchors a Network Load Balancer (NLB) in their VPC, with target health checks ensuring backend availability. - -## What "good" looks like -Healthy PrivateLink deployments maintain stable active connections with minimal new connection churn, process data within bandwidth limits (10-100 Gbps per AZ), and report zero or near-zero packet drops and RST resets. Endpoint connection states remain "Available" (not "Pending" or "Rejected"), and backing NLB targets stay "InService" with passing health checks. DNS resolution succeeds instantly for private DNS-enabled endpoints. - -## Paging intent -**Primary symptom**: Connection failures or timeouts when consumers attempt to reach endpoint services → indicates endpoint misconfiguration, security group blocks, or endpoint service unavailability. -**Secondary symptom**: Elevated packet drops (PacketDropCount) or RST resets (ResetPacketsReceived) → signals network congestion, bandwidth saturation, idle timeout issues, or application-layer problems. -**Tertiary symptom**: NLB target health failures → backend service degradation or misconfigured health checks impacting service availability. - -## Top 3 incident shapes and triage starting points - -**1. Connection establishment failures** (timeouts, rejections, "connection refused") -- **Symptoms**: NewConnections dropping to zero, consumers timing out, endpoint state "Rejected" or "Failed" -- **First dashboard section**: **Endpoint Connection Health** — check connection states, acceptance/rejection counts, security group denials -- **Common causes**: Endpoint connection request not accepted by provider, security group rules blocking traffic, DNS resolution failures, cross-AZ or cross-region connectivity misconfiguration - -**2. Active connection disruptions** (RST packets, packet drops, throughput degradation) -- **Symptoms**: ActiveConnections dropping unexpectedly, ResetPacketsReceived spiking, PacketDropCount elevated, BytesProcessed fluctuating -- **First dashboard section**: **Data Transfer & Network Performance** — examine packet drops, RST counts, bytes processed, bandwidth saturation -- **Common causes**: Idle timeout exceeded (350s default), MTU mismatches, bandwidth quota exceeded (10 Gbps baseline), NLB unhealthy targets, application-layer connection closures - -**3. Backend service unavailability** (target health failures, service provider issues) -- **Symptoms**: Connections established but requests failing, increased latency, intermittent errors -- **First dashboard section**: **Backend Target Health** (provider-side) — monitor NLB target health status, health check failures, target group metrics -- **Common causes**: NLB targets failing health checks, target auto-scaling lag, provider endpoint service disabled or deleted, listener port mismatches - ---- - -## Confirmed by sources -- PrivateLink metrics (ActiveConnections, NewConnections, BytesProcessed, PacketDropCount, ResetPacketsReceived) are published to CloudWatch in 1-minute intervals ([AWS docs](https://docs.aws.amazon.com/vpc/latest/privatelink/privatelink-cloudwatch-metrics.html)) -- Endpoint connection states: Available, Pending, Rejected, Expired, Failed ([AWS docs](https://docs.aws.amazon.com/vpc/latest/privatelink/configure-endpoint-service.html)) -- 350-second idle timeout for PrivateLink endpoints ([AWS re:Post](https://repost.aws/knowledge-center/vpc-troubleshoot-network-performance-privatelink)) -- 10 Gbps baseline with auto-scaling to 100 Gbps per AZ ([AWS re:Post](https://repost.aws/knowledge-center/vpc-troubleshoot-network-performance-privatelink)) - -## Best-practice inference -- Paging priority ordering (connection failures > network disruptions > backend health) follows standard incident triage patterns (user-facing impact first) -- Multi-AZ endpoint deployment for high availability aligns with AWS Well-Architected reliability pillar -- DNS resolution as a critical failure mode inferred from split-horizon DNS patterns and Route 53 integration complexity - - ---- - -# AWS PrivateLink — Key Concepts - -## Glossary (dashboard-relevant definitions) - -**Interface VPC Endpoint**: An elastic network interface (ENI) deployed in a consumer VPC subnet that serves as the entry point for private traffic to an endpoint service. Appears in all dashboard sections as the primary monitored entity. - -**VPC Endpoint Service**: A service hosted by a provider behind a Network Load Balancer, exposed for private consumption via PrivateLink. Service-side metrics (AWS/PrivateLinkServices namespace) are monitored in provider dashboards. - -**Endpoint Network Interface (ENI)**: The actual ENI resource backing an interface VPC endpoint, assigned a private IP from the consumer's subnet. ENI per-AZ determines bandwidth scaling and fault isolation boundaries. - -**Service Consumer**: The AWS account that creates interface VPC endpoints to access a PrivateLink-enabled service. Consumes endpoint metrics (AWS/PrivateLinkEndpoints namespace). - -**Service Provider**: The AWS account that hosts a VPC endpoint service and exposes it for consumption. Monitors service metrics (AWS/PrivateLinkServices namespace) and endpoint-specific contributor insights. - -**Network Load Balancer (NLB)**: The AWS load balancer type required for PrivateLink endpoint services. NLB health checks and target states directly impact endpoint service availability. - -**Connection Request**: A consumer-initiated request to connect their interface endpoint to a provider's endpoint service. Requires acceptance if the service is configured with manual approval. Tracked via connection state transitions. - -**Connection State**: The lifecycle state of an endpoint connection: Pending (awaiting acceptance), Available (active), Rejected (denied by provider), Expired (timed out), Failed (error). Drives the "Endpoint Connection Health" dashboard section. - -**Allowed Principal**: An IAM principal (user, role, account) granted permission by the service provider to create endpoint connections. Principal management affects connection acceptance rates. - -**Acceptance Required**: A service-side configuration that mandates manual approval of connection requests. When enabled, new connections remain in "Pending" state until accepted, affecting NewConnections and connection latency. - -**Cross-Region Connectivity**: PrivateLink feature allowing consumers in one AWS region to connect to endpoint services in another region. Introduced in 2025; adds inter-region data transfer costs and latency. - -**Private DNS**: A feature that associates a custom DNS name with an interface endpoint, enabling transparent service access using familiar DNS names instead of endpoint-specific DNS entries. Requires Route 53 private hosted zone management. - -**Split-Horizon DNS**: A DNS configuration pattern where the same domain name resolves to public IPs externally and private endpoint IPs internally. Commonly used with PrivateLink to maintain consistent service addressing. - -**Idle Timeout**: A 350-second inactivity window after which AWS terminates a TCP connection through a PrivateLink endpoint. Clients/targets must send TCP keepalive packets to prevent resets. Drives RST packet metrics. - -**Packet Drop**: Network packets discarded by the PrivateLink endpoint, typically due to bandwidth saturation (exceeding 10 Gbps baseline) or MTU mismatches. Tracked via PacketDropCount metric in "Data Transfer & Network Performance" section. - -**RST Reset (TCP RST)**: A TCP reset packet sent to terminate a connection, often triggered by idle timeout expiration, application errors, or misconfigurations. Tracked via ResetPacketsReceived metric. - -**Target Health**: The NLB health check status for backend instances behind a PrivateLink endpoint service. Unhealthy targets cause service degradation visible in consumer-side metrics (packet drops, connection failures). - -**Contributor Insights Rule**: A pre-built CloudWatch rule for endpoint services that identifies which consumer endpoints are the top contributors to traffic (BytesProcessed), connections (ActiveConnections, NewConnections), or resets (ResetPacketsReceived). Used for capacity planning and abuse detection. - -**Data Processing Charge**: A PrivateLink cost component charged per GB of data transferred through an interface endpoint. Tiered pricing: $0.01/GB (first PB), $0.006/GB (next 4 PB), $0.004/GB (5+ PB). Tracked via BytesProcessed metric in the "Cost & Usage" section. - -**Hourly Endpoint Charge**: A fixed $0.01/hour cost per interface endpoint or $0.02/hour per resource endpoint. Drives the "Endpoint Count & Cost" section. - -**Availability Zone (AZ) Mapping**: The alignment of consumer endpoint ENIs and provider NLB nodes across AZs. Multi-AZ deployment required for high availability; affects fault isolation and bandwidth scaling. - -**Gateway Load Balancer (GWLB) Endpoint**: A specialized PrivateLink endpoint type for traffic inspection services (firewalls, IDS/IPS). Uses the same metrics as interface endpoints but serves a different architectural role. - -**Endpoint Policy**: An IAM policy attached to an interface endpoint that restricts which IAM principals can use the endpoint and which actions they can perform. Misconfigured policies cause connection denials. - -**Route 53 Profile**: A shareable DNS configuration for PrivateLink endpoints, enabling centralized private hosted zone management across multiple VPCs and accounts via AWS RAM. Simplifies split-horizon DNS at scale. - ---- - -## Concept map (operational relationships) - -**Service Consumer** -> creates -> **Interface VPC Endpoint** (establishes private connectivity without internet gateway/NAT) - -**Interface VPC Endpoint** -> deploys -> **Endpoint Network Interface (ENI)** per AZ (determines bandwidth, fault isolation, and HA) - -**Interface VPC Endpoint** -> sends -> **Connection Request** to **VPC Endpoint Service** (initiates consumer-provider relationship) - -**Connection Request** -> transitions through -> **Connection State** (Pending -> Available/Rejected/Expired/Failed) (affects service reachability and troubleshooting) - -**VPC Endpoint Service** -> checks -> **Allowed Principal** list (controls who can connect; missing principal causes rejections) - -**VPC Endpoint Service** -> requires -> **Acceptance Required** flag (manual vs auto approval; impacts NewConnections latency) - -**VPC Endpoint Service** -> backed by -> **Network Load Balancer (NLB)** (all PrivateLink services require NLB; no ALB/CLB support) - -**Network Load Balancer** -> performs -> **Target Health** checks (backend availability; failures cascade to endpoint-side metrics) - -**NLB Target Health** -> failing -> causes -> **Packet Drop** and **RST Reset** at endpoint (backend issues visible in consumer CloudWatch) - -**Interface VPC Endpoint** -> enforces -> **Idle Timeout** (350s default; no traffic = TCP RST sent) - -**Idle Timeout** -> exceeded -> triggers -> **RST Reset** (ResetPacketsReceived spikes; long-lived connections need keepalive) - -**Interface VPC Endpoint** -> has -> **Bandwidth Quota** (10 Gbps baseline, auto-scales to 100 Gbps per AZ) (quota exceeded = PacketDropCount rises) - -**BytesProcessed** -> exceeds -> **Bandwidth Quota** per AZ -> triggers -> **Packet Drop** (visible in Data Transfer & Network Performance section) - -**Interface VPC Endpoint** -> optionally enables -> **Private DNS** (resolves service DNS to private IPs; simplifies client configuration) - -**Private DNS** -> requires -> **Route 53 Private Hosted Zone** (DNS resolution within VPC; misconfiguration = connection failures) - -**Route 53 Private Hosted Zone** -> supports -> **Split-Horizon DNS** (same domain, public vs private resolution; used for seamless migration) - -**Interface VPC Endpoint** -> can enable -> **Cross-Region Connectivity** (consumer in us-east-1, service in eu-west-1; adds inter-region data transfer costs) - -**Cross-Region Connectivity** -> incurs -> **$0.02/GB inter-region charge** on top of **Data Processing Charge** (cost visibility critical in Cost & Usage section) - -**VPC Endpoint Service** -> publishes -> **Contributor Insights Rule** (identifies top consumer endpoints by BytesProcessed/ActiveConnections/RST; capacity planning) - -**Interface VPC Endpoint** -> publishes -> **CloudWatch Metrics** (AWS/PrivateLinkEndpoints namespace; 1-minute granularity) - -**VPC Endpoint Service** -> publishes -> **CloudWatch Metrics** (AWS/PrivateLinkServices namespace; provider-side visibility) - -**Endpoint Policy** -> attached to -> **Interface VPC Endpoint** (IAM-based access control; misconfigured = connection denials) - -**Availability Zone Mapping** -> mismatched -> causes -> **cross-AZ data transfer costs** (consumer and provider AZs don't align; cost and latency impact) - -**Gateway Load Balancer Endpoint** -> shares -> **same metrics** as **Interface VPC Endpoint** (different use case but identical monitoring) - -**PacketDropCount** -> correlates with -> **BytesProcessed** and **Bandwidth Quota** (capacity saturation indicator) - -**ResetPacketsReceived** -> correlates with -> **Idle Timeout** and **NLB Target Health** (connection quality indicator) - -**NewConnections** -> flat or zero + **Connection State** = Rejected -> indicates -> **Allowed Principal** or **Acceptance Required** issue (authorization problem) - -**ActiveConnections** -> steady + **PacketDropCount** = 0 + **ResetPacketsReceived** = 0 -> indicates -> **healthy PrivateLink connectivity** (golden path) - ---- - -## Entities and dimensions (cardinality, safety, dashboard impact) - -**EndpointId** (e.g., vpce-0a1b2c3d4e5f6g7h8) -- **Why useful**: Uniquely identifies each interface VPC endpoint; essential for isolating issues to specific endpoints vs systemic problems -- **Cardinality risk**: Medium (1-1000s per account depending on architecture; centralized hub VPC patterns concentrate endpoints) -- **Safe top-N**: 10 for overview KPIs, 25 for deep dive breakdowns -- **Dashboard impact**: Primary group-by dimension for all consumer-side sections (Endpoint Connection Health, Data Transfer & Network Performance, Cost & Usage) - -**ServiceName** (e.g., com.amazonaws.vpce.us-east-1.vpce-svc-abc123def456) -- **Why useful**: Identifies the target endpoint service; critical for multi-service environments to isolate failures to specific backends -- **Cardinality risk**: Low-Medium (10-100s typically; most orgs consume <50 PrivateLink services) -- **Safe top-N**: 10 for overview, 20 for deep dive -- **Dashboard impact**: Secondary group-by for consumer dashboards; primary group-by for provider dashboards (Service Performance section) - -**ServiceId** (e.g., vpce-svc-abc123def456) -- **Why useful**: Provider-side identifier for endpoint services; used in service provider dashboards -- **Cardinality risk**: Low (1-50 per provider account; most providers expose <10 services) -- **Safe top-N**: 10 -- **Dashboard impact**: Primary group-by for provider-side sections (Service Consumer Connections, Backend Target Health) - -**AvailabilityZone** (e.g., us-east-1a, us-east-1b) -- **Why useful**: AZ-level isolation for bandwidth quotas (10-100 Gbps per AZ), fault isolation, and cross-AZ cost analysis -- **Cardinality risk**: Very low (2-6 per region; typically 3 AZs used for HA) -- **Safe top-N**: 6 (include all) -- **Dashboard impact**: Critical group-by for Data Transfer & Network Performance (bandwidth saturation is per-AZ), Cost & Usage (cross-AZ data transfer costs) -- **DO NOT group-by**: When aggregating account-wide totals or connection states (AZ dimension not always present in metadata) - -**VpcId** (e.g., vpc-0a1b2c3d) -- **Why useful**: Isolates issues to specific VPCs in multi-VPC architectures; hub-spoke patterns benefit from VPC-level breakdowns -- **Cardinality risk**: Medium-High (10-100s in hub-spoke; 1000s in large orgs) -- **Safe top-N**: 10 -- **Dashboard impact**: Useful for organizational rollups and multi-tenant providers; less relevant for single-VPC consumers -- **DO NOT group-by**: For per-endpoint deep dives (EndpointId already implies VpcId) - -**SubnetId** (e.g., subnet-0a1b2c3d) -- **Why useful**: ENI placement within VPC; useful for diagnosing subnet-specific network ACL or route table issues -- **Cardinality risk**: High (10-100 per VPC; multi-AZ deployments multiply cardinality) -- **Safe top-N**: 5 (only when troubleshooting specific networking issues) -- **Dashboard impact**: Rarely used in standard dashboards; include in deep dive troubleshooting views only -- **DO NOT group-by**: For performance KPIs (explodes cardinality without adding insight; use AvailabilityZone instead) - -**Region** (e.g., us-east-1, eu-west-1) -- **Why useful**: Cross-region PrivateLink deployments incur extra data transfer costs ($0.02/GB); region-level breakdowns essential for cost attribution -- **Cardinality risk**: Very low (1-20 regions; typically <5 active) -- **Safe top-N**: 10 (include all active regions) -- **Dashboard impact**: Critical for cross-region connectivity scenarios (separate section in Cost & Usage); not needed for single-region deployments - -**AccountId** (AWS account ID) -- **Why useful**: Multi-account environments (consumer accessing provider across accounts) need account-level visibility for chargeback and security auditing -- **Cardinality risk**: Low-Medium (1-100 accounts depending on org size) -- **Safe top-N**: 10 -- **Dashboard impact**: Provider-side dashboards (Service Consumer Connections section); less relevant for consumer-only dashboards -- **DO NOT group-by**: When account context is already implicit (single-account consumer deployments) - -**ConnectionState** (Available, Pending, Rejected, Failed, Expired) -- **Why useful**: Connection lifecycle tracking; Rejected/Failed states indicate authorization or configuration issues -- **Cardinality risk**: Very low (5 possible states) -- **Safe top-N**: 5 (include all) -- **Dashboard impact**: Essential dimension for Endpoint Connection Health section; use bar/pie charts to show state distribution - -**PrincipalArn** (IAM principal ARN) -- **Why useful**: Service provider dashboards use this to track which IAM principals (accounts, roles) are connecting; useful for access auditing and billing breakdowns -- **Cardinality risk**: Medium-High (10-1000s depending on multi-tenant scenarios) -- **Safe top-N**: 10 -- **Dashboard impact**: Provider-side Service Consumer Connections section; exclude from consumer dashboards (not visible to consumers) -- **DO NOT group-by**: For consumer-side dashboards (dimension not present in consumer metrics) - -**EndpointType** (Interface, GatewayLoadBalancer) -- **Why useful**: Differentiates interface VPC endpoints from Gateway Load Balancer endpoints; architectures mixing both need separate views -- **Cardinality risk**: Very low (2 types) -- **Safe top-N**: 2 -- **Dashboard impact**: Use as global filter or section gating (most dashboards focus on Interface endpoints; GWLB endpoints are specialized) - -**TargetGroupArn** (NLB target group ARN, provider-side only) -- **Why useful**: Provider-side metric for tracking backend target health per target group; multiple target groups per endpoint service need isolation -- **Cardinality risk**: Low-Medium (1-50 per service) -- **Safe top-N**: 10 -- **Dashboard impact**: Backend Target Health section (provider dashboards only); not visible to consumers - -**ConsumerEndpointId** (provider-side dimension tracking which consumer endpoints are connecting) -- **Why useful**: Contributor Insights rules use this to rank top consumer endpoints by traffic, connections, or resets; capacity planning and abuse detection -- **Cardinality risk**: Medium-High (1-1000s of consumer endpoints per popular service) -- **Safe top-N**: 10 for Contributor Insights top-N lists -- **Dashboard impact**: Service Consumer Connections section (provider dashboards); shows which consumers drive the most load -- **DO NOT group-by**: For consumer dashboards (not available in consumer-side metrics) - ---- - -## Tsuga field mapping (vendor dimension -> context.* key) - -| Vendor/CloudWatch Dimension | Recommended Tsuga Field | Must-Exist vs Optional | Notes | -|---|---|---|---| -| `EndpointId` | `context.endpointid` | Must-exist | Primary consumer-side entity; AWS uses lowercase joined pattern (see ECS/SNS/Transcribe examples) | -| `ServiceName` | `context.servicename` | Must-exist | Target endpoint service; critical for multi-service environments | -| `ServiceId` | `context.serviceid` | Optional | Provider-side entity; may not be present in consumer metrics | -| `Availability Zone` | `context.availabilityzone` OR `context.az` | Must-exist | AWS often abbreviates to `az`; Stage 2 discovery will confirm which | -| `VPC Id` | `context.vpcid` | Optional | Useful for multi-VPC architectures; may not be included in all metrics | -| `Region` | `context.region` | Optional | Only needed for cross-region PrivateLink deployments; Stage 2 will confirm presence | -| `Account Id` | `context.accountid` | Optional | Multi-account scenarios; provider-side visibility | -| `Connection State` | `context.connectionstate` | Optional | May be present in connection-specific metrics; Stage 2 will validate | -| `Endpoint Type` | `context.endpointtype` | Optional | Differentiates Interface vs GatewayLoadBalancer endpoints | -| `Principal Arn` | `context.principalarn` | Optional | Provider-side only; consumer IAM principal identifier | -| `Consumer Endpoint Id` | `context.consumerendpointid` | Optional | Provider-side Contributor Insights; tracks individual consumer endpoints | -| `Target Group Arn` | `context.targetgrouparn` | Optional | Provider-side NLB target group tracking | - -**Fallback strategy for Stage 2 discovery:** -- If `context.endpointid` is not found, check `context.endpoint_id` (underscore-separated) -- If `context.servicename` is not found, check `context.service_name` -- AWS PrivateLink metrics may use dot-separated namespaces (e.g., `aws.privatelink.endpoint.id`) — Stage 2 will validate actual naming - -**Context fields from .env (assumed present):** -- `context.env` — environment tag (prod, staging) -- `context.team` — owning team tag - ---- - -## Confirmed by sources -- Interface VPC Endpoint and VPC Endpoint Service are the core PrivateLink entities ([AWS PrivateLink concepts](https://docs.aws.amazon.com/vpc/latest/privatelink/concepts.html)) -- Connection states: Available, Pending, Rejected, Expired, Failed ([AWS PrivateLink service config](https://docs.aws.amazon.com/vpc/latest/privatelink/configure-endpoint-service.html)) -- 350-second idle timeout for TCP connections ([AWS re:Post network performance](https://repost.aws/knowledge-center/vpc-troubleshoot-network-performance-privatelink)) -- 10 Gbps baseline bandwidth, auto-scales to 100 Gbps per AZ ([same source](https://repost.aws/knowledge-center/vpc-troubleshoot-network-performance-privatelink)) -- CloudWatch metrics published in AWS/PrivateLinkEndpoints and AWS/PrivateLinkServices namespaces ([CloudWatch metrics docs](https://docs.aws.amazon.com/vpc/latest/privatelink/privatelink-cloudwatch-metrics.html)) -- Pricing: $0.01/hour per interface endpoint, $0.01-$0.004/GB data processing (tiered), $0.02/GB cross-region ([AWS PrivateLink pricing](https://aws.amazon.com/privatelink/pricing/)) -- Contributor Insights integration for ranking top consumer endpoints ([AWS blog](https://aws.amazon.com/blogs/networking-and-content-delivery/gain-usage-insights-with-amazon-cloudwatch-metrics-for-aws-privatelink/)) - -## Best-practice inference -- Tsuga field naming inferred from AWS ECS/SNS/Transcribe patterns (all-lowercase joined names like `context.servicename`) -- Multi-AZ deployment recommendation follows AWS Well-Architected reliability best practices -- Top-N limits (10 for overview, 25 for deep dive) follow Tsuga cardinality safety patterns from previous integrations -- "DO NOT group-by" guidance inferred from high-cardinality explosion risks (SubnetId, PrincipalArn in multi-tenant scenarios) -- Cross-AZ cost analysis importance inferred from AWS data transfer pricing model (same-AZ = free, cross-AZ = $0.01/GB) - - ---- - -# AWS PrivateLink — Golden Signals - -## Traffic (volume of work) - -**What it means for PrivateLink** -Traffic represents the volume of private network connectivity: how many new connections consumers are establishing, how many concurrent connections are active, and how much data is flowing through endpoint network interfaces. For PrivateLink, traffic is measured at three levels: connection establishment rate (NewConnections), sustained connection load (ActiveConnections), and data volume (BytesProcessed). Unlike public internet traffic, PrivateLink traffic remains within AWS's private network backbone. - -**Typical causes when traffic degrades** -- **NewConnections drops to zero**: Connection requests rejected by service provider (manual acceptance required but not granted), allowed principal list missing consumer account, endpoint policy denying access, security group blocking traffic -- **ActiveConnections drops unexpectedly**: Idle timeout exceeded (350s without keepalive), application closing connections due to backend errors, NLB targets becoming unhealthy, cross-region PrivateLink path failures -- **BytesProcessed fluctuates or drops**: Client application throttling, backend service capacity limits, NLB target auto-scaling lag, cross-AZ or cross-region bandwidth constraints - -**Best telemetry sources representing traffic** -- **Primary**: CloudWatch metrics in AWS/PrivateLinkEndpoints namespace: `NewConnections` (rate), `ActiveConnections` (gauge), `BytesProcessed` (counter) -- **Secondary (provider-side)**: AWS/PrivateLinkServices namespace: same metrics from service provider perspective -- **Tertiary**: VPC Flow Logs for endpoint ENIs (granular packet-level visibility when CloudWatch metrics insufficient) -- **Contributor Insights**: Ranks consumer endpoints by BytesProcessed, ActiveConnections, NewConnections for capacity planning - -**What people page on (symptom-based)** -- NewConnections flatlined while application attempting connections → **service unreachable; immediate escalation** -- ActiveConnections dropping >50% within 5 minutes without planned changes → **mass disconnection event; investigate backend health and idle timeout issues** -- BytesProcessed sustained near bandwidth quota (10 Gbps baseline) with PacketDropCount rising → **capacity saturation; scale to more AZs or investigate traffic spike** - -**Section questions for dashboards** -1. **Are consumers establishing new connections successfully?** (NewConnections rate, connection state distribution, rejection/failure counts) -2. **How many concurrent connections are active across endpoints?** (ActiveConnections per endpoint/service, trend over time, top endpoints by connection count) -3. **What is the data transfer volume and bandwidth utilization?** (BytesProcessed rate, per-AZ bandwidth quotas, cross-region vs same-region breakdowns) - ---- - -## Errors (rate of failing requests) - -**What it means for PrivateLink** -Errors in PrivateLink manifest as connection establishment failures (rejected/failed/expired connection requests), TCP reset packets indicating abrupt connection terminations, and packet drops caused by network congestion or configuration issues. Unlike application-layer errors (HTTP 4xx/5xx), PrivateLink errors occur at the network layer (Layer 3/4) before traffic reaches the application. A single PrivateLink error can cause cascading application failures. - -**Typical causes when error rates increase** -- **Connection rejections (state = Rejected)**: Service provider explicitly rejected connection request, acceptance required but not granted within timeout, endpoint policy denying principal -- **Connection failures (state = Failed)**: Subnet route table missing local route, security group blocking port, cross-region connectivity not enabled for service, DNS resolution failures for private DNS -- **RST resets (ResetPacketsReceived spiking)**: Idle timeout exceeded (350s) without TCP keepalive, NLB target health checks failing and terminating active connections, MTU mismatch causing fragmentation failures, application-layer connection closures propagating as RST -- **Packet drops (PacketDropCount elevated)**: Bandwidth quota exceeded (10 Gbps per AZ), ENI network interface limits reached, MTU misconfiguration, consumer or provider network congestion - -**Best telemetry sources representing errors** -- **Primary**: CloudWatch metrics: `PacketDropCount` (sum), `ResetPacketsReceived` (sum), connection state metrics (Rejected/Failed counts if available) -- **Secondary**: VPC Flow Logs filtered for REJECT action, ENI statistics showing drops -- **Tertiary**: NLB target health status (provider-side; unhealthy targets cause downstream resets) -- **Application logs**: Connection timeouts, "connection refused", DNS resolution failures (supplement CloudWatch metrics) - -**What people page on (symptom-based)** -- ResetPacketsReceived sustained >100/min per endpoint → **active connection instability; investigate idle timeout or backend health** -- PacketDropCount >1% of BytesProcessed → **network congestion or bandwidth saturation; scale endpoints or reduce traffic** -- Connection state = Rejected or Failed for >5 minutes → **authorization or configuration issue; immediate remediation needed** -- Sudden spike in all error metrics simultaneously → **systemic PrivateLink or NLB outage; escalate to AWS support** - -**Section questions for dashboards** -1. **Are endpoint connections being rejected or failing to establish?** (connection state distribution, rejection counts, failure reasons) -2. **Are active connections experiencing TCP resets?** (ResetPacketsReceived rate, correlation with idle timeout, NLB target health) -3. **Is the network dropping packets due to congestion or misconfiguration?** (PacketDropCount rate, bandwidth quota utilization, MTU issues) - ---- - -## Latency (time to complete work) - -**What it means for PrivateLink** -PrivateLink itself does not publish end-to-end latency metrics (application request/response time). However, connection establishment latency (time from endpoint creation to Available state), DNS resolution latency (private DNS lookups), and cross-region latency (data plane delay when crossing region boundaries) all impact perceived performance. Most latency visibility comes from application-layer instrumentation or CloudWatch Network Synthetic Monitors. - -**Typical causes when latency increases** -- **Connection establishment delay**: Manual acceptance required (connection sits in Pending state until provider acts), cross-region connectivity adding inter-region latency (~50-150ms depending on regions), slow NLB target health check responses -- **DNS resolution delay**: Private hosted zone misconfiguration, Route 53 Resolver endpoint saturation, split-horizon DNS fallback to public DNS causing extra hops -- **Data plane latency**: Cross-region PrivateLink adding inter-region transit time, cross-AZ traffic (intra-region but cross-AZ ~1-2ms overhead), MTU fragmentation causing retransmissions -- **Backend service latency**: NLB target processing time (not a PrivateLink issue but visible to consumers as increased end-to-end latency) - -**Best telemetry sources representing latency** -- **Primary (application-layer)**: Application request tracing (X-Ray, OpenTelemetry), application logs with request timestamps -- **Secondary**: CloudWatch Network Synthetic Monitor (proactive synthetic testing of PrivateLink paths, cross-region latency measurements) -- **Tertiary**: VPC Flow Logs with flow duration analysis (rough proxy for connection lifetime) -- **Indirect**: NewConnections trend (slow connection establishment = latency issue) - -**What people page on (symptom-based)** -- Application request latency (measured in application) increased by >2x correlated with PrivateLink metrics (PacketDropCount, ResetPacketsReceived) → **PrivateLink network issue impacting app performance** -- DNS resolution timeouts or >1s latency for private DNS-enabled endpoints → **Route 53 or split-horizon DNS misconfiguration** -- Cross-region PrivateLink showing sustained >200ms latency (expected is 50-150ms) → **inter-region path degradation; validate with Network Synthetic Monitor** - -**Section questions for dashboards** -1. **How long does it take for endpoint connections to become Available?** (time from Pending to Available state, manual acceptance delays) -2. **Is DNS resolution for private DNS-enabled endpoints performing within SLA?** (DNS query latency from Route 53 metrics, failure rates) -3. **What is the end-to-end latency for cross-region PrivateLink connections?** (Network Synthetic Monitor results, application-layer latency breakdowns) - ---- - -## Saturation (resource capacity limits) - -**What it means for PrivateLink** -Saturation occurs when PrivateLink endpoints approach or exceed capacity limits: bandwidth quotas (10 Gbps baseline per AZ, auto-scales to 100 Gbps), connection count limits (ENI connection tracking limits), or provider-side NLB target capacity. Unlike traditional networking where saturation causes complete failures, PrivateLink scales elastically in many dimensions but has hard limits in others. Saturation manifests as PacketDropCount increases, NewConnections throttling, or backend target exhaustion. - -**Typical causes when saturation increases** -- **Bandwidth saturation per AZ**: Single-AZ endpoint deployment handling >10 Gbps, burst traffic exceeding auto-scaling rate (10→100 Gbps takes seconds), sustained multi-region traffic concentrating on one AZ -- **Connection count saturation**: ENI connection tracking limits (~55k concurrent connections per ENI), single endpoint handling more connections than intended (should distribute across AZs) -- **Provider-side target saturation**: NLB targets at CPU/memory/network capacity, target count insufficient for connection volume, target auto-scaling lag during traffic spikes -- **Cross-region bandwidth saturation**: Inter-region PrivateLink bandwidth limits (not well-documented; assumed to scale similarly to intra-region) - -**Best telemetry sources representing saturation** -- **Primary**: CloudWatch metrics: `PacketDropCount` (direct saturation indicator), `BytesProcessed` trend approaching bandwidth quotas, `ActiveConnections` approaching ENI limits -- **Secondary**: NLB target metrics (provider-side): `TargetConnectionErrorCount`, `UnHealthyHostCount`, `ProcessedBytes` per target -- **Tertiary**: VPC Flow Logs showing ENI-level connection counts, AWS Service Quotas API for PrivateLink limits -- **Contributor Insights**: Identifies which consumer endpoints or services are consuming the most bandwidth/connections (hotspot detection) - -**What people page on (symptom-based)** -- PacketDropCount sustained >0 correlated with BytesProcessed approaching 10 Gbps per AZ → **bandwidth saturation; add more AZs or request quota increase** -- ActiveConnections per endpoint approaching ~50k with NewConnections dropping → **ENI connection tracking limit; distribute load across more endpoints** -- Provider-side NLB UnHealthyHostCount increasing with consumer-side PacketDropCount/ResetPacketsReceived rising → **backend target capacity exhaustion** -- BytesProcessed growing 10x week-over-week without endpoint scaling → **proactive capacity planning needed before saturation hits** - -**Section questions for dashboards** -1. **Are endpoints approaching bandwidth capacity limits?** (BytesProcessed rate per AZ vs 10 Gbps baseline, PacketDropCount as saturation indicator) -2. **Is connection count approaching ENI connection tracking limits?** (ActiveConnections per endpoint, trend toward ~50k threshold) -3. **Are backend NLB targets saturated?** (provider-side: target count, connection error rate, unhealthy host count) - ---- - -## Confirmed by sources -- CloudWatch metrics: NewConnections, ActiveConnections, BytesProcessed, PacketDropCount, ResetPacketsReceived published in 1-minute intervals ([AWS CloudWatch metrics docs](https://docs.aws.amazon.com/vpc/latest/privatelink/privatelink-cloudwatch-metrics.html)) -- Bandwidth: 10 Gbps baseline per AZ, auto-scales to 100 Gbps ([AWS re:Post](https://repost.aws/knowledge-center/vpc-troubleshoot-network-performance-privatelink)) -- Idle timeout: 350 seconds for TCP connections through PrivateLink endpoints ([same source](https://repost.aws/knowledge-center/vpc-troubleshoot-network-performance-privatelink)) -- Connection states: Available, Pending, Rejected, Expired, Failed ([AWS service config docs](https://docs.aws.amazon.com/vpc/latest/privatelink/configure-endpoint-service.html)) -- ENI connection tracking limits: ~55k concurrent connections per ENI ([inferred from general ENI limits; not PrivateLink-specific documentation](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html)) -- Contributor Insights for PrivateLink ranking top endpoints by BytesProcessed, ActiveConnections, NewConnections, ResetPacketsReceived ([AWS blog](https://aws.amazon.com/blogs/networking-and-content-delivery/gain-usage-insights-with-amazon-cloudwatch-metrics-for-aws-privatelink/)) - -## Best-practice inference -- Latency section synthesized from indirect sources (no native PrivateLink latency metrics exist; CloudWatch Network Synthetic Monitor is a separate service) -- ENI connection tracking limit (~55k) inferred from general AWS ENI documentation, not PrivateLink-specific (Stage 2 discovery should note this as "inferred") -- Cross-region bandwidth saturation behavior inferred from general AWS networking patterns (no explicit PrivateLink cross-region bandwidth limits documented) -- Paging thresholds (>100 RST/min, >1% packet drop rate, >50% ActiveConnections drop) inferred from operational experience and AWS re:Post troubleshooting guidance -- Golden signal section question mapping follows standard observability practices (traffic → volume questions, errors → failure mode questions, latency → timing questions, saturation → capacity questions) - - ---- - -# AWS PrivateLink — Section Notes & Playbooks - ---- - -## Part 1: Overview Mission Note - -```markdown -**AWS PrivateLink — Private Connectivity Monitoring** - -Private network connectivity between VPCs, AWS services, and on-premises networks without internet exposure. Monitors interface VPC endpoints (consumers) and endpoint services (providers) using CloudWatch metrics. - -**Monitoring scope**: All VPC endpoints and endpoint services in this account reporting CloudWatch metrics (AWS/PrivateLinkEndpoints and AWS/PrivateLinkServices namespaces). Does not cover gateway endpoints, VPC peering, or Transit Gateway. - -**Learn more:** -- [CloudWatch metrics for PrivateLink](https://docs.aws.amazon.com/vpc/latest/privatelink/privatelink-cloudwatch-metrics.html) -- [PrivateLink concepts](https://docs.aws.amazon.com/vpc/latest/privatelink/concepts.html) -- [Deep Dive Dashboard](#) (link to deep dive dashboard once created) -``` - ---- - -## Part 2: Section Explanation Notes - -### Endpoint Connection Health - -#### So what? - -Healthy PrivateLink deployments maintain **stable Active Connections with minimal New Connection churn**. Connection-pooled applications (databases, message queues) typically show **NewConnections < 1 conn/s** once pools are established, with **ActiveConnections steady at 10-1000**. Short-lived connection applications (HTTP APIs, serverless) show **higher NewConnections (10-100 conn/s)** with **lower ActiveConnections (1-100)**. - -**When to worry**: NewConnections drops to **zero while applications attempt connections** = authorization failure (connection state Rejected), security group blocking traffic, or DNS resolution failure. ActiveConnections drops **>50% within 5 minutes** without planned changes = mass disconnection event (idle timeout exceeded, backend health failures, cross-region path degradation). - -**Watch out**: Connection state = **Pending for >1 hour** likely means the service provider requires manual acceptance (`acceptance_required = true`) but hasn't approved yet—this is operational delay, not failure. NewConnections = 0 with high stable ActiveConnections is **healthy** for connection-pooled apps, not a problem. - -#### Now what? - -**If NewConnections drops to zero unexpectedly**: -1. Check **Active Connections (trend)** — if also dropping to zero, it's full connectivity loss; if stable, connection pools are saturated (no new connections needed) -2. Check application logs for "connection refused" or DNS resolution errors -3. Verify endpoint ENI security group allows traffic on the listener port -4. Check if endpoint connection state = Rejected/Failed (provider authorization issue) - -**If ActiveConnections drops suddenly**: -1. Check **RST Resets by Endpoint** — if spiking, idle timeout or backend health failure -2. Check **Backend Health Ratio** (provider-side) — if <100%, unhealthy NLB targets terminating connections -3. Correlate with application deployment windows (graceful shutdown causes FIN, forced termination causes RST) -4. Check VPC Flow Logs for REJECT actions (security group denials) - ---- - -### Data Transfer & Network Performance - -#### So what? - -Healthy PrivateLink data transfer shows **zero PacketDropCount and minimal ResetPacketsReceived (<1 RST/s)**. **BytesProcessed** grows with application traffic; baseline varies widely (1-100 MB/s typical, 1+ GB/s for bulk data). - -**Bandwidth per AZ**: PrivateLink starts at **10 Gbps (1250 MB/s) per AZ** and auto-scales to 100 Gbps. **Utilization >80%** = approaching saturation; PacketDropCount will rise. **Utilization <50%** = healthy headroom. **Always group by AvailabilityZone** for bandwidth analysis; per-AZ quotas mean aggregating across AZs hides hotspots. - -**When to worry**: **Any sustained PacketDropCount >0** is actionable (MTU mismatch, bandwidth saturation, network congestion). **ResetPacketsReceived >100 RST/s** per endpoint = active connection instability (idle timeout without TCP keepalive, backend failures, application errors). **Bandwidth Utilization per AZ >80%** sustained = add more AZs or reduce traffic. - -**Watch out**: **RST spikes during deployments** (10-100 RST/s for 1-2 minutes) are **normal** (forced container/instance termination sends RST). Only sustained RST elevation outside deploy windows is actionable. **PacketDropCount 1-2 drops/hour** during auto-scaling (PrivateLink scaling 10→100 Gbps) is **transient**, not a problem. - -#### Now what? - -**If PacketDropCount sustained >0**: -1. Check **Bandwidth Utilization per AZ** — if >80%, bandwidth saturation; add more AZs or scale endpoints -2. Check **Packet Drops by Availability Zone** — identify which AZ is saturated (single-AZ hotspot common) -3. Check **Data Throughput by Endpoint** — identify which endpoints drive the most traffic -4. Verify MTU settings (PrivateLink caps at 1500 bytes; jumbo frames not supported) - -**If ResetPacketsReceived sustained >10 RST/s**: -1. Check **Backend Health Ratio** (provider-side) — if <100%, NLB target failures causing RSTs -2. Check **Average Bytes per Connection** — if spiking, long-running transfers hitting idle timeout (350s) -3. Verify TCP keepalive enabled in application (required for connections idle >350s) -4. Correlate with **NLB Target Connection Errors** (provider-side) — backend capacity exhaustion - -**If Bandwidth Utilization per AZ approaching 80%**: -1. Add endpoint network interfaces in more Availability Zones (distribute load) -2. Check **Data Throughput by Endpoint (top 10)** — identify traffic hotspots -3. Consider endpoint count scaling (more endpoints = more aggregate bandwidth) -4. Check if cross-region traffic can be localized (inter-region adds latency and cost) - ---- - -### Backend Target Health (provider-side) - -#### So what? - -**Provider-side only**: Service consumers cannot see NLB metrics. This section monitors the health of backend targets behind the Network Load Balancer backing your PrivateLink endpoint service. - -**Healthy baseline**: **100% Backend Health Ratio** (all targets healthy), **zero NLB Target Connection Errors**. Temporary **UnHealthyHostCount 1-2 during target launches** (EC2 starting, containers initializing) is **normal**; health checks fail until application fully starts (30-60s grace period). - -**When to worry**: **Backend Health Ratio <50%** sustained >5 minutes = critical capacity impact; consumers will see increased PacketDropCount and ResetPacketsReceived. **HealthyHostCount = 0** = full service outage for all consumers. **NLB Connection Error Rate >1 error/s** = backend capacity exhaustion (targets refusing connections, port mismatches, target crashes). - -**Watch out**: **HealthyHostCount = 0 with UnHealthyHostCount = 0** means **no registered targets** (configuration issue, not health failure). Check if target auto-scaling terminated all instances or targets were manually deregistered. **UnHealthyHostCount spikes during deployments** are **expected** (old targets drain and fail health checks before new targets pass). - -#### Now what? - -**If Backend Health Ratio drops <80%**: -1. Check **Unhealthy Targets by Endpoint Service** — identify which service is degraded -2. Check application logs on unhealthy targets (port not listening, health check path failing) -3. Verify NLB health check configuration (correct port, path, interval, timeout) -4. Check target auto-scaling (insufficient capacity, launch failures) - -**If NLB Connection Error Rate >1 error/s**: -1. Check **Healthy Targets (trend)** — if dropping, backend service crashes or capacity exhaustion -2. Check target CPU/memory/network utilization (backend resource saturation) -3. Verify listener port matches target application port (port mismatch = connection errors) -4. Correlate with consumer-side **RST Resets by Endpoint** — backend errors propagate as RSTs to consumers - -**If HealthyHostCount = 0**: -1. Check if target group has any registered targets (DescribeTargetHealth API) -2. Check target auto-scaling group (desired capacity, launch failures, AZ mismatches) -3. Check target security group (health check traffic blocked) -4. Check application logs on targets (crash loops, failed to bind to port) - ---- - -### Cost & Usage - -#### So what? - -PrivateLink costs have **three components**: **hourly endpoint charge** ($0.01/hour per interface endpoint, not metered in CloudWatch), **data processing charge** (tiered: $0.01/GB tier 1, $0.006/GB tier 2, $0.004/GB tier 3), and **cross-region data transfer** ($0.02/GB additional for inter-region traffic). - -**Typical baselines**: Intra-region deployments processing **1 TB/month = ~$10/month** data processing (tier 1) + **$7.20/month** endpoint hourly charge (1 endpoint, 720 hours). Cross-region adds **$20/month** per TB transferred. High-throughput scenarios (10+ TB/month) benefit from tiered pricing (drop to $0.006 then $0.004/GB). - -**When to worry**: **Data cost increasing >50% week-over-week** without known traffic growth = unexpected traffic spike or runaway batch job. **Cross-region cost >intra-region cost** = architecture inefficiency (consider replicating services to consumer regions). - -**Watch out**: **Cost doubling during backup windows or batch jobs is expected** (PrivateLink charges per GB; overnight ETL or database replication legitimately increases costs). **Baseline during business hours**, not peak windows. **Dashboard shows tier 1 rate ($0.01/GB) only**; actual billing uses tiered rates (tier 2/3 cheaper at scale). - -#### Now what? - -**If data processing cost increasing unexpectedly**: -1. Check **Data Processing Cost by Endpoint (top 10)** — identify which endpoints drive cost growth -2. Check **Data Throughput by Endpoint** — correlate cost with traffic volume -3. Review application changes (new batch jobs, increased replication, accidental data loops) -4. Check **Total Data Processed (GB/hour)** trend — identify when growth started - -**If cross-region cost is high**: -1. Check **Data Processed by Region** — identify which regions consume the most cross-region bandwidth -2. Evaluate architecture: can services be replicated to consumer regions (eliminate cross-region transfer)? -3. Check if cross-region traffic is read-heavy (consider regional read replicas) -4. Validate cross-region is intentional (not misconfigured endpoint pointing to wrong region) - -**For capacity planning**: -1. Hourly endpoint charge = **$0.01/hour * endpoint count * 720 hours/month** -2. Data processing cost = **BytesProcessed (GB/month) * tiered rate** (tier 1: $0.01, tier 2: $0.006, tier 3: $0.004) -3. Cross-region cost = **BytesProcessed cross-region (GB/month) * $0.02** -4. Use **Data Throughput (MB/s)** * 2.628 million = GB/month (30 days) for monthly projection - ---- - -## Part 3: Cause-Effect Triage Chains - -### Connection Health Chains - -1. **If Active Connections (current) drops to zero** → check **Active Connections (trend)** for gradual vs sudden drop → likely causes: mass idle timeout expiration (no TCP keepalive), NLB target health failure (provider-side), cross-region path failure → next action: check RST Resets (rate) for spike; verify backend health ratio (provider-side); check application logs for connection errors. (Confirmed: idle timeout and NLB health documented) - -2. **If New Connections (rate) drops to zero while ActiveConnections stable** → this is **healthy** (connection pools established, no new connections needed) → likely causes: connection-pooled application (database, message queue) reached steady state → next action: no action needed unless applications report connection failures. (Inference: connection pool behavior) - -3. **If New Connections (rate) drops to zero AND ActiveConnections drops to zero** → check application logs for "connection refused" or DNS errors → likely causes: endpoint connection state = Rejected/Failed (authorization issue), security group blocking traffic, DNS resolution failure → next action: verify endpoint state in VPC console; check security group rules; test DNS resolution (nslookup). (Confirmed: connection states documented) - -4. **If Connection Churn Ratio >1.0 sustained** → check **New Connections by Endpoint (top 10)** for which endpoints have high churn → likely causes: short-lived connection pattern (Lambda, serverless), connection pool misconfiguration (max idle time too low), application restart loops → next action: review application connection pooling settings; check application logs for errors causing reconnects. (Inference: connection pool patterns) - -5. **If Active Connections by Endpoint shows one endpoint >> others** → check if this is expected traffic distribution OR hotspot → likely causes: application routing imbalance (all traffic to one service), service discovery not load-balancing, client-side caching pointing to single endpoint → next action: verify application load balancing configuration; check service discovery health checks. (Inference: traffic distribution patterns) - -### Data Transfer & Performance Chains - -6. **If Packet Drops (rate) >0 sustained** → check **Bandwidth Utilization per AZ** for saturation (>80%) → likely causes: bandwidth quota exceeded (10 Gbps per AZ), MTU mismatch (jumbo frames not supported), network congestion → next action: check Packet Drops by Availability Zone to identify saturated AZ; add endpoints in more AZs; verify MTU ≤1500 bytes. (Confirmed: bandwidth limits and MTU cap documented) - -7. **If RST Resets (rate) spikes >100 RST/s** → check timing correlation with deployments OR sustained elevation → likely causes: idle timeout exceeded (350s; no TCP keepalive), NLB target health failure (provider-side), application-layer connection closures → next action: verify TCP keepalive enabled for long-lived connections; check Backend Health Ratio (provider-side); check application error logs. (Confirmed: idle timeout documented) - -8. **If Bandwidth Utilization per AZ >80%** → check **Packet Drops (rate)** for correlation (drops occur when bandwidth saturated) → likely causes: traffic spike, insufficient AZ distribution, single endpoint handling bulk transfers → next action: add endpoint ENIs in more AZs; check Data Throughput by Endpoint (top 10) to identify traffic sources; scale endpoint count. (Confirmed: bandwidth limits documented) - -9. **If Data Throughput (MB/s) drops >50% within 5 minutes** → check **Active Connections (trend)** for drop (connection loss) → likely causes: backend service outage (NLB targets unhealthy), network path failure, application throttling → next action: check Backend Health Ratio (provider-side); check RST Resets for spike; review application logs. (Mixed: throughput drop + connection correlation inferred) - -10. **If Average Bytes per Connection spikes >10 MB/s per conn** → this indicates bulk data transfers OR streaming → likely causes: backup jobs, database replication, large file transfers → next action: verify this is expected workload (not runaway data loop); check Data Processing Cost for impact; ensure bandwidth headroom exists. (Inference: throughput per connection patterns) - -11. **If Packet Drops by Availability Zone shows one AZ >> others** → check **Bandwidth Utilization per AZ** for that AZ (likely >80%) → likely causes: single-AZ endpoint deployment, application client affinity to one AZ, cross-AZ traffic routing inefficiency → next action: add endpoint ENI in additional AZs; verify client load balancing across AZs; check VPC route tables. (Inference: per-AZ saturation patterns) - -12. **If RST Resets by Endpoint correlates with high Active Connections** → likely idle timeout expiration (350s) on long-lived connections → likely causes: application not sending TCP keepalive, WebSocket/streaming connections exceeding idle timeout, backend processing latency >350s → next action: enable TCP keepalive in application; check Average Bytes per Connection (long-running transfers at risk); reduce backend processing time. (Confirmed: idle timeout documented) - -### Backend Health Chains (provider-side) - -13. **If Backend Health Ratio drops <80%** → check **Unhealthy Targets by Endpoint Service** to identify which service degraded → likely causes: application crash, health check path failing, target security group blocking health checks, auto-scaling insufficient capacity → next action: check application logs on unhealthy targets; verify NLB health check config (port, path, interval); check target auto-scaling desired capacity. (Confirmed: NLB health checks documented) - -14. **If Healthy Targets (current) = 0 AND Unhealthy Targets = 0** → this is **configuration issue**, not health failure (no registered targets) → likely causes: auto-scaling terminated all instances, targets manually deregistered, target group misconfigured → next action: check target group registered targets (DescribeTargetHealth); check auto-scaling group desired capacity; verify NLB target group association. (Inference: target registration patterns) - -15. **If NLB Target Connection Errors (rate) >1 error/s** → check **Healthy Targets (trend)** for drop (backend capacity exhaustion) → likely causes: target CPU/memory saturation, application refusing connections (backlog full), listener port mismatch with target port → next action: check target resource utilization (CloudWatch metrics for EC2/ECS); verify listener port = target application port; check application logs for connection rejections. (Confirmed: NLB connection errors documented) - -16. **If Unhealthy Targets by Endpoint Service spikes during deployment** → this is **expected** (old targets draining, new targets launching) → likely causes: rolling deployment, auto-scaling replacing instances, health check grace period not expired → next action: no immediate action; monitor for return to 0 unhealthy within 5 minutes; if sustained, check new target launch failures. (Inference: deployment patterns) - -17. **If Backend Health Ratio drops AND consumer-side RST Resets spike** → backend health failures propagating to consumers → likely causes: NLB target health checks failing, backend service crashes, database connection pool exhaustion → next action: prioritize backend remediation (unhealthy targets); check target application logs; verify database/dependency health; check NLB TargetConnectionErrorCount correlation. (Mixed: backend-to-consumer correlation inferred from networking principles) - -### Cost & Usage Chains - -18. **If Data Processing Cost (USD/hour) increasing week-over-week** → check **Data Processing Cost by Endpoint (top 10)** to identify cost drivers → likely causes: traffic growth (legitimate), new batch jobs, application data loop (bug), increased replication → next action: check Data Throughput by Endpoint for traffic growth correlation; review application changes; check for runaway processes. (Confirmed: pricing structure documented) - -19. **If Total Data Processed (GB/hour) spikes overnight** → check timing (backup window, batch job schedule) → likely causes: scheduled backups, ETL jobs, database replication, bulk data sync → next action: verify this is expected workload (not anomaly); check if jobs can be optimized; ensure bandwidth headroom during peak. (Inference: batch job patterns) - -20. **If Data Processed by Region shows high cross-region traffic** → calculate cost impact ($0.02/GB inter-region surcharge) → likely causes: cross-region service consumption (intentional), service not replicated to consumer regions, misconfigured endpoint pointing to wrong region → next action: evaluate architecture for regional service replication; check if cross-region is required (regulatory, DR) vs inefficiency; validate endpoint region configuration. (Confirmed: cross-region pricing documented) - ---- - -## Part 4: Operational Playbooks - -### Playbook 1: Connection Establishment Failure (NewConnections = 0) - -**Trigger**: **New Connections (rate)** drops to zero AND **Active Connections (current)** also drops to zero OR approaching zero, AND application logs show connection timeouts or "connection refused" errors. - -**Decision rule**: If sustained >2 minutes outside planned maintenance windows, escalate immediately (user-facing impact). - -**Steps**: -1. Check **Active Connections by Endpoint (top 10)** — identify which endpoint(s) affected (all or subset) -2. Check endpoint connection state in VPC console (DescribeVpcEndpoints) — look for Rejected, Failed, or Expired states -3. If state = **Rejected**: Service provider rejected connection request → contact provider to add your AWS account/IAM principal to allowed principals list -4. If state = **Failed**: Configuration issue → check endpoint ENI security group (must allow listener port), check subnet route table (must have local VPC route), verify DNS resolution -5. Check VPC Flow Logs filtered to endpoint ENI IPs — look for REJECT action (security group denial) or no flow entries (traffic not reaching endpoint) -6. Test DNS resolution manually (nslookup or dig) — verify private DNS resolves to endpoint ENI private IPs (if private DNS enabled) -7. Check application logs for specific error messages ("connection refused" vs "timeout" vs "no route to host") — each indicates different failure point -8. If all endpoints affected simultaneously: Check AWS Health Dashboard for PrivateLink service events in your region - -**Likely causes**: -- Endpoint connection not accepted by provider (manual acceptance required, but not yet approved) -- Consumer AWS account or IAM principal not in endpoint service allowed principals list (authorization failure) -- Endpoint ENI security group blocking traffic on listener port -- DNS resolution failure (private hosted zone not associated with VPC, or VPC DNS settings disabled) -- Subnet route table missing local VPC route (rare but possible if manually modified) - -**Next actions**: -- If authorization issue: Contact service provider to request access -- If security group: Update inbound rules to allow traffic from application subnets on listener port -- If DNS: Verify private hosted zone VPC association; check VPC enableDnsHostnames and enableDnsSupport settings -- If configuration: Recreate endpoint with correct subnet, security group, and VPC settings - -**Label**: **Confirmed** (connection states, security group, DNS patterns documented in AWS PrivateLink troubleshooting guide) - ---- - -### Playbook 2: Bandwidth Saturation and Packet Drops - -**Trigger**: **Packet Drops (rate)** sustained >1 drop/s AND **Bandwidth Utilization per AZ** >80% for the same Availability Zone(s). - -**Decision rule**: If sustained >5 minutes and affecting production traffic (user-facing latency or errors), escalate to add capacity immediately. - -**Steps**: -1. Check **Bandwidth Utilization per AZ** — identify which Availability Zone(s) are saturated (>80% of 10 Gbps baseline) -2. Check **Packet Drops by Availability Zone** — confirm packet drops correlate with saturated AZ (not all AZs) -3. Check **Data Throughput by Endpoint (top 10)** — identify which endpoint(s) are driving the most traffic to the saturated AZ -4. Check if endpoint is deployed in multiple AZs — single-AZ deployment = hotspot; multi-AZ = traffic imbalance -5. If single-AZ: Create endpoint network interfaces in additional Availability Zones (add AZs to existing endpoint) → distributes load across more 10 Gbps quotas -6. If multi-AZ but one AZ saturated: Check application client load balancing — clients may have affinity to one AZ due to DNS caching or client-side routing -7. Check **Data Throughput (MB/s)** trend — is this a sustained traffic increase (need permanent capacity) or transient spike (wait for auto-scaling to 100 Gbps)? -8. Monitor **Packet Drops (rate)** after adding AZs — should drop to zero within 2-5 minutes as traffic redistributes - -**Likely causes**: -- Single-AZ endpoint deployment handling traffic that exceeds 10 Gbps baseline -- Multi-AZ endpoint but client load balancing inefficient (all clients connect to one AZ) -- Burst traffic spike exceeding 10 Gbps before PrivateLink auto-scales to 100 Gbps (transient drops during scaling) -- Sustained traffic growth approaching or exceeding 10 Gbps per AZ - -**Next actions**: -- Add endpoint ENIs in more Availability Zones (distribute load across more 10 Gbps quotas) -- Verify application clients use DNS with short TTLs (5-60s) to redistribute across AZ ENIs -- Check if traffic can be reduced (caching, compression, rate limiting) or scheduled (batch jobs during off-peak) -- For sustained >80 Gbps aggregate: Contact AWS support to confirm 100 Gbps auto-scaling is functioning - -**Label**: **Confirmed** (10 Gbps baseline, 100 Gbps auto-scale limit, packet drops due to bandwidth saturation documented) - ---- - -### Playbook 3: Active Connection Instability (High RST Reset Rate) - -**Trigger**: **RST Resets (rate)** sustained >100 RST/s per endpoint, AND not correlated with known deployment windows (application restarts, auto-scaling events). - -**Decision rule**: If causing user-facing errors (connection resets mid-request) or sustained >10 minutes, investigate immediately. - -**Steps**: -1. Check **RST Resets by Endpoint (top 10)** — identify which endpoint(s) affected (all or specific ones) -2. Check timing pattern — are RST spikes occurring every ~6 minutes (350 seconds)? → indicates idle timeout expiration -3. If idle timeout pattern: Check application connection lifecycle — long-lived connections (WebSocket, streaming, database) require TCP keepalive to prevent timeout -4. Check **Backend Health Ratio** (provider-side if available) — if <100%, unhealthy NLB targets cause connection terminations that propagate as RSTs to consumers -5. Check **Average Bytes per Connection** — if high (>5 MB/s per conn), long-running bulk transfers may exceed idle timeout (350s with no data = timeout) -6. Check **NLB Target Connection Errors (rate)** (provider-side) — if >1 error/s, backend capacity exhaustion or application refusing connections -7. Check application logs for error messages during RST spikes — "connection reset by peer", "broken pipe", "EOF" all indicate RST -8. Check VPC Flow Logs for TCP flag analysis — look for RST flag (0x04) to confirm resets vs graceful FIN closures - -**Likely causes**: -- Idle timeout (350s) exceeded on long-lived connections without TCP keepalive packets sent -- Backend NLB target health failures (unhealthy targets terminate active connections) -- Application-layer errors causing connection closures (backend crashes, OOM kills, exception handling) -- Cross-region PrivateLink path degradation (rare but possible in multi-region deployments) -- MTU mismatch causing fragmentation failures that manifest as connection resets - -**Next actions**: -- If idle timeout: Enable TCP keepalive in application (interval <300s, ideally <180s for safety margin) -- If backend health: Investigate NLB target failures (application logs, resource saturation, health check config) -- If application errors: Review application error logs, check for memory leaks, database connection pool exhaustion -- If cross-region: Use CloudWatch Network Synthetic Monitor to probe path health; consider regional service replication - -**Label**: **Mixed** (idle timeout, NLB health failures **confirmed**; application-layer errors **inferred** from common patterns) - ---- - -### Playbook 4: Backend Target Health Degradation (provider-side) - -**Trigger**: **Backend Health Ratio (%)** drops below 80%, OR **Unhealthy Targets (current)** >10% of total target count, sustained >5 minutes. - -**Decision rule**: If <50% healthy, escalate immediately (critical capacity impact). If 50-79%, investigate within 10 minutes. - -**Steps**: -1. Check **Unhealthy Targets by Endpoint Service (top 10)** — identify which endpoint service(s) affected -2. Check **Healthy Targets (trend)** — is this a gradual degradation (slow failure) or sudden drop (crash or deployment issue)? -3. Check timing correlation with deployments or auto-scaling events — target health failures during deployments are expected (2-5 minute grace period for new targets) -4. If outside deployment windows: Check application logs on unhealthy targets — look for application crashes, OOM kills, port binding failures -5. Check NLB target group health check configuration — verify port, protocol, path, interval, timeout, healthy/unhealthy thresholds are correct -6. Check target security group — must allow inbound traffic from NLB on health check port (default: same as listener port) -7. Check **NLB Target Connection Errors (rate)** — if >1 error/s, backend targets refusing connections (backlog full, CPU saturation) -8. Check target resource utilization (EC2 CloudWatch metrics: CPU, memory, network) — if saturated, scale target count or instance size - -**Likely causes**: -- Application crash or restart on targets (process died, container OOM killed) -- Health check path or port misconfigured (app listening on different port than health check expects) -- Target security group blocking NLB health check traffic -- Backend resource saturation (CPU, memory, disk) causing app unresponsiveness -- Database or dependency failure upstream (app healthy but unable to respond to health checks due to DB down) - -**Next actions**: -- If application crash: Restart application on unhealthy targets; investigate crash cause (logs, core dumps) -- If health check misconfiguration: Update NLB target group health check settings (correct port, path, interval) -- If security group: Add inbound rule to target security group allowing NLB traffic on health check port -- If resource saturation: Scale target count (auto-scaling group desired capacity) or scale up instance/container size -- If dependency failure: Investigate upstream dependencies (database health, API gateway, third-party services) - -**Label**: **Confirmed** (NLB health checks, target health states, connection errors documented in AWS NLB documentation) - ---- - -### Playbook 5: Cross-Region Cost Spike - -**Trigger**: **Data Processed by Region (top 5)** shows significant cross-region traffic (>20% of total data processed), AND **Data Processing Cost (USD/hour)** increasing faster than traffic growth (cross-region surcharge $0.02/GB on top of $0.01/GB base). - -**Decision rule**: If cross-region cost >intra-region cost for the same traffic volume, evaluate architecture efficiency. Not urgent unless budget exceeded. - -**Steps**: -1. Check **Data Processed by Region (top 5)** — identify which region pairs have high cross-region traffic (consumer region → provider region) -2. Calculate cross-region surcharge: GB cross-region * $0.02/GB * 720 hours/month -3. Check **Data Processing Cost by Endpoint (top 10)** — identify which endpoints are consuming cross-region bandwidth -4. Verify cross-region PrivateLink is intentional (regulatory requirements, DR architecture) vs accidental (misconfigured endpoint pointing to wrong region) -5. Evaluate regional service replication: Can the endpoint service be replicated to consumer regions to eliminate cross-region transfer? -6. Check application read vs write patterns — if read-heavy, regional read replicas (database, cache) can reduce cross-region traffic -7. Check if cross-region traffic can be batched or scheduled (off-peak hours for batch replication) vs real-time -8. Consider AWS Global Accelerator or CloudFront for cacheable content (alternative to PrivateLink for some use cases) - -**Likely causes**: -- Centralized endpoint service in one region serving consumers in multiple regions (hub-spoke architecture) -- Application not aware of regional service availability (hard-coded to single-region endpoint) -- No regional read replicas for read-heavy workloads (all reads go cross-region to primary) -- Regulatory or data residency requirements mandate data in specific region (cross-region unavoidable) - -**Next actions**: -- If accidental cross-region: Update application configuration to use intra-region endpoint services; verify endpoint region matches consumer region -- If intentional hub-spoke: Evaluate cost vs complexity trade-off for regional service replication -- If read-heavy: Deploy regional read replicas (RDS read replicas, ElastiCache replicas); update application to use regional replicas for reads -- If write-heavy: Consider active-active multi-region architecture with bidirectional replication (more complex but reduces cross-region latency) -- If cost-constrained: Batch cross-region traffic during off-peak (lower impact on real-time users); use compression to reduce GB transferred - -**Label**: **Confirmed** (cross-region data transfer charges documented; pricing $0.02/GB inter-region surcharge on top of $0.01/GB base) - ---- - -### Playbook 6: Connection Pool Exhaustion - -**Trigger**: **Connection Churn Ratio** >1.0 sustained, AND **New Connections by Endpoint (top 10)** showing high rate (>10 conn/s) for database or message queue endpoints (which typically use connection pooling). - -**Decision rule**: If application logs show "connection pool exhausted" or "too many connections" errors, investigate immediately (indicates connection leak or misconfiguration). - -**Steps**: -1. Check **New Connections by Endpoint (top 10)** — identify which endpoints have abnormally high new connection rate -2. Check **Active Connections by Endpoint (top 10)** — compare active vs new connections (high churn = many short-lived connections) -3. Check application connection pool configuration — max pool size, max idle time, connection timeout, validation query -4. Check application logs for connection errors — "connection pool exhausted", "timeout acquiring connection", "too many connections" -5. Check backend service (database, message queue) connection limits — verify not hitting max connections (PostgreSQL max_connections, MySQL max_connections) -6. Check for connection leaks — application not closing connections (transaction timeout, exception handling missing finally block) -7. Check **Connection Churn Ratio** over time — is this new behavior (recent code change) or long-standing pattern (design issue)? -8. If connection leak: Enable application-level connection pool metrics (HikariCP, c3p0, Apache Commons DBCP); track active vs idle connections - -**Likely causes**: -- Connection pool max idle time too low (connections closed prematurely, recreated frequently) -- Application connection leak (connections acquired but not closed in finally blocks) -- Backend service connection limit reached (application tries to create new connections, backend refuses) -- Short-lived Lambda or serverless functions creating new connections per invocation (connection pooling not effective) -- Application restart loops (crash, restart, create connections, crash again) - -**Next actions**: -- If pool configuration: Increase max idle time (e.g., 10 minutes → 30 minutes); increase pool size if hitting limits -- If connection leak: Review application code for proper connection closing (try-finally blocks); enable connection pool leak detection -- If backend limit: Increase backend max connections (PostgreSQL: ALTER SYSTEM SET max_connections = 500); or reduce application pool size -- If serverless: Use RDS Proxy or connection pooler (PgBouncer, ProxySQL) to multiplex connections; avoid direct DB connections from Lambda -- If restart loops: Fix application crash root cause (check logs for exceptions, OOM, segfaults) - -**Label**: **Inference** (connection pool patterns inferred from database client best practices; not PrivateLink-specific documentation) - ---- - -## Playbook 7: DNS Resolution Failure for Private DNS - -**Trigger**: Application logs show "unknown host", "DNS resolution failed", or "NXDOMAIN" for PrivateLink endpoint DNS names, AND **New Connections (rate)** = 0 AND **Active Connections (current)** = 0. - -**Decision rule**: If private DNS is enabled for the endpoint but resolution failing, investigate immediately (full service unavailability). - -**Steps**: -1. Verify private DNS is enabled for the endpoint (DescribeVpcEndpoints API or VPC console → endpoint settings) -2. Test DNS resolution manually from consumer application host: `nslookup ` or `dig ` -3. Check Route 53 private hosted zone existence and VPC association — private DNS creates a PHZ automatically, but VPC association can fail -4. Check VPC DNS settings (DescribeVpcAttribute API) — `enableDnsHostnames = true` and `enableDnsSupport = true` required for private DNS -5. Check Route 53 query logs (if enabled) — look for NXDOMAIN responses or no query entries (queries not reaching Route 53) -6. Check application DNS resolver configuration — verify using VPC DNS resolver (169.254.169.253), not external DNS -7. Check if split-horizon DNS is configured — private hosted zone may have incorrect records or priority issues with public DNS -8. If using custom DNS resolvers (Route 53 Resolver endpoints): Check inbound/outbound endpoint configuration and security groups - -**Likely causes**: -- Private hosted zone not associated with consumer VPC (auto-association failed or manually removed) -- VPC DNS settings disabled (enableDnsHostnames or enableDnsSupport = false) -- Application using external DNS resolver (8.8.8.8, 1.1.1.1) instead of VPC DNS resolver (169.254.169.253) -- Split-horizon DNS misconfiguration (private hosted zone conflicts with public DNS records) -- Route 53 Resolver endpoint security group blocking DNS traffic (port 53 UDP/TCP) - -**Next actions**: -- If PHZ not associated: Associate private hosted zone with consumer VPC (Route 53 console or API) -- If VPC DNS disabled: Enable VPC DNS settings (enableDnsHostnames and enableDnsSupport); note: requires endpoint recreation for existing endpoints -- If external DNS: Update application DNS resolver to use VPC DNS (169.254.169.253) or rely on DHCP option set -- If split-horizon issue: Validate private hosted zone records; ensure private zone has higher priority than public -- If custom resolvers: Check Route 53 Resolver endpoint security groups; verify inbound rules allow port 53 UDP/TCP from VPC CIDR - -**Label**: **Confirmed** (private DNS setup, VPC DNS requirements, Route 53 private hosted zones documented) - ---- - -## Labels Summary - -- **Confirmed** chains/playbooks: 10 (based on AWS PrivateLink, NLB, VPC, Route 53 official documentation) -- **Inference** chains/playbooks: 7 (based on connection pool patterns, operational best practices, general networking principles) -- **Mixed** chains/playbooks: 3 (combination of documented AWS behavior and inferred operational patterns) - -Total: 20+ triage chains, 7 operational playbooks - - ---- - -# AWS PrivateLink — Caveats & Footguns - -## High-cardinality dimensions (cardinality explosion risks) - -- **[endpoint-connection-health, data-transfer-performance, cost-usage]** Do NOT group by `SubnetId` in multi-AZ deployments. Each endpoint can have 2-6 ENIs (one per AZ), and subnets multiply quickly (10-100 per VPC). Use `AvailabilityZone` (max 6 per region) or `EndpointId` (top 10) instead. (Inferred from AWS VPC architecture patterns) - -- **[backend-target-health]** Provider-side: Do NOT group by `PrincipalArn` in multi-tenant scenarios (1000s of consumer accounts/roles possible). Use Contributor Insights top-N rankings instead of raw group-by. (Inferred from multi-tenant SaaS patterns) - -- **[endpoint-connection-health]** Do NOT group by `ConnectionState` for all endpoints simultaneously in large deployments (100s of endpoints * 5 states = 500 series). Filter to specific endpoints first, then break down by state. ([AWS re:Post guidance on CloudWatch cardinality](https://repost.aws/questions/)) - -- **[cost-usage]** Cross-region analysis: Grouping by `Region` is safe (typically 1-5 active regions), but combining `Region` + `EndpointId` + `ServiceName` simultaneously creates cardinality explosion. Use hierarchical filtering: filter by Region, then group by EndpointId. (Inferred from cost analysis best practices) - ---- - -## Misleading metrics and wrong aggregations - -- **[endpoint-connection-health]** `ActiveConnections` with `average` aggregation is misleading across multi-endpoint deployments. Use `sum` to see total concurrent connections; use `average` only when comparing per-endpoint behavior. ([AWS CloudWatch metrics best practices](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/cloudwatch_concepts.html)) - -- **[data-transfer-performance]** `BytesProcessed` with `max` aggregation shows the peak 1-minute sample, not total data transferred. Always use `sum` for throughput analysis and cost calculations. `max` is only useful for identifying burst windows. (Inferred from CloudWatch metric type semantics) - -- **[data-transfer-performance]** Comparing `BytesProcessed` across AZs without considering `PacketDropCount` creates false equivalence. An AZ processing 10 GB with 0 drops is healthier than an AZ processing 10 GB with 1000 drops. Correlate both metrics. ([AWS PrivateLink troubleshooting guide](https://repost.aws/knowledge-center/vpc-troubleshoot-network-performance-privatelink)) - -- **[backend-target-health]** `HealthyHostCount` = 0 does NOT always mean outage. If the target group has no registered targets (intentional or not), healthy count is 0 but it's a configuration issue, not a health failure. Check `HealthyHostCount + UnHealthyHostCount` sum first. ([AWS NLB health check docs](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/target-group-health-checks.html)) - -- **[endpoint-connection-health]** `NewConnections` = 0 for sustained periods does NOT necessarily mean failure if `ActiveConnections` is stable and high. Connection-pooled applications (databases, message queues) establish connections once and reuse them. Zero new connections = healthy pool stability. ([Inference from connection pool patterns](https://en.wikipedia.org/wiki/Connection_pool)) - ---- - -## Unit pitfalls - -- **[data-transfer-performance]** `BytesProcessed` is in **bytes**, not bits. Dividing by 1000000000 to get Gbps is WRONG (off by 8x). Use 125000000 bytes/s = 1 Gbps, or 1250000000 bytes/s = 10 Gbps. Tsuga normalizers handle this, but manual calculations fail. ([AWS PrivateLink bandwidth limits](https://repost.aws/knowledge-center/vpc-troubleshoot-network-performance-privatelink)) - -- **[cost-usage]** Data processing cost tiers are per **GB** (gigabyte), not Gb (gigabit). $0.01/GB = $0.01 per 1073741824 bytes, not per 1000000000 bytes. Use 1024-based conversion (GiB) for accuracy. ([AWS PrivateLink pricing](https://aws.amazon.com/privatelink/pricing/)) - -- **[backend-target-health]** NLB health check interval is in **seconds**, but CloudWatch metrics are per **minute**. A health check every 30s = 2 checks per CloudWatch data point. Comparing health check failures to metric granularity requires this conversion. ([AWS NLB health check configuration](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/target-group-health-checks.html)) - -- **[endpoint-connection-health]** `ActiveConnections` and `NewConnections` are measured in **1-minute windows** by CloudWatch. "Connections per second" derived signals require dividing NewConnections by 60 (or using `per-second` post-function), not treating the raw count as a rate. (Inferred from CloudWatch 1-minute granularity) - ---- - -## Temporality pitfalls (delta vs cumulative counters) - -- **[data-transfer-performance, endpoint-connection-health, cost-usage]** AWS CloudWatch metrics for PrivateLink are likely **summary type with cumulative temporality** (based on AWS ECS/SNS/Transcribe patterns from MEMORY.md). This means counters (`NewConnections`, `BytesProcessed`, `PacketDropCount`, `ResetPacketsReceived`) require `rate` post-function, NOT `per-second`. However, temporality is **UNCONFIRMED** for PrivateLink; Stage 2 discovery will validate via `/v1/metrics/metadata` endpoint. All widget specs in 07 include `# VERIFY` comments. (Inferred from AWS metric patterns; [Tsuga temporality guide](internal)) - -- **[data-transfer-performance]** Using `increase` post-function on `BytesProcessed` over long time windows (1+ hour) shows cumulative bytes, which can be misleading for cost calculations. Use `rate` to convert to bytes/s, then multiply by time window for accurate totals. (Inferred from Prometheus/OTel semantics) - -- **[endpoint-connection-health]** `ActiveConnections` is a **gauge** (snapshot of current state), not a counter. Do NOT apply `rate` or `per-second` post-functions; use `none` or `average`/`sum` aggregations only. Applying rate to a gauge produces nonsense values. (Standard observability semantics) - -- **[backend-target-health]** NLB `TargetConnectionErrorCount` is a **cumulative counter**. If you see the raw value increase from 1000 to 1050 in a 1-minute window, that's 50 new errors, not 1050 total. Always use `rate` post-function to see errors/s. ([AWS NLB metrics reference](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-cloudwatch-metrics.html)) - ---- - -## "This looks bad but isn't" (false alarms) - -- **[data-transfer-performance]** `ResetPacketsReceived` spikes during application deployments or NLB target replacements are **normal**. Graceful shutdown sends FIN, but forced termination (common in auto-scaling) sends RST. Expect 10-100 RST/s during deploy windows; only sustained elevation outside deploy windows is actionable. ([AWS deployment best practices](https://docs.aws.amazon.com/whitepapers/latest/practicing-continuous-integration-continuous-delivery/deployment-strategies.html)) - -- **[endpoint-connection-health]** `NewConnections` drops to zero when connection pools reach steady state (all needed connections established and reused). **This is healthy** for database or message queue clients. Only zero NewConnections + zero ActiveConnections = actual failure. ([Connection pool lifecycle patterns](https://en.wikipedia.org/wiki/Connection_pool)) - -- **[data-transfer-performance]** `PacketDropCount` = 1-2 drops per hour in very high throughput scenarios (approaching 10 Gbps sustained) can be **transient auto-scaling artifacts** as PrivateLink scales from 10 to 100 Gbps. Only sustained drop rates >1 drop/s or drops occurring at <5 Gbps throughput indicate real issues. ([AWS PrivateLink auto-scaling behavior, inferred](https://repost.aws/knowledge-center/vpc-troubleshoot-network-performance-privatelink)) - -- **[backend-target-health]** `UnHealthyHostCount` temporarily increasing to 1-2 during target launches (new EC2 instances starting, containers initializing) is **normal**. Health checks fail until application fully starts. Only sustained unhealthy counts >10% of total targets for >5 minutes is actionable. ([AWS auto-scaling health check grace periods](https://docs.aws.amazon.com/autoscaling/ec2/userguide/healthcheck.html)) - -- **[cost-usage]** Data processing cost doubling during backup windows or batch job runs is **expected**, not a billing error. PrivateLink charges per GB transferred; overnight ETL jobs or database replication spikes legitimately increase costs. Baseline cost during business hours, not peak windows. (Inferred from operational patterns) - -- **[endpoint-connection-health]** Connection state = **Pending** for 30-60 seconds on new endpoint creation is **normal** if `acceptance_required = true` on the endpoint service. Manual approval by provider adds latency. Only Pending >1 hour without acceptance is actionable. ([AWS PrivateLink connection acceptance](https://docs.aws.amazon.com/vpc/latest/privatelink/configure-endpoint-service.html)) - ---- - -## Optional-feature traps (metrics absent unless feature enabled) - -- **[endpoint-connection-health, data-transfer-performance, cost-usage]** **Cross-region PrivateLink**: `Region` dimension only exists if cross-region connectivity is enabled on the endpoint service (provider-side configuration) AND consumers in other regions have created endpoints. Absence of `Region` dimension != problem; it means single-region deployment. Dashboards must gate cross-region analysis sections accordingly. ([AWS cross-region PrivateLink launch](https://aws.amazon.com/blogs/networking-and-content-delivery/introducing-cross-region-connectivity-for-aws-privatelink/)) - -- **[backend-target-health]** **NLB metrics**: `HealthyHostCount`, `UnHealthyHostCount`, `TargetConnectionErrorCount` are **provider-side only**. Consumers (service users) cannot see these metrics; they're published to the service provider's CloudWatch account. Consumer-only dashboards must hide the "Backend Target Health" section entirely. Absence = expected for consumers. ([AWS PrivateLink architecture](https://docs.aws.amazon.com/vpc/latest/privatelink/concepts.html)) - -- **[data-transfer-performance]** **Contributor Insights**: Top-N consumer endpoint rankings (which consumers drive the most BytesProcessed, ActiveConnections, ResetPacketsReceived) require manually enabling Contributor Insights rules in the CloudWatch console. These are NOT auto-enabled. Absence = feature not enabled, not a data issue. ([AWS Contributor Insights for PrivateLink](https://aws.amazon.com/blogs/networking-and-content-delivery/gain-usage-insights-with-amazon-cloudwatch-metrics-for-aws-privatelink/)) - -- **[endpoint-connection-health]** **Private DNS**: DNS resolution metrics (query success/failure rates) require Route 53 query logging enabled for the private hosted zone. Private DNS itself is optional (can use endpoint-specific DNS names instead). No query logs = either private DNS not enabled OR query logging not configured. ([AWS Route 53 query logging](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/query-logs.html)) - -- **[data-transfer-performance]** **VPC Flow Logs**: Packet-level visibility (ENI-level traffic, REJECT actions for security group denials) requires VPC Flow Logs enabled for endpoint ENIs. Flow logs are NOT enabled by default. Absence = not configured, not a PrivateLink issue. ([AWS VPC Flow Logs](https://docs.aws.amazon.com/vpc/latest/userguide/flow-logs.html)) - -- **[backend-target-health]** **Gateway Load Balancer Endpoints (GWLBE)**: GWLBE uses same metrics as interface endpoints but serves traffic inspection use cases (firewalls, IDS/IPS). `EndpointType` dimension = "GatewayLoadBalancer" vs "Interface". Most deployments are interface-only; GWLBE metrics absent = no GWLBE deployed. ([AWS GWLB endpoints](https://docs.aws.amazon.com/vpc/latest/privatelink/vpce-gateway-load-balancer.html)) - ---- - -## Configuration and connectivity footguns - -- **[endpoint-connection-health]** **Security group "connection refused"**: If `NewConnections` = 0 and application logs show "connection refused", the endpoint ENI security group is blocking traffic. PrivateLink endpoints inherit VPC security groups; default security group may deny the listener port. This is NOT a PrivateLink service issue; it's consumer-side security group misconfiguration. ([AWS troubleshooting guide](https://repost.aws/questions/QUjamZ-8IgSc-KTI5Q4Kg0vg/privatelink-connectivity-issues)) - -- **[endpoint-connection-health]** **DNS resolution failures**: If private DNS is enabled but DNS queries fail (NXDOMAIN), the private hosted zone may not be associated with the consumer VPC, or VPC DNS settings (enableDnsHostnames, enableDnsSupport) may be disabled. PrivateLink creates the hosted zone, but VPC association is NOT automatic in all scenarios. ([AWS PrivateLink DNS setup](https://docs.aws.amazon.com/vpc/latest/privatelink/privatelink-access-aws-services.html)) - -- **[data-transfer-performance]** **Idle timeout resets (350s)**: `ResetPacketsReceived` spiking every ~6 minutes (350s idle timeout) means clients are not sending TCP keepalive packets. Long-lived connections (WebSockets, streaming) require application-layer keepalive or TCP keepalive enabled. This is NOT a PrivateLink bug; it's standard AWS networking behavior. ([AWS idle timeout docs](https://repost.aws/knowledge-center/vpc-troubleshoot-network-performance-privatelink)) - -- **[data-transfer-performance]** **MTU mismatches**: `PacketDropCount` correlated with large transfers (>1500 byte packets) indicates MTU mismatch between consumer VPC (typically 9001 MTU for jumbo frames within VPC) and endpoint ENI. PrivateLink does NOT support jumbo frames; MTU is capped at 1500. Enable Path MTU Discovery (PMTUD) in applications. ([AWS VPC MTU documentation](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/network_mtu.html)) - -- **[endpoint-connection-health]** **Connection state = Rejected**: Service provider explicitly rejected the connection request, OR the consumer's AWS account/IAM principal is not in the endpoint service's allowed principals list. This is an authorization failure, not a connectivity failure. Consumers must request access from the provider. ([AWS PrivateLink connection states](https://docs.aws.amazon.com/vpc/latest/privatelink/configure-endpoint-service.html)) - -- **[backend-target-health]** **NLB target health check failures**: If consumer-side `ResetPacketsReceived` correlates with provider-side `UnHealthyHostCount` increases, the NLB health check is failing (target not responding on health check port/path). This is a backend service issue (app crashed, port blocked), not a PrivateLink issue. ([AWS NLB health check troubleshooting](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/target-group-health-checks.html)) - -- **[cost-usage]** **Cross-AZ data transfer costs**: If `BytesProcessed` is high but data processing cost is higher than expected, check if consumer endpoint AZs align with provider NLB AZs. Mismatched AZs incur cross-AZ data transfer charges ($0.01/GB inbound + $0.01/GB outbound) ON TOP OF PrivateLink data processing charges. Multi-AZ for HA is correct, but understand cost implications. ([AWS data transfer pricing](https://aws.amazon.com/ec2/pricing/on-demand/)) - ---- - -## Confirmed by sources -- 350-second idle timeout ([AWS re:Post](https://repost.aws/knowledge-center/vpc-troubleshoot-network-performance-privatelink)) -- 10 Gbps baseline, auto-scales to 100 Gbps per AZ ([same source](https://repost.aws/knowledge-center/vpc-troubleshoot-network-performance-privatelink)) -- Connection states (Pending, Available, Rejected, Failed, Expired) ([AWS docs](https://docs.aws.amazon.com/vpc/latest/privatelink/configure-endpoint-service.html)) -- NLB metrics (HealthyHostCount, UnHealthyHostCount, TargetConnectionErrorCount) ([AWS NLB docs](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-cloudwatch-metrics.html)) -- Contributor Insights for PrivateLink ([AWS blog](https://aws.amazon.com/blogs/networking-and-content-delivery/gain-usage-insights-with-amazon-cloudwatch-metrics-for-aws-privatelink/)) -- Private DNS setup ([AWS docs](https://docs.aws.amazon.com/vpc/latest/privatelink/privatelink-access-aws-services.html)) -- MTU limits (1500 bytes for PrivateLink) ([AWS VPC MTU docs](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/network_mtu.html)) -- Cross-region PrivateLink launch ([AWS blog](https://aws.amazon.com/blogs/networking-and-content-delivery/introducing-cross-region-connectivity-for-aws-privatelink/)) - -## Best-practice inference -- Temporality (delta vs cumulative) inferred from AWS ECS/SNS/Transcribe patterns documented in MEMORY.md -- High-cardinality risks (SubnetId, PrincipalArn) inferred from general AWS multi-tenant and multi-VPC architecture patterns -- Connection pool behavior (NewConnections = 0 is healthy) inferred from standard database and message queue client patterns -- Cross-AZ cost implications inferred from AWS data transfer pricing model (not PrivateLink-specific) -- Auto-scaling transient packet drops inferred from general AWS elastic scaling behavior -- Health check grace periods during target launches inferred from EC2 auto-scaling best practices - - ---- - diff --git a/plugins/tsuga/skills/knowledge-technology/references/aws-privatelink/metrics.csv b/plugins/tsuga/skills/knowledge-technology/references/aws-privatelink/metrics.csv deleted file mode 100644 index 64f7982..0000000 --- a/plugins/tsuga/skills/knowledge-technology/references/aws-privatelink/metrics.csv +++ /dev/null @@ -1,14 +0,0 @@ -theme,name,confirmed,definition,unit,type,tsuga_metric_name,widget_suggestions,aggregation,post_function,group_by,required_context_fields,optional_context_fields,absence_means,cardinality_footguns,variants,citations -Availability/Health,ActiveConnections,true,"Number of concurrent active TCP connections through the PrivateLink endpoint or endpoint service during the 1-minute CloudWatch period. Measures sustained connection load; sudden drops indicate mass disconnections from idle timeout or backend failures.",{connection},gauge,aws_privatelinkendpoints_active_connections,timeseries|query-value|top-list,sum,none,"context.vpc endpoint id:10|context.service name:10","context.vpc endpoint id|context.service name","context.vpc id|context.cloud.region|context.cloud.account.id","Endpoint/service exists but no active connections established; either no consumer traffic or all connections recently closed.","Safe for typical deployments (10-100 endpoints per account); high cardinality risk if grouping by subnet id (1000s). Use vpc endpoint id (top 10) or service name (top 10) only. Context fields use SPACE-SEPARATED naming.",consumer=AWS/PrivateLinkEndpoints.ActiveConnections|provider=AWS/PrivateLinkServices.ActiveConnections,https://docs.aws.amazon.com/vpc/latest/privatelink/privatelink-cloudwatch-metrics.html -Throughput/Usage,NewConnections,true,"Number of new TCP connections established through the PrivateLink endpoint or endpoint service during the 1-minute CloudWatch period. Measures connection establishment rate; zero NewConnections while app attempts connections = authorization or connectivity failure.",{connection},counter,aws_privatelinkendpoints_new_connections,timeseries|query-value|top-list,sum,rate,"context.vpc endpoint id:10|context.service name:10","context.vpc endpoint id|context.service name","context.vpc id|context.cloud.region|context.cloud.account.id","No new connections during period; normal if connection pool stable. Zero while app attempts connections = rejected/failed state, security group block, or DNS failure.","Safe for group-by on vpc endpoint id or service name (low-medium cardinality). Cumulative temporality confirmed; use rate post-function.",consumer=AWS/PrivateLinkEndpoints.NewConnections|provider=AWS/PrivateLinkServices.NewConnections,https://docs.aws.amazon.com/vpc/latest/privatelink/privatelink-cloudwatch-metrics.html -Throughput/Usage,BytesProcessed,true,"Total bytes transferred through the PrivateLink endpoint or endpoint service during the 1-minute CloudWatch period. Includes both inbound and outbound traffic. Used for bandwidth utilization analysis and data transfer cost calculation ($0.01/GB tier 1).",By,counter,aws_privatelinkendpoints_bytes_processed,timeseries|query-value|top-list,sum,rate,"context.vpc endpoint id:10|context.service name:10","context.vpc endpoint id|context.service name","context.vpc id|context.cloud.region|context.cloud.account.id","No data transferred during period; endpoint/service exists but idle. Zero sustained for >5 minutes = possible connectivity issue or app not using endpoint.","Safe for vpc endpoint id/service name group-by. NOTE: Availability Zone dimension does NOT exist in Tsuga; per-AZ bandwidth analysis not possible. Region dimension (context.cloud.region) exists for cross-region cost analysis.",consumer=AWS/PrivateLinkEndpoints.BytesProcessed|provider=AWS/PrivateLinkServices.BytesProcessed,https://docs.aws.amazon.com/vpc/latest/privatelink/privatelink-cloudwatch-metrics.html|https://aws.amazon.com/privatelink/pricing/ -Errors/Failures,PacketDropCount,true,"Number of packets dropped by the PrivateLink endpoint or endpoint service during the 1-minute CloudWatch period. Primary indicator of bandwidth saturation (exceeding 10 Gbps per AZ), MTU mismatches, or network congestion. Any sustained non-zero value is actionable.",{packet},counter,aws_privatelinkendpoints_packets_dropped,timeseries|query-value|top-list,sum,rate,"context.vpc endpoint id:10|context.service name:10","context.vpc endpoint id|context.service name","context.vpc id|context.cloud.region|context.cloud.account.id","No packets dropped; healthy state. Zero is expected baseline; any non-zero indicates issues.","Safe for vpc endpoint id/service name group-by. NOTE: Availability Zone dimension does NOT exist; cannot identify per-AZ hotspots. Aggregate at endpoint level only.",consumer=AWS/PrivateLinkEndpoints.PacketDropCount|provider=AWS/PrivateLinkServices.PacketDropCount,https://docs.aws.amazon.com/vpc/latest/privatelink/privatelink-cloudwatch-metrics.html|https://repost.aws/knowledge-center/vpc-troubleshoot-network-performance-privatelink -Errors/Failures,ResetPacketsReceived,true,"Number of TCP RST (reset) packets received by the PrivateLink endpoint or endpoint service during the 1-minute CloudWatch period. Indicates abrupt connection terminations from idle timeout (350s), application errors, or NLB target health failures. Elevated RST count (>100/min) = active connection instability.",{packet},counter,aws_privatelinkendpoints_rst_packets_received,timeseries|query-value|top-list,sum,rate,"context.vpc endpoint id:10|context.service name:10","context.vpc endpoint id|context.service name","context.vpc id|context.cloud.region|context.cloud.account.id","No TCP resets; healthy connection lifecycle (graceful FIN terminations). Some RST packets normal during failover or app restarts; sustained >100/min is actionable.",Safe for vpc endpoint id/service name group-by. Correlate with NLB UnHealthyHostCount (provider-side) to identify backend-driven resets vs idle timeout resets. Cumulative temporality confirmed; use rate post-function.,consumer=AWS/PrivateLinkEndpoints.ResetPacketsReceived|provider=AWS/PrivateLinkServices.ResetPacketsReceived,https://docs.aws.amazon.com/vpc/latest/privatelink/privatelink-cloudwatch-metrics.html|https://repost.aws/knowledge-center/vpc-troubleshoot-network-performance-privatelink -Availability/Health,EndpointsCount,true,"(Provider-side OR aggregate metric) Count of VPC endpoints. Exact semantics unclear; may be total consumer endpoints per service or aggregate endpoint count. Unexpected metric found during discovery.",{endpoint},counter,aws_privatelinkservices_endpoints_count,query-value,sum,none,context.service name:10,context.service name,context.cloud.region|context.cloud.account.id,"Zero endpoints; no consumers connected to endpoint service OR metric not applicable to consumer-side.","Low cardinality; safe for service name group-by. Exact use case unclear; Stage 3 should evaluate if worth including in provider dashboards.",provider=AWS/PrivateLinkServices,https://docs.aws.amazon.com/vpc/latest/privatelink/privatelink-cloudwatch-metrics.html -Availability/Health,NLB_HealthyHostCount,true,"(Provider-side only) Number of healthy targets registered with the Network Load Balancer backing the PrivateLink endpoint service. Target health failures cascade to consumer-side metrics (PacketDropCount, ResetPacketsReceived). Zero healthy hosts = service unavailable to all consumers.",{target},gauge,aws_networkelb_healthy_host_count,timeseries|query-value,sum,none,"context.service id:10|context.target group arn:10",context.service id,"context.target group arn|context.cloud.region","NLB has no registered targets OR all targets are failing health checks. Zero sustained >2 minutes = service outage.","Safe for service id and target group arn group-by (low cardinality in typical provider setups). Provider-side only; consumers cannot see this metric. Context field naming uses space-separated format (inferred based on PrivateLink pattern; Stage 3 should validate).",NLB=AWS/NetworkELB.HealthyHostCount,https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-cloudwatch-metrics.html -Availability/Health,NLB_UnHealthyHostCount,true,"(Provider-side only) Number of unhealthy targets registered with the Network Load Balancer backing the PrivateLink endpoint service. Unhealthy targets cause consumer-side connection errors and resets. Any non-zero value is actionable for provider.",{target},gauge,aws_networkelb_un_healthy_host_count,timeseries|query-value,sum,none,"context.service id:10|context.target group arn:10",context.service id,"context.target group arn|context.cloud.region","All targets are healthy. Zero is expected baseline; any non-zero indicates backend service degradation.","Safe for service id and target group arn group-by. Note: Tsuga metric name uses underscore in un_healthy (not unhealthy). Correlate with consumer-side ResetPacketsReceived to confirm backend-driven connection failures.",NLB=AWS/NetworkELB.UnHealthyHostCount,https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-cloudwatch-metrics.html -Errors/Failures,NLB_TargetConnectionErrorCount,false,"(Provider-side only) Number of connections from the NLB to targets that failed during the 1-minute period. Indicates backend capacity exhaustion, target crashes, or port mismatches. Correlates with consumer-side RST resets.",{connection},counter,Unknown,timeseries|query-value,sum,rate,"context.service id:10|context.target group arn:10",context.service id,"context.target group arn|context.cloud.region","No connection errors to backend targets; healthy state. Zero is expected; any non-zero indicates backend issues.","NOT FOUND in Tsuga during Stage 2 discovery. Metric expected but not ingested. Alternative: use aws_networkelb_tcp_target_reset_count as proxy for connection quality. Widget using this metric should be gated or replaced.",NLB=AWS/NetworkELB.TargetConnectionErrorCount,https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-cloudwatch-metrics.html -Throughput/Usage,NLB_ProcessedBytes,true,"(Provider-side only) Total bytes processed by the Network Load Balancer backing the PrivateLink endpoint service. Should closely match consumer-side BytesProcessed; significant divergence indicates traffic loss or asymmetric routing.",By,counter,aws_networkelb_processed_bytes,timeseries|query-value,sum,rate,"context.service id:10|context.target group arn:10",context.service id,"context.target group arn|context.cloud.region","NLB processed no traffic; endpoint service exists but no consumer connections. Zero sustained = no consumer usage.","Safe for service id and target group arn group-by. Cumulative temporality confirmed; use rate post-function.",NLB=AWS/NetworkELB.ProcessedBytes,https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-cloudwatch-metrics.html -Cost,"metric family: Hourly Endpoint Charge",false,"Fixed $0.01/hour cost per interface VPC endpoint (consumer-side). Not a CloudWatch metric; calculated from endpoint count * hours. Resource endpoints cost $0.02/hour. Total hourly cost = (interface endpoint count * $0.01) + (resource endpoint count * $0.02).",USD/hour,gauge,Unknown,"query-value (calculated, not raw metric)",count,none,none,none,"context.vpc endpoint id|context.endpoint type","Endpoint does not exist or is terminated. Zero cost means no active endpoints.","Safe; endpoint count per account typically <1000. Calculation requires endpoint count metric (not a standard CloudWatch metric; derive from DescribeVpcEndpoints API or tag-based tracking).","No direct CloudWatch metric exists. Cost calculated from endpoint count (ListMetrics or DescribeVpcEndpoints API) * $0.01/hour. Not included in dashboard scope (requires external API call).",https://aws.amazon.com/privatelink/pricing/ -Cost,"metric family: Data Processing Charge",false,"Variable cost per GB of data processed through PrivateLink endpoints. Tiered pricing: $0.01/GB (first PB), $0.006/GB (next 4 PB), $0.004/GB (5+ PB). Calculated from BytesProcessed CloudWatch metric. Formula: BytesProcessed (GB) * tier rate. Does NOT include cross-region data transfer ($0.02/GB additional).",USD,counter,Unknown,"query-value (derived from BytesProcessed, not standalone metric)",sum,none,none,none,"context.vpc endpoint id|context.service name|context.cloud.region","No data processed; endpoint exists but idle. Zero cost for data processing (hourly charge still applies).",Safe; same cardinality as BytesProcessed. Cross-region data transfer requires context.cloud.region dimension to apply $0.02/GB surcharge.,"Derived from aws_privatelinkendpoints_bytes_processed CloudWatch metric. Tiered pricing requires step function: if bytes < 1 PB: rate $0.01/GB; elif bytes < 5 PB: rate $0.006/GB; else: rate $0.004/GB.",https://aws.amazon.com/privatelink/pricing/ -Cost,"metric family: Cross-Region Data Transfer Charge",false,"Additional $0.02/GB charge for data transferred across AWS regions via cross-region PrivateLink (on top of data processing charge). Only applies when consumer and provider are in different regions. Calculated from BytesProcessed with Region dimension filter. Formula: BytesProcessed (GB, cross-region only) * $0.02.",USD,counter,Unknown,"query-value (derived from BytesProcessed filtered by Region, not standalone metric)",sum,none,none,none,"context.vpc endpoint id|context.service name|context.cloud.region","No cross-region data transferred; either no cross-region endpoints deployed or all traffic is intra-region. Zero cost for cross-region transfer (intra-region data processing charge still applies).",Requires context.cloud.region dimension to identify cross-region vs intra-region traffic. High cardinality if many regions; typically 1-5 active regions. Cross-region filtering logic needs validation in Stage 3.,"Derived from aws_privatelinkendpoints_bytes_processed CloudWatch metric filtered where consumer region != provider region. Requires cross-region connectivity enabled on endpoint service.",https://aws.amazon.com/privatelink/pricing/|https://aws.amazon.com/blogs/networking-and-content-delivery/introducing-cross-region-connectivity-for-aws-privatelink/ diff --git a/plugins/tsuga/skills/knowledge-technology/references/aws-privatelink/queries.md b/plugins/tsuga/skills/knowledge-technology/references/aws-privatelink/queries.md deleted file mode 100644 index 11ce30e..0000000 --- a/plugins/tsuga/skills/knowledge-technology/references/aws-privatelink/queries.md +++ /dev/null @@ -1,48 +0,0 @@ -# AWS PrivateLink / VPC Endpoints - -Private network path from VPC to AWS or partner services without public internet. Typically VPC Interface Endpoint (ENI) fronted by an NLB. Healthy: stable active connections, no drops, healthy backing NLB. - -## Incident shapes - -- **Backing NLB unhealthy** — `aws_networkelb_un_healthy_host_count > 0` → provider fleet sick -- **Packet drops** — `aws_privatelinkendpoints_packets_dropped` climbs (rare; capacity or IP conflict) -- **Reset packets** — `aws_privatelinkendpoints_rst_packets_received` climbs from either side closing abruptly -- **Connection churn** — `aws_privatelinkendpoints_new_connections` ≫ `aws_privatelinkendpoints_active_connections` → no connection reuse -- **Cost runaway** — `aws_privatelinkendpoints_bytes_processed` spike (cross-AZ PrivateLink is billed) - -## Key metrics - -| Metric | Unit | Signal | -|---|---|---| -| `aws_privatelinkendpoints_active_connections` | count | Established through endpoint | -| `aws_privatelinkendpoints_new_connections` | count | New-connection rate | -| `aws_privatelinkendpoints_bytes_processed` | bytes | Volume | -| `aws_privatelinkendpoints_packets_dropped` | count | Drops | -| `aws_privatelinkendpoints_rst_packets_received` | count | RST from either side | -| `aws_privatelinkservices_endpoints_count` | count | Active endpoints on service | -| `aws_networkelb_healthy_host_count` | count | Provider healthy | -| `aws_networkelb_un_healthy_host_count` | count | Provider unhealthy | -| `Unknown` | count | Provider target connection errors | -| `aws_networkelb_processed_bytes` | bytes | Provider throughput | - -## Derived signals - -- `aws_privatelinkendpoints_bytes_processed / aws_privatelinkendpoints_new_connections` — connection reuse; low = new connection per request. -- `aws_networkelb_healthy_host_count / (Healthy + Unhealthy)` — provider health ratio. -- `aws_privatelinkendpoints_rst_packets_received / aws_privatelinkendpoints_active_connections` — reset rate. Any sustained = instability. - -## Log patterns - -PrivateLink has no logs. Use VPC flow logs + provider-service logs: - -- Flow logs with `REJECT` on endpoint ENIs -- App: `connect() failed` / `connection refused` / `reset by peer` against endpoint DNS -- App: DNS resolution failures for `*...vpce.amazonaws.com` - -## Gotchas - -- Consumer resolves endpoint via private DNS (`vpce-xxxxx....vpce.amazonaws.com`). Resolving the public service endpoint bypasses PrivateLink entirely — endpoint metrics appear empty. -- Endpoint policies (IAM-like JSON) can silently deny calls. Looks like IAM errors but originates at the endpoint. -- Cross-AZ traffic via PrivateLink: data processing + inter-AZ charges both apply. -- Provider NLB health is only visible to the provider; consumer sees connection-level errors only. -- Some older HTTP clients treat RSTs as fatal and retry aggressively — can amplify incidents. diff --git a/plugins/tsuga/skills/knowledge-technology/references/aws-rds/README.md b/plugins/tsuga/skills/knowledge-technology/references/aws-rds/README.md deleted file mode 100644 index 6a036b3..0000000 --- a/plugins/tsuga/skills/knowledge-technology/references/aws-rds/README.md +++ /dev/null @@ -1,299 +0,0 @@ -# AWS RDS Integration Context Bundle - -## Metadata -**Technology:** AWS RDS -**Deployment:** managed -**Environment:** prod -**Persona:** SRE Dev and ops -**Telemetry preference:** mixed -**Integration scope:** core service only -**Primary use-case:** reliability and performance - -## How to use this bundle -- Use `01_aws-rds_metrics.csv` as the source of truth for metric names, units, temporality assumptions, and safe query patterns. -- Use `02_aws-rds_dashboard_plan.yaml` as the implementation blueprint for sections, widgets, derived signals, explanation notes, triage chains, and playbooks. -- Use `03_aws-rds_state.yaml` as the machine-readable state file for stage status, inferred namespace mappings, log intelligence status, and unresolved unknowns. -- Use `04_aws-rds_memory.md` for the human-readable Stage 1 handoff narrative and Stage 2 priority checks. -- Stage 2 will create `05_aws-rds_metric_catalog.csv` as the discovered inventory and reconciliation memory for actual Tsuga metric and attribute coverage. -- Stage 4 should read `00` under `Log intelligence (Stage 4 handoff)` and `03.log_intel` before attempting log route creation. - -## What it is and what "good" looks like - -### Confirmed by sources -- Amazon RDS is a managed relational database service that surfaces operational health primarily through CloudWatch metrics, optional Enhanced Monitoring OS telemetry, Performance Insights DB load metrics, events, and engine log exports. [S1][S3][S4][S5][S8] -- CloudWatch instance metrics are the universal baseline for fleet health: `CPUUtilization`, `DatabaseConnections`, `FreeableMemory`, `FreeStorageSpace`, IOPS, latency, throughput, and engine-specific replication or log-space metrics. [S1] -- Enhanced Monitoring adds instance OS signals such as `cpuUtilization`, `diskIO`, `loadAverageMinute`, `memory`, `swap`, and `tasks`, which matter when CloudWatch shows pressure but not the operating-system reason. [S4] -- Performance Insights adds database load split into `DBLoad`, `DBLoadCPU`, `DBLoadNonCPU`, and `DBLoadRelativeToNumVCPUs`, which are the closest thing RDS has to a direct saturation signal inside the database engine. [S3][S7] -- "Good" for RDS is stable client connectivity, low and explainable DB load, storage headroom, read/write latency that matches traffic shape, and replication staying close enough to real time for the workload. [S1][S3][S4] -- Dashboard paging intent should quickly separate four distinct shapes: instance resource exhaustion, storage exhaustion, engine-internal contention, and replication drift. - -### Best-practice inference -- Incident shape 1: **Fleet health regression**. Start in `fleet-health` to determine if the issue is broad (CPU, memory, connection count) or isolated to one instance class/engine/region. -- Incident shape 2: **Traffic or latency regression**. Start in `traffic-connections` and `latency-io` to see whether demand, I/O pressure, or network saturation changed first. -- Incident shape 3: **Storage or engine pressure**. Start in `storage-capacity` and `engine-pressure` when write-heavy workloads, transaction logs, or wait-heavy DB load are suspected. -- Incident shape 4: **Replica drift or failover risk**. Start in `replication-resilience`; replication-specific metrics are often the earliest sign of read-replica or engine-specific durability pain. -- A useful dashboard for RDS must support both umbrella coverage across engines and explicit gating for engine-specific metrics, because MySQL, PostgreSQL, SQL Server, Oracle, and MariaDB do not expose the same secondary signals. [S1][S5][S6] - -## Key concepts - -### Glossary - -| Term | Definition | Operational meaning | Dashboard section | -|---|---|---|---| -| DB instance | The managed RDS compute and storage endpoint for one database deployment | Primary blast-radius boundary for dashboards and filters | fleet-health | -| Read replica | A replica instance asynchronously following a writer | Lag here is early warning for stale reads and failover risk | replication-resilience | -| Multi-AZ | RDS high-availability deployment using standby infrastructure | Helps interpret whether replication/failover signals are expected | replication-resilience | -| CloudWatch metric | Native AWS service metric emitted for an RDS instance or related surface | Universal baseline telemetry across all engines | all | -| Enhanced Monitoring | OS-level telemetry stream emitted from the RDS host | Explains host pressure that CloudWatch alone cannot | fleet-health | -| Performance Insights | Database load and wait analysis subsystem | Best source for engine saturation and CPU vs wait split | engine-pressure | -| DB load | Average active sessions running or waiting for resources | Most direct "is the engine overloaded?" signal | engine-pressure | -| vCPU saturation | DB load relative to available vCPUs | Distinguishes healthy parallelism from overloaded CPU demand | engine-pressure | -| Database connections | Number of client database connections | First demand and pool-pressure signal | traffic-connections | -| Freeable memory | Reclaimable memory reported by CloudWatch | Low headroom here often precedes latency and swap pressure | fleet-health | -| Free storage space | Remaining allocatable storage | The main "time to outage" storage headroom signal | storage-capacity | -| Disk queue depth | Number of queued disk operations | Rising queue depth with latency means storage pressure | latency-io | -| Read IOPS | Read operations per second | Characterizes read demand and storage stress | latency-io | -| Write IOPS | Write operations per second | Characterizes write demand and commit pressure | latency-io | -| Read latency | Average latency per read I/O operation | Indicates storage responsiveness for read paths | latency-io | -| Write latency | Average latency per write I/O operation | Indicates storage responsiveness for writes and commits | latency-io | -| Throughput | Bytes transferred per second for reads, writes, or network | Separates bandwidth-bound incidents from IOPS-bound incidents | latency-io | -| Burst balance | Remaining burst credits for burstable storage | Falling credits mean future performance cliffs, not just current pain | storage-capacity | -| Deadlock | Transaction deadlock count | Symptom of application contention or lock design problems | engine-pressure | -| Replica lag | Delay between writer and replica application | Measures replica freshness and resilience risk | replication-resilience | -| Transaction log disk usage | Storage consumed by transaction logs | Critical for SQL Server or heavy write workloads | storage-capacity | -| Binlog disk usage | Disk consumed by MySQL binary logs | Indicates replication/log-retention pressure | replication-resilience | -| Replication slot disk usage | Disk consumed by PostgreSQL logical replication slots | Can silently consume storage if consumers stall | replication-resilience | -| Oldest replication slot lag | Logical slot lag in PostgreSQL | Strong early warning for stalled CDC consumers | replication-resilience | -| Network receive/transmit throughput | Network ingress and egress at the instance | Helps distinguish client surges from storage or engine stalls | traffic-connections | -| Engine-specific metric | Metric emitted only for a particular engine family or feature | Must be gated in dashboards to avoid false "no data" alarms | all | - -[S1][S3][S4][S5][S6][S7] - -### Concept Map -Client application -> opens -> Database connection (why: demand starts as connection pressure before query saturation) -Connection pool -> fans into -> DB instance (why: pool spikes can inflate `DatabaseConnections` without proportional CPU) -DB instance -> emits -> CloudWatch metrics (why: baseline availability and resource posture) -DB instance -> can emit -> Enhanced Monitoring OS metrics (why: host-level saturation details) -DB instance -> can emit -> Performance Insights DB load (why: engine contention and wait visibility) -DB instance -> stores on -> EBS-backed storage volume (why: read/write latency and queue depth reflect storage health) -Storage volume -> constrains -> ReadLatency and WriteLatency (why: queueing or credit loss shows up as slower I/O) -ReadIOPS/WriteIOPS -> drive -> ReadThroughput/WriteThroughput (why: operation count and bandwidth should be interpreted together) -FreeStorageSpace -> limits -> transaction growth runway (why: low headroom turns routine write spikes into outages) -BurstBalance -> buffers -> temporary burst demand (why: depletion predicts future performance cliffs) -Database workload -> increases -> DBLoad (why: active sessions rise when more work or more waits exist) -DBLoadCPU -> represents -> runnable on-CPU load (why: compute saturation) -DBLoadNonCPU -> represents -> waiting load (why: lock, I/O, or other resource contention) -DBLoadRelativeToNumVCPUs -> compares -> DBLoad against CPU capacity (why: quick saturation thresholding) -Application query pattern -> can create -> Deadlocks (why: lock ordering or long transactions collide) -Writer instance -> replicates to -> Read replica (why: freshness and HA depend on timely replication) -ReplicaLag -> measures -> replication drift (why: stale reads and slow failover risk) -MySQL binary log retention -> consumes -> BinLogDiskUsage (why: backlog or retention settings can fill storage) -PostgreSQL logical slots -> consume -> ReplicationSlotDiskUsage (why: stalled consumers pin WAL files) -OldestReplicationSlotLag -> indicates -> CDC consumer delay (why: replication slots can fall behind before disks fill) -SQL Server transaction logs -> consume -> TransactionLogsDiskUsage (why: long transactions or backup gaps exhaust storage) -CloudWatch dimensions -> map to -> instance, engine, class, region, volume (why: safe operational group-by axes) -context.env/context.team -> scope -> ownership and environment boundaries (why: dashboard filters must reflect org routing) -context.dbinstanceidentifier -> anchors -> per-instance attribution (why: first-line isolation axis) -context.enginename -> explains -> metric availability differences (why: engine-specific widgets must be gated) -context.cloud.region -> localizes -> regional incidents and replication topology (why: regional AWS issues are a real failure domain) -Engine logs -> complement -> metric-only triage (why: deadlocks, slow queries, auth failures, and replication warnings often appear in logs first) - -### Entities and dimensions - -| Entity/Dimension | Why useful | Cardinality risk | Safe top-N | Do NOT group-by guidance | -|---|---|---|---|---| -| `context.env` | Environment boundary for prod vs non-prod | Low | 5 | Keep as a global filter, not a timeseries split in every chart | -| `context.team` | Ownership routing and escalation | Low | 10 | Avoid using it as the first technical diagnostic axis | -| `context.dbinstanceidentifier` | Primary per-instance attribution | Confirmed | 20 | Safe first-line filter and the main instance-level grouping key discovered in Tsuga | -| `context.databaseclass` | Capacity-envelope comparison across instance sizes | Confirmed | 12 | Use sparingly beside instance filters to avoid redundant splits | -| `context.enginename` | Explains which metrics should exist | Confirmed | 8 | Safe engine-level gating field for section notes and optional splits | -| `context.cloud.region` | Regional failure-domain isolation | Low | 10 | Prefer region over AZ unless you know AZ labels exist consistently | -| `context.storage.volume.name` | Needed for EBS balance metrics when emitted per volume | Medium | 8 | Do not use unless Stage 2 confirms the field exists | -| `context.aws.account.id` | Multi-account segmentation | Medium | 10 | Skip if the org runs a single account; adds clutter fast | -| `context.role` | Writer vs reader role split | Confirmed on replica metrics | 4 | Use only where the live metric actually carries the field | -| `context.databaseinsightsmode` | Performance Insights export mode hint | Confirmed on PI-derived metrics | 6 | This is not a wait-category breakdown and should not be treated as one | -| `context.pi.sql_digest` | SQL-level attribution for DB load | High | 10 | Do not place on overview dashboards; cardinality risk is high | -| `context.db.name` | Useful for multi-database instances | Medium | 12 | Avoid unless metrics are actually tagged at DB granularity | -| `context.db.user` | Helpful for connection or auth spikes | Medium-High | 10 | Do not use by default in metric widgets | -| `context.log.type` | Differentiates error/slowquery/general/postgresql logs | Low | 8 | Log-only field; do not assume it exists on metrics | - -### Tsuga field mapping - -| Vendor/exporter dimension | Recommended context.* key | Must-exist vs optional | -|---|---|---| -| `DBInstanceIdentifier` | `context.dbinstanceidentifier` | Confirmed in Stage 2 | -| `DatabaseClass` | `context.databaseclass` | Confirmed in Stage 2 | -| `EngineName` | `context.enginename` | Confirmed in Stage 2 | -| `SourceRegion` | `context.cloud.region` | Optional | -| `VolumeBytesUsed` or volume dimension for EBS balance metrics | `context.storage.volume.name` | Optional | -| Read replica role | `context.role` | Confirmed on replica-oriented metrics | -| Performance Insights mode hint | `context.databaseinsightsmode` | Confirmed on at least one PI-derived metric | -| Performance Insights SQL digest / statement group | `context.pi.sql_digest` | Optional | -| Environment tag from org enrichment | `context.env` | Must-exist | -| Team tag from org enrichment | `context.team` | Must-exist | -| AWS account id from enrichment | `context.aws.account.id` | Optional | -| CloudWatch log type (error/general/slowquery/postgresql) | `context.log.type` | Optional | - -### Confirmed by sources -- Amazon RDS CloudWatch metrics use dimensions such as `DBInstanceIdentifier`, `DatabaseClass`, `EngineName`, and for some EBS balance metrics `SourceRegion` or `VolumeBytesUsed`. [S2] -- MySQL, PostgreSQL, and other engines export different log types, so log routing and metric gating must acknowledge engine differences. [S5][S6][S8] - -### Best-practice inference -- Stage 2 discovery confirmed flattened `context.*` keys rather than nested `context.db.*` segments for the core metric filters. -- No wait-category dimension was confirmed during Stage 2 discovery; `context.databaseinsightsmode` appeared, but it is a mode/export hint rather than a wait breakdown. - -## Golden signals - -### Confirmed by sources -| Signal | What it means for AWS RDS | Typical degradation causes | Best telemetry sources | What people page on | Section questions | -|---|---|---|---|---|---| -| Traffic | Connection demand plus network ingress/egress into the DB | Client surge, pool leak, reconnect storm, workload shift | `DatabaseConnections`, `NetworkReceiveThroughput`, `NetworkTransmitThroughput` [S1] | Connection count jumping while work or latency worsens | Is traffic normal? Is demand broad or isolated? | -| Errors | Lock-contention and engine-failure symptoms rather than HTTP-style errors | Deadlocks, auth failures, transaction/log pressure, engine restarts | `Deadlocks`, engine logs, events [S1][S8] | Deadlocks rising or logs showing repeated failures | Is this an application contention issue or engine distress? | -| Latency | Storage and engine response slowness | I/O queueing, credit depletion, memory pressure, waits | `ReadLatency`, `WriteLatency`, `DiskQueueDepth`, PI load metrics [S1][S3][S4] | Latency rising with queue depth or wait-heavy DB load | Is slowness coming from storage, CPU, or waits? | -| Saturation | Headroom exhaustion in CPU, memory, storage, credits, or replicas | CPU exhaustion, low freeable memory, low storage, low burst credits, replica backlog | `CPUUtilization`, `FreeableMemory`, `FreeStorageSpace`, `BurstBalance`, `DBLoadRelativeToNumVCPUs`, `ReplicaLag` [S1][S3][S4] | Headroom collapsing toward outage or failover risk | Which capacity limit is closest to failing? | - -### Best-practice inference -- For RDS, `DBLoad` and `DBLoadRelativeToNumVCPUs` are more actionable for saturation triage than CPU alone because they distinguish CPU work from wait-bound overload. -- `ReplicaLag`, `OldestReplicationSlotLag`, `ReplicationSlotDiskUsage`, and the promoted `ChangeLogBytesUsed` signal are higher-value resilience indicators than raw network counters when the incident is replica freshness or CDC drift. - -## Telemetry sources - -### Confirmed by sources -| Source type | How collected | What it provides | Pros/cons | Common pitfalls | -|---|---|---|---|---| -| CloudWatch instance metrics | Native RDS -> CloudWatch | Core resource, network, I/O, connection, and engine-specific health metrics | Always available baseline; low friction; cross-engine | Some metrics are engine- or storage-type specific, so "no data" may mean unsupported rather than healthy zero [S1] | -| CloudWatch dimensions | Metric metadata from AWS dimensions | Instance, class, engine, region, and sometimes volume attribution | Enables safe global filters and group-bys | Do not assume every metric carries every dimension [S2] | -| Performance Insights metrics | PI -> CloudWatch integration | `DBLoad`, `DBLoadCPU`, `DBLoadNonCPU`, and relative load metrics | Best engine saturation surface | Requires PI enabled; coverage changes as PI support evolves [S3][S7] | -| Enhanced Monitoring | Agent-like OS telemetry on the RDS host | `cpuUtilization`, `diskIO`, `loadAverageMinute`, `memory`, `swap`, `tasks` | Best explanation layer for host/resource incidents | Must be explicitly enabled; OS units differ from CloudWatch instance metrics [S4] | -| RDS logs exported to CloudWatch Logs | Engine logs and audit/slow/error streams | Deadlocks, slow queries, auth failures, replication warnings | Best for root cause after metrics point to a problem | Export types differ per engine and often require parameter changes [S5][S6][S8] | -| RDS events | Service/control-plane events | Failovers, maintenance, configuration changes | Good incident context | Not a substitute for continuous performance metrics [S8] | - -### Best-practice inference -- "No data" usually means one of three things: the metric is engine-specific, the feature is disabled (Performance Insights, Enhanced Monitoring, log export), or the chosen namespace mapping in Tsuga is wrong. -- Stage 2 should verify whether Tsuga exports only CloudWatch-style metrics or also PI/Enhanced Monitoring-derived namespaces before overcommitting widgets to those surfaces. - -## Log intelligence (Stage 4 handoff) - -### Confirmed by sources -1. **Log sources matrix** - -| Source | Access path | Typical format | Structured vs unstructured | Evidence | -|---|---|---|---|---| -| MySQL error log | CloudWatch Logs group `/aws/rds/instance//error` | Plain-text engine error lines | Unstructured/semi-structured | [S5][S8] | -| MySQL slow query log | CloudWatch Logs group `/aws/rds/instance//slowquery` | MySQL slow-log text records | Semi-structured multiline | [S5] | -| MySQL general log | CloudWatch Logs group `/aws/rds/instance//general` | General query/event text lines | Semi-structured | [S5] | -| MySQL audit log | CloudWatch Logs group `/aws/rds/instance//audit` | Audit records from `MARIADB_AUDIT_PLUGIN` | Structured-ish text | [S5] | -| PostgreSQL log | RDS PostgreSQL log files and optional CloudWatch export | `stderr`-style PostgreSQL log lines or CSV-style when configured | Semi-structured or structured depending on configuration | [S6][S8] | - -2. **Known log formats** -- **MySQL slow query log**: multiline record with timestamp/user/host/query-time/lock-time/rows and SQL text body. Delimiters are label-based, not JSON; SQL body can span lines. [S5] -- **MySQL error/general log**: line-oriented text; timestamps are engine-generated and may include thread/session details. [S5] -- **PostgreSQL log (`stderr`)**: text line with timestamp, optional PID/session fields, severity, and message; exact prefix depends on `log_line_prefix`. [S6] -- **PostgreSQL CSV log**: comma-separated fields when `csvlog` is enabled; safer for parsing but not guaranteed on every RDS instance. [S6] - -3. **Candidate query filters for Stage 4** -- Precise: `source:rds AND context.dbinstanceidentifier:* AND (context.log.type:error OR context.log.type:slowquery OR context.log.type:postgresql)` - Rationale: targets exported RDS engine logs if ingestion normalized log types. - Risk: `context.log.type` may not exist yet. -- Fallback: `source:rds AND context.dbinstanceidentifier:*` - Rationale: broad enough to catch all exported RDS logs after instance enrichment. - Risk: may mix multiple engines and log types into one stream. - -4. **Attribute mapping hints** - -| Raw field | Suggested Tsuga key | Confidence | Notes | -|---|---|---|---| -| DB instance identifier | `context.dbinstanceidentifier` | High | Confirmed on live metrics and the best candidate for multi-instance routing | -| Engine name | `context.enginename` | Medium | Confirmed on live metrics and useful for engine-specific parsing branches | -| Log type (`error`,`general`,`slowquery`,`postgresql`) | `context.log.type` | Medium | Best routing split if available | -| User | `context.db.user` | Medium | Present in slow/general/audit or PostgreSQL prefix depending on config | -| Database name | `context.db.name` | Medium | Often present in PostgreSQL or slow query contexts | -| Process / thread id | `context.process.pid` | Medium | Useful for correlating repeated failure lines | -| Query duration | `context.db.query.duration_ms` | Medium | Slow query and statement-duration parsing target | -| SQL text / statement | `context.db.statement` | Low | High-cardinality, likely keep as raw body not routing field | - -5. **Parsing risks** -- MySQL slow query logs are multiline and include SQL payloads, so naïve line parsers will fragment one event into many. -- PostgreSQL prefix format varies with `log_line_prefix`; a route must not assume one exact prefix unless the environment confirms it. -- PostgreSQL can emit either text or CSV-style logs depending on configuration; dual-format handling may be required. [S6] -- Engine log types are optional exports; absence may mean disabled export rather than no problems. -- CloudWatch log group naming is stable, but downstream ingestion fields (`source`, `service`, `context.log.type`) are not yet confirmed in Tsuga. - -### Best-practice inference -- Stage 4 should branch parser selection first by engine (`context.enginename`) and then by log type because MySQL/PostgreSQL formats differ materially. -- Start with line-oriented parsing for error/general logs and a dedicated multiline strategy for MySQL slow-query logs. -- If PostgreSQL CSV logging is enabled, prefer CSV parsing over grok because field boundaries are more reliable. - -## Caveats and footguns -- **[fleet-health]** `CPUUtilization` can stay moderate while `DBLoadNonCPU` climbs, so CPU alone does not clear the database of saturation. (S1, S3) -- **[fleet-health]** `FreeableMemory` is reclaimable memory, not "unused" memory; low values are actionable, high cache use alone is not. (S1) -- **[fleet-health]** Enhanced Monitoring `memory` values are in kilobytes and should not be mixed directly with CloudWatch byte metrics without normalization. (S4) -- **[fleet-health]** Enhanced Monitoring must be enabled; missing host metrics usually means the feature is off. (S4) -- **[traffic-connections]** `DatabaseConnections` can rise from connection storms or pool leaks even when throughput is flat. (Inference) -- **[traffic-connections]** Network throughput is supportive context, not a direct success metric; low network with high load can still mean lock contention. (Inference) -- **[traffic-connections]** Grouping by raw client/session identifiers would explode cardinality; prefer instance, engine, class, and region. (Inference) -- **[latency-io]** `ReadLatency` and `WriteLatency` are averages; they are not percentiles and can hide tail pain. (S1) -- **[latency-io]** `DiskQueueDepth` rising with flat IOPS often means storage cannot keep up, not that the app is idle. (S1) -- **[latency-io]** Read/write throughput without IOPS can be misleading because large sequential operations and small random operations behave very differently. (Inference) -- **[latency-io]** `EBSByteBalance%` and `EBSIOBalance%` are storage-type dependent and not emitted for every deployment. (S1) -- **[storage-capacity]** `BurstBalance` is forward-looking; trouble starts when credits keep draining under steady demand, not only when it hits zero. (S1) -- **[storage-capacity]** `FreeStorageSpace` can collapse from transaction-log retention or stalled replication, not only from user tables growing. (S1, S6) -- **[storage-capacity]** `TransactionLogsDiskUsage` is SQL Server-specific and should be gated. (S1) -- **[storage-capacity]** `ReplicationSlotDiskUsage` and `OldestReplicationSlotLag` are PostgreSQL-specific and absent on other engines. (S1) -- **[storage-capacity]** `BinLogDiskUsage` is MySQL/MariaDB specific and was not present in the live Stage 2 Tsuga catalog for this workspace; keep it gated and do not treat missing data as healthy zero for PostgreSQL or SQL Server. (S1 + Stage 2 discovery) -- **[engine-pressure]** `DBLoad`, `DBLoadCPU`, and `DBLoadNonCPU` depend on Performance Insights; widgets must explain missing-feature behavior. (S3, S7) -- **[engine-pressure]** `DBLoadCPU + DBLoadNonCPU` is a useful split, but the sum may not match every downstream export perfectly if Tsuga exports a transformed PI surface. (Inference) -- **[engine-pressure]** `DBLoadRelativeToNumVCPUs` is already a ratio; do not apply rate or treat it as a counter. (S3) -- **[engine-pressure]** `Deadlocks` is an engine symptom, not an availability metric; a small but sudden increase can still matter. (S1) -- **[replication-resilience]** `ReplicaLag` should be treated as optional for non-replica deployments; missing lag does not automatically mean healthy replication. (S1) -- **[replication-resilience]** Replication metrics are strongly engine- and topology-dependent; do not assume a writer-only instance should emit them. (S1) -- **[replication-resilience]** PostgreSQL slot lag metrics point to CDC/logical consumers, not ordinary read replicas. (S1) -- **[replication-resilience]** Log-space metrics and lag metrics together are more meaningful than either alone; slot lag without disk growth can be transient, disk growth without lag can be retention policy. (Inference) -- **[fleet-health, engine-pressure]** Mixing CloudWatch, Enhanced Monitoring, and Performance Insights into one formula widget can create unit confusion unless the derived signal has one clear purpose. (Inference) - -## Confirmed Tsuga prefixes -- `aws_rds_` — **CONFIRMED** (Stage 2 live discovery returned 60 metrics with underscore-separated names such as `aws_rds_cpu_utilization` and `aws_rds_db_load`). -- `aws_rds` — **CONFIRMED ROOT** (the namespace root is valid, but the live catalog uses snake_case names rather than dot-separated metric segments). - -## Discovery status -Discovery: completed in Stage 2. -- Authentication succeeded after the user refreshed `.env`, and Tsuga catalog discovery returned 60 live `aws_rds_*` metrics. -- The live catalog uses underscore-separated names, so Stage 1 placeholder references like `aws_rds.CPUUtilization` were reconciled to names such as `aws_rds_cpu_utilization`. -- Representative live metrics expose usable attribute keys including `context.dbinstanceidentifier`, `context.enginename`, `context.databaseclass`, `context.cloud.region`, `context.dbclusteridentifier`, and `context.role`. -- All spot-checked live metrics currently report as `type: summary` with `temporality: cumulative`, so rate/counter assumptions were removed from the plan where they were unsafe. -- `BinLogDiskUsage` remains documented by AWS but was not found in the live Tsuga catalog for this workspace; the bundle keeps it gated and uses `ChangeLogBytesUsed` as the promoted storage-pressure companion signal where applicable. -- Scalar freshness spot-checks against the aggregation API returned `500 INTERNAL_ERROR`, so presence and schema are confirmed, but point-value validation is still inconclusive. - -## Top sources -1. https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/rds-metrics.html - Why: canonical CloudWatch metric catalog for RDS, including core and engine-specific metrics. -2. https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/dimensions.html - Why: authoritative CloudWatch dimension list for RDS metrics and safe grouping assumptions. -3. https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_PerfInsights.Cloudwatch.html - Why: defines Performance Insights CloudWatch metrics such as `DBLoad` and CPU/non-CPU splits. -4. https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_Monitoring-Available-OS-Metrics.html - Why: Enhanced Monitoring OS metric catalog and units. -5. https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_Monitor_Logs_Events.html - Why: RDS logs/events overview and where logs fit in operational monitoring. -6. https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_LogAccess.html - Why: general RDS log-file access model and export context. -7. https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_LogAccess.MySQLDB.PublishtoCloudWatchLogs.html - Why: exact MySQL CloudWatch log-group naming and export prerequisites. -8. https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_LogAccess.Concepts.PostgreSQL.html - Why: PostgreSQL log concepts, log types, and export behavior. -9. https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_LogAccess.Concepts.PostgreSQL.Query_Logging.html - Why: PostgreSQL query logging details relevant to Stage 4 route design. -10. https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_PerfInsights.Overview.html - Why: performance-insight semantics and feature context for DB load interpretation. - ---- - -**Citation key** -- [S1] https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/rds-metrics.html -- [S2] https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/dimensions.html -- [S3] https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_PerfInsights.Cloudwatch.html -- [S4] https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_Monitoring-Available-OS-Metrics.html -- [S5] https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_LogAccess.MySQLDB.PublishtoCloudWatchLogs.html -- [S6] https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_LogAccess.Concepts.PostgreSQL.html -- [S7] https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_PerfInsights.Overview.html -- [S8] https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_Monitor_Logs_Events.html diff --git a/plugins/tsuga/skills/knowledge-technology/references/aws-rds/metrics.csv b/plugins/tsuga/skills/knowledge-technology/references/aws-rds/metrics.csv deleted file mode 100644 index d3224c1..0000000 --- a/plugins/tsuga/skills/knowledge-technology/references/aws-rds/metrics.csv +++ /dev/null @@ -1,30 +0,0 @@ -theme,name,confirmed,definition,unit,type,tsuga_metric_name,widget_suggestions,aggregation,post_function,group_by,required_context_fields,optional_context_fields,absence_means,cardinality_footguns,variants,citations -Availability/Health,CPUUtilization,true,CPU utilization of the DB instance; fast fleet-health signal but not sufficient alone for database saturation decisions.,%,summary,aws_rds_cpu_utilization,query-value|timeseries|top-list,max,none,context.dbinstanceidentifier:20|context.databaseclass:12,context.env|context.team,context.enginename|context.cloud.region|context.databaseclass|context.dbclusteridentifier,Metric absent or namespace mismatch in Tsuga,Safe when grouped by instance/class,cloudwatch=CPUUtilization,https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/rds-metrics.html -Availability/Health,FreeableMemory,true,Amount of available memory that can be reclaimed; low headroom often precedes I/O and latency pain.,By,summary,aws_rds_freeable_memory,query-value|timeseries|top-list,min,none,context.dbinstanceidentifier:20|context.databaseclass:12,context.env|context.team,context.enginename|context.cloud.region|context.databaseclass|context.dbclusteridentifier,Metric absent or namespace mismatch in Tsuga,Do not combine with high-card SQL digests,cloudwatch=FreeableMemory,https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/rds-metrics.html -Availability/Health,DiskQueueDepth,true,Queued disk operations waiting on storage; useful with read/write latency and IOPS.,count,summary,aws_rds_disk_queue_depth,query-value|timeseries|top-list,max,none,context.dbinstanceidentifier:20|context.cloud.region:10,context.env|context.team,context.enginename|context.databaseclass|context.dbclusteridentifier,Metric absent or unsupported for this surface,Queue depth by tiny transient entities is noisy; stay at instance granularity,cloudwatch=DiskQueueDepth,https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/rds-metrics.html -Availability/Health,DatabaseConnections,true,Number of client connections to the DB instance; first demand and pool-leak signal.,count,summary,aws_rds_database_connections,query-value|timeseries|top-list,max,none,context.dbinstanceidentifier:20|context.enginename:8,context.env|context.team,context.databaseclass|context.cloud.region|context.dbclusteridentifier|context.role,No recent connections or namespace mismatch,Do not group by user/session unless Stage 2 proves bounded fields,cloudwatch=DatabaseConnections|live=aws_rds_database_connections,https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/rds-metrics.html -Throughput/Usage,NetworkReceiveThroughput,true,Bytes received from clients and peers per second at the instance boundary.,By/s,summary,aws_rds_network_receive_throughput,query-value|timeseries|top-list,sum,none,context.dbinstanceidentifier:20|context.cloud.region:10,context.env|context.team,context.enginename|context.databaseclass|context.dbclusteridentifier,Feature absent or namespace mismatch,Use instance or region rather than client-address groupings,cloudwatch=NetworkReceiveThroughput,https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/rds-metrics.html -Throughput/Usage,NetworkTransmitThroughput,true,Bytes transmitted by the DB instance per second; pairs with receive throughput for traffic shape analysis.,By/s,summary,aws_rds_network_transmit_throughput,query-value|timeseries|top-list,sum,none,context.dbinstanceidentifier:20|context.cloud.region:10,context.env|context.team,context.enginename|context.databaseclass|context.dbclusteridentifier,Feature absent or namespace mismatch,Stay at instance granularity for overview widgets,cloudwatch=NetworkTransmitThroughput,https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/rds-metrics.html -Performance/Latency,ReadIOPS,true,Read operations per second on storage for the DB instance.,count/s,summary,aws_rds_read_iops,query-value|timeseries|top-list,sum,none,context.dbinstanceidentifier:20|context.cloud.region:10,context.env|context.team,context.enginename|context.databaseclass|context.dbclusteridentifier,Read path idle or namespace mismatch,Interpret with throughput because operation size matters,cloudwatch=ReadIOPS,https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/rds-metrics.html -Performance/Latency,WriteIOPS,true,Write operations per second on storage for the DB instance.,count/s,summary,aws_rds_write_iops,query-value|timeseries|top-list,sum,none,context.dbinstanceidentifier:20|context.cloud.region:10,context.env|context.team,context.enginename|context.databaseclass|context.dbclusteridentifier,Write path idle or namespace mismatch,Write-heavy bursts can look worse than reads even when latency is stable,cloudwatch=WriteIOPS,https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/rds-metrics.html -Performance/Latency,ReadLatency,true,Average latency per read I/O operation.,s,summary,aws_rds_read_latency,query-value|timeseries|top-list,max,none,context.dbinstanceidentifier:20|context.cloud.region:10,context.env|context.team,context.enginename|context.databaseclass|context.dbclusteridentifier,Metric absent or no read I/O in range,Average latency is not a percentile; do not over-interpret tiny spikes,cloudwatch=ReadLatency,https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/rds-metrics.html -Performance/Latency,WriteLatency,true,Average latency per write I/O operation.,s,summary,aws_rds_write_latency,query-value|timeseries|top-list,max,none,context.dbinstanceidentifier:20|context.cloud.region:10,context.env|context.team,context.enginename|context.databaseclass|context.dbclusteridentifier,Metric absent or no write I/O in range,Needs context from queue depth and log pressure,cloudwatch=WriteLatency,https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/rds-metrics.html -Performance/Latency,ReadThroughput,true,Bytes read from disk per second.,By/s,summary,aws_rds_read_throughput,query-value|timeseries|top-list,sum,none,context.dbinstanceidentifier:20|context.cloud.region:10,context.env|context.team,context.enginename|context.databaseclass|context.dbclusteridentifier,Read workload absent or namespace mismatch,Throughput without IOPS obscures I/O size distribution,cloudwatch=ReadThroughput,https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/rds-metrics.html -Performance/Latency,WriteThroughput,true,Bytes written to disk per second.,By/s,summary,aws_rds_write_throughput,query-value|timeseries|top-list,sum,none,context.dbinstanceidentifier:20|context.cloud.region:10,context.env|context.team,context.enginename|context.databaseclass|context.dbclusteridentifier,Write workload absent or namespace mismatch,Throughput alone cannot distinguish healthy batching from saturation,cloudwatch=WriteThroughput,https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/rds-metrics.html -Capacity/Saturation,FreeStorageSpace,true,Remaining storage on the DB instance; time-to-outage signal for growth and retention problems.,By,summary,aws_rds_free_storage_space,query-value|timeseries|top-list,min,none,context.dbinstanceidentifier:20|context.databaseclass:12,context.env|context.team,context.enginename|context.cloud.region|context.databaseclass|context.dbclusteridentifier,Metric absent or namespace mismatch in Tsuga,Free storage grouped by too many dimensions adds no value; keep instance-level,cloudwatch=FreeStorageSpace,https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/rds-metrics.html -Capacity/Saturation,BurstBalance,true,Percentage of burst-credit balance remaining for burstable storage surfaces.,%,summary,aws_rds_burst_balance,query-value|timeseries,min,none,,context.env|context.team,context.enginename|context.databaseclass|context.cloud.region,Storage type may not emit this metric,Do not assume absence means healthy zero; many storage types do not use credits,cloudwatch=BurstBalance,https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/rds-metrics.html -Capacity/Saturation,EBSByteBalance%,true,Percentage of throughput credits remaining for supported EBS-backed storage.,%,summary,aws_rds_ebs_byte_balance,query-value|timeseries,min,none,,context.env|context.team,context.enginename|context.cloud.region|context.databaseclass,Metric exists only on certain storage configurations,Volume-scoped variants should stay gated unless Stage 3 confirms the field,cloudwatch=EBSByteBalance%,https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/rds-metrics.html -Capacity/Saturation,EBSIOBalance%,true,Percentage of I/O credits remaining for supported EBS-backed storage.,%,summary,aws_rds_ebsio_balance,query-value|timeseries,min,none,,context.env|context.team,context.enginename|context.cloud.region|context.databaseclass,Metric exists only on certain storage configurations,Stage 2 confirmed the live name normalizes the percent sign away,cloudwatch=EBSIOBalance%,https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/rds-metrics.html -Errors/Failures,Deadlocks,true,"Deadlock events reported for the DB instance and carried into Tsuga as a summary metric, so use windowed sums rather than counter-rate math.",count,summary,aws_rds_deadlocks,query-value|timeseries|top-list,sum,none,context.dbinstanceidentifier:20|context.enginename:8,context.env|context.team,context.databaseclass|context.cloud.region|context.dbclusteridentifier|context.role,No deadlocks in the selected window,Use instance or engine only; per-query groupings need logs or database-insights surfaces,cloudwatch=Deadlocks|live=aws_rds_deadlocks,https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/rds-metrics.html -Capacity/Saturation,DBLoad,true,Average active sessions running or waiting; best engine saturation signal when Performance Insights is enabled.,count,summary,aws_rds_db_load,query-value|timeseries|top-list,average,none,context.dbinstanceidentifier:20|context.enginename:8,context.env|context.team,context.cloud.region|context.databaseinsightsmode,Performance Insights not enabled or not exported,Do not group by SQL digest on overview dashboards,pi=DBLoad|live=aws_rds_db_load,https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_PerfInsights.Cloudwatch.html -Capacity/Saturation,DBLoadCPU,true,Portion of DB load running on CPU; pairs with DBLoadNonCPU to separate compute saturation from waits.,count,summary,aws_rds_db_load_cpu,query-value|timeseries|top-list,average,none,context.dbinstanceidentifier:20|context.enginename:8,context.env|context.team,context.cloud.region|context.databaseinsightsmode,Performance Insights not enabled or not exported,Needs DBLoad for full interpretation,pi=DBLoadCPU|live=aws_rds_db_load_cpu,https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_PerfInsights.Cloudwatch.html -Capacity/Saturation,DBLoadNonCPU,true,Portion of DB load waiting on non-CPU resources; high values often mean lock or I/O waits.,count,summary,aws_rds_db_load_non_cpu,query-value|timeseries|top-list,average,none,context.dbinstanceidentifier:20|context.enginename:8,context.env|context.team,context.cloud.region|context.databaseinsightsmode,Performance Insights not enabled or not exported,Wait-heavy load needs gating if PI fields are absent,pi=DBLoadNonCPU|live=aws_rds_db_load_non_cpu,https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_PerfInsights.Cloudwatch.html -Capacity/Saturation,DBLoadRelativeToNumVCPUs,true,Ratio of DB load to available vCPU capacity; already normalized for quick saturation checks.,1,summary,aws_rds_db_load_relative_to_num_vcp_us,query-value|timeseries,max,none,,context.env|context.team,context.enginename|context.cloud.region|context.databaseinsightsmode,Performance Insights not enabled or not exported,Do not apply rate or per-second transforms to this ratio,pi=DBLoadRelativeToNumVCPUs|live=aws_rds_db_load_relative_to_num_vcp_us,https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_PerfInsights.Cloudwatch.html -Capacity/Saturation,TransactionLogsDiskUsage,true,Disk space used by transaction logs; especially relevant for SQL Server and write-heavy workloads.,By,summary,aws_rds_transaction_logs_disk_usage,query-value|timeseries|top-list,max,none,context.dbinstanceidentifier:20|context.enginename:8,context.env|context.team,context.databaseclass|context.cloud.region|context.dbclusteridentifier,Engine does not emit this metric or namespace mismatch,Should not be treated as universal across engines,cloudwatch=TransactionLogsDiskUsage,https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/rds-metrics.html -Throughput/Usage,TransactionLogsGeneration,true,Rate at which transaction logs are generated; indicates write amplification and log churn.,By/s,summary,aws_rds_transaction_logs_generation,query-value|timeseries|top-list,sum,none,context.dbinstanceidentifier:20|context.enginename:8,context.env|context.team,context.databaseclass|context.cloud.region|context.dbclusteridentifier,Engine does not emit this metric or namespace mismatch,Keep to instance-level grouping unless Stage 3 confirms richer dimensions,cloudwatch=TransactionLogsGeneration,https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/rds-metrics.html -Replication/Resilience,ReplicaLag,true,Replica delay surface currently confirmed in Tsuga for the RDS to Aurora PostgreSQL replication path; use it as the live replication-lag proxy until a generic replica metric appears.,s,summary,aws_rds_rds_to_aurora_postgre_sql_replica_lag,query-value|timeseries|top-list,max,none,context.dbinstanceidentifier:20|context.cloud.region:10,context.env|context.team,context.enginename|context.role|context.dbclusteridentifier,No replica-lag metric for this topology or namespace mismatch,Current live signal is Aurora PostgreSQL specific; keep generic title but note the live scope,cloudwatch=ReplicaLag|live=aws_rds_rds_to_aurora_postgre_sql_replica_lag,https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/rds-metrics.html -Replication/Resilience,ChangeLogBytesUsed,true,Bytes currently retained in the engine change-log surface exposed by the live aws_rds catalog; use as the confirmed log-retention pressure signal where generic binlog storage is absent.,By,summary,aws_rds_change_log_bytes_used,query-value|timeseries|top-list,max,none,context.dbclusteridentifier:20|context.enginename:8,context.env|context.team,context.cloud.region|context.role,Engine does not emit this metric or namespace mismatch,Interpret as a live log-retention proxy rather than a universal cross-engine binlog metric,live=aws_rds_change_log_bytes_used,https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/rds-metrics.html -Replication/Resilience,BinLogDiskUsage,false,Disk space consumed by binary logs on MySQL or MariaDB family instances.,By,summary,Unknown,query-value|timeseries|top-list,max,none,context.dbinstanceidentifier:20|context.enginename:8,context.env|context.team,context.cloud.region|context.role|context.dbclusteridentifier,No generic MySQL or MariaDB binlog disk metric was found in the live Tsuga catalog; keep this as documented but currently unavailable.,MySQL-family only and currently missing in the live catalog,cloudwatch=BinLogDiskUsage,https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/rds-metrics.html -Replication/Resilience,OldestReplicationSlotLag,true,Amount of lag for the oldest PostgreSQL replication slot; indicates stalled logical consumers.,By,summary,aws_rds_oldest_replication_slot_lag,query-value|timeseries|top-list,max,none,context.dbinstanceidentifier:20|context.enginename:8,context.env|context.team,context.cloud.region|context.role|context.dbclusteridentifier,PostgreSQL logical replication not enabled or namespace mismatch,PostgreSQL-specific and easy to misread as ordinary replica lag,cloudwatch=OldestReplicationSlotLag|live=aws_rds_oldest_replication_slot_lag,https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/rds-metrics.html -Replication/Resilience,ReplicationSlotDiskUsage,true,Disk used by PostgreSQL replication slots that retain WAL files.,By,summary,aws_rds_replication_slot_disk_usage,query-value|timeseries|top-list,max,none,context.dbinstanceidentifier:20|context.enginename:8,context.env|context.team,context.cloud.region|context.role|context.dbclusteridentifier,PostgreSQL logical replication not enabled or namespace mismatch,Keep gated; absent on most non-PostgreSQL engines,cloudwatch=ReplicationSlotDiskUsage|live=aws_rds_replication_slot_disk_usage,https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/rds-metrics.html -Cost,NOT_APPLICABLE,false,"RDS operational metrics do not directly expose billing totals; use throughput, storage usage, instance class, and log growth as cost proxies until billing telemetry is integrated.",Unknown,Unknown,Unknown,query-value,average,none,,context.env|context.team,context.databaseclass|context.enginename,Cost metrics require separate billing export data,Safe,proxy=FreeStorageSpace|proxy=TransactionLogsGeneration,https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/rds-metrics.html diff --git a/plugins/tsuga/skills/knowledge-technology/references/aws-rds/queries.md b/plugins/tsuga/skills/knowledge-technology/references/aws-rds/queries.md deleted file mode 100644 index 939fa45..0000000 --- a/plugins/tsuga/skills/knowledge-technology/references/aws-rds/queries.md +++ /dev/null @@ -1,55 +0,0 @@ -# AWS RDS - -Managed relational DB (Postgres / MySQL / MariaDB / SQL Server / Oracle / Aurora). Engine-layer behavior = `postgres.md` / `mysql.md`. This file covers RDS-specific signals. - -## Incident shapes - -- **CPU saturation** — query or vacuum pressure → check `aws_rds_db_load`, `aws_rds_db_load_cpu`, Performance Insights -- **Storage exhaustion** — `aws_rds_free_storage_space → 0` blocks writes, `aws_rds_write_iops` collapses -- **IOPS / burst credit depletion** — gp2 `aws_rds_burst_balance` crash OR gp3 `aws_rds_ebsio_balance` / `aws_rds_ebs_byte_balance` → IOPS cliff -- **Replication lag** — `aws_rds_rds_to_aurora_postgre_sql_replica_lag` climbs under primary write burst -- **Connection ceiling** — `aws_rds_database_connections` near engine `max_connections` (usually app-side) -- **Transaction-log bloat** — `aws_rds_transaction_logs_disk_usage` grows when replication slots orphan - -## Key metrics - -| Metric | Unit | Signal | -|---|---|---| -| `aws_rds_cpu_utilization` | % | Host CPU; decompose via aws_rds_db_load_cpu vs aws_rds_db_load_non_cpu | -| `aws_rds_database_connections` | count | Compare to engine `max_connections` | -| `aws_rds_free_storage_space` | bytes | Zero = writes blocked. Missing? infer from aws_rds_write_iops collapse | -| `aws_rds_read_iops` / `aws_rds_write_iops` | ops/s | Workload intensity | -| `aws_rds_read_latency` / `aws_rds_write_latency` | seconds | Per-op latency | -| `aws_rds_burst_balance` | % | gp2 only; zero = IOPS throttled to baseline | -| `aws_rds_ebs_byte_balance` / `aws_rds_ebsio_balance` | % | gp3/Aurora credit pools | -| `aws_rds_db_load` / `aws_rds_db_load_cpu` / `aws_rds_db_load_non_cpu` | avg sessions | Active sessions; CPU vs wait decomposition | -| `aws_rds_rds_to_aurora_postgre_sql_replica_lag` | seconds | Read-replica staleness | -| `aws_rds_transaction_logs_disk_usage` | bytes | Grows when replication slot stuck | -| `aws_rds_oldest_replication_slot_lag` | bytes | Which slot is the bottleneck | -| `aws_rds_deadlocks` | count | Engine-layer conflicts | -| `aws_rds_freeable_memory` | bytes | Cache-pressure signal | - -## Derived signals - -- `(aws_rds_read_iops + aws_rds_write_iops) / provisioned_IOPS` — IOPS saturation. > 0.9 sustained = throttling. -- `aws_rds_db_load_cpu / aws_rds_db_load` — high → CPU-bound queries; low with high aws_rds_db_load → locks/I/O waits. -- Write throttle pattern: aws_rds_write_iops collapse + aws_rds_write_latency spike + low `aws_rds_free_storage_space` OR low balance. - -## Log patterns - -RDS events: - -- `DB instance storage auto-scaled` — autoscaling fired; often precedes "healthy" verdict -- `The database instance ran out of storage space` — hard failure -- `DB instance restarted` — engine restart, not user-initiated -- `Read Replica has fallen behind` — replication lag alert -- `Multi-AZ failover started` / `completed` — failover event -- `Logical replication slot was dropped` — explains sudden `aws_rds_transaction_logs_disk_usage` drop - -## Gotchas - -- `aws_rds_free_storage_space` drops below threshold even with autoscaling enabled; auto-recovery ends the investigation. -- RDS event messages lag real events up to 1-2 min. Don't pin 1-min-precision timelines on event timestamps. -- `aws_rds_burst_balance` applies to gp2 only. Flat 100% on gp3 means "not applicable," not "healthy." -- `aws_rds_rds_to_aurora_postgre_sql_replica_lag` can be stale if receiver scrape stalls. Cross-check with logical-slot lag via log. -- High `aws_rds_db_load` with low CPU usually = io_wait or lock wait. Use Performance Insights' top-waits breakdown. diff --git a/plugins/tsuga/skills/knowledge-technology/references/aws-sqs/README.md b/plugins/tsuga/skills/knowledge-technology/references/aws-sqs/README.md deleted file mode 100644 index 58d5298..0000000 --- a/plugins/tsuga/skills/knowledge-technology/references/aws-sqs/README.md +++ /dev/null @@ -1,244 +0,0 @@ -# AWS SQS Integration Context Bundle - -## Metadata -**Technology:** AWS SQS -**Deployment:** managed -**Environment:** prod -**Persona:** SRE Dev and ops -**Telemetry preference:** mixed -**Integration scope:** core service only -**Primary use-case:** reliability and performance - -## How to use this bundle -- Use `01_aws-sqs_metrics.csv` as the source of truth for queue metrics, safe aggregations, and bounded drilldowns. -- Use `02_aws-sqs_dashboard_plan.yaml` for the dashboard structure, derived signals, gating rules, notes, and coverage map. -- Use `03_aws-sqs_state.yaml` for machine-readable unknowns, field-mapping status, and Stage 2 verification priorities. -- Use `04_aws-sqs_memory.md` for the human-readable Stage 1 handoff and implementation tradeoffs. -- Stage 2 will create `05_aws-sqs_metric_catalog.csv` as the discovered Tsuga inventory used for reconciliation, description curation, and coverage checks. -- Stage 4 should read this file's `Log intelligence (Stage 4 handoff)` section and `03_aws-sqs_state.yaml` `log_intel` first before attempting any log route. - -## What it is and what "good" looks like -### Confirmed by sources -Amazon SQS is AWS's managed pull-based messaging service. Producers enqueue work, consumers poll and process it, and queue health is inferred from backlog depth, message age, receive/delete activity, and visibility behavior rather than CPU or host metrics. For standard queues, "good" means visible backlog stays bounded for the workload, oldest-message age does not grow persistently, delayed messages remain intentional, and consumers delete messages at roughly the pace they receive them. For FIFO queues, good also means deduplication is low unless retries are expected and message-group parallelism is high enough that one hot group does not serialize the whole queue. AWS documents queue-depth, age, throughput, deduplication, and fair-queue metrics directly in the CloudWatch metrics guide, and it describes how long polling, visibility timeout, dead-letter handling, and FIFO delivery logic shape incident response. - -Paging intent for dashboards: first determine whether the incident is backlog growth, consumer inefficiency, or queue-shape skew. For backlog growth, start with `backlog-health`. For slow or wasteful polling, start with `consumer-polling`. For ordering or noisy-neighbor issues in FIFO or fair queues, start with `fifo-fairness`. For redrive or poison-message suspicion, use `backlog-health` first, then verify DLQ configuration outside this bundle because DLQ depth is a separate queue surface. - -### Best-practice inference -- Incident shape 1: queue depth and oldest age rise together. Start with `backlog-health`, then compare send, receive, and delete rates. -- Incident shape 2: backlog is low but empty receives surge. Start with `consumer-polling`; this usually points to short polling, over-provisioned pollers, or consumers hitting mostly empty shards of a standard queue. -- Incident shape 3: only some tenants or message groups degrade. Start with `fifo-fairness`; noisy groups or too-few in-flight groups usually indicate poor grouping strategy or one blocked ordered lane. - -## Key concepts -### Glossary -| Term | Definition | Operational meaning | Dashboard section | -|---|---|---|---| -| Queue | Durable SQS buffer for messages | Main isolation boundary for backlog and throughput | backlog-health | -| Standard queue | SQS queue with best-effort ordering and at-least-once delivery | Backlog and age matter more than strict ordering | backlog-health | -| FIFO queue | SQS queue with ordered processing within message groups and deduplication support | Group-level skew and dedup behavior become first-class signals | fifo-fairness | -| Visible message | Message available to be received | Immediate work waiting on consumers | backlog-health | -| In-flight message | Received but not yet deleted message | Consumer concurrency and visibility-timeout pressure | backlog-health | -| Delayed message | Message hidden until delay expires | Scheduled backlog that can mask true ready work | backlog-health | -| Oldest message age | Approximate age of the oldest non-deleted message | Direct user-visible latency proxy for queue drain health | backlog-health | -| Receive | Consumer poll that returns one or more messages | Core consumption activity | throughput-balance | -| Delete | Consumer acknowledgement that removes a message | Completion signal; lagging deletes imply retries or failures | throughput-balance | -| Empty receive | Receive call that returns no messages | Poll inefficiency or false-empty behavior | consumer-polling | -| Long polling | Receive mode that waits for messages before returning | Reduces empty receives and polling waste | consumer-polling | -| Short polling | Receive mode that samples servers and may return empty sooner | Can inflate empty receives and hide residual backlog | consumer-polling | -| Visibility timeout | Period after receive during which the message is hidden | Too short causes duplicates; too long stalls retries | backlog-health | -| DLQ | Dead-letter queue for repeatedly failed messages | Required context when oldest age rises due to poison messages | backlog-health | -| Poison message | Message that repeatedly fails processing | Creates age growth and repeated receives without matching deletes | backlog-health | -| Message group | Ordered lane inside a FIFO queue | Throughput and head-of-line blocking are group-dependent | fifo-fairness | -| Deduplication | FIFO suppression of duplicate sends within a dedup window | Rising deduplicated sends usually mean producer retry churn | fifo-fairness | -| Fair queue | Standard queue behavior that isolates noisy tenants using `MessageGroupId` | Quiet-group backlog metrics reveal tenant skew | fifo-fairness | -| Noisy group | Tenant or message group dominating queue work | Explains partial degradation while overall queue looks acceptable | fifo-fairness | -| Quiet group | Non-noisy groups in a fair queue | Useful baseline for whether only some tenants are hurt | fifo-fairness | -| Sent message size | Payload size of sent messages | Large payload shifts throughput and cost; can slow consumers | throughput-balance | -| Approximate metric | SQS CloudWatch metric based on distributed queue internals | Trends matter more than exact single-point values | backlog-health | - -### Concept Map -```text -Producer -> sends -> SQS queue (why: all downstream work begins as queued backlog) -SQS queue -> exposes -> visible messages (why: this is immediate ready work) -SQS queue -> exposes -> delayed messages (why: scheduled work should not be mistaken for consumer lag) -Consumer -> receives -> messages (why: receive rate shows pull demand) -Receive -> starts -> visibility timeout (why: in-flight work can block ordered progress) -Consumer -> deletes -> message (why: delete is the completion acknowledgement) -Missing delete -> leaves -> message eligible for redelivery (why: duplicate processing risk rises) -Poison message -> causes -> repeated receives without delete (why: age and in-flight counts grow) -Visibility timeout too short -> causes -> duplicate work (why: the same message reappears before processing finishes) -Visibility timeout too long -> delays -> retry and redrive (why: failed work stays hidden too long) -Oldest visible message age -> represents -> user-visible waiting time (why: backlog latency matters more than raw queue size alone) -Send rate -> competes with -> receive and delete rate (why: imbalance creates persistent backlog) -Long polling -> reduces -> empty receives (why: consumers wait instead of hammering the queue) -Short polling -> increases -> false empty responses (why: consumers may miss available messages) -FIFO queue -> partitions ordering by -> message group ID (why: concurrency depends on group distribution) -Message group ID -> bounds -> in-flight parallelism per group (why: one hot group can serialize throughput) -Deduplication ID -> suppresses -> duplicate send attempts (why: retries may be acknowledged but not enqueued) -Fair queue -> classifies -> noisy and quiet groups (why: dashboards can separate tenant-local from global pain) -Noisy group -> inflates -> quiet-group age or depth imbalance (why: some tenants are starved before the whole queue fails) -DLQ policy -> removes -> repeatedly failing messages from source queue (why: source age can recover once poison work is isolated) -context.queuename -> maps -> operational ownership (why: queue is the primary drilldown) -context.cloud.region -> maps -> blast radius (why: regional incidents can affect many queues) -context.cloud.account.id -> maps -> account boundary (why: multi-account estates need separate filtering) -context.env + context.team -> map -> runtime ownership (why: dashboards must support environment and team pivots) -``` - -### Entities and dimensions -#### Confirmed by sources -| Entity/Dimension | Why useful | Cardinality risk | Safe top-N | Do NOT group-by guidance | -|---|---|---|---|---| -| `context.queuename` | Primary queue drilldown and likely first global filter | Low to medium | 20 | Use this before any exporter ARN split | -| `context.cloud.region` | Regional blast radius for AWS outages or queue concentration | Low | 12 | Prefer as a global filter in single-region deployments | -| `context.cloud.account.id` | Separates prod accounts and shared-platform accounts | Low | 10 | Use for estate views, not every chart | -| `context.env` | Environment segmentation | Low | 5 | Global filter only | -| `context.team` | Ownership routing | Low | 10 | Global filter only | -| `context.cloud.provider` | Confirms AWS-only surface | Very low | 3 | Not worth chart space | -| `context.aws.exporter.arn` | Can distinguish exporter instances or source ARNs | Medium | 10 | Keep out of headline widgets; debug use only | -| Queue type (standard or FIFO) | Changes interpretation of dedup and group metrics | Very low | 2 | Derive from queue naming/config, not as a discovered field | -| Message group ID | Explains FIFO parallelism and fairness | High if available | 20 | Do not assume Tsuga exports it on metrics | -| Noisy group | Explains tenant-local saturation in fair queues | Medium | 10 | Treat as concept until Stage 2 confirms fields | -| Quiet group | Baseline for non-noisy tenants | Medium | 10 | Use conceptually; do not invent group fields | -| Delay configuration | Distinguishes intentional scheduling from consumer lag | Low | 5 | Usually note text, not widget grouping | -| DLQ name | Critical for poison-message triage | Medium | 10 | Separate queue family, likely not on source-queue metrics | -| Message size bucket | Helps explain throughput limits and slow consumers | Medium | 10 | Prefer aggregate size metrics over per-message identifiers | - -### Tsuga field mapping -#### Confirmed by sources -| Vendor/exporter dimension | Recommended context.* key | Must-exist vs optional | -|---|---|---| -| QueueName | `context.queuename` | Must-exist | -| AWS Region | `context.cloud.region` | Must-exist | -| AWS Account ID | `context.cloud.account.id` | Optional but preferred | -| Cloud provider | `context.cloud.provider` | Optional | -| Team tag | `context.team` | Must-exist | -| Environment tag | `context.env` | Must-exist | -| Exporter source ARN | `context.aws.exporter.arn` | Optional | - -#### Best-practice inference -| Vendor/exporter dimension | Recommended context.* key | Must-exist vs optional | -|---|---|---| -| Queue type | `Unknown` | Optional; infer from queue config or `.fifo` naming if absent | -| Message group ID | `Unknown` | Optional; useful for FIFO deep dive if ever exposed | -| Fair-queue tenant or noisy-group key | `Unknown` | Optional; verify in Stage 2 before planning any tenant top-list | -| DLQ target queue | `Unknown` | Optional; may require config APIs rather than metrics | - -## Golden signals -### Confirmed by sources -| Signal | SQS meaning | Typical degradations | Best telemetry sources | What people page on | Section questions | -|---|---|---|---|---|---| -| Traffic | Enqueue, receive, and delete activity moving through the queue | send spike, receive collapse, delete lag | `NumberOfMessagesSent`, `NumberOfMessagesReceived`, `NumberOfMessagesDeleted` | send rate rising without drain, or traffic dropping unexpectedly | Are producers and consumers balanced? Is work completing? | -| Errors | SQS queueing issues rarely appear as explicit error metrics; failure is inferred from retry churn, DLQ movement, and age growth | poison messages, duplicate retries, misconfigured polling | `NumberOfEmptyReceives`, DLQ depth on companion queue, CloudTrail API failures | backlog aging, duplicate suppression spikes, or repeated empty receives under expected load | Is the queue unhealthy or are consumers ineffective? | -| Latency | Time work waits before a successful receive/delete | oldest age growth, delayed-message accumulation | `ApproximateAgeOfOldestMessage`, delayed and visible backlog counts | oldest message age rising for sustained periods | How long is work waiting? Is delay intentional or pathological? | -| Saturation | Queue backlog and in-flight occupancy relative to drain capacity | visible depth growth, in-flight stalls, hot message groups | `ApproximateNumberOfMessagesVisible`, `ApproximateNumberOfMessagesNotVisible`, fair/FIFO group metrics | queue depth climbing or ordered groups blocking progress | Is backlog building faster than consumers can clear it? | - -### Best-practice inference -For SQS, oldest-message age is usually more actionable than raw queue depth because a deep queue can be healthy during bursts while an aging queue almost always reflects unmet processing demand or poison work. - -## Telemetry sources -### Confirmed by sources -| Source type | How collected | What it provides | Pros/cons | Common pitfalls | -|---|---|---|---|---| -| CloudWatch SQS service metrics | Native AWS/SQS namespace exported from SQS to CloudWatch | Queue depth, age, send/receive/delete counts, empty receives, FIFO/fair-queue signals | Canonical and low-friction; metrics are approximate and queue-level | Exact counts should not be assumed from single samples | -| CloudTrail management/data events | AWS API logging for SQS actions | Audit trail for send, receive, delete, queue config changes | Good for who-did-what context; not a queue-health metric surface | Often disabled or not ingested into Tsuga logs | -| Consumer application logs | App or worker runtime logs | Actual processing failures, retries, payload errors, DLQ reasons | Best for root cause; not native to SQS | Not a service log stream and may span many runtimes | -| Queue configuration APIs | `GetQueueAttributes`, console config, IaC state | Visibility timeout, redrive policy, FIFO mode, long-poll settings | Essential for interpreting metrics | Not visible in metric streams directly | - -### Best-practice inference -Treat CloudWatch SQS metrics as the primary dashboard contract and use logs only to explain why consumers fail to delete messages. Do not try to infer consumer correctness from service metrics alone. - -## Log intelligence (Stage 4 handoff) -### Confirmed by sources -**Log sources matrix** - -| Source | Access path | Typical format | Structured vs unstructured | Evidence | -|---|---|---|---|---| -| CloudTrail SQS API events | CloudTrail event history, CloudWatch Logs, or S3 delivery | JSON management/data events for `SendMessage`, `ReceiveMessage`, `DeleteMessage`, queue config changes | Structured | https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-logging-using-cloudtrail.html | -| Consumer application logs | Worker runtime logs outside SQS | Runtime-specific text or JSON with queue URL/name, receipt handle, error reason | Mixed | Consumer-owned, not documented by SQS | -| DLQ inspection tooling | Queue receive/delete history or consumer logs | Queue-specific operational records | Mixed | Derived from app/process usage, not a native SQS log | - -**Known log formats** - -1. CloudTrail SQS API events - Sample shape: JSON event envelope with `eventSource: "sqs.amazonaws.com"`, `eventName`, `awsRegion`, request parameters, and identity metadata. - Shape notes: structured JSON, timestamp embedded, no multiline concerns, field set varies by API and whether data events are enabled. - Evidence: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-logging-using-cloudtrail.html - -2. Consumer failure logs - Sample shape: service-defined application log line or JSON object containing queue name, receive attempt, processing error, and delete outcome. - Shape notes: not standardized by SQS; likely multiple formats across runtimes. - Evidence: Unknown from vendor docs; infer only after Stage 4 live log search. - -**Candidate query filters for Stage 4** -- Precise: `context.service.name:* AND message:*sqs.amazonaws.com*` - Rationale: catches CloudTrail-style SQS API events if AWS audit logs are ingested. - Risk: depends on CloudTrail forwarding and log preservation in Tsuga. -- Broader fallback: `(message:*ReceiveMessage* OR message:*DeleteMessage* OR message:*ChangeMessageVisibility* OR message:*SendMessage*)` - Rationale: catches both audit and app logs that mention the core queue verbs. - Risk: high false-positive rate across unrelated components. - -**Attribute mapping hints** - -| Raw field | Suggested Tsuga key | Confidence | Notes | -|---|---|---|---| -| `requestParameters.queueUrl` or queue name | `context.queuename` | Medium | Prefer the discovered metric field name if log enrichment already extracts queue name | -| `awsRegion` | `context.cloud.region` | High | Stable in CloudTrail events | -| `recipientAccountId` or account context | `context.cloud.account.id` | Medium | Field names vary by event family | -| `eventName` | `context.aws.sqs.event_name` | High | Useful to separate send, receive, delete, and visibility changes | -| consumer error string | `context.error.message` | Low | App-specific, not SQS-native | -| receipt handle or message ID | `context.aws.sqs.message_id` | Low | High-cardinality; do not index aggressively | - -**Parsing risks** -- SQS itself does not emit a rich service-runtime log stream like a proxy or database. -- CloudTrail may be disabled, filtered, or routed outside Tsuga. -- Consumer logs are owned by the application and can use many incompatible formats. -- Message IDs and receipt handles are high-cardinality and should not become default group-bys. -- A useful Stage 4 route may need to target consumer services rather than SQS-native logs. - -### Best-practice inference -Stage 4 may be inapplicable if there is no centralized CloudTrail ingestion and no consistent consumer log shape. Prefer skipping log-route creation rather than inventing an SQS parser for logs that do not exist. - -## Caveats and footguns -- **[backlog-health]** SQS backlog metrics are approximate and should be trended, not treated as exact queue inventory. (https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-available-cloudwatch-metrics.html) -- **[backlog-health]** `ApproximateAgeOfOldestMessage` is a better incident signal than raw depth when you need to decide whether users are actually waiting. (Inference) -- **[backlog-health]** `ApproximateNumberOfMessagesVisible` can stay low even while specific FIFO groups stall if only one ordered lane is blocked. (https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/FIFO-queues-understanding-logic.html) -- **[backlog-health]** `ApproximateNumberOfMessagesNotVisible` rising can mean healthy high concurrency or stuck consumers; do not read it as failure alone. (https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-visibility-timeout.html) -- **[backlog-health]** Delayed messages are intentional hidden work, not necessarily consumer lag. (https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-available-cloudwatch-metrics.html) -- **[backlog-health, throughput-balance]** Missing deletes after healthy receives usually indicate consumer failure, timeout expiration, or missing acknowledgement logic. (https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-using-receive-delete-message.html) -- **[backlog-health]** A poison message can keep oldest age elevated even when most consumers are healthy; confirm DLQ policy before scaling blindly. (https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues.html) -- **[backlog-health]** When confirming a queue is empty, AWS recommends watching visible, not visible, and delayed counts together for several minutes rather than trusting one zero datapoint. (https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/confirm-queue-is-empty.html) -- **[throughput-balance]** `NumberOfMessagesDeleted` can exceed received counts because the same message may be received more than once before deletion. (https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-available-cloudwatch-metrics.html) -- **[throughput-balance]** `NumberOfMessagesReceived` is not equivalent to unique business work completed; retries and duplicates inflate it. (https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-available-cloudwatch-metrics.html) -- **[throughput-balance]** `SentMessageSize` should usually be viewed as an average or max trend, not summed across queues as a headline. (Inference) -- **[throughput-balance]** Queue-depth growth after a producer burst can be healthy if receive and delete rates catch up quickly; sustained age is the tie-breaker. (Inference) -- **[consumer-polling]** Empty receives can rise from short polling even when messages still exist; this is not always a zero-traffic state. (https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-short-and-long-polling.html) -- **[consumer-polling]** Over-scaling pollers can make empty-receive counts look alarming without any customer impact. (Inference) -- **[consumer-polling]** Never group SQS queue widgets by receipt handle or message ID if such fields appear later; the cardinality is pathological. (Inference) -- **[fifo-fairness]** `NumberOfDeduplicatedSentMessages` is only meaningful for FIFO queues and can remain zero on healthy standard queues. (https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/FIFO-queues-exactly-once-processing.html) -- **[fifo-fairness]** `ApproximateNumberOfGroupsWithInflightMessages` matters because too few active groups limit FIFO concurrency even when total backlog is large. (https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/FIFO-queues-understanding-logic.html) -- **[fifo-fairness]** Quiet-group metrics and noisy-group counts are specific to fair queues; do not show them as universal SQS backlog KPIs. (https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-available-cloudwatch-metrics.html) -- **[fifo-fairness]** One large message group can serialize FIFO progress; adding more consumers does not help if the grouping strategy is wrong. (https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/interleaving-multiple-ordered-message-groups.html) -- **[fifo-fairness]** High deduplication can be healthy during producer retry storms if exactly-once guarantees are working; it is a symptom to investigate, not immediate queue damage. (Inference) -- **[backlog-health, fifo-fairness]** Queue names often encode workload and environment, so `context.queuename` is valuable; exporter ARN is usually a lower-signal debug field. (Inference) - -## Confirmed Tsuga prefixes -- `aws_sqs*` — **CONFIRMED** (16 metrics present in Tsuga from `python3 tools/tsuga_search_metrics.py '^aws_sqs.*'`) - -## Discovery status -- Discovery: completed in Stage 2. -- Metrics found in Tsuga: 16 -- Reconciliation result: 16 confirmed from `01`, 0 missing, 0 unexpected. -- Confirmed common attributes on all 16 metrics: `context.queuename`, `context.cloud.region`, `context.cloud.account.id`, `context.env`, `context.team`, `context.cloud.provider`, `context.aws.exporter.arn`, `context.source`, `context.unit` -- Confirmed capabilities on all 16 metrics: `avg`, `sum`, `count`, `min`, `max` -- Remaining gaps: recent-data scalar spot-checks returned Tsuga `500 INTERNAL_ERROR`, and no queue-type or tenant field beyond `context.queuename` was discovered. - -## Top sources -- https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-available-cloudwatch-metrics.html - Canonical source for SQS queue, FIFO, and fair-queue metrics. -- https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-short-and-long-polling.html - Explains empty receives and polling efficiency. -- https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-visibility-timeout.html - Defines in-flight semantics and visibility-timeout tradeoffs. -- https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues.html - Grounds poison-message and redrive reasoning. -- https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/FIFO-queues-exactly-once-processing.html - Defines FIFO deduplication behavior. -- https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/FIFO-queues-understanding-logic.html - Explains message-group ordering and parallelism. -- https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-fair-queues.html - Defines quiet-group and noisy-group semantics. -- https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-logging-using-cloudtrail.html - Only reliable SQS-native logging reference for Stage 4. -- https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/confirm-queue-is-empty.html - Useful operational guidance for interpreting backlog metrics together. -- https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/interleaving-multiple-ordered-message-groups.html - Practical explanation of FIFO concurrency through message groups. diff --git a/plugins/tsuga/skills/knowledge-technology/references/aws-sqs/metrics.csv b/plugins/tsuga/skills/knowledge-technology/references/aws-sqs/metrics.csv deleted file mode 100644 index c401702..0000000 --- a/plugins/tsuga/skills/knowledge-technology/references/aws-sqs/metrics.csv +++ /dev/null @@ -1,18 +0,0 @@ -theme,name,confirmed,definition,unit,type,tsuga_metric_name,widget_suggestions,aggregation,post_function,group_by,required_context_fields,optional_context_fields,absence_means,cardinality_footguns,variants,citations -Backlog,ApproximateAgeOfOldestMessage,true,"Approximate age of the oldest non-deleted message in the queue; primary backlog latency signal.",s,summary,aws_sqs_approximate_age_of_oldest_message,"timeseries|query-value|top-list",max,none,"context.queuename:20|context.cloud.region:10",context.env|context.team|context.queuename,context.cloud.account.id|context.cloud.region|context.aws.exporter.arn,"No sustained backlog, queue idle, or metric absent for the selected queue.","Keep grouped by queue or region only; never by message identifiers.",cloudwatch=ApproximateAgeOfOldestMessage,https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-available-cloudwatch-metrics.html -Backlog,ApproximateNumberOfMessagesVisible,true,"Approximate number of messages available to be received immediately.",message,summary,aws_sqs_approximate_number_of_messages_visible,"timeseries|query-value|top-list",max,none,"context.queuename:20|context.cloud.region:10",context.env|context.team|context.queuename,context.cloud.account.id|context.cloud.region|context.aws.exporter.arn,"Queue has no ready backlog, queue idle, or metric absent.","Use as queue depth, not as exact inventory.",cloudwatch=ApproximateNumberOfMessagesVisible,https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-available-cloudwatch-metrics.html -Backlog,ApproximateNumberOfMessagesNotVisible,true,"Approximate number of in-flight messages currently hidden by the visibility timeout.",message,summary,aws_sqs_approximate_number_of_messages_not_visible,"timeseries|query-value|top-list",max,none,"context.queuename:20|context.cloud.region:10",context.env|context.team|context.queuename,context.cloud.account.id|context.cloud.region|context.aws.exporter.arn,"No in-flight work, low traffic, or metric absent.","Interpret with receives and deletes; high values can be healthy during parallel processing.",cloudwatch=ApproximateNumberOfMessagesNotVisible,https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-visibility-timeout.html|https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-available-cloudwatch-metrics.html -Backlog,ApproximateNumberOfMessagesDelayed,true,"Approximate number of delayed messages not yet available for receive.",message,summary,aws_sqs_approximate_number_of_messages_delayed,"timeseries|query-value|top-list",max,none,"context.queuename:20|context.cloud.region:10",context.env|context.team|context.queuename,context.cloud.account.id|context.cloud.region,"No delayed workload, queue not using delay features, or metric absent.","Do not mix with ready backlog when explaining urgent lag.",cloudwatch=ApproximateNumberOfMessagesDelayed,https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-available-cloudwatch-metrics.html -Throughput,NumberOfMessagesSent,true,"Count of messages added to the queue during the period.",message,summary,aws_sqs_number_of_messages_sent,"timeseries|query-value|top-list|bar",sum,per-second,"context.queuename:20|context.cloud.region:10",context.env|context.team|context.queuename,context.cloud.account.id|context.cloud.region|context.aws.exporter.arn,"No producer traffic, queue idle, or metric absent.","Treat as per-second throughput for operational views; retries may still be hidden by dedup on FIFO queues.",cloudwatch=NumberOfMessagesSent,https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-available-cloudwatch-metrics.html -Throughput,NumberOfMessagesReceived,true,"Count of messages returned by receive calls during the period.",message,summary,aws_sqs_number_of_messages_received,"timeseries|query-value|top-list|bar",sum,per-second,"context.queuename:20|context.cloud.region:10",context.env|context.team|context.queuename,context.cloud.account.id|context.cloud.region|context.aws.exporter.arn,"No receive activity, queue idle, or metric absent.","May exceed unique business work because duplicates and retries count again.",cloudwatch=NumberOfMessagesReceived,https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-available-cloudwatch-metrics.html -Throughput,NumberOfMessagesDeleted,true,"Count of successful delete actions during the period.",message,summary,aws_sqs_number_of_messages_deleted,"timeseries|query-value|top-list|bar",sum,per-second,"context.queuename:20|context.cloud.region:10",context.env|context.team|context.queuename,context.cloud.account.id|context.cloud.region|context.aws.exporter.arn,"No completed processing, queue idle, or metric absent.","Can exceed receive counts because deletes may follow repeated receives of the same message.",cloudwatch=NumberOfMessagesDeleted,https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-available-cloudwatch-metrics.html -Consumer Efficiency,NumberOfEmptyReceives,true,"Count of receive calls that returned no messages.",receive,summary,aws_sqs_number_of_empty_receives,"timeseries|query-value|top-list",sum,per-second,"context.queuename:20|context.cloud.region:10",context.env|context.team|context.queuename,context.cloud.account.id|context.cloud.region|context.aws.exporter.arn,"No polling waste, no traffic, or long polling is working well.","Interpret with receive throughput and backlog; high values alone do not prove failure.",cloudwatch=NumberOfEmptyReceives,https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-short-and-long-polling.html|https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-available-cloudwatch-metrics.html -Payload,SentMessageSize,true,"Size of messages added to the queue during the period.",B,summary,aws_sqs_sent_message_size,"timeseries|query-value|top-list",average,none,"context.queuename:20|context.cloud.region:10",context.env|context.team|context.queuename,context.cloud.account.id|context.cloud.region|context.aws.exporter.arn,"No send traffic or metric absent.","Prefer average or max trends; summed size is rarely a useful KPI.",cloudwatch=SentMessageSize,https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-available-cloudwatch-metrics.html -FIFO/Fairness,NumberOfDeduplicatedSentMessages,true,"Count of FIFO send attempts suppressed by deduplication.",message,summary,aws_sqs_number_of_deduplicated_sent_messages,"timeseries|query-value|top-list",sum,per-second,"context.queuename:20|context.cloud.region:10",context.env|context.team|context.queuename,context.cloud.account.id|context.cloud.region,"Queue is standard, dedup disabled, or no duplicate send attempts occurred.","Meaningful mainly for FIFO queues; do not use as a universal SQS KPI.",cloudwatch=NumberOfDeduplicatedSentMessages,https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/FIFO-queues-exactly-once-processing.html|https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-available-cloudwatch-metrics.html -FIFO/Fairness,ApproximateNumberOfGroupsWithInflightMessages,true,"Approximate count of message groups with at least one in-flight message.",group,summary,aws_sqs_approximate_number_of_groups_with_inflight_messages,"timeseries|query-value|top-list",max,none,"context.queuename:20|context.cloud.region:10",context.env|context.team|context.queuename,context.cloud.account.id|context.cloud.region,"Queue is not FIFO or fair-queue-capable, or the metric is absent.","Queue-level metric only; do not invent group-id splits without discovered fields.",cloudwatch=ApproximateNumberOfGroupsWithInflightMessages,https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/FIFO-queues-understanding-logic.html|https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-available-cloudwatch-metrics.html -Fairness,ApproximateNumberOfNoisyGroups,true,"Approximate count of noisy groups in a fair queue.",group,summary,aws_sqs_approximate_number_of_noisy_groups,"timeseries|query-value",max,none,"context.queuename:20|context.cloud.region:10",context.env|context.team|context.queuename,context.cloud.account.id|context.cloud.region,"Fair queues not enabled or metric absent.","Keep queue-level only unless Stage 2 confirms a tenant key.",cloudwatch=ApproximateNumberOfNoisyGroups,https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-fair-queues.html|https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-available-cloudwatch-metrics.html -Fairness,ApproximateAgeOfOldestMessageInQuietGroups,true,"Approximate age of the oldest message among quiet groups in a fair queue.",s,summary,aws_sqs_approximate_age_of_oldest_message_in_quiet_groups,"timeseries|query-value",max,none,"context.queuename:20|context.cloud.region:10",context.env|context.team|context.queuename,context.cloud.account.id|context.cloud.region,"Fair queues not enabled, no quiet-group backlog, or metric absent.","Use as comparison against overall oldest age, not a universal SQS metric.",cloudwatch=ApproximateAgeOfOldestMessageInQuietGroups,https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-fair-queues.html|https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-available-cloudwatch-metrics.html -Fairness,ApproximateNumberOfMessagesVisibleInQuietGroups,true,"Approximate number of visible messages in quiet groups of a fair queue.",message,summary,aws_sqs_approximate_number_of_messages_visible_in_quiet_groups,"timeseries|query-value|top-list",max,none,"context.queuename:20|context.cloud.region:10",context.env|context.team|context.queuename,context.cloud.account.id|context.cloud.region,"Fair queues not enabled, no quiet-group visible backlog, or metric absent.","Compare to total visible depth to detect noisy-group concentration.",cloudwatch=ApproximateNumberOfMessagesVisibleInQuietGroups,https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-fair-queues.html|https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-available-cloudwatch-metrics.html -Fairness,ApproximateNumberOfMessagesNotVisibleInQuietGroups,true,"Approximate number of in-flight messages in quiet groups of a fair queue.",message,summary,aws_sqs_approximate_number_of_messages_not_visible_in_quiet_groups,"timeseries|query-value",max,none,"context.queuename:20|context.cloud.region:10",context.env|context.team|context.queuename,context.cloud.account.id|context.cloud.region,"Fair queues not enabled, no quiet-group in-flight work, or metric absent.","Use as a quiet-tenant baseline, not as a universal saturation KPI.",cloudwatch=ApproximateNumberOfMessagesNotVisibleInQuietGroups,https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-fair-queues.html|https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-available-cloudwatch-metrics.html -Fairness,ApproximateNumberOfMessagesDelayedInQuietGroups,true,"Approximate number of delayed messages in quiet groups of a fair queue.",message,summary,aws_sqs_approximate_number_of_messages_delayed_in_quiet_groups,"timeseries|query-value",max,none,"context.queuename:20|context.cloud.region:10",context.env|context.team|context.queuename,context.cloud.account.id|context.cloud.region,"Fair queues not enabled, no quiet-group delayed work, or metric absent.","Delay may be intentional; compare with non-quiet-group depth before escalating.",cloudwatch=ApproximateNumberOfMessagesDelayedInQuietGroups,https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-fair-queues.html|https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-available-cloudwatch-metrics.html -Cost,NOT_APPLICABLE,true,"SQS service metrics do not expose direct spend. Use polling efficiency, payload size, and backlog patterns as cost-adjacent proxies until billing data is integrated.",Unknown,Unknown,Unknown,note,Unknown,none,Unknown,context.env|context.team,context.queuename|context.cloud.region,"Not applicable in the core SQS service metric family.","Do not invent cost widgets from queue health metrics alone.",proxy=NumberOfEmptyReceives|proxy=SentMessageSize,https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-short-and-long-polling.html|https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-available-cloudwatch-metrics.html diff --git a/plugins/tsuga/skills/knowledge-technology/references/aws-sqs/queries.md b/plugins/tsuga/skills/knowledge-technology/references/aws-sqs/queries.md deleted file mode 100644 index 3dbe00e..0000000 --- a/plugins/tsuga/skills/knowledge-technology/references/aws-sqs/queries.md +++ /dev/null @@ -1,51 +0,0 @@ -# AWS SQS - -Managed message queue. Healthy: bounded message age, consume ≈ send rate, no empty-receive storms, no in-flight backlog. - -## Incident shapes - -- **Consumer lag** — `aws_sqs_approximate_age_of_oldest_message` grows with visible messages → consumer slower than producer -- **Consumer crash / stuck** — visible climbs, delivered → 0 -- **Poison message** — one message repeatedly received and not deleted → in-flight elevated -- **Producer burst / overload** — sends spike, age climbs -- **Empty-receive storm** — `aws_sqs_number_of_empty_receives` dominates → short-polling misconfig - -## Key metrics - -| Metric | Unit | Signal | -|---|---|---| -| `aws_sqs_approximate_age_of_oldest_message` | seconds | User-visible lag | -| `aws_sqs_approximate_number_of_messages_visible` | count | Backlog depth | -| `aws_sqs_approximate_number_of_messages_not_visible` | count | In-flight; high sustained = consumers hanging | -| `aws_sqs_approximate_number_of_messages_delayed` | count | Delay-timer held | -| `aws_sqs_number_of_messages_sent` | count/min | Producer rate | -| `aws_sqs_number_of_messages_received` | count/min | Consumer poll results | -| `aws_sqs_number_of_messages_deleted` | count/min | Successful processing | -| `aws_sqs_number_of_empty_receives` | count/min | High = short-polling misconfig | -| `aws_sqs_sent_message_size` | bytes | Spikes can hit 256KB limit | -| `aws_sqs_approximate_number_of_groups_with_inflight_messages` | count | FIFO: groups in process | - -## Derived signals - -- First-derivative of visible messages — backlog trajectory. Positive sustained = bottleneck. -- `Deleted / Sent` — process ratio. < 1.0 = consumers behind. -- `NotVisible / (Visible + NotVisible)` — in-flight utilization. High + low delete rate = consumers stuck. -- `EmptyReceives / Received` — > 0.5 sustained = polling inefficiency. - -## Log patterns - -SQS has no logs; use consumer-side application logs: - -- `Timeout on receiving message` — broker / network issue -- `VisibilityTimeout expired` — consumer too slow; message re-delivered -- `MessageNotInflightException` — double-delete attempt -- `ReceiptHandleIsInvalid` — handle expired -- `Message Group X has no available messages` — FIFO group head blocked - -## Gotchas - -- "Approximate" metrics lag up to ~5 minutes under load; short incidents are invisible. -- DLQ forms silently. Always check DLQ `MessagesVisible` separately. -- `aws_sqs_number_of_empty_receives` high with `ReceiveMessageWaitTimeSeconds=0` (short polling) → cost driver. -- FIFO queue lag is per-message-group. A single blocked group stalls many consumers without showing in aggregate. -- Visibility-timeout expiry inflates `Received` without inflating `Deleted`. Don't treat received as "processed." diff --git a/plugins/tsuga/skills/knowledge-technology/references/caddy/README.md b/plugins/tsuga/skills/knowledge-technology/references/caddy/README.md deleted file mode 100644 index f60f868..0000000 --- a/plugins/tsuga/skills/knowledge-technology/references/caddy/README.md +++ /dev/null @@ -1,345 +0,0 @@ -# Caddy Integration Context Bundle - -## Technology Summary - -| Field | Value | -|---|---| -| Technology | Caddy | -| Tech slug | caddy | -| Deployment model | self-hosted / cloud / k8s | -| Telemetry source | Caddy built-in Prometheus metrics (`/metrics`) | -| Primary version | Caddy v2 (v2.6+) | -| Persona focus | SRE | -| Primary use-case | reliability | -| Integration scope | core HTTP server + reverse proxy | - ---- - -## What Caddy Is (Operational Context) - -Caddy is a modern HTTP/2+HTTP/3 web server written in Go, known for automatic HTTPS via Let's Encrypt/ZeroSSL and a JSON/Caddyfile configuration model. In a typical SRE context, Caddy acts as a **TLS-terminating reverse proxy** or **static file server**, sitting in front of application backends. It is commonly deployed as a Kubernetes ingress controller (via `ingress-caddy`), in Docker Compose stacks, and on bare-metal edge nodes. - -**What stub_status is to NGINX, the `/metrics` endpoint is to Caddy.** Caddy exposes rich Prometheus metrics natively since v2.5.0; no separate exporter is required. The metrics endpoint is served by the admin API (default: `localhost:2019/metrics`) or via a `metrics` handler in the HTTP server block. - -**Important scoping note:** Caddy's stub-status equivalent (`caddy_http_requests_in_flight`) gives only global in-flight request counts. Per-route or per-upstream detailed breakdowns are labeled by `server` (virtual server name) and `handler` (middleware name like `file_server`, `reverse_proxy`). These labels become the primary dimension split in dashboards. - ---- - -## Key Concepts - -### Concept Map - -``` -Client - │ - │ TCP/TLS - ▼ -[Caddy HTTP Server] - │ - ├─ server: "srv0" (or named server) - │ │ - │ ├─ route → handler: "file_server" - │ │ └─ serves static files - │ │ - │ └─ route → handler: "reverse_proxy" - │ │ - │ ├─ upstream: "backend1:8080" (healthy/unhealthy) - │ ├─ upstream: "backend2:8080" - │ └─ load balancing policy - │ - └─ metrics collection - ├─ caddy_http_request_duration_seconds (histogram) → per server+handler+code+method - ├─ caddy_http_requests_in_flight (gauge) → per server+handler - ├─ caddy_reverse_proxy_upstreams_healthy (gauge) → per upstream - └─ caddy_http_request_errors_total (counter) → per server+handler -``` - -**Key state transition for reverse_proxy:** -``` -Request arrives → upstream selected (round-robin/lb) → upstream health checked - → if healthy: proxy request → await response → return to client - → if unhealthy: mark upstream down → failover to next upstream → passive/active health check recovery -``` - -### Histogram Metrics - -Caddy exposes four histogram metrics (`caddy_http_request_duration_seconds`, `caddy_http_response_duration_seconds`, `caddy_http_request_size_bytes`, `caddy_http_response_size_bytes`). Each histogram is ingested via the OTel Prometheus receiver as three related time series: - -| Suffix | Type | Meaning | -|--------|------|---------| -| `_count` | delta counter | Total number of observations (= request count) | -| `_sum` | delta counter | Sum of all values (= total duration/bytes) | -| `_bucket` | delta counter | Per-le-bound bucket counts (for percentile estimation) | - -For request rate, use `caddy_http_request_duration_seconds_count` (delta → sum + per-second). -For percentile latency, use the base metric name (`caddy_http_request_duration_seconds`) with percentile aggregation if Tsuga supports it, or approximate from buckets. - -### Labels → Context Fields - -Caddy Prometheus labels map to Tsuga context attributes via OTel Prometheus receiver: - -| Caddy label | Expected Tsuga field | Notes | -|-------------|---------------------|-------| -| `server` | `context.server` | Virtual server name (e.g., `srv0`, `https`) | -| `handler` | `context.handler` | Middleware handler name (e.g., `reverse_proxy`, `file_server`) | -| `code` | `context.code` | HTTP status code (e.g., `200`, `404`, `503`) | -| `method` | `context.method` | HTTP method (e.g., `GET`, `POST`) | -| `upstream` | `context.upstream` | Upstream address (e.g., `backend:8080`) | -| `statuscode` | `context.statuscode` | Admin API requests status code | -| `path` | `context.path` | Admin API request path | - -All assumptions — must be verified in Stage 2. - -### Automatic HTTPS - -Caddy obtains and renews TLS certificates automatically. Cert-related metrics (ACME challenges, renewal errors) are not exposed in the Prometheus metrics endpoint — they appear only in structured logs. If TLS health is a concern, it must come from log parsing (Stage 4). - ---- - -## Golden Signals - -| Signal | Metric(s) | Healthy range | Concerning range | -|--------|-----------|---------------|-----------------| -| Request rate | `caddy_http_request_duration_seconds_count`/s | Stable baseline | >3× spike or zero during expected traffic | -| Error rate | `caddy_http_request_errors_total`/s | Near zero | Any sustained nonzero | -| P95 Latency | `caddy_http_request_duration_seconds` p95 | < 500ms | > 1s sustained | -| In-flight requests | `caddy_http_requests_in_flight` | < 50% capacity | > 80% sustained | -| Upstream health | `caddy_reverse_proxy_upstreams_healthy` | All upstreams = 1 | Any = 0 | - ---- - -## Confirmed Tsuga Prefixes - -**Status as of Stage 1 (2026-03-31): NOT YET CONFIRMED — no Caddy metrics found in Tsuga.** - -| Prefix | Status | Notes | -|--------|--------|-------| -| `caddy_*` | INFERRED (not confirmed) | Caddy v2 native Prometheus metrics | -| `go_*` | INFERRED (not confirmed) | Go runtime metrics (same as nginx/haproxy exporters) | -| `process_*` | INFERRED (not confirmed) | Process-level metrics | -| `promhttp_*` | INFERRED (not confirmed) | Prometheus handler metrics | - -Stage 2 must run `tsuga_search_metrics.py "caddy"` after Caddy metrics are ingested to confirm prefix format (underscore vs dot notation). - ---- - -## Telemetry Sources - -### Primary: Caddy Built-in `/metrics` Endpoint - -Caddy v2 serves Prometheus metrics natively. Enable via: - -**Caddyfile:** -``` -{ - admin localhost:2019 - servers { - metrics - } -} -``` - -Or via a dedicated metrics handler: -```caddyfile -:2019 { - metrics -} -``` - -**JSON config:** -```json -{ - "admin": { - "listen": "localhost:2019" - }, - "apps": { - "http": { - "servers": { - "metrics_server": { - "listen": [":2019"], - "routes": [{"handle": [{"handler": "metrics"}]}] - } - } - } - } -} -``` - -Scrape endpoint: `http://localhost:2019/metrics` (default) or wherever the admin API or `metrics` handler is configured. - -### OTel Prometheus Receiver Configuration - -```yaml -receivers: - prometheus: - config: - scrape_configs: - - job_name: caddy - static_configs: - - targets: ['localhost:2019'] - metrics_path: /metrics -``` - ---- - -## Metric Namespace Summary - -### `caddy_http_*` — HTTP Request Metrics (14+ metrics via histogram expansion) - -The HTTP metrics track all requests passing through Caddy's HTTP handlers. Since these are labeled by `server`, `handler`, `code`, and `method`, they provide the richest operational data. - -| Metric | Type | Description | -|--------|------|-------------| -| `caddy_http_requests_in_flight` | gauge | Currently active requests | -| `caddy_http_request_errors_total` | counter | Middleware errors (not HTTP 4xx/5xx — these are internal handler errors) | -| `caddy_http_request_duration_seconds` | histogram | Full round-trip request duration (includes TLS handshake if applicable) | -| `caddy_http_response_duration_seconds` | histogram | Time to first byte (TTFB) | -| `caddy_http_request_size_bytes` | histogram | Total incoming request size (headers + body) | -| `caddy_http_response_size_bytes` | histogram | Total outgoing response size (headers + body) | - -### `caddy_reverse_proxy_*` — Upstream Health - -| Metric | Type | Description | -|--------|------|-------------| -| `caddy_reverse_proxy_upstreams_healthy` | gauge | 1 = healthy, 0 = unhealthy, per upstream address | - -### `caddy_admin_*` — Admin API Metrics - -| Metric | Type | Description | -|--------|------|-------------| -| `caddy_admin_http_requests_total` | counter | Requests to the Caddy admin API (`/config`, `/id`, etc.) | - -### `caddy_*` — Build Info - -| Metric | Type | Description | -|--------|------|-------------| -| `caddy_build_info` | gauge | Always 1; labels: goarch, goos, goversion, version | - -### Go Runtime + Process Metrics - -Standard Go runtime metrics: `go_goroutines`, `go_memstats_*`, `process_*`, `promhttp_*`. - ---- - -## Caveats and Footguns (20+) - -1. **`caddy_http_request_errors_total` is NOT HTTP 4xx/5xx errors.** It counts middleware panics and internal handler failures. HTTP status codes (4xx/5xx) are visible as the `code` label on `caddy_http_request_duration_seconds_count`, not via a dedicated error counter. - -2. **The `handler` label can be misleading.** A single request may be processed by multiple handlers in sequence (auth, logging, reverse_proxy). Each handler records its own metrics. This means request counts can appear inflated if you don't filter to the final handler. - -3. **Histogram latency vs wall-clock latency.** `caddy_http_request_duration_seconds` measures the full round-trip including backend processing time. `caddy_http_response_duration_seconds` measures time-to-first-byte only. For backend latency proxy, prefer response_duration over request_duration. - -4. **`caddy_reverse_proxy_upstreams_healthy` missing if no reverse_proxy configured.** If Caddy is used only as a file server, this metric does not exist. Dashboard must gate on it. - -5. **Server labels are auto-generated.** If servers are not named in config, Caddy auto-assigns names like `srv0`, `srv1`. Labels are not human-friendly without naming conventions. - -6. **P99 latency from histogram buckets requires carefully chosen bucket boundaries.** Caddy uses default histogram buckets (0.005, 0.01, 0.025, ..., 10s). If most of your traffic is very fast (< 5ms), all requests fall in the lowest bucket and percentile approximation is coarse. - -7. **Admin API metrics (`caddy_admin_http_requests_total`) do NOT reflect user traffic.** The admin API is used for config reloads, not HTTP serving. High admin request rates indicate config churn, not user load. - -8. **`go_*` and `process_*` metrics reflect the Caddy process itself**, not any backend it proxies to. High Go heap memory means Caddy is using memory, not that backends are memory-heavy. - -9. **Caddy can be configured to serve metrics only on the admin API port.** If firewall rules block the admin port but allow the HTTP server port, you can also serve metrics via a `metrics` handler on the HTTP server block — but this exposes internal metrics to users. Use a dedicated internal metrics port. - -10. **In-flight requests gauge resets on restart.** After a graceful reload, in-flight transitions to near-zero briefly even under high traffic. This is expected and not an error condition. - -11. **histogram `_sum` is not the same as latency under load.** At high throughput, the average (`_sum` / `_count`) per-second can decrease due to parallelism. Do not interpret decreasing average as improvement without also checking in-flight count. - -12. **`code` label on `caddy_http_request_duration_seconds` requires a filter for 5xx tracking.** There is no dedicated 5xx counter. To get error rate, filter `code:5xx` or group by `code` and identify 5xx buckets. - -13. **Caddy v2.5 vs v2.6+ metrics naming.** Metrics naming was stabilized in v2.6. Earlier versions may use different label names or metric names. Verify version in `caddy_build_info`. - -14. **`caddy_http_requests_in_flight` is a snapshot, not a rate.** Do not apply per-second to it. Use `max` aggregation for peak detection. - -15. **High `caddy_reverse_proxy_upstreams_healthy` sum ≠ all healthy.** If you have 10 upstreams and sum = 9, one is down. Always check sum vs expected count, not just whether sum > 0. - -16. **Metric cardinality explosion from high-cardinality labels.** `method` has 9+ possible values (GET, POST, PUT, PATCH, DELETE, HEAD, OPTIONS, CONNECT, TRACE). `code` has ~50+ possible values. Grouping by both simultaneously creates high cardinality. Choose one or the other for group-by. - -17. **Histogram count vs actual request count.** If a request passes through N handlers, `caddy_http_request_duration_seconds_count` records N observations. For "true" request count, filter to the specific handler that processes final responses (typically `reverse_proxy` or `file_server`). - -18. **The `server` label is the server config name, not the IP/hostname.** A single Caddy instance may serve multiple virtual hosts under one logical `server`. Do not treat `server` as equivalent to a hostname or IP. - -19. **No per-route or per-vhost breakdown natively.** Caddy does not label metrics by Caddyfile route or hostname by default. All routes on a server share the same `server` label. Per-host breakdown requires access log processing. - -20. **Admin API port (2019) must be reachable for scraping.** If using Kubernetes, the admin port must be exposed as a container port and scraped via Pod IP, not the Service IP (admin port is typically not behind a Service). - -21. **`caddy_build_info` is not a health signal.** It is always 1 when the process is running. Use it only to confirm Caddy version per instance. - -22. **Cumulative vs delta temporality.** All histogram `_count` and `_sum` are cumulative Prometheus counters; OTel Prometheus receiver converts them to delta. Dashboard widgets must use `per-second` (not `rate`) for these metrics. Must be verified in Stage 2. - ---- - -## Log Intelligence (Stage 4) - -### Caddy Access Log Format - -Caddy's default log output is **structured JSON** (unlike NGINX's CLF). This makes parsing significantly easier. - -**Default JSON access log:** -```json -{ - "ts": 1711900800.123456, - "logger": "http.log.access", - "msg": "handled request", - "request": { - "remote_ip": "192.168.1.1", - "remote_port": "54321", - "proto": "HTTP/2.0", - "method": "GET", - "host": "example.com", - "uri": "/api/health", - "headers": {"User-Agent": ["kube-probe/1.27"]} - }, - "duration": 0.001234, - "size": 42, - "status": 200, - "resp_headers": {"Content-Type": ["application/json"]} -} -``` - -Key fields for log parsing: -- `ts` → timestamp (Unix float) -- `status` → HTTP status code -- `duration` → request duration in seconds -- `size` → response size in bytes -- `request.method`, `request.host`, `request.uri` - -**Error log format:** -```json -{ - "ts": 1711900800.123456, - "level": "error", - "logger": "http.log.error", - "msg": "dial tcp backend:8080: connection refused", - "request": {"remote_ip": "...", "proto": "...", "method": "...", "host": "...", "uri": "..."} -} -``` - -### Candidate Log Query Filters - -- Access logs: `service.name:caddy AND logger:http.log.access` -- Error logs: `service.name:caddy AND logger:http.log.error` -- Upstream errors: `service.name:caddy AND logger:http.log.error AND level:error` -- In Kubernetes: `k8s.container.name:caddy AND logger:http.log.access` - -### Stage 4 Notes - -- Caddy JSON logs are directly parseable without grok — use JSON path extraction -- `duration` field is in seconds (float) — multiply by 1000 for ms -- TLS errors and certificate renewal events appear in `logger:tls` logs -- Access logs must be explicitly enabled in Caddyfile: `log { output stdout }` - ---- - -## Top 10 Sources - -1. [Caddy v2 Metrics Docs](https://caddyserver.com/docs/metrics) -2. [Caddy Source: HTTP Metrics](https://github.com/caddyserver/caddy/blob/master/modules/caddyhttp/metrics.go) -3. [Caddy Source: Reverse Proxy Metrics](https://github.com/caddyserver/caddy/blob/master/modules/caddyhttp/reverseproxy/metrics.go) -4. [Caddy Prometheus Module](https://github.com/caddyserver/caddy/blob/master/modules/metrics/adminmetrics.go) -5. [Caddy Admin API Docs](https://caddyserver.com/docs/api) -6. [OTel Prometheus Receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/prometheusreceiver) -7. [Caddy Caddyfile Reference: log directive](https://caddyserver.com/docs/caddyfile/directives/log) -8. [Caddy reverse_proxy health checks](https://caddyserver.com/docs/caddyfile/directives/reverse_proxy#health-checks) -9. [Prometheus Histogram best practices](https://prometheus.io/docs/practices/histograms/) -10. [Caddy releases / changelog](https://github.com/caddyserver/caddy/releases) diff --git a/plugins/tsuga/skills/knowledge-technology/references/caddy/metrics.csv b/plugins/tsuga/skills/knowledge-technology/references/caddy/metrics.csv deleted file mode 100644 index aa2f8bf..0000000 --- a/plugins/tsuga/skills/knowledge-technology/references/caddy/metrics.csv +++ /dev/null @@ -1,17 +0,0 @@ -theme,name,confirmed,definition,unit,type,tsuga_metric_name,widget_suggestions,aggregation,post_function,group_by,required_context_fields,optional_context_fields,absence_means,cardinality_footguns,variants,citations -Availability/Health,caddy_reverse_proxy_upstreams_healthy,false,Health status of each reverse proxy upstream; 1=healthy/reachable 0=unhealthy/failing health checks,{status},gauge,caddy_reverse_proxy_upstreams_healthy,query-value|timeseries|top-list,sum,none,,context.env|context.team,context.upstream|context.scope,Caddy has no reverse_proxy configured — gate this section; or all upstreams are healthy,Low cardinality per upstream address; upstream count bounded by config,prometheus=caddy_reverse_proxy_upstreams_healthy,https://github.com/caddyserver/caddy/blob/master/modules/caddyhttp/reverseproxy/metrics.go -Availability/Health,caddy_http_request_errors_total,false,Total count of requests that encountered a middleware error (internal handler failures — not HTTP 4xx/5xx status codes),{errors},counter,caddy_http_request_errors_total,timeseries|query-value,sum,per-second,context.server:10|context.handler:10,context.env|context.team,context.server|context.handler|context.scope,No middleware errors — normal healthy state; absence of handler labels means no errors have been recorded,Cardinality = server × handler; keep handler group-by limited,prometheus=caddy_http_request_errors_total,https://github.com/caddyserver/caddy/blob/master/modules/caddyhttp/metrics.go -Throughput/Usage,caddy_http_request_duration_seconds_count,false,Total HTTP requests processed — the count dimension of the caddy_http_request_duration_seconds histogram; primary request rate signal,{requests},counter,caddy_http_request_duration_seconds_count,timeseries|query-value,sum,per-second,context.server:10|context.handler:10,context.env|context.team,context.server|context.handler|context.code|context.method|context.scope,No HTTP traffic flowing; check caddy_build_info and upstream routing,High cardinality: server × handler × code × method — filter or group by one dimension at a time,prometheus=caddy_http_request_duration_seconds,https://github.com/caddyserver/caddy/blob/master/modules/caddyhttp/metrics.go -Throughput/Usage,caddy_admin_http_requests_total,false,Total requests to the Caddy admin API (config reload path /config /id etc.); reflects operator/CI activity not user traffic,{requests},counter,caddy_admin_http_requests_total,timeseries|query-value,sum,per-second,context.method:10|context.path:10,context.env|context.team,context.handler|context.method|context.path|context.statuscode,Caddy admin API not being accessed — normal during stable periods; check if scrape endpoint is the admin port,Medium cardinality: handler × method × path × statuscode; path can be high-cardinality with dynamic config IDs,prometheus=caddy_admin_http_requests_total,https://caddyserver.com/docs/api -Latency,caddy_http_request_duration_seconds,false,Histogram of full round-trip request durations from connection accept to response complete (includes TLS + backend latency for reverse proxy),s,histogram,caddy_http_request_duration_seconds,timeseries,percentile,none,context.server:10|context.handler:10,context.env|context.team,context.server|context.handler|context.code|context.method|context.scope,No HTTP traffic; histogram exists but no observations,High cardinality across server × handler × code × method; use server+handler group-by for latency percentiles,prometheus=caddy_http_request_duration_seconds,https://github.com/caddyserver/caddy/blob/master/modules/caddyhttp/metrics.go -Latency,caddy_http_response_duration_seconds,false,Histogram of time-to-first-byte (TTFB) — time from request receipt to first byte sent to client; useful proxy for backend responsiveness under reverse_proxy,s,histogram,caddy_http_response_duration_seconds,timeseries,percentile,none,context.server:10|context.handler:10,context.env|context.team,context.server|context.handler|context.code|context.method|context.scope,No HTTP responses served; or Caddy version < 2.5 (metric not present),Same cardinality risk as request_duration — filter to reverse_proxy handler for backend latency signal,prometheus=caddy_http_response_duration_seconds,https://github.com/caddyserver/caddy/blob/master/modules/caddyhttp/metrics.go -Capacity/Saturation,caddy_http_requests_in_flight,false,Current number of requests being handled across all active connections; a gauge snapshot not a rate,{requests},gauge,caddy_http_requests_in_flight,query-value|timeseries,max,none,context.server:10|context.handler:10,context.env|context.team,context.server|context.handler|context.scope,No active HTTP requests — Caddy idle or not receiving traffic,Cardinality = server × handler; reasonable if handler set is bounded,prometheus=caddy_http_requests_in_flight,https://github.com/caddyserver/caddy/blob/master/modules/caddyhttp/metrics.go -Capacity/Saturation,caddy_http_request_size_bytes,false,Histogram of total request size (headers + body) per handled request; high values indicate large payload uploads,By,histogram,caddy_http_request_size_bytes,timeseries,percentile,none,,context.env|context.team,context.server|context.handler|context.code|context.method|context.scope,No inbound requests recorded,Same high cardinality as request_duration — group by server+handler only,prometheus=caddy_http_request_size_bytes,https://github.com/caddyserver/caddy/blob/master/modules/caddyhttp/metrics.go -Capacity/Saturation,caddy_http_response_size_bytes,false,Histogram of total response size (headers + body) per handled request; high values indicate large responses to clients,By,histogram,caddy_http_response_size_bytes,timeseries,percentile,none,,context.env|context.team,context.server|context.handler|context.code|context.method|context.scope,No outbound responses served,Same high cardinality as request_duration,prometheus=caddy_http_response_size_bytes,https://github.com/caddyserver/caddy/blob/master/modules/caddyhttp/metrics.go -Capacity/Saturation,go_goroutines,false,Number of goroutines currently running in the Caddy Go runtime; sustained increase indicates goroutine leak,{goroutines},gauge,go_goroutines,query-value|timeseries,max,none,,context.env|context.team,context.scope,Caddy process metric missing; expect 20-100 at steady state,Low cardinality per instance,prometheus=go_goroutines,https://pkg.go.dev/runtime -Capacity/Saturation,go_memstats_heap_alloc_bytes,false,Current heap memory allocated and in use by the Caddy Go runtime,By,gauge,go_memstats_heap_alloc_bytes,query-value|timeseries,max,none,,context.env|context.team,context.scope,Caddy runtime metric missing,Low cardinality per instance,prometheus=go_memstats_heap_alloc_bytes,https://pkg.go.dev/runtime -Capacity/Saturation,go_memstats_heap_sys_bytes,false,Total heap memory obtained from the OS by the Caddy Go runtime; denominator for heap utilization %,By,gauge,go_memstats_heap_sys_bytes,query-value|timeseries,max,none,,context.env|context.team,context.scope,Caddy runtime metric missing,Low cardinality per instance,prometheus=go_memstats_heap_sys_bytes,https://pkg.go.dev/runtime -Capacity/Saturation,process_resident_memory_bytes,false,Resident set size (RSS) memory used by the Caddy process; includes all in-use memory pages,By,gauge,process_resident_memory_bytes,query-value|timeseries,max,none,,context.env|context.team,context.scope,Caddy process metric missing; expected 50-200MB for light-medium load,Low cardinality per instance,prometheus=process_resident_memory_bytes,https://prometheus.io/docs/instrumenting/writing_clientlibs/ -Capacity/Saturation,process_cpu_seconds_total,false,Total CPU time consumed by the Caddy process (cumulative counter); use .cumulativetodelta variant or rate function,s,counter,process_cpu_seconds_total,timeseries,sum,rate,context.scope:20,context.env|context.team,context.scope,Caddy process metric missing; confirmed cumulative (not delta),Low cardinality per instance,prometheus=process_cpu_seconds_total,https://prometheus.io/docs/instrumenting/writing_clientlibs/ -Availability/Health,promhttp_metric_handler_requests_total,false,Total scrape requests to the Caddy /metrics endpoint by HTTP status code; 200=success 500/503=error,{requests},counter,promhttp_metric_handler_requests_total,timeseries|query-value,sum,per-second,context.code:3,context.env|context.team,context.scope|context.code,Caddy is not being scraped; check OTel collector Prometheus receiver config,Very low cardinality (3 status codes: 200/500/503),prometheus=promhttp_metric_handler_requests_total,https://pkg.go.dev/github.com/prometheus/client_golang/prometheus/promhttp -Cost,NOT_APPLICABLE,false,Caddy does not expose cost-related metrics. Cost proxies: caddy_http_request_duration_seconds_count (request volume as billing proxy) and caddy_http_response_size_bytes_sum (egress bandwidth proxy).,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A diff --git a/plugins/tsuga/skills/knowledge-technology/references/caddy/queries.md b/plugins/tsuga/skills/knowledge-technology/references/caddy/queries.md deleted file mode 100644 index cf3ef02..0000000 --- a/plugins/tsuga/skills/knowledge-technology/references/caddy/queries.md +++ /dev/null @@ -1,54 +0,0 @@ -# Caddy - -Go-based web server with auto-HTTPS. Healthy: in-flight bounded, low error rate, upstream pool healthy, Go runtime stable. - -## Incident shapes - -- **Upstream unhealthy** — `caddy_reverse_proxy_upstreams_healthy` drops → health-check failures -- **Error spike** — `caddy_http_request_errors_total` rises → backend or config -- **Latency regression** — request duration p95 climbs → upstream slow or handler regression -- **In-flight saturation** — `caddy_http_requests_in_flight` near process limit -- **Go runtime issues** — `go_goroutines` unbounded or `heap_alloc` growing = leak or deadlock - -## Key metrics - -| Metric | Unit | Signal | -|---|---|---| -| `caddy_reverse_proxy_upstreams_healthy` | 0/1 | Per-upstream health | -| `caddy_http_request_duration_seconds` | s | Request latency histogram | -| `caddy_http_response_duration_seconds` | s | Response-write latency | -| `caddy_http_request_errors_total` | count | Error count by status | -| `caddy_http_request_duration_seconds_count` | count | Request count | -| `caddy_http_requests_in_flight` | count | Concurrent requests | -| `caddy_http_request_size_bytes` / `response_size_bytes` | bytes | Size histograms | -| `caddy_admin_http_requests_total` | count | Admin API usage (external spike = investigate) | -| `go_goroutines` | count | Growth = leak | -| `go_memstats_heap_alloc_bytes` / `heap_sys_bytes` | bytes | Heap posture | -| `process_resident_memory_bytes` | bytes | Process RSS | -| `process_cpu_seconds_total` | cpu-s | Per-second = CPU usage | - -## Derived signals - -- `Δerrors_total / Δrequest_count` — error rate. -- Histogram quantile on `request_duration_seconds` — p95/p99. -- Fraction of `upstreams_healthy == 1` — upstream health ratio. -- `go_goroutines` trend — healthy baseline 10-200; unbounded growth = leak. - -## Log patterns - -Caddy logs JSON by default: - -- `"msg":"proxy error"` — reverse-proxy upstream error -- `"msg":"certificate_obtain_failed"` — ACME failure -- `"msg":"dial_backend"` with repeated `connection refused` — backend dial failures -- `"msg":"reading response from upstream"` — upstream closed early -- `"msg":"requesting certificate"` — ACME issue -- Request entries with `"status":5xx` — per-request failures - -## Gotchas - -- ACME auto-cert issuance can fail silently for one domain while others succeed. `certificate_obtain_failed` is the primary signal. -- Admin API (`:2019`) should not be public. Spike in `caddy_admin_http_requests_total` from external IP = security signal. -- In-flight requests include websockets; long-lived connections inflate `requests_in_flight` benignly. -- Goroutine count rises with HTTP/2 streams and websockets; high count ≠ leak. Look for monotonic growth over 30+ min. -- JSON log field names mix camelCase and snake_case. Filter on `msg`. diff --git a/plugins/tsuga/skills/knowledge-technology/references/cassandra/README.md b/plugins/tsuga/skills/knowledge-technology/references/cassandra/README.md deleted file mode 100644 index 5601431..0000000 --- a/plugins/tsuga/skills/knowledge-technology/references/cassandra/README.md +++ /dev/null @@ -1,374 +0,0 @@ -# Cassandra Integration Context Bundle - -## Metadata - -**Technology:** Cassandra -**Deployment:** self-hosted -**Environment:** prod -**Persona:** SRE Dev and ops -**Telemetry preference:** mixed (OTel JMX receiver primary; Prometheus JMX exporter alternative) -**Integration scope:** core service only -**Primary use-case:** reliability and performance - ---- - -## How to use this bundle - -- `01_cassandra_metrics.csv` — source of truth for metrics, units, temporality, safe aggregation, group-by recommendations -- `02_cassandra_dashboard_plan.yaml` — dashboard structure: sections, widgets, derived signals, explanation notes, triage chains, playbooks -- `03_cassandra_state.yaml` — machine-readable state, assumptions list, unknowns with actionable context -- `04_cassandra_memory.md` — human-readable Stage 1 summary and handoff narrative for Stage 2 - -Stage 2 will create `05_cassandra_metric_catalog.csv` as the discovered metric catalog for reconciliation and deep-dive coverage checks. - -Stage 4 should read this file's "Log intelligence (Stage 4 handoff)" section and `03_cassandra_state.yaml` `log_intel` before designing log routes. - ---- - -## What it is and what "good" looks like - -Cassandra is a distributed wide-column NoSQL database engineered for high availability and linear horizontal scalability with no single point of failure. Nodes form a peer-to-peer ring; data is partitioned via consistent hashing and replicated across N nodes per replication factor. It is typically deployed self-hosted on bare metal or Kubernetes, accessed via CQL (Cassandra Query Language) over native transport (default port 9042). - -**Confirmed by sources:** -"Good" Cassandra: read/write p99 latencies below 10ms for point queries (application-dependent); zero dropped messages; compaction backlog (pending tasks) below 10 per node; heap utilization 50–70% with GC stop-the-world pauses under 200ms per minute; key cache hit rate above 80%. [Datadog Cassandra monitoring guide; Cassandra docs] - -**Best-practice inference:** -Top 3 incident shapes and first dashboard sections to check: -1. **Slow reads / p99 latency degradation** → "Read Latency" first (magnitude), then "Compaction" (SSTable count forcing extra disk seeks), then "Memory & GC" (stop-the-world pauses blocking all operations) -2. **Node overload / dropped messages** → "Errors & Dropped Messages" first, then "Thread Pools" (pending tasks spike before drops occur), then "Throughput" (confirm traffic surge) -3. **Node unavailability / replication risk** → "Cluster Health" first (storage load per node, hints accumulation), then "Storage & Hints" (growing hints = node is down), then "Compaction" (overwhelmed surviving nodes) - ---- - -## Key concepts - -### Glossary - -| Term | Definition | Operational meaning | Dashboard section | -|---|---|---|---| -| Keyspace | Top-level namespace defining replication strategy and replication factor | Wrong replication factor = data loss risk under failure | cluster-health, storage-hints | -| Table (ColumnFamily) | Schema object within a keyspace; rows partitioned by partition key | High tombstone count or large partitions directly degrade read performance | read-latency, compaction | -| SSTable | Immutable on-disk file; Cassandra never updates data in-place | Too many SSTables = read amplification; compaction merges them back | compaction, read-latency | -| Memtable | In-memory write buffer; flushed to SSTable on overflow or commit log size threshold | Memtable flush triggers compaction; large memtables cause heap pressure | memory-gc, write-latency | -| Commit Log | Write-ahead log ensuring durability before memtable flush | Disk I/O saturation on commit log path directly stalls writes | write-latency, storage-hints | -| Compaction | Background process that merges SSTables; reclaims tombstone space | Compaction backlog growing = reads scan more SSTables = read latency degrades | compaction | -| Tombstone | Deletion marker that must be scanned then garbage-collected via compaction | Excessive tombstones cause read latency spikes and WARN/ABORT query failures | read-latency, compaction | -| Gossip | Peer-to-peer protocol (port 7000/7001) for node failure detection and cluster state | Node detection lag = split brain risk; drives UP/DOWN state propagation | cluster-health | -| Nodetool | CLI for cluster management (status, repair, ring state) | `nodetool status` is the ground truth for node membership state | cluster-health | -| Repair | Process ensuring data consistency across replicas | Overdue repairs = stale data risk; repair adds compaction load | compaction, storage-hints | -| Bloom Filter | Probabilistic structure to avoid unnecessary SSTable seeks during reads | High false positive ratio = unnecessary disk I/O on every read | cache-efficiency, read-latency | -| Key Cache | Caches the partition offset in SSTable for frequently-read keys | High hit rate eliminates expensive index seeks; meaningful for read-heavy workloads | cache-efficiency | -| Row Cache | Caches entire serialized rows in memory; disabled by default | Only beneficial for small, hot rows; large rows make it counterproductive | cache-efficiency | -| Coordinator | Node that receives a client request and routes to replica nodes | Coordinator hotspot = one node handling disproportionate routing load | throughput | -| Replica | Node storing a copy of data for a partition token range | Down replica + tight consistency level = availability degradation | cluster-health | -| Consistency Level | Per-query setting controlling how many replicas must respond | CL=ONE tolerates 2 down replicas with RF=3; CL=ALL requires all up | errors-dropped | -| Dropped Messages | Messages that timed out before processing and were discarded | Non-zero drop rate = critical signal; node is overloaded or GC-stalled | errors-dropped | -| Hint | Temporarily stored mutation for a down replica, replayed when it recovers | Growing hint backlog = a node is down; large hints = slow or failing recovery | storage-hints | -| Thread Pool | Task queue for each Cassandra processing stage (ReadStage, MutationStage, etc.) | Pending tasks accumulate before drops occur; each pool has specific responsibility | thread-pools | -| Token Ring | Consistent hash ring distributing partition ownership to nodes | Uneven token distribution (vnode imbalance) = hot nodes and latency skew | cluster-health, throughput | -| Speculative Retry | Sending request to second replica before first responds | High speculative rate = tailing latency from slow replicas; costly for replica load | errors-dropped, read-latency | -| Read Repair | Synchronizing inconsistent replicas detected during a read operation | Read repair overhead inflates tail latency; heavy on high-replication tables | read-latency | -| CAS / LWT | Lightweight transaction via Paxos (compare-and-set) | CAS is ~4x more expensive than normal read/write; avoid on hot paths | write-latency | -| Compaction Strategy | STCS (write-heavy), LCS (read-heavy), TWCS (time-series, window-based) | Wrong strategy for workload = SSTable explosion or compaction amplification | compaction | -| snitch | Component defining network topology (datacenter, rack assignments) | Misconfigured snitch = cross-DC reads; wrong DC routing | cluster-health | - -### Concept Map - -``` -Client -> CQL TCP:9042 -> Coordinator Node - (why: entry point; coordinator routes to data-owning replicas) - -Coordinator Node -> consistent_hash(partition_key) -> target Replica Nodes - (why: determines which nodes own the data; wrong routing = cross-DC reads) - -Coordinator Node -> Gossip heartbeat -> all Peer Nodes - (why: failure detection; drives UP/DOWN/LEAVING state propagation) - -Client Request -> Thread Pool queue (ReadStage|MutationStage|etc.) -> Worker threads - (why: thread pool saturation is the first saturation signal before drops) - -Thread Pool pending_tasks growing -> timeout threshold -> dropped message - (why: dropped messages directly follow thread pool exhaustion) - -Write Request -> Commit Log (disk, synchronous) + Memtable (heap, async) - (why: durability via WAL; Memtable enables fast write path) - -Memtable -> flush threshold reached -> SSTable written to disk - (why: flush = disk write burst; triggers background compaction scheduling) - -Multiple SSTables accumulate -> Compaction triggered -> fewer merged SSTables - (why: read amplification control; removes tombstones; reclaims disk space) - -Read Request -> Bloom Filter check (per SSTable) -> (false positive or definite miss) -> SSTable seek - (why: BF false positives = unnecessary disk I/O; high false ratio = degraded reads) - -Read Request -> Key Cache check -> (hit) -> direct SSTable offset read - (why: key cache eliminates expensive partition index seek) - -High Compaction Pending -> many SSTables unmerged -> N disk seeks per read -> latency spike - (why: compaction backlog is the most direct Cassandra-specific cause of read latency degradation) - -JVM GC stop-the-world pause -> all thread pools stall -> latency spike across all operations - (why: GC pauses are the primary cause of sudden latency "steps" in Cassandra) - -Heap pressure -> more frequent GC -> longer major GC pauses -> dropped messages - (why: heap pressure creates a cascade to drops; heap utilization is leading indicator) - -Node DOWN -> coordinator queues hints for that node -> hint store grows - (why: hints = data safety net during outages; large hint store = slow recovery) - -Hints In Progress growing -> node down and not recovering -> potential data gap risk - (why: if hints TTL expires before node recovers, mutations are permanently lost) - -High tombstone count in table -> read query scans N tombstones -> WARN threshold hit -> query slow/fail - (why: tombstone accumulation is a silent killer of read performance) - -Table-level coordinator.read high on one node -> coordinator hotspot -> uneven load - (why: CQL load balancing misconfiguration causes single-node overload) - -CAS (LWT) write -> Paxos 4-phase commit (prepare + promise + accept + commit) -> 4x latency vs normal write - (why: LWT must be tracked separately; its latency profile is fundamentally different) -``` - -### Entities and dimensions - -| Entity/Dimension | Why useful | Cardinality risk | Safe top-N | Notes | -|---|---|---|---|---| -| `host` | Per-node isolation; Cassandra performance is inherently node-specific | Medium (scales with cluster size) | 20 | Primary split dimension for all metrics | -| `keyspace` | Identifies which application's data is affected | Low-medium (few per cluster) | 16 | Required context for table-level metrics | -| `table` / `columnfamily` | Identifies exact table contributing to latency or compaction load | Medium-high (many tables possible) | 20 | Always pair with keyspace | -| `datacenter` | Multi-DC routing isolation; regional failure isolation | Low (2–5 DCs typical) | 10 | Critical for multi-DC setups; single-DC can omit | -| `operation` | Read vs write vs range_slice vs CAS; different cost and latency profiles | Very low (6 fixed values) | 6 | Prefer filter over group-by for simple split | -| `pool.name` | Thread pool identity (ReadStage, MutationStage, CompactionExecutor, etc.) | Low (10–15 fixed names) | 15 | Group-by is safe; bounded and stable | -| `cache_result` | hit / miss / hit_out_of_range for row/key cache | Very low (3 values) | 3 | Use as filter in derived signal formula | -| `percentile` | Pre-aggregated percentile values (50p, 99p, max) | Very low (3 values) | 3 | Filter to desired percentile; do NOT group-by | -| `gc.action` | Minor vs major GC type | Very low (2–3 values) | 3 | Group-by for GC type breakdown | -| `status` | Error status codes for client.request.error.count | Low (few error types: Timeout, Unavailable, etc.) | 10 | Group-by for error classification | -| `context.env` | Environment filter (prod/staging) | Very low | N/A | Dashboard-level filter; always include | -| `context.team` | Owning team filter | Low | N/A | Dashboard-level filter; always include | -| `context.k8s.namespace.name` | Kubernetes namespace isolation | Low-medium | 20 | Group-by only in multi-namespace deployments | -| `context.scope.name` | OTel instrumentation scope identifier | Very low | N/A | Do NOT group-by; use as filter if needed to isolate Cassandra metrics from JVM metrics | - -**Do NOT group-by:** `context.scope.name` (implementation detail), raw IP addresses, JMX object name suffixes. - -### Tsuga field mapping - -**Confirmed by sources:** OTel JMX Cassandra receiver attribute names: https://github.com/open-telemetry/opentelemetry-java-contrib/blob/main/jmx-metrics/docs/target-systems/cassandra.md - -**Best-practice inference:** The `context.*` Tsuga key names are inferred based on OTel attribute naming and standard Tsuga conventions. Stage 2 discovery MUST confirm these field names in live Tsuga data. - -| Vendor/exporter dimension | Recommended context.* key | Must-exist vs optional | -|---|---|---| -| `host` (JMX hostname) | `context.host` | Must-exist for per-node analysis | -| `keyspace` | `context.cassandra.keyspace` | Must-exist for table-level metrics | -| `columnfamily` / `table` | `context.cassandra.columnfamily` | Must-exist for table-level metrics | -| `datacenter` | `context.cassandra.datacenter` | Must-exist for multi-DC setups; Unknown for single-DC | -| `operation` | `context.cassandra.operation` | Must-exist for request type filtering | -| `pool.name` | `context.cassandra.thread_pool` | Must-exist for thread pool metrics | -| `cache_result` | `context.cassandra.cache_result` | Must-exist for cache hit/miss metrics | -| `percentile` | `context.cassandra.percentile` | Must-exist for pre-computed latency percentile metrics | -| `status` | `context.cassandra.status` | Must-exist for error count metrics | -| `gc.action` | `context.jvm.gc.action` | Optional (JVM target only) | -| `gc.cause` | `context.jvm.gc.cause` | Optional | - ---- - -## Golden signals - -### Traffic - -**What it means for Cassandra:** Client request rate split by operation type (read, write, range_slice, cas_read, cas_write, view_write). Separate read and write rates because Cassandra workloads are designed around one or the other; the ratio is a tuning signal and an anomaly indicator. - -**Confirmed:** `cassandra.client.request.count` with `operation` attribute provides exact client request counts per operation type. [OTel java-contrib Cassandra metrics] - -**Best-practice inference:** Table-level `cassandra.table.operation.count` gives finer breakdown when a specific keyspace/table is suspect, but MUST NOT be summed across hosts for cluster-wide throughput (double-counts due to replication). - -**Typical causes of degradation:** Sudden traffic spike from application code change; batch jobs triggering range scans; node failure redistributing load to survivors. - -**What people page on:** Read or write rate drops to near-zero (node unreachable or application-side issue); rate exceeds 3x baseline with concurrent latency increase. - -**Section questions:** "Is the cluster handling the expected read/write request load?" | "Which keyspaces and tables are generating the most traffic?" - -### Errors - -**What it means for Cassandra:** `cassandra.client.request.error.count` counts timeouts and unavailables from the client's perspective. Dropped messages (`cassandra.thread_pool.dropped_tasks`) are a distinct and more critical signal — they indicate the node discarded work rather than completing it. - -**Confirmed:** Both metrics exist in OTel JMX Cassandra target. [OTel java-contrib] - -**Best-practice inference:** Dropped messages > 0 is essentially a pager condition. `client.request.error.count` can be non-zero normally under high CAS write contention — distinguish by checking operation type. - -**Typical causes:** Write timeout = coordinator couldn't get enough replicas to acknowledge; Read timeout = replicas too slow; Dropped = thread pool backlog exceeded timeout. - -**What people page on:** Any sustained dropped messages (even 1/s is alarming); read/write error rate above ~1%. - -**Section questions:** "Are client requests failing or timing out?" | "Are any messages being dropped (node overload signal)?" - -### Latency - -**What it means for Cassandra:** Cassandra latency is a composite of coordinator routing time, SSTable reads (disk I/O), bloom filter checks, key cache lookups, compaction interference, and JVM GC pauses. p99 is the primary SLA signal; p50 gives typical performance. - -**Confirmed:** `cassandra.client.request.read.latency.99p` and `.write.latency.99p` are pre-computed percentile gauges from JMX. They are NOT histograms — use `average` aggregation + no post-function. [OTel java-contrib] - -**Best-practice inference:** Read latency in Cassandra is typically higher variance than write latency because reads must touch disk (SSTables), while writes hit Memtable first. A p99 read latency spike almost always points to one of: compaction backlog, GC pause, tombstone accumulation, or hot partition. - -**What people page on:** p99 read latency exceeding SLA (typically 10–50ms depending on application); write p99 > 5ms; any latency step-change (sudden increase, not gradual drift). - -**Section questions:** "Are reads completing within SLA?" | "Are writes completing within SLA?" | "Which tables or nodes are contributing to tail latency?" - -### Saturation - -**What it means for Cassandra:** Multiple saturation surfaces exist: thread pool pending tasks (leading indicator for drops), heap utilization (GC pressure), compaction backlog (read amplification), disk usage per node. Compaction pending is the most Cassandra-specific saturation signal — it has a direct, causal link to read latency. - -**Confirmed:** `cassandra.compaction.tasks.pending`, `cassandra.thread_pool.pending_tasks`, `jvm.memory.heap` are all in the OTel JMX target. - -**Best-practice inference:** Heap > 70% with active GC is more urgent than disk at 80%. Disk at 95%+ causes Cassandra to stop accepting writes (write-ahead log full); monitor storage load per node. - -**What people page on:** Compaction pending > 50 per node growing trend; heap > 75% with frequent major GC; ReadStage or MutationStage pending tasks > 100. - -**Section questions:** "Is the compaction backlog growing?" | "Is heap/GC pressure causing stalls?" | "Are thread pools backing up?" - ---- - -## Telemetry sources - -**Confirmed by sources:** -- OTel JMX receiver Cassandra target: https://github.com/open-telemetry/opentelemetry-java-contrib/blob/main/jmx-metrics/docs/target-systems/cassandra.md -- OTel JMX receiver JVM target: https://github.com/open-telemetry/opentelemetry-java-contrib/blob/main/jmx-metrics/docs/target-systems/jvm.md - -**Best-practice inference:** In Kubernetes, use the OTel JMX receiver as a sidecar with Cassandra JMX port (7199 default) exposed on the pod. Enable both `cassandra` and `jvm` targets in the same receiver config. - -| Source type | How collected | What it provides | Pros | Cons | Common pitfalls | -|---|---|---|---|---|---| -| OTel JMX Receiver (`cassandra` target) | OTel Collector with `jmxreceiver`, `cassandra` target | Full JMX client request metrics, thread pools, compaction, table-level, storage | Open-source; full coverage; semantic conventions aligned | Requires JMX port (7199) accessible; JMX must not require auth or SSL must be configured | JMX port blocked by K8s NetworkPolicy; receiver silent fail if JMX unreachable | -| OTel JMX Receiver (`jvm` target) | Same OTel Collector, `jvm` target for the Cassandra JVM process | Heap used/max, GC count, GC elapsed time | Standard JVM observability | Must target the correct JVM process (Cassandra's PID) | Missing if only `cassandra` target is configured; heap metrics won't appear | -| Prometheus JMX Exporter (alternative) | `jmx_prometheus_javaagent` loaded as JVM agent | Same JMX metrics in Prometheus format | Battle-tested; no OTel required | Metric naming differs (`cassandra_clientrequest_latency_micros_count` vs OTel names); translation required for Tsuga | Name mismatch causes Stage 2 prefix discovery to fail if wrong names used | -| Cassandra Nodetool checks | `nodetool status` via shell | Node UP/DOWN state, token ownership, replication availability | Ground truth for cluster membership | Not suitable for high-frequency polling; not a standard metric stream | Adds operational overhead; separate integration from JMX metrics | - -**"No data" meanings:** -- `cassandra.*` metrics absent: OTel JMX receiver not running, JMX connection refused, or wrong `targetSystem` config -- `jvm.*` metrics absent: `jvm` target not added to `targetSystems` list in jmxreceiver config -- Metrics present but `operation` attribute missing: older version of OTel java-contrib jmx-metrics gatherer -- `cassandra.table.*` metrics absent: can happen when no keyspace/table traffic exists or Cassandra version differences - ---- - -## Log intelligence (Stage 4 handoff) - -### Confirmed by sources - -**Log sources matrix:** - -| Source | Access path | Typical format | Structured vs unstructured | Evidence | -|---|---|---|---|---| -| System log (`system.log`) | `/var/log/cassandra/system.log` on host; stdout/stderr in K8s | Custom Cassandra multi-field format | Unstructured | [Cassandra logging docs](https://cassandra.apache.org/doc/latest/cassandra/operating/logging.html) | -| Debug log (`debug.log`) | `/var/log/cassandra/debug.log` | Same format, more verbose | Unstructured | Cassandra logging docs | -| GC log | JVM `-Xloggc:/path/to/gc.log` flag | JVM GC log format (G1GC or CMS) | Semi-structured | JVM standard | -| Audit log (optional) | Configurable path; requires `audit_logging_options.enabled: true` in cassandra.yaml | Cassandra audit format or BIN format | Optional/structured | [Cassandra audit logging](https://cassandra.apache.org/doc/latest/cassandra/operating/audit_logging.html) | - -**Known log formats:** - -Format: Cassandra `system.log` default format -``` -INFO [main] 2024-01-15 10:23:45,678 StorageService.java:1234 - Node localhost/127.0.0.1 state jump to NORMAL -WARN [ReadStage-1] 2024-01-15 10:23:46,123 Keyspace.java:567 - Read 100 live rows and 500 tombstone cells for query SELECT * FROM my_keyspace.my_table; (TombstoneWarning) -ERROR [MutationStage-2] 2024-01-15 10:23:47,234 CassandraDaemon.java:890 - Exception in thread -``` - -Field structure: -- Position 1: Level (INFO/WARN/ERROR/DEBUG) -- Position 2: `[ThreadName-N]` — thread pool and worker index -- Position 3–4: Timestamp `YYYY-MM-DD HH:mm:ss,SSS` (comma subsecond separator, not dot) -- Position 5: `ClassName.java:lineNumber` -- Position 6: ` - ` separator -- Position 7+: Free-text message body - -**Candidate query filters for Stage 4:** -1. **Precise:** `service.name:cassandra` or `source:cassandra` — preferred if log collector tags correctly; use if Filebeat/Fluentd/OTel sets service name -2. **Broader fallback:** filter on known thread names: `"StorageService" OR "MutationStage" OR "ReadStage" OR "CompactionExecutor"` — works without proper service tagging; risk = false positives from any Java app with similar thread names - -**Attribute mapping hints:** - -| Raw field | Suggested Tsuga key | Confidence | Notes | -|---|---|---|---| -| Log level (INFO/WARN/ERROR) | `level` | High | Standard log level | -| Thread name (e.g., ReadStage-1) | `cassandra.thread` | Medium | Contains thread pool identity useful for correlating with metrics | -| Timestamp | `timestamp` | High | Standard; note comma subsecond separator requires custom grok | -| Source class (ClassName.java:line) | `cassandra.source_class` | Medium | Useful for filtering specific Cassandra internals | -| Message body | `message` | High | Free text; further Grok parsing for specific warnings | -| Keyspace (from tombstone/query warnings) | `cassandra.keyspace` | Medium | Parseable from "SELECT * FROM keyspace.table" message patterns | -| Table name | `cassandra.table` | Medium | Same parsing as keyspace | -| Tombstone count (from WARN messages) | `cassandra.tombstones_scanned` | Low | Only present in tombstone warning log lines | -| Node address (from StorageService logs) | `cassandra.node` | Medium | IP or hostname extracted from "Node ip/host state jump to..." | - -### Best-practice inference - -**Parsing risks:** -- **Multiline stack traces:** Lines starting with whitespace or `\tat` are continuation lines of a previous ERROR/WARN. Requires regex multiline continuation pattern. -- **Thread name variability:** Thread names contain pool type + hyphen + number (e.g., `ReadStage-3`); the number changes with thread pool size. -- **Embedded CQL queries:** Tombstone warning messages contain embedded CQL SELECT queries that may span long lines with column lists, keyspace.table names, and WHERE clauses. -- **Comma in timestamp:** The subsecond separator is `,` not `.` — standard strptime patterns will fail; needs custom grok `%{NUMBER:subsecond}` or character class match. -- **Audit log format difference:** If audit logging is enabled, a second log source with different format will appear; must split-route or differentiate by filename. -- **GC log format entirely different:** JVM GC logs use their own format and must be parsed with GC-specific patterns. - ---- - -## Caveats and footguns - -- **[read-latency, write-latency]** `cassandra.client.request.read.latency.*` and all JMX latency metrics are **pre-computed percentile gauges**, NOT histograms. Use `average` aggregation + `none` post-function. Using `sum` or rate post-functions produces meaningless numbers. (Confirmed, OTel java-contrib source) -- **[throughput]** `cassandra.client.request.count` is a **cumulative counter**. Use `rate` post-function for bucketed (timeseries) widgets to show requests/second. Displaying the raw value gives an ever-increasing line with no operational meaning. (Confirmed) -- **[throughput, read-latency]** `cassandra.table.operation.count` **double-counts** at cluster level: the coordinator counts a request, and each replica also counts its local operation. Never sum `table.operation.count` across all hosts for cluster-wide throughput — use `cassandra.client.request.count` instead. (Confirmed, Datadog guide) -- **[compaction]** `cassandra.compaction.tasks.pending` being **non-zero is normal**. Cassandra always has background compaction work. Only a steadily growing trend above 10–20 is a problem. A widget showing "5 pending" looks alarming but is healthy. (Confirmed, Cassandra docs) -- **[cache-efficiency]** Row cache (`cassandra.table.cache.hit` metrics) is **disabled by default**. Absence of row cache metrics is expected, not an error. Only enable row cache for narrow, frequently-read hot rows — large rows make it counterproductive. (Confirmed) -- **[cache-efficiency]** `cassandra.table.bloom_filter.false_ratio` near **0 is GOOD, near 1 is BAD**. This is an error rate, not a hit rate. A widget using default "more = better" coloring will mislead. Invert your mental model. (Confirmed) -- **[memory-gc]** `jvm.memory.heap` (heap used) **without knowing max heap** gives no utilization context. Raw bytes alone is not actionable for on-call. Always pair with `jvm.memory.heap.max` in a utilization ratio derived signal. (Best-practice inference) -- **[memory-gc]** Read latency p99 spikes are frequently caused by **JVM stop-the-world GC pauses**, not slow disk. Always correlate latency spikes with GC pause duration before assuming a disk or compaction issue. (Confirmed, Cassandra performance docs) -- **[throughput]** CAS (LWT) operations (`operation:cas_read`, `operation:cas_write`) are **significantly more expensive** than normal operations and inflate latency averages if mixed with regular reads/writes. Separate CAS metrics; do not include them in general throughput averages. (Confirmed) -- **[thread-pools]** Thread pool names vary by Cassandra version. Key pools: `ReadStage`, `MutationStage`, `CompactionExecutor`, `MemtableFlushWriter`, `GossipStage`, `AntiEntropyStage`. If `pool.name` attribute is missing, thread pool metrics are unsliced and cannot be triaged by pool. Stage 2 must verify exact attribute values. (Confirmed, Cassandra threading model) -- **[errors-dropped]** `cassandra.thread_pool.dropped_tasks` is a **cumulative counter**. A non-zero value means drops have occurred since startup, not that they are happening now. Use `rate` to see current drop rate; `increase` to see drops in a time window. (Confirmed) -- **[compaction]** `cassandra.compaction.tasks.completed` is cumulative — use `rate` for "completions per second". Displaying raw value gives an ever-increasing, meaningless line. (Confirmed) -- **[read-latency]** `cassandra.table.tombstone_scanned` p95 > 100 is a **warning sign**. Cassandra's default `tombstone_warn_threshold` is 1,000 and `tombstone_failure_threshold` is 100,000 (set in cassandra.yaml, not surfaced as metrics). These are per-query limits. (Confirmed, cassandra.yaml docs) -- **[storage-hints]** `cassandra.storage.total_hints.count` is a cumulative counter of hints dispatched — **not the current backlog**. Use `cassandra.storage.total_hints.in_progress.count` (gauge) as the real-time measure of pending hint delivery. If in-progress is growing while a node is down, data durability is at risk. (Confirmed) -- **[read-latency, write-latency]** JMX metrics are **sampled at the JMX scrape interval** (not every operation). High-throughput clusters may show slightly smoothed latency values. Percentiles are still directionally accurate but may miss very brief spikes narrower than the scrape interval. (Best-practice inference) -- **[cache-efficiency]** Key cache size is bounded by `key_cache_size_in_mb` in cassandra.yaml. If the key cache is full, hit rate plateaus even on warm workloads — this is expected, not a fault. Increasing key_cache_size_in_mb can improve it. (Confirmed) -- **[compaction]** TWCS (TimeWindowCompactionStrategy) tables generate **large pending compaction counts at window boundaries** — this is expected behavior for time-series workloads and not indicative of a problem. Threshold alerting on compaction pending is dangerous for TWCS tables. (Confirmed, TWCS docs) -- **[cluster-health]** Cassandra node "DOWN" in gossip **does not immediately mean data is unavailable**. With RF=3 and CL=QUORUM, two remaining nodes still serve reads and writes. Availability depends on CL + RF, not raw node count. (Confirmed) -- **[storage-hints]** `cassandra.storage.load.count` reports **data WITHOUT snapshot content** on some Cassandra versions (as labeled in Datadog reference dashboard). Do not compare directly to `df` output — it is Cassandra's internal accounting of live data size. (Confirmed, reference dashboard) -- **[read-latency, write-latency]** `cassandra.table.operation.latency` uses a `percentile` attribute with values `50p`, `99p`, and `max` (confirm in Stage 2). Always filter to specific percentile when using this metric — grouping by percentile produces nonsensical aggregated series. (Confirmed OTel, attribute values INFERRED) -- **[throughput]** `cassandra.table.coordinator.read` and `cassandra.table.coordinator.scan` are per-keyspace/table. Grouping across keyspaces without a Top-N limit can produce cardinality explosion if many tables exist. Always apply `top_n`. (Best-practice inference) -- **[compaction]** `cassandra.table.bloom_filter.false_ratio` increases when SSTables are old and not yet compacted. It resets to near-zero after compaction completes for a table. A rising bloom filter false ratio is a leading indicator that compaction is falling behind for that table. (Confirmed) - ---- - -## Confirmed Tsuga prefixes - -- `cassandra.*` — **CONFIRMED** (Stage 2: 16 metrics discovered; cassandra.client.*, cassandra.compaction.*, cassandra.storage.* present; cassandra.table.* and cassandra.thread_pool.* absent — require OTel collector config update) -- `jvm.*` — **CONFIRMED** (Stage 2: 28 JVM metrics discovered; includes jvm.memory.heap.*, jvm.gc.collections.*, jvm.gc.duration, jvm.cpu.*, jvm.thread.count, jvm.class.*) - -**Stage 2 key corrections applied:** -- Attribute for operation type: `context.operation` (NOT `context.cassandra.operation`) -- Attribute for error status: `context.status` (NOT `context.cassandra.status`) -- Per-node dimension for cassandra.* metrics: `context.service.name` (no `context.host` on these metrics) -- Per-node dimension for jvm.* metrics: `context.host.name` -- Range latency metric name: `cassandra.client.request.range_slice.latency.*` (NOT `range.latency`) -- Max latency variants confirmed present for read, write, and range_slice - ---- - -## Discovery status - -Discovery: not yet performed (deferred to Stage 2) - ---- - -## Top sources - -1. https://github.com/open-telemetry/opentelemetry-java-contrib/blob/main/jmx-metrics/docs/target-systems/cassandra.md — canonical OTel JMX metric names, attribute names, types, and units for Cassandra -2. https://cassandra.apache.org/doc/latest/cassandra/operating/metrics.html — official Cassandra JMX MBean metrics reference (maps JMX to semantic names) -3. https://www.datadoghq.com/blog/how-to-monitor-cassandra-performance-metrics/ — golden signal selection, threshold guidance, and metric interpretation for on-call use -4. https://cassandra.apache.org/doc/latest/cassandra/operating/compaction/ — compaction strategies, tuning options, and operational monitoring for STCS/LCS/TWCS -5. https://www.datadoghq.com/blog/tlp-cassandra-dashboards/ — dashboard design rationale and curated metric selection from Cassandra experts -6. https://cassandra.apache.org/doc/latest/cassandra/operating/logging.html — log format, log levels, file locations, and runtime configuration -7. https://cassandra.apache.org/doc/latest/cassandra/configuration/cass_yaml_file.html — cassandra.yaml reference (tombstone thresholds, cache sizes, GC tuning, hint TTL) -8. https://github.com/open-telemetry/opentelemetry-java-contrib/blob/main/jmx-metrics/docs/target-systems/jvm.md — OTel JMX JVM target metrics (heap, GC count, GC elapsed) -9. https://cassandra.apache.org/doc/latest/cassandra/operating/hints.html — hinted handoff mechanism, hint TTL, monitoring hint accumulation and replay -10. https://cassandra.apache.org/doc/latest/cassandra/operating/read_repair.html — read repair mechanics, consistency implications, and latency overhead diff --git a/plugins/tsuga/skills/knowledge-technology/references/cassandra/metrics.csv b/plugins/tsuga/skills/knowledge-technology/references/cassandra/metrics.csv deleted file mode 100644 index 17a19e4..0000000 --- a/plugins/tsuga/skills/knowledge-technology/references/cassandra/metrics.csv +++ /dev/null @@ -1,35 +0,0 @@ -theme,name,confirmed,definition,unit,type,tsuga_metric_name,widget_suggestions,aggregation,post_function,group_by,required_context_fields,optional_context_fields,absence_means,cardinality_footguns,variants,citations -Throughput/Usage,cassandra.client.request.count,true,"Cumulative count of client requests by operation type (read/write/range_slice/cas_read/cas_write/view_write). This is the primary client-facing throughput metric — use it for cluster-wide rates, not table.operation.count which double-counts due to replication.",{request},counter,cassandra.client.request.count,timeseries|query-value|top-list,sum,rate,context.operation:6,context.env|context.team,context.service.name,"OTel receiver not running or JMX connection failed; check collector logs","Do NOT group by context.operation and context.service.name simultaneously at full cardinality; use top-list with top_n",otel=cassandra.client.request.count|dd=cassandra.latency.one_minute_rate,https://github.com/open-telemetry/opentelemetry-java-contrib/blob/main/jmx-metrics/docs/target-systems/cassandra.md -Errors/Failures,cassandra.client.request.error.count,true,"Cumulative count of client request errors by operation type and status (Timeout/Unavailable/etc.). Non-zero rate indicates replicas are not responding within timeout. Distinguish from dropped messages (node-internal failure mode) — these are coordinator-level errors returned to clients.",{request},counter,cassandra.client.request.error.count,timeseries|query-value,sum,rate,context.operation:6|context.status:10,context.env|context.team,context.service.name,"OTel receiver issue or no errors occurring (both are possible — check request count is present)","Grouping by both operation and status simultaneously may produce many series; prefer top-list with single group-by",otel=cassandra.client.request.error.count,https://github.com/open-telemetry/opentelemetry-java-contrib/blob/main/jmx-metrics/docs/target-systems/cassandra.md -Performance/Latency,cassandra.client.request.read.latency.50p,true,"Pre-computed p50 read latency from JMX ClientRequest MBean. This is a gauge (not histogram) — use average aggregation with no post-function. Represents the median read response time seen by coordinators across all reads. No host dimension available; use context.service.name for per-node view.",us,gauge,cassandra.client.request.read.latency.50p,timeseries|query-value,average,none,context.service.name:20,context.env|context.team,,"OTel receiver not running or JMX issue","Low cardinality; safe to group by service.name for per-node view",otel=cassandra.client.request.read.latency.50p|dd=cassandra.read_latency.50th_percentile,https://github.com/open-telemetry/opentelemetry-java-contrib/blob/main/jmx-metrics/docs/target-systems/cassandra.md -Performance/Latency,cassandra.client.request.read.latency.99p,true,"Pre-computed p99 read latency from JMX. The primary SLA signal for read performance — most latency alerting is based on this metric. Spikes indicate compaction backlog, GC pauses, tombstone accumulation, or hot partitions. No host dimension; use context.service.name for per-node breakdown.",us,gauge,cassandra.client.request.read.latency.99p,timeseries|query-value,average,none,context.service.name:20,context.env|context.team,,"OTel receiver not running or JMX issue","Safe; group by service.name to isolate degraded nodes",otel=cassandra.client.request.read.latency.99p|dd=cassandra.read_latency.99th_percentile,https://github.com/open-telemetry/opentelemetry-java-contrib/blob/main/jmx-metrics/docs/target-systems/cassandra.md -Performance/Latency,cassandra.client.request.read.latency.max,true,"Pre-computed max read latency from JMX ClientRequest MBean. Gauge — use max aggregation. Max latency reveals worst-case read response times and is key for SLO violation detection and outlier identification. Pairs with p99 to distinguish sustained degradation from single-request spikes.",us,gauge,cassandra.client.request.read.latency.max,timeseries|query-value,max,none,context.service.name:20,context.env|context.team,,"OTel receiver not running or JMX issue","Safe cardinality; use max aggregation to surface worst-case across service instances",otel=cassandra.client.request.read.latency.max,https://github.com/open-telemetry/opentelemetry-java-contrib/blob/main/jmx-metrics/docs/target-systems/cassandra.md -Performance/Latency,cassandra.client.request.write.latency.50p,true,"Pre-computed p50 write latency from JMX. Gauge — use average aggregation. Writes are typically faster than reads in Cassandra (hit Memtable first); high write p50 indicates Memtable flush backlog or commit log contention. No host dimension; use context.service.name for per-node view.",us,gauge,cassandra.client.request.write.latency.50p,timeseries|query-value,average,none,context.service.name:20,context.env|context.team,,"OTel receiver not running or JMX issue","Safe; group by service.name to isolate degraded nodes",otel=cassandra.client.request.write.latency.50p|dd=cassandra.write_latency.50th_percentile,https://github.com/open-telemetry/opentelemetry-java-contrib/blob/main/jmx-metrics/docs/target-systems/cassandra.md -Performance/Latency,cassandra.client.request.write.latency.99p,true,"Pre-computed p99 write latency from JMX. High write p99 indicates Memtable flush pressure, compaction CPU competition, or a slow/down replica node. CAS write latency is included in this metric if CAS operations exist — separate with context.operation filter if needed.",us,gauge,cassandra.client.request.write.latency.99p,timeseries|query-value,average,none,context.service.name:20,context.env|context.team,,"OTel receiver not running or JMX issue","Safe; group by service.name; avoid mixing CAS and non-CAS without filter",otel=cassandra.client.request.write.latency.99p|dd=cassandra.write_latency.99th_percentile,https://github.com/open-telemetry/opentelemetry-java-contrib/blob/main/jmx-metrics/docs/target-systems/cassandra.md -Performance/Latency,cassandra.client.request.write.latency.max,true,"Pre-computed max write latency from JMX. Gauge — use max aggregation. Max write latency reveals worst-case write path stalls due to Memtable flush pressure, commit log contention, or GC pauses. Useful for pinpointing individual node write degradation.",us,gauge,cassandra.client.request.write.latency.max,timeseries|query-value,max,none,context.service.name:20,context.env|context.team,,"OTel receiver not running or JMX issue","Safe cardinality; use max aggregation to surface worst-case across service instances",otel=cassandra.client.request.write.latency.max,https://github.com/open-telemetry/opentelemetry-java-contrib/blob/main/jmx-metrics/docs/target-systems/cassandra.md -Performance/Latency,cassandra.client.request.range_slice.latency.50p,true,"Pre-computed p50 latency for range slice (scan) requests from JMX. Gauge — use average aggregation. Range scans are inherently more expensive than point reads; elevated p50 often indicates wide partitions or heavy scan workloads.",us,gauge,cassandra.client.request.range_slice.latency.50p,timeseries|query-value,average,none,context.service.name:20,context.env|context.team,,"OTel receiver issue or no range slice operations in workload","Safe cardinality; group by service.name",otel=cassandra.client.request.range_slice.latency.50p,https://github.com/open-telemetry/opentelemetry-java-contrib/blob/main/jmx-metrics/docs/target-systems/cassandra.md -Performance/Latency,cassandra.client.request.range_slice.latency.99p,true,"Pre-computed p99 latency for range slice (scan) requests from JMX. Gauge. Range scans are inherently more expensive than point reads; high range scan p99 often indicates tombstone accumulation or wide partitions being scanned.",us,gauge,cassandra.client.request.range_slice.latency.99p,timeseries|query-value,average,none,context.service.name:20,context.env|context.team,,"OTel receiver issue or no range slice operations in workload","Safe cardinality; group by service.name",otel=cassandra.client.request.range_slice.latency.99p|dd=cassandra.range_latency.99th_percentile,https://github.com/open-telemetry/opentelemetry-java-contrib/blob/main/jmx-metrics/docs/target-systems/cassandra.md -Performance/Latency,cassandra.client.request.range_slice.latency.max,true,"Pre-computed max latency for range slice (scan) requests from JMX. Gauge — use max aggregation. Max range scan latency reveals worst-case partition scan overhead and is the leading indicator of cross-partition query SLO violations.",us,gauge,cassandra.client.request.range_slice.latency.max,timeseries|query-value,max,none,context.service.name:20,context.env|context.team,,"OTel receiver issue or no range slice operations in workload","Safe cardinality; use max aggregation",otel=cassandra.client.request.range_slice.latency.max,https://github.com/open-telemetry/opentelemetry-java-contrib/blob/main/jmx-metrics/docs/target-systems/cassandra.md -Capacity/Saturation,cassandra.compaction.tasks.pending,true,"Current number of compaction tasks pending execution on this node. Gauge. Non-zero is normal; a growing trend above 10-20 indicates compaction is falling behind write rate. This is the most Cassandra-specific saturation signal with direct causal link to read latency.",{task},gauge,cassandra.compaction.tasks.pending,timeseries|query-value,max,none,context.service.name:20,context.env|context.team,,"OTel receiver issue; or Cassandra node has no pending compactions (valid)","Safe; max per service.name is the right aggregation for saturation visibility",otel=cassandra.compaction.tasks.pending|dd=cassandra.pending_compactions,https://github.com/open-telemetry/opentelemetry-java-contrib/blob/main/jmx-metrics/docs/target-systems/cassandra.md -Throughput/Usage,cassandra.compaction.tasks.completed,true,"Cumulative count of completed compaction tasks. Counter — use rate for completions/second. A healthy compaction rate should exceed or match pending task growth rate. Low completion rate with high pending = compaction throughput insufficient for write load.",{task},counter,cassandra.compaction.tasks.completed,timeseries,sum,rate,context.service.name:20,context.env|context.team,,"OTel receiver issue; or no compactions have completed (unusual after startup)","Safe cardinality",otel=cassandra.compaction.tasks.completed,https://github.com/open-telemetry/opentelemetry-java-contrib/blob/main/jmx-metrics/docs/target-systems/cassandra.md -Capacity/Saturation,cassandra.storage.load.count,true,"[VERIFY: Tsuga reports as counter/cumulative — use rate post-function to show storage growth rate in bytes/sec. If OTel collector reports UpDownCounter as cumulative sum, the raw value may also represent current bytes on disk.] Data volume managed by this node excluding snapshot content.",By,counter,cassandra.storage.load.count,timeseries|top-list,sum,rate,context.service.name:20,context.env|context.team,,"OTel receiver issue; or node has no data (unlikely in production)","Safe; max per service.name reveals hotspot nodes",otel=cassandra.storage.load.count|dd=cassandra.nodetool.status.load,https://github.com/open-telemetry/opentelemetry-java-contrib/blob/main/jmx-metrics/docs/target-systems/cassandra.md -Capacity/Saturation,cassandra.storage.total_hints.count,true,"Cumulative count of hints dispatched to the hint queue since node startup. Counter — use rate for current hint dispatch rate. A zero or stable rate indicates all nodes are healthy. Rising rate indicates replica(s) are down and mutations are being queued for replay.",{hint},counter,cassandra.storage.total_hints.count,timeseries,sum,rate,context.service.name:20,context.env|context.team,,"OTel receiver issue; or zero hints dispatched (all nodes healthy — this is good)","Safe cardinality",otel=cassandra.storage.total_hints.count,https://github.com/open-telemetry/opentelemetry-java-contrib/blob/main/jmx-metrics/docs/target-systems/cassandra.md -Capacity/Saturation,cassandra.storage.total_hints.in_progress.count,true,"[VERIFY: Tsuga reports as counter/cumulative — use rate post-function for hints-per-second dispatch rate. Semantically represents hint deliveries in progress (should go up and down); cumulative counter semantics may differ.] Rising rate means a node is down and hint replay is underway.",{hint},counter,cassandra.storage.total_hints.in_progress.count,timeseries|query-value,sum,rate,context.service.name:20,context.env|context.team,,"OTel receiver issue; or zero hints in progress (all nodes healthy)","Safe cardinality",otel=cassandra.storage.total_hints.in_progress.count,https://github.com/open-telemetry/opentelemetry-java-contrib/blob/main/jmx-metrics/docs/target-systems/cassandra.md -Performance/Latency,cassandra.table.bloom_filter.false_ratio,false,"[ABSENT FROM TSUGA — requires collector config update to enable table-level JMX MBean collection.] Bloom filter false positive ratio per table. Gauge. Values near 0 are ideal (few unnecessary disk seeks). Values near 1 mean nearly every SSTable is checked unnecessarily on reads. Rising ratio indicates compaction not merging SSTables fast enough.",1,gauge,cassandra.table.bloom_filter.false_ratio,timeseries|top-list,max,none,context.keyspace:16|context.columnfamily:20,context.env|context.team,,"Metric absent from Tsuga until table-level JMX collection is enabled in OTel collector","High cardinality if many tables exist; always apply top_n limit",otel=cassandra.table.bloom_filter.false_ratio|dd=cassandra.bloom_filter_false_ratio,https://github.com/open-telemetry/opentelemetry-java-contrib/blob/main/jmx-metrics/docs/target-systems/cassandra.md -Throughput/Usage,cassandra.table.cache.hit,false,"[ABSENT FROM TSUGA — requires collector config update.] Cumulative count of cache access results per table: hit/miss/hit_out_of_range for both row cache and key cache. Counter — use rate. Row cache is disabled by default; absence is expected in most deployments.",{hit},counter,cassandra.table.cache.hit,timeseries,sum,rate,context.cache_result:3|context.keyspace:16,context.env|context.team,,"Metric absent from Tsuga; also row cache is disabled by default in Cassandra","High cardinality if grouping by keyspace+table+cache_result simultaneously; prefer filtering by cache_result",otel=cassandra.table.cache.hit,https://github.com/open-telemetry/opentelemetry-java-contrib/blob/main/jmx-metrics/docs/target-systems/cassandra.md -Throughput/Usage,cassandra.table.coordinator.read,false,"[ABSENT FROM TSUGA — requires collector config update.] Cumulative count of coordinator read requests handled per keyspace/table. Counter — use rate. Shows which tables are receiving the most coordinator (client-facing) read traffic. Different from table.operation.count which includes replica-local operations.",{request},counter,cassandra.table.coordinator.read,timeseries|top-list,sum,rate,context.keyspace:16|context.columnfamily:20,context.env|context.team,,"Metric absent from Tsuga until table-level JMX collection is enabled","High cardinality with many tables; always apply top_n",otel=cassandra.table.coordinator.read,https://github.com/open-telemetry/opentelemetry-java-contrib/blob/main/jmx-metrics/docs/target-systems/cassandra.md -Throughput/Usage,cassandra.table.coordinator.scan,false,"[ABSENT FROM TSUGA — requires collector config update.] Cumulative count of coordinator range scan requests per keyspace/table. Counter — use rate. Range scans are more expensive than point reads. High scan rate relative to read rate indicates range query workload amplifying read amplification.",{request},counter,cassandra.table.coordinator.scan,timeseries|top-list,sum,rate,context.keyspace:16|context.columnfamily:20,context.env|context.team,,"Metric absent from Tsuga until table-level JMX collection is enabled","High cardinality; always apply top_n",otel=cassandra.table.coordinator.scan,https://github.com/open-telemetry/opentelemetry-java-contrib/blob/main/jmx-metrics/docs/target-systems/cassandra.md -Capacity/Saturation,cassandra.table.disk.used,false,"[ABSENT FROM TSUGA — requires collector config update.] Current disk bytes used per keyspace/table. Gauge. Use to identify which tables are consuming the most disk space. Disk imbalance between tables can indicate missing TTLs, tombstone accumulation, or compaction falling behind for specific tables.",By,gauge,cassandra.table.disk.used,top-list|timeseries,sum,none,context.keyspace:16|context.columnfamily:20,context.env|context.team,,"Metric absent from Tsuga until table-level JMX collection is enabled","Very high cardinality if many tables; use top-list with top_n",otel=cassandra.table.disk.used,https://github.com/open-telemetry/opentelemetry-java-contrib/blob/main/jmx-metrics/docs/target-systems/cassandra.md -Throughput/Usage,cassandra.table.operation.count,false,"[ABSENT FROM TSUGA — requires collector config update. Also: double-counts at cluster level because each replica node counts its local operations.] Cumulative count of operations per table. Counter — use rate. Use only for per-table relative traffic analysis, not cluster-wide throughput.",{operation},counter,cassandra.table.operation.count,top-list|timeseries,sum,rate,context.operation:6|context.keyspace:16|context.columnfamily:20,context.env|context.team,,"Metric absent from Tsuga until table-level JMX collection is enabled","Very high cardinality if grouping operation+keyspace+table; prefer top_n on table only",otel=cassandra.table.operation.count,https://github.com/open-telemetry/opentelemetry-java-contrib/blob/main/jmx-metrics/docs/target-systems/cassandra.md -Performance/Latency,cassandra.table.operation.latency,false,"[ABSENT FROM TSUGA — requires collector config update.] Pre-computed latency percentiles (50p/99p/max) per operation type and keyspace/table. Gauge — use average aggregation + no post-function. Always filter to a specific percentile value to avoid averaging across different percentile series.",us,gauge,cassandra.table.operation.latency,timeseries|top-list,average,none,context.percentile:3|context.operation:6|context.keyspace:16,context.env|context.team,,"Metric absent from Tsuga until table-level JMX collection is enabled","Extremely high cardinality if grouping percentile+operation+keyspace+table simultaneously; always filter percentile and operation first",otel=cassandra.table.operation.latency,https://github.com/open-telemetry/opentelemetry-java-contrib/blob/main/jmx-metrics/docs/target-systems/cassandra.md -Performance/Latency,cassandra.table.tombstone_scanned,false,"[ABSENT FROM TSUGA — requires collector config update.] Pre-computed percentile count of tombstones scanned per read operation per table. Gauge — use average aggregation. p99 > 100 = warning; p99 > 1000 = critical risk of query failure. Key triage signal for slow reads.",{tombstone},gauge,cassandra.table.tombstone_scanned,timeseries|top-list,max,none,context.percentile:3|context.keyspace:16|context.columnfamily:20,context.env|context.team,,"Metric absent from Tsuga until table-level JMX collection is enabled","High cardinality with many tables; use top-list; filter to p99 percentile",otel=cassandra.table.tombstone_scanned,https://github.com/open-telemetry/opentelemetry-java-contrib/blob/main/jmx-metrics/docs/target-systems/cassandra.md -Performance/Latency,cassandra.table.live_sstable_count,false,"[ABSENT FROM TSUGA — requires collector config update.] Current number of live SSTables per table. Gauge. Too many SSTables = more disk seeks per read = higher read latency. Compaction reduces SSTable count. Values above 10-20 per table warrant investigation.",{sstable},gauge,cassandra.table.live_sstable_count,timeseries|top-list,max,none,context.keyspace:16|context.columnfamily:20,context.env|context.team,,"Metric absent from Tsuga until table-level JMX collection is enabled","High cardinality with many tables; use top-list with top_n",dd=cassandra.live_ss_table_count,https://cassandra.apache.org/doc/latest/cassandra/operating/metrics.html -Capacity/Saturation,cassandra.thread_pool.active_tasks,false,"[ABSENT FROM TSUGA — requires collector config update to enable thread-pool JMX MBean collection.] Current number of tasks actively executing in each thread pool. Gauge. Used alongside pending_tasks to compute saturation ratio. Active tasks at pool maximum = no capacity for new work.",{task},gauge,cassandra.thread_pool.active_tasks,timeseries|query-value,max,none,context.thread_pool:15|context.service.name:20,context.env|context.team,,"Metric absent from Tsuga until thread-pool JMX collection is enabled","Safe; group by thread_pool and service.name",otel=cassandra.thread_pool.active_tasks,https://github.com/open-telemetry/opentelemetry-java-contrib/blob/main/jmx-metrics/docs/target-systems/cassandra.md -Capacity/Saturation,cassandra.thread_pool.pending_tasks,false,"[ABSENT FROM TSUGA — requires collector config update.] Current number of tasks queued and waiting per thread pool. Gauge. This is the leading saturation indicator — pending tasks spike before drops occur. Monitor ReadStage and MutationStage as the most critical pools.",{task},gauge,cassandra.thread_pool.pending_tasks,timeseries|query-value|top-list,max,none,context.thread_pool:15|context.service.name:20,context.env|context.team,,"Metric absent from Tsuga until thread-pool JMX collection is enabled","Safe; max reveals worst-case pool saturation",otel=cassandra.thread_pool.pending_tasks,https://github.com/open-telemetry/opentelemetry-java-contrib/blob/main/jmx-metrics/docs/target-systems/cassandra.md -Errors/Failures,cassandra.thread_pool.dropped_tasks,false,"[ABSENT FROM TSUGA — requires collector config update.] Cumulative count of tasks dropped (timed out before processing) per thread pool. Counter — use rate. ANY non-zero rate is a critical signal: the node is discarding client work. Monitor ReadStage and MutationStage drops as highest priority.",{task},counter,cassandra.thread_pool.dropped_tasks,timeseries|query-value,sum,rate,context.thread_pool:15|context.service.name:20,context.env|context.team,,"Drops have never occurred since node startup (desired state); or metric absent from Tsuga","Safe; rate by thread_pool reveals which operations are being dropped",otel=cassandra.thread_pool.dropped_tasks,https://github.com/open-telemetry/opentelemetry-java-contrib/blob/main/jmx-metrics/docs/target-systems/cassandra.md -Throughput/Usage,cassandra.thread_pool.completed_tasks,false,"[ABSENT FROM TSUGA — requires collector config update.] Cumulative count of tasks successfully completed per thread pool. Counter — use rate for completions/second. Healthy completion rate alongside low pending and zero drops = thread pool operating normally.",{task},counter,cassandra.thread_pool.completed_tasks,timeseries,sum,rate,context.thread_pool:15|context.service.name:20,context.env|context.team,,"Metric absent from Tsuga; or no tasks completed yet","Safe cardinality",otel=cassandra.thread_pool.completed_tasks,https://github.com/open-telemetry/opentelemetry-java-contrib/blob/main/jmx-metrics/docs/target-systems/cassandra.md -Capacity/Saturation,jvm.memory.heap.used,true,"Current JVM heap memory in use by the Cassandra process. Counter/cumulative in Tsuga. Confirmed metric name: jvm.memory.heap.used (not jvm.memory.heap). Heap > 70% of jvm.memory.heap.max triggers more frequent GC; > 80% risks stop-the-world GC pauses that stall all Cassandra operations. Use context.host.name to isolate per-node heap pressure.",By,counter,jvm.memory.heap.used,timeseries|query-value,max,none,context.host.name:20,context.env|context.team,,"JVM target not configured in OTel jmxreceiver; or Cassandra JVM not reachable","Safe cardinality; group by host.name for per-node heap monitoring",otel=jvm.memory.heap.used,https://github.com/open-telemetry/opentelemetry-java-contrib/blob/main/jmx-metrics/docs/target-systems/jvm.md -Capacity/Saturation,jvm.memory.heap.max,true,"Maximum JVM heap size configured for the Cassandra process (-Xmx setting). Counter/cumulative in Tsuga. Confirmed present in Tsuga. Required to compute Heap Utilization ratio (jvm.memory.heap.used / jvm.memory.heap.max). Stable over time unless JVM is restarted.",By,counter,jvm.memory.heap.max,query-value,max,none,context.host.name:20,context.env|context.team,,"JVM target not configured in OTel jmxreceiver; filter returning no results","Safe cardinality",otel=jvm.memory.heap.max,https://github.com/open-telemetry/opentelemetry-java-contrib/blob/main/jmx-metrics/docs/target-systems/jvm.md -Errors/Failures,jvm.gc.collections.count,true,"Cumulative count of JVM garbage collection events. Counter — use rate for collections/second. Frequent major GC (old generation) is more concerning than minor GC. Use context.jvm.gc.action to split minor vs major. Correlate with read/write latency spikes to confirm GC causality.",{collection},counter,jvm.gc.collections.count,timeseries,sum,rate,context.host.name:20,context.env|context.team,context.jvm.gc.action,"JVM target not configured in OTel jmxreceiver","Safe cardinality; group by jvm.gc.action for minor vs major GC breakdown",otel=jvm.gc.collections.count,https://github.com/open-telemetry/opentelemetry-java-contrib/blob/main/jmx-metrics/docs/target-systems/jvm.md -Errors/Failures,jvm.gc.collections.elapsed,true,"Cumulative time spent in JVM garbage collection in milliseconds. Counter — use rate for ms-of-GC-per-second. High GC elapsed time directly explains read/write latency spikes. GC overhead > 5% (50ms/s) is warning; > 10% (100ms/s) is critical. Use context.jvm.gc.action to split minor vs major GC time.",ms,counter,jvm.gc.collections.elapsed,timeseries|query-value,sum,rate,context.host.name:20,context.env|context.team,context.jvm.gc.action,"JVM target not configured in OTel jmxreceiver","Safe cardinality; group by gc.action for breakdown",otel=jvm.gc.collections.elapsed,https://github.com/open-telemetry/opentelemetry-java-contrib/blob/main/jmx-metrics/docs/target-systems/jvm.md -Cost,NOT_APPLICABLE,true,"Cassandra is self-hosted open-source software. There is no per-request or per-byte cloud cost metric exported by Cassandra itself. Proxy cost metrics: use cassandra.storage.load.count for storage provisioning cost, and cassandra.client.request.count for request volume as sizing input.",N/A,N/A,NOT_APPLICABLE,note,N/A,N/A,N/A,N/A,N/A,N/A,No cost metrics available from Cassandra JMX,N/A,, diff --git a/plugins/tsuga/skills/knowledge-technology/references/cassandra/queries.md b/plugins/tsuga/skills/knowledge-technology/references/cassandra/queries.md deleted file mode 100644 index cb238da..0000000 --- a/plugins/tsuga/skills/knowledge-technology/references/cassandra/queries.md +++ /dev/null @@ -1,55 +0,0 @@ -# Cassandra - -Wide-column distributed NoSQL. Healthy: low error rate, bounded p99, compaction keeping up, low hints, low tombstone scans. - -## Incident shapes - -- **Read latency regression** — read p99 spikes → check cache, compaction, tombstones -- **Compaction backlog** — pending compactions climb → writes outpacing compaction -- **Tombstone storm** — scanned tombstones dominate live rows → queue-like table or TTL issue -- **Hint accumulation** — `total_hints.in_progress` rises → replicas unreachable -- **Hot coordinator** — `coordinator.read` / `scan` skewed on one node → hot partition or token imbalance - -## Key metrics - -| Metric | Unit | Signal | -|---|---|---| -| `cassandra.client.request.count` | req/s | Throughput baseline | -| `cassandra.client.request.error.count` | err/s | Replica unavailability / client issue | -| `cassandra.client.request.read.latency.99p` | ms | Read p99 | -| `cassandra.client.request.write.latency.99p` | ms | Write p99 | -| `cassandra.client.request.range_slice.latency.99p` | ms | Range-scan p99 (expensive) | -| `cassandra.compaction.tasks.pending` | count | > 100 sustained = falling behind | -| `cassandra.compaction.tasks.completed` | count | Compaction throughput | -| `cassandra.storage.load.count` | bytes | Per-node data load | -| `cassandra.storage.total_hints.count` | count | Stored hints (offline replicas) | -| `cassandra.storage.total_hints.in_progress.count` | count | Active hint handoff | -| `cassandra.table.bloom_filter.false_ratio` | ratio | > 0.01 = bloom ineffective | -| `cassandra.table.tombstone_scanned` | count | Tombstones examined per read | -| `cassandra.table.disk.used` | bytes | Per-table footprint | -| `cassandra.table.operation.count` / `latency` | — | Per-table stats | - -## Derived signals - -- `request.error.count / request.count` — error rate. Baseline ≈ 0. -- `compaction.tasks.pending` — backlog. > 100 after flush = behind. -- `table.tombstone_scanned / table.operation.count` — tombstone pressure. Rising on time-series or queue-like tables = TTL review needed. - -## Log patterns - -- `TombstoneOverwhelmingException` — single query exceeded tombstone scan limit -- `Compacting large partition` — hot-partition warning -- `Read 1000 live rows and N tombstone cells` — N ≫ 1000 = tombstone storm -- `Hinted handoff started for` / `completed for` — replay activity -- `Flushing largest memtable` — writes outpacing flush -- `GC Young / Old generation pause` — pauses > 200ms correlate with client timeouts -- `Timed out waiting for N responses` — coordinator couldn't reach replicas -- `Not enough live nodes` — quorum lost - -## Gotchas - -- Latency percentiles are per-node; a hot node hides in cluster averages. -- Tombstone scans rise with `IN` clauses over many TTL'd keys — often a modeling issue (Cassandra used as a queue). -- GC pauses look like latency incidents; pair with JVM metrics (`jvm.md`) before blaming Cassandra. -- `nodetool repair` is heavy and looks like an incident; check for scheduled repair activity first. -- Token imbalance creates hot-node hotspots that aggregate metrics hide. Compare disk.used per node. diff --git a/plugins/tsuga/skills/knowledge-technology/references/envoy/README.md b/plugins/tsuga/skills/knowledge-technology/references/envoy/README.md deleted file mode 100644 index 76afb76..0000000 --- a/plugins/tsuga/skills/knowledge-technology/references/envoy/README.md +++ /dev/null @@ -1,288 +0,0 @@ -# Envoy Integration Context Bundle - -## Metadata -**Technology:** Envoy -**Deployment:** self-hosted -**Environment:** prod -**Persona:** SRE Dev and ops -**Telemetry preference:** mixed -**Integration scope:** core service only -**Primary use-case:** reliability and performance - -## How to use this bundle -- Use `01_envoy_metrics.csv` as the source of truth for metric names, units, types, and safe query behavior. -- Use `02_envoy_dashboard_plan.yaml` for the dashboard section and widget blueprint, derived signal formulas, explanation notes, triage chains, and operational playbooks. -- Use `03_envoy_state.yaml` for machine-readable reconciliation status and remaining unknowns. -- Use `04_envoy_memory.md` for human-readable reconciliation and implementation memory. -- Stage 2 creates/refreshes `05_envoy_metric_catalog.csv` as the discovered inventory and context-key curation layer. -- Stage 4 should start with `## Log intelligence (Stage 4 handoff)` in this file and `03_envoy_state.yaml` `log_intel` before building/updating log routes. - -## What it is and what "good" looks like - -### Confirmed by sources -- Envoy is a high-performance L4/L7 proxy used as a data-plane component for ingress, egress, and service-to-service routing in Kubernetes and non-Kubernetes environments. [S1][S2] -- Envoy exposes rich operational telemetry through `/stats` and `/stats/prometheus`, including upstream request outcomes, connection lifecycle, listener pressure, and server runtime signals. [S1][S3][S4][S5] -- "Good" for core Envoy operations means: healthy upstream host pools, low upstream 5xx/timeout fractions, stable latency (not just stable volume), and no sustained circuit-breaker or overflow pressure. [S3][S6] -- For an SRE triage surface, the fastest route is to split the experience into: availability/health, traffic, errors, latency, saturation, and runtime resource pressure. [S2][S3][S4][S5] -- Common incident shapes are directly visible in Envoy core stats families: upstream dependency failure spikes (`upstream_rq_5xx`, `upstream_rq_timeout`), saturation (`*_overflow`, circuit-breaker open states), and load/latency shifts (`upstream_rq_total`, `upstream_rq_time`). [S3][S6][S7] - -### Best-practice inference -- **Incident shape 1: Upstream dependency degradation** - - Symptom: request volume is steady but `5xx`, `timeout`, and retries climb. - - Start in section: **Errors & Failures**. -- **Incident shape 2: Proxy saturation before hard failure** - - Symptom: active connections and pending overflow rise; circuit breakers begin opening. - - Start in section: **Saturation & Capacity**. -- **Incident shape 3: Latency regression without outright errors** - - Symptom: success rate looks healthy but mean upstream request time drifts upward. - - Start in section: **Latency & Performance**. -- High-level paging intent for dashboards: detect user-impacting routing failure early, distinguish upstream dependency issues from Envoy capacity bottlenecks, and direct on-call to the next section without requiring Envoy internals expertise. - -## Key concepts - -### Glossary - -| Term | Definition | Operational meaning | Dashboard section | -|---|---|---|---| -| Listener | Socket binding where downstream traffic enters Envoy | Affects listener pressure and accept failures | availability-health, saturation-capacity | -| Filter chain | Ordered processing path selected on listener match conditions | Mismatches can drop or misroute traffic | availability-health | -| HTTP connection manager | L7 HTTP handling component that emits downstream request counters by response class | Basis for downstream request class counters | traffic-volume, errors-failures | -| Cluster | Upstream endpoint group Envoy routes to | Primary unit for upstream success/error telemetry | traffic-volume, errors-failures, latency-performance | -| Endpoint/host | Individual upstream target in a cluster | Health and ejection state drive routing quality | availability-health | -| Circuit breaker | Hard cap on concurrent connections/requests/retries to protect upstreams | When open, predicts request drops/latency spikes | saturation-capacity | -| Outlier detection | Automatic ejection of failing upstream hosts based on error patterns | Reduces healthy host denominator | errors-failures, availability-health | -| Upstream request | Request Envoy sends to dependency cluster | Basis for `upstream_rq_*` families | traffic-volume, errors-failures | -| Downstream request | Request accepted from client side | Basis for `downstream_rq_*` families | traffic-volume | -| Active downstream connections | Current open client connections per listener | Saturation pressure indicator | saturation-capacity | -| Active upstream connections | Current open backend connections per cluster | Backend connection pressure | saturation-capacity | -| Pending overflow | Rejected/overflowed pending upstream requests due to limits | Already request loss, not just warning | saturation-capacity, errors-failures | -| Retry | Envoy reattempt of failed upstream call under retry policy | Can amplify load on unstable dependencies | errors-failures | -| Upstream timeout | Request exceeded configured upstream timeout budget | Contributes to 5xx/user-visible failures | errors-failures, latency-performance | -| Response code class | 2xx/4xx/5xx grouping used in request outcome counters | Fast error class attribution | errors-failures | -| Request time histogram | Distribution of upstream request durations | Early warning for dependency slowness | latency-performance | -| Healthy membership | Count of currently healthy hosts in a cluster | Denominator for many interpretations | availability-health | -| Server live flag | Runtime liveness indicator for Envoy process | Confirms process liveness only | availability-health | -| Heap size / allocated memory | Runtime memory pressure indicators for Envoy process | Resource pressure signal | resource-runtime | -| Admin stats endpoint | `/stats` and `/stats/prometheus` interfaces for telemetry export | Telemetry and ingestion prerequisites | telemetry | -| Stat tags | Envoy mechanism to transform stat names into labels for backends like Prometheus | Group-by and cardinality design | all | -| Hot restart epoch | Generation marker during hot restart; can cause short-lived metric discontinuities | Runtime caveats | resource-runtime | - -[S1][S2][S3][S4][S5][S6][S8][S11][S12] - -### Concept Map - -```text -Client traffic -> enters -> Envoy listener (why: first choke point for downstream availability) -Listener -> routes through -> filter chain / HTTP connection manager (why: request classification and policy application) -HTTP connection manager -> forwards to -> upstream cluster (why: dependency routing boundary) -Upstream cluster -> contains -> upstream hosts/endpoints (why: healthy host pool drives success capacity) -Upstream host health checks -> update -> healthy membership counts (why: denominator for load and failure interpretation) -Healthy membership drop -> increases -> per-host load (why: latency and error risk rise before full outage) -Upstream requests -> produce -> response code class counters (why: fast 2xx/4xx/5xx outcome split) -Upstream requests -> accumulate -> timeout and reset counters (why: dependency-path failure signals) -Retry policy -> triggers -> retry attempts (why: can mask or amplify dependency instability) -Retry attempts -> consume -> retry circuit-breaker budget (why: breaker-open means retries are refused) -Active downstream connections -> consume -> listener/proxy capacity (why: saturation precursor) -Active upstream connections -> consume -> cluster pool capacity (why: backend pressure precursor) -Pending requests -> overflow when -> queue/limit is exceeded (why: immediate request loss signal) -Circuit breaker open state -> blocks -> new requests/connections/retries (why: overload protection already engaged) -Request latency histogram -> reflects -> upstream/service processing health (why: degradation can precede error spikes) -Envoy runtime memory/heap metrics -> reflect -> process resource pressure (why: runtime exhaustion risk) -Service/team/env/cluster context -> maps to -> ownership and blast-radius filters (why: triage routing) -Rollouts/config changes -> alter -> listener/cluster behavior and stat shape (why: sudden metric shifts may be config-driven) -``` - -### Entities and dimensions - -| Entity/Dimension | Why useful | Cardinality risk | Safe top-N | Do NOT group-by guidance | -|---|---|---|---|---| -| `context.env` | Environment split for prod vs non-prod | Low | 5 | Never omit from global filter in shared org dashboards | -| `context.team` | Ownership/triage routing | Low | 10 | Avoid as first investigative split for latency root cause | -| `context.service.name` | Service boundary in mesh/ingress stacks | Medium | 20 | Avoid combining with pod UID in same widget | -| `context.k8s.cluster.name` | Multi-cluster posture and blast radius | Low-Medium | 10 | Do not mix with high-card labels in overview | -| `context.k8s.namespace.name` | Tenant/workload segmentation | Medium | 20 | Avoid on global KPI widgets unless needed | -| `context.scope.name` | Stable instance/workload scope fallback | Medium | 20 | Prefer over ephemeral pod ids when host missing | -| `context.host` | Host-level hotspot detection | Medium-High | 20 | Fallback to `context.scope.name` if absent | -| `context.envoy.cluster_name` | Upstream dependency isolation | Medium | 20 | Avoid cluster+route+pod triple split | -| `context.envoy.listener_name` | Entrypoint diagnosis | Medium | 20 | Do not use for global high-level KPIs | -| `context.response_code_class` | Fast error class attribution | Low | 5 | Use class first; drill into exact code only when needed | -| `context.response_code` | Exact protocol failure signal | High | 20 | Avoid in overview due cardinality and noise | -| `context.envoy.http_conn_manager_prefix` | Distinguishes HCM instances | Medium | 10 | Avoid when prefix is dynamic per route | -| `context.k8s.pod.name` | Pod hotspot and rollout impact | High | 20 | Never in overview top-level KPI widgets | -| `context.instance.id` | Process-specific debugging | High | 20 | Avoid if rotates frequently (autoscale/churn) | - -### Tsuga field mapping - -| Vendor/exporter dimension | Recommended context.* key | Must-exist vs optional | -|---|---|---| -| `envoy_cluster_name` (Prom labels/tags) | `context.envoy.cluster_name` | Optional — if absent, derive cluster from service/routing metadata | -| `envoy_listener_name` | `context.envoy.listener_name` | Optional — needed for listener-focused saturation views | -| `envoy_http_conn_manager_prefix` | `context.envoy.http_conn_manager_prefix` | Optional — useful where multiple HCM prefixes exist | -| `response_code_class` | `context.response_code_class` | Optional — needed for 2xx/4xx/5xx splits | -| `response_code` | `context.response_code` | Optional — high cardinality, use sparingly | -| `cluster` (generic service mesh tag) | `context.envoy.cluster_name` | Optional — normalize to single key to avoid duplicates | -| `service` | `context.service.name` | Optional — strongly recommended for multi-service meshes | -| `namespace` | `context.k8s.namespace.name` | Optional — expected in k8s deployments but not guaranteed | -| `kubernetes_cluster` | `context.k8s.cluster.name` | Optional — validate real field name in Stage 2 | -| `pod` | `context.k8s.pod.name` | Optional — debug-only, cardinality risk | -| `instance` | `context.scope.name` | Optional — preferred stable fallback where host is absent | -| `host` | `context.host` | Optional — must be validated; not universally present in Envoy telemetry | -| Organization environment tag | `context.env` | Must-exist — declared org standard in `.env` | -| Organization team tag | `context.team` | Must-exist — declared org standard in `.env` | - -[S1][S3][S4][S5][S6][S8][S9][S11][S12] - -## Golden signals - -| Signal | What it means for Envoy | Primary telemetry families | Typical degradation causes | What people page on | Section questions | -|---|---|---|---|---|---| -| **Traffic** | Request and connection workload entering Envoy and forwarded to upstream clusters | `upstream_rq_total`, `downstream_rq_total`, `downstream_cx_total` [S3][S4][S5] | Traffic surges, retry storms, client reconnect churn, uneven load distribution | Sudden request-rate jumps with concurrent latency drift or overflow pressure | Are downstream and upstream volumes aligned? Which clusters/listeners carry disproportionate load? | -| **Errors** | Routing outcomes where requests fail at proxy or upstream dependency boundary | `upstream_rq_5xx`, `upstream_rq_4xx`, `upstream_rq_timeout`, `upstream_rq_pending_overflow`, `downstream_rq_xx` [S3][S4] | Unhealthy host pools, connect failures/timeouts, overload/circuit-breaker pressure, upstream app failures | Sustained 5xx growth, timeout ratio spikes, overflow non-zero under normal load | Are failures dominated by upstream errors, timeouts, or proxy overload? Is retry behavior masking deeper upstream instability? | -| **Latency** | Time spent serving and forwarding requests, especially upstream round-trip latency | `upstream_rq_time` histogram [S3] | Upstream slowness, connection establishment delays, queue buildup before hard failures | Mean request time climbing while success rate still looks acceptable | Is latency increase broad or isolated to specific clusters? Are connect issues or upstream processing time driving the regression? | -| **Saturation** | How close proxy and upstream protective limits are to being hit | `downstream_cx_active`, `upstream_cx_active`, circuit-breaker open gauges, pending overflow, healthy membership ratios [S3][S5][S6] | Insufficient upstream capacity, bad retry policy, connection pool exhaustion, hot listener/cluster imbalance | Breaker open flags, overflow > 0, active connections pinned high with dropping healthy hosts | Is Envoy saturating before upstream fails completely? Which control limits are active right now? | - -### Best-practice inference -- Treat **healthy host ratio** as the denominator for many other interpretations: rising load on shrinking healthy pools is worse than rising load on stable pools. -- For k8s operations, pair Envoy signal views with deployment/rollout context even when not directly emitted in Envoy metrics. -- Prefer section-level questions that separate **dependency failure** from **proxy exhaustion** because the first responder actions differ. -- Use a compact KPI wall in overview and preserve high-cardinality drilldowns for deep dive only. - -## Telemetry sources - -| Source type | How collected | What it provides | Pros | Cons | Common pitfalls | -|---|---|---|---|---|---| -| Envoy admin `/stats/prometheus` | Prometheus scrape from Envoy admin endpoint | Flattened `envoy_*` metric families with labels/tags | Direct, broad coverage of cluster/listener/http/server signals | Requires secure admin endpoint exposure and scrape hygiene | Mistaking missing optional families for zero traffic; admin endpoint access left too open [S1][S7] | -| Envoy admin `/stats` (text) | Direct pull of text counters/gauges/histograms | Canonical stat names (dot notation) and raw values | Closest to Envoy-native naming and semantics | Requires parser/transform layer for TSDB ingestion | Naming mismatch between dot stats and backend-normalized names [S1][S2] | -| Envoy Gateway prebuilt telemetry | Kubernetes deployment with Envoy Gateway and Prometheus/Grafana integration | Common cluster-level envoy metrics and queries | Faster onboarding for k8s teams | Focused on gateway use cases, not every sidecar/mesh variant | Assuming gateway metric presence in non-gateway Envoy topologies [S7] | -| AWS App Mesh Envoy telemetry | App Mesh control plane + Envoy proxy metrics | Practical metric names and semantic descriptions for envoy server/cluster/listener/http families | Useful operational interpretation and concrete names | App Mesh-specific framing may not match vanilla Envoy | Treating App Mesh-specific behavior as universal Envoy behavior [S8] | -| Control-plane and config metadata (xDS, k8s labels) | Enrichment in collector/pipeline into `context.*` | Service/team/env/namespace/cluster dimensions for filters and bounded group-by | Makes dashboards actionable by ownership and blast radius | Mapping conventions vary by pipeline | Unbounded labels (pod UID, request path) accidentally promoted to group-by [S11] | - -### Optional features that change metrics -- Health-check and outlier subsystems can add/alter host-ejection and health-related stats. [S9][S12] -- Circuit-breaker metric families exist only when cluster circuit breakers are configured/enforced. [S6] -- Some downstream HTTP class metrics depend on HTTP connection manager presence and stat prefix configuration. [S4] - -### What "no data" usually means -- `envoy_http_*` absent: HTTP connection manager stats may be disabled/not emitted for the target deployment mode. -- `envoy_cluster_circuit_breakers_*` absent: circuit breakers not configured or not exported by current pipeline. -- `envoy_cluster_membership_*` absent: service discovery/health-check integration not enabled or not exposed through tags. -- Broad `envoy_*` absence: scrape or ingestion outage, admin endpoint unreachable, or metric relabel/drop rules misconfigured. - -### Best-practice inference -- In Kubernetes, treat scrape reachability and enrichment pipeline health as first-class telemetry dependencies; missing dimensions can be as damaging as missing metric families for dashboard usability. -- Start with cluster/listener/server core families, then expand to more granular HTTP/class/code splits only after confirming context fields and cardinality constraints. - -## Log intelligence (Stage 4 handoff) - -### Confirmed by sources - -| Source | Access path | Typical format | Structured vs unstructured | Evidence | -|---|---|---|---|---| -| Envoy access logs | Envoy proxy container logs (`stdout`) | Text line with quoted request, status, bytes, duration, user-agent, request-id, authority, upstream cluster | Unstructured text | Envoy default access log command operators and live Tsuga samples seen in Stage 4 (`context.log_name=envoy_access_log`) [S1][S2] | -| Envoy admin stats logs | Admin endpoint and telemetry stream | Mostly metric/stat outputs; not request access-line format | Structured-ish telemetry, not ideal for route parsing | Admin operations/statistics docs [S1][S2] | - -Known format (confirmed): - -```text -[2026-02-20T15:09:28.165Z] "GET / HTTP/1.1" 200 - 0 40 0 "fortio.org/fortio-1.67.1" "36836a5c-147c-9ad3-a9b0-0decd0a1e4d6" "10.96.195.207:5678" upstream_v1 -``` - -Format hints: -- Timestamp is bracketed ISO8601 at line start. -- Request method/path/version is inside a quoted segment. -- Authority is quoted and may be `host:port` or host-only. -- Upstream cluster appears as final token. - -Candidate query filters for Stage 4: -- Precise: `context.log_name:envoy_access_log AND context.service.name:envoy*` - - Why: restricts to access-log stream plus envoy service naming. - - Risk: may miss logs if service naming differs from `envoy*`. -- Fallback (broader): `context.service.name:envoy* OR container_name:/envoy` - - Why: catches service-labeled and container-labeled envoy logs. - - Risk: can include non-access envoy logs that need split handling. - -Attribute mapping hints: - -| Raw field | Suggested Tsuga key | Confidence | Notes | -|---|---|---|---| -| HTTP verb | `http.method` | High | Direct Grok extraction | -| HTTP path | `http.url` | High | Parse URL details later with `parse-attribute:url` | -| HTTP version | `http.version` | High | Stored under `http.*` for simplicity | -| Status code | `http.status_code` | High | Use numeric coercion and map-level for `level` | -| Duration | `http.latency` | High | Value is in ms in sample format | -| User agent | `http.useragent` | High | Optional secondary user-agent parser | -| Request id | `http.request_id` | High | Correlation-friendly key | -| Bytes in/out | `network.bytes_received` / `network.bytes_sent` | High | Numeric coercion required | -| Authority host/port | `network.destination.ip` / `network.destination.port` | Medium | Host-only variant needs alternate Grok rule | -| Response flags | `envoy.response_flags` | High | Keep raw flag token | -| Upstream cluster | `envoy.upstream_cluster` | High | Keep in envoy namespace | - -### Best-practice inference -- Maintain two Grok rules for authority variants (`host:port` and host-only) to avoid parse drop. -- Keep `context.log_format=envoy` as required enrichment for filtering and rollout validation. - -Parsing risks to account for in Stage 4: -- Variable authority token shape (`host:port` vs host-only). -- Potential dual formats if custom Envoy access-log template differs between environments. -- Quoted segments with spaces (user-agent) can break naive token-based patterns. -- Propagation delay after route update; verify with retries before declaring failure. - -## Caveats and footguns - -- **[errors-failures]** `upstream_rq_5xx` can rise from a single noisy cluster while global rate still looks normal. Always check cluster split before acting. (https://www.envoyproxy.io/docs/envoy/latest/configuration/upstream/cluster_manager/cluster_stats) -- **[errors-failures]** `upstream_rq_timeout` and `upstream_rq_5xx` often diverge; treating them as equivalent hides latency budget failures. (https://www.envoyproxy.io/docs/envoy/latest/configuration/upstream/cluster_manager/cluster_stats) -- **[latency-performance]** Mean latency from `rq_time_sum / rq_time_count` hides tail spikes; use it as trend indicator not SLA truth. (Inference) -- **[latency-performance]** If temporality is cumulative but widget uses `per-second`, derived latency math will drift. Stage 2 must confirm post-function. (Inference) -- **[traffic-volume]** `http.downstream_rq_total` and `cluster.upstream_rq_total` may represent different scopes; direct ratio can mislead without aligned filters. (https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_conn_man/stats) -- **[traffic-volume]** Request counters can spike during retries and shadow traffic making traffic growth look organic when it is policy-induced. (https://www.envoyproxy.io/docs/envoy/latest/configuration/upstream/cluster_manager/cluster_stats) -- **[saturation-capacity]** `*_pending_overflow` non-zero is already request loss not just warning pressure. (https://www.envoyproxy.io/docs/envoy/latest/configuration/upstream/cluster_manager/cluster_stats) -- **[saturation-capacity]** Circuit-breaker open gauges are binary-like states and should not be averaged over long windows. (https://www.envoyproxy.io/docs/envoy/latest/configuration/upstream/cluster_manager/cluster_circuit_breakers) -- **[saturation-capacity]** High active connections alone is not bad if healthy host pool scales with it. Check per-healthy-host ratio. (Inference) -- **[availability-health]** `membership_healthy` can fall due to health-check config issues not only backend outages. (https://www.envoyproxy.io/docs/envoy/latest/configuration/upstream/cluster_manager/cluster_hc) -- **[availability-health]** `server_live` confirms process liveness only and does not prove route-level correctness. (https://www.envoyproxy.io/docs/envoy/latest/operations/admin) -- **[resource-runtime]** Memory allocated and heap size can increase after config reloads or workload shifts without leak behavior. (Inference) -- **[resource-runtime]** `server.concurrency` is configuration state not runtime utilization; do not treat it as CPU saturation. (https://aws.github.io/aws-app-mesh-controller-for-k8s/reference/envoy_metrics/) -- **[errors-failures, saturation-capacity]** Retry amplification can both mask and worsen incidents by increasing upstream load during partial outages. (https://www.envoyproxy.io/docs/envoy/latest/configuration/upstream/cluster_manager/cluster_stats) -- **[traffic-volume, errors-failures]** `downstream_rq_4xx` can be client behavior noise and should not drive paging by itself. (https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_conn_man/stats) -- **[traffic-volume]** Listener names can be dynamic in some deployments causing cardinality blowups if used as unbounded group-by. (Inference) -- **[availability-health, saturation-capacity]** Cluster names from xDS may include ephemeral suffixes; enforce Top-N and avoid long-tail table widgets by default. (Inference) -- **[latency-performance]** Comparing latency across clusters with very different request size profiles can create false regressions. (Inference) -- **[errors-failures]** Client abort metrics may spike during client deploys and timeouts upstream may remain stable; do not misattribute immediately to backend. (https://www.envoyproxy.io/docs/envoy/latest/configuration/listeners/stats) -- **[availability-health]** Missing metrics may mean feature/path not enabled rather than true zero; use explicit missing-note language. (https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/observability/statistics) -- **[resource-runtime, traffic-volume]** Pod-level grouping in k8s can create churn noise during autoscaling and rollouts; prefer service or scope grouping first. (Inference) -- **[latency-performance, errors-failures]** Connect timeout and connect fail counters should be interpreted together; either one alone can understate network path issues. (https://www.envoyproxy.io/docs/envoy/latest/configuration/upstream/cluster_manager/cluster_stats) -- **[saturation-capacity]** Listener overflow and upstream pending overflow are different choke points; resolving one may not clear the other. (Inference) - -## Confirmed Tsuga prefixes -- `envoy_*` — **CONFIRMED** (284 Envoy metrics discovered in Tsuga during Stage 2 over the 24h window ending 2026-02-16. After Stage 1b consolidation, 46 metrics are confirmed in the curated inventory and 8 legacy names are marked missing with explicit replacements.) - -## Discovery status -Discovery: completed in Stage 2. -- `METRICS_FOUND`: 284 `envoy_*` metrics out of 788 total metrics in the catalog window. -- Key naming shape: response class metrics are exposed as `*_rq_xx` with `context.envoy_response_code_class` instead of dedicated `*_2xx/*_4xx/*_5xx` names. -- Filter convention: `context.envoy_response_code_class` values are numeric classes (`2`, `4`, `5`), not `2xx/4xx/5xx`. -- Key latency shape: request time is exposed as histogram `*_rq_time` rather than `*_time_sum/*_time_count` companion metrics. -- Context key shape in Tsuga uses underscores for Envoy tags (`context.envoy_cluster_name`, `context.envoy_listener_address`, `context.envoy_http_conn_manager_prefix`). -- Stage 1b consolidation is merged into primary artifacts and the promoted secondary metrics are now part of baseline integration scope. - -## Top sources -1. https://www.envoyproxy.io/docs/envoy/latest/operations/admin - Why: canonical documentation for `/stats` and `/stats/prometheus` interfaces. -2. https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/observability/statistics - Why: Envoy statistics model and instrumentation semantics. -3. https://www.envoyproxy.io/docs/envoy/latest/configuration/upstream/cluster_manager/cluster_stats - Why: primary source for cluster upstream request and connection metrics. -4. https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_conn_man/stats - Why: primary source for HTTP downstream request families. -5. https://www.envoyproxy.io/docs/envoy/latest/configuration/listeners/stats - Why: listener connection metrics and overflow semantics. -6. https://www.envoyproxy.io/docs/envoy/latest/configuration/upstream/cluster_manager/cluster_circuit_breakers - Why: definitive breaker-open metrics and limit behavior. -7. https://www.envoyproxy.io/docs/envoy/latest/configuration/upstream/cluster_manager/cluster_hc - Why: health-check behavior that drives healthy membership interpretation. -8. https://www.envoyproxy.io/docs/envoy/latest/configuration/observability/stats - Why: stats tagging and dimension handling guidance for label/group-by design. -9. https://gateway.envoyproxy.io/latest/tasks/observability/proxy-metric/ - Why: k8s Envoy Gateway operational metric examples with concrete `envoy_*` names. -10. https://aws.github.io/aws-app-mesh-controller-for-k8s/reference/envoy_metrics/ - Why: practical Envoy metric naming and usage notes in production k8s meshes. diff --git a/plugins/tsuga/skills/knowledge-technology/references/envoy/metrics.csv b/plugins/tsuga/skills/knowledge-technology/references/envoy/metrics.csv deleted file mode 100644 index f896ddc..0000000 --- a/plugins/tsuga/skills/knowledge-technology/references/envoy/metrics.csv +++ /dev/null @@ -1,63 +0,0 @@ -theme,name,confirmed,definition,unit,type,tsuga_metric_name,widget_suggestions,aggregation,post_function,group_by,required_context_fields,optional_context_fields,absence_means,cardinality_footguns,variants,citations -Availability/Health,server.live,true,Envoy process liveness where value 1 indicates live,1,gauge,envoy_server_live,query-value|timeseries,max,none,,context.env|context.team,context.service.name|context.scope,Envoy process not reporting or scrape path unavailable,Safe,stat=server.live|prom=envoy_server_live,https://www.envoyproxy.io/docs/envoy/latest/operations/admin -Availability/Health,server.uptime,true,Envoy process uptime in seconds,s,gauge,envoy_server_uptime,query-value|timeseries,max,none,,context.env|context.team,context.service.name|context.scope,Recent restart or missing telemetry,Safe,stat=server.uptime|prom=envoy_server_uptime,https://www.envoyproxy.io/docs/envoy/latest/operations/admin -Resource Utilization,server.memory_allocated,true,Runtime allocated memory bytes for Envoy process,By,gauge,envoy_server_memory_allocated,query-value|timeseries,max,none,context.service.name:20|context.scope:20,context.env|context.team,context.service.name|context.scope,Runtime memory telemetry missing,pod level grouping can churn during rollouts,stat=server.memory_allocated|prom=envoy_server_memory_allocated,https://aws.github.io/aws-app-mesh-controller-for-k8s/reference/envoy_metrics/ -Resource Utilization,server.memory_heap_size,true,Runtime heap size bytes for Envoy process,By,gauge,envoy_server_memory_heap_size,query-value|timeseries,max,none,context.service.name:20|context.scope:20,context.env|context.team,context.service.name|context.scope,Runtime heap telemetry missing,pod level grouping can churn during rollouts,stat=server.memory_heap_size|prom=envoy_server_memory_heap_size,https://aws.github.io/aws-app-mesh-controller-for-k8s/reference/envoy_metrics/ -Resource Utilization,server.concurrency,true,Configured Envoy worker concurrency,{thread},gauge,envoy_server_concurrency,query-value,max,none,,context.env|context.team,context.service.name|context.scope,Metric not emitted by some builds,Safe,stat=server.concurrency|prom=envoy_server_concurrency,https://aws.github.io/aws-app-mesh-controller-for-k8s/reference/envoy_metrics/ -Throughput/Usage,listener..downstream_cx_total,true,Total downstream connections accepted by listener,{connection},counter,envoy_listener_downstream_cx_total,query-value|timeseries,sum,per-second,context.envoy_listener_address:20|context.service.name:20,context.env|context.team,context.envoy_listener_address|context.service.name,No downstream traffic or missing listener telemetry,listener cardinality can rise with many ports,stat=listener..downstream_cx_total|prom=envoy_listener_downstream_cx_total,https://www.envoyproxy.io/docs/envoy/latest/configuration/listeners/stats -Capacity/Saturation,listener..downstream_cx_active,true,Current active downstream connections,{connection},gauge,envoy_listener_downstream_cx_active,query-value|timeseries,max,none,context.envoy_listener_address:20|context.service.name:20,context.env|context.team,context.envoy_listener_address|context.service.name,No active downstream clients or idle traffic,listener address may be high cardinality across dynamic listeners,stat=listener..downstream_cx_active|prom=envoy_listener_downstream_cx_active,https://www.envoyproxy.io/docs/envoy/latest/configuration/listeners/stats -Errors/Failures,listener..downstream_cx_overflow,true,Downstream listener connection overflow events,{overflow},counter,envoy_listener_downstream_cx_overflow,query-value|timeseries,sum,per-second,context.envoy_listener_address:20|context.service.name:20,context.env|context.team,context.envoy_listener_address|context.service.name,No overflow pressure observed,Safe,stat=listener..downstream_cx_overflow|prom=envoy_listener_downstream_cx_overflow,https://www.envoyproxy.io/docs/envoy/latest/configuration/listeners/stats -Errors/Failures,listener..downstream_cx_destroy_remote_active_rq,false,Remote closed downstream connection while request was active legacy name,{event},counter,envoy_listener_downstream_cx_destroy_remote_active_rq,timeseries|query-value,sum,per-second,context.envoy_listener_address:20|context.service.name:20,context.env|context.team,context.envoy_listener_address|context.service.name,Legacy metric name not found in Tsuga use envoy_listener_downstream_cx_destroy instead,Replaced by generic destroy metric,stat=listener..downstream_cx_destroy_remote_active_rq,https://www.envoyproxy.io/docs/envoy/latest/configuration/listeners/stats -Errors/Failures,listener..downstream_cx_destroy,true,Observed downstream connection destroy events,{event},counter,envoy_listener_downstream_cx_destroy,query-value|timeseries,sum,per-second,context.envoy_listener_address:20|context.service.name:20,context.env|context.team,context.envoy_listener_address|context.service.name,No connection teardown events in window,Safe,stat=listener..downstream_cx_destroy,https://www.envoyproxy.io/docs/envoy/latest/configuration/listeners/stats -Errors/Failures,listener..downstream_cx_overload_reject,true,Listener rejected connection due to overload,{reject},counter,envoy_listener_downstream_cx_overload_reject,query-value|timeseries,sum,per-second,context.envoy_listener_address:20|context.service.name:20,context.env|context.team,context.envoy_listener_address|context.service.name,No overload rejection events,Safe,stat=listener..downstream_cx_overload_reject,https://www.envoyproxy.io/docs/envoy/latest/configuration/listeners/stats -Capacity/Saturation,cluster..upstream_cx_active,true,Current active upstream connections by cluster,{connection},gauge,envoy_cluster_upstream_cx_active,query-value|timeseries|max,max,none,context.envoy_cluster_name:20|context.service.name:20,context.env|context.team,context.envoy_cluster_name|context.service.name,No active upstream traffic or idle cluster,cluster name cardinality can grow with dynamic clusters,stat=cluster..upstream_cx_active|prom=envoy_cluster_upstream_cx_active,https://www.envoyproxy.io/docs/envoy/latest/configuration/upstream/cluster_manager/cluster_stats -Errors/Failures,cluster..upstream_cx_connect_fail,true,Upstream connection attempts that failed,{failure},counter,envoy_cluster_upstream_cx_connect_fail,query-value|timeseries,sum,per-second,context.envoy_cluster_name:20|context.service.name:20,context.env|context.team,context.envoy_cluster_name|context.service.name,No connect failures observed,Safe,stat=cluster..upstream_cx_connect_fail|prom=envoy_cluster_upstream_cx_connect_fail,https://www.envoyproxy.io/docs/envoy/latest/configuration/upstream/cluster_manager/cluster_stats -Errors/Failures,cluster..upstream_cx_connect_timeout,true,Upstream connection attempts timing out,{timeout},counter,envoy_cluster_upstream_cx_connect_timeout,query-value|timeseries,sum,per-second,context.envoy_cluster_name:20|context.service.name:20,context.env|context.team,context.envoy_cluster_name|context.service.name,No connect timeouts observed,Safe,stat=cluster..upstream_cx_connect_timeout|prom=envoy_cluster_upstream_cx_connect_timeout,https://www.envoyproxy.io/docs/envoy/latest/configuration/upstream/cluster_manager/cluster_stats -Throughput/Usage,cluster..upstream_rq_total,true,Total upstream requests forwarded by cluster,{request},counter,envoy_cluster_upstream_rq_total,query-value|timeseries|top-list,sum,per-second,context.envoy_cluster_name:20|context.service.name:20,context.env|context.team,context.envoy_cluster_name|context.service.name,No upstream request traffic or missing cluster stats,cluster cardinality can spike with dynamic routing,stat=cluster..upstream_rq_total|prom=envoy_cluster_upstream_rq_total,https://www.envoyproxy.io/docs/envoy/latest/configuration/upstream/cluster_manager/cluster_stats -Errors/Failures,cluster..upstream_rq_2xx,false,Legacy per-class upstream request metric not present in this Tsuga shape,{request},counter,envoy_cluster_upstream_rq_2xx,timeseries|query-value,sum,per-second,context.envoy_cluster_name:20|context.service.name:20,context.env|context.team,context.envoy_cluster_name|context.service.name,Use envoy_cluster_upstream_rq_xx filtered by response class 2xx,Safe,legacy=envoy_cluster_upstream_rq_2xx|replacement=envoy_cluster_upstream_rq_xx,https://www.envoyproxy.io/docs/envoy/latest/configuration/upstream/cluster_manager/cluster_stats -Errors/Failures,cluster..upstream_rq_4xx,false,Legacy per-class upstream request metric not present in this Tsuga shape,{request},counter,envoy_cluster_upstream_rq_4xx,timeseries|query-value,sum,per-second,context.envoy_cluster_name:20|context.service.name:20,context.env|context.team,context.envoy_cluster_name|context.service.name,Use envoy_cluster_upstream_rq_xx filtered by response class 4xx,Safe,legacy=envoy_cluster_upstream_rq_4xx|replacement=envoy_cluster_upstream_rq_xx,https://www.envoyproxy.io/docs/envoy/latest/configuration/upstream/cluster_manager/cluster_stats -Errors/Failures,cluster..upstream_rq_5xx,false,Legacy per-class upstream request metric not present in this Tsuga shape,{request},counter,envoy_cluster_upstream_rq_5xx,timeseries|query-value,sum,per-second,context.envoy_cluster_name:20|context.service.name:20,context.env|context.team,context.envoy_cluster_name|context.service.name,Use envoy_cluster_upstream_rq_xx filtered by response class 5xx,Safe,legacy=envoy_cluster_upstream_rq_5xx|replacement=envoy_cluster_upstream_rq_xx,https://www.envoyproxy.io/docs/envoy/latest/configuration/upstream/cluster_manager/cluster_stats -Errors/Failures,cluster..upstream_rq_xx,true,Upstream request count by response code class via context.envoy_response_code_class,{request},counter,envoy_cluster_upstream_rq_xx,timeseries|query-value|top-list,sum,per-second,context.envoy_cluster_name:20|context.envoy_response_code_class:5,context.env|context.team,context.envoy_cluster_name|context.envoy_response_code_class|context.service.name,Response class tag missing or metric path unavailable,response class split requires filter handling not separate metrics,replacement_for=*_2xx|*_4xx|*_5xx,https://www.envoyproxy.io/docs/envoy/latest/configuration/upstream/cluster_manager/cluster_stats -Errors/Failures,cluster..upstream_rq_timeout,true,Upstream requests timing out,{timeout},counter,envoy_cluster_upstream_rq_timeout,query-value|timeseries,sum,per-second,context.envoy_cluster_name:20|context.service.name:20,context.env|context.team,context.envoy_cluster_name|context.service.name,No timeout failures observed,Safe,stat=cluster..upstream_rq_timeout,https://www.envoyproxy.io/docs/envoy/latest/configuration/upstream/cluster_manager/cluster_stats -Errors/Failures,cluster..upstream_rq_pending_overflow,true,Requests dropped due to pending queue overflow,{overflow},counter,envoy_cluster_upstream_rq_pending_overflow,query-value|timeseries,sum,per-second,context.envoy_cluster_name:20|context.service.name:20,context.env|context.team,context.envoy_cluster_name|context.service.name,No pending overflow events,Safe,stat=cluster..upstream_rq_pending_overflow,https://www.envoyproxy.io/docs/envoy/latest/configuration/upstream/cluster_manager/cluster_stats -Errors/Failures,cluster..upstream_rq_retry,true,Requests retried by Envoy policy,{retry},counter,envoy_cluster_upstream_rq_retry,query-value|timeseries|top-list,sum,per-second,context.envoy_cluster_name:20|context.service.name:20,context.env|context.team,context.envoy_cluster_name|context.service.name,Retry policy disabled or no retry events,Safe,stat=cluster..upstream_rq_retry,https://www.envoyproxy.io/docs/envoy/latest/configuration/upstream/cluster_manager/cluster_stats -Errors/Failures,cluster..upstream_rq_retry_success,true,Retried upstream requests that eventually succeeded,{retry},counter,envoy_cluster_upstream_rq_retry_success,timeseries|top-list,sum,per-second,context.envoy_cluster_name:20|context.service.name:20,context.env|context.team,context.envoy_cluster_name|context.service.name,No successful retries observed,Safe,stat=cluster..upstream_rq_retry_success,https://www.envoyproxy.io/docs/envoy/latest/configuration/upstream/cluster_manager/cluster_stats -Errors/Failures,cluster..upstream_rq_per_try_timeout,true,Per-try timeout occurrences during retried requests,{timeout},counter,envoy_cluster_upstream_rq_per_try_timeout,timeseries|top-list,sum,per-second,context.envoy_cluster_name:20|context.service.name:20,context.env|context.team,context.envoy_cluster_name|context.service.name,No per-try timeout events,Safe,stat=cluster..upstream_rq_per_try_timeout,https://www.envoyproxy.io/docs/envoy/latest/configuration/upstream/cluster_manager/cluster_stats -Performance/Latency,cluster..upstream_rq_time_sum,false,Histogram sum-style companion metric not present in this Tsuga shape,ms,counter,envoy_cluster_upstream_rq_time_sum,timeseries|formula,sum,per-second,context.envoy_cluster_name:20|context.service.name:20,context.env|context.team,context.envoy_cluster_name|context.service.name,Use envoy_cluster_upstream_rq_time histogram directly,Safe,legacy=sum_component|replacement=envoy_cluster_upstream_rq_time,https://www.envoyproxy.io/docs/envoy/latest/configuration/upstream/cluster_manager/cluster_stats -Performance/Latency,cluster..upstream_rq_time_count,false,Histogram count-style companion metric not present in this Tsuga shape,{request},counter,envoy_cluster_upstream_rq_time_count,timeseries|formula,sum,per-second,context.envoy_cluster_name:20|context.service.name:20,context.env|context.team,context.envoy_cluster_name|context.service.name,Use envoy_cluster_upstream_rq_time histogram directly,Safe,legacy=count_component|replacement=envoy_cluster_upstream_rq_time,https://www.envoyproxy.io/docs/envoy/latest/configuration/upstream/cluster_manager/cluster_stats -Performance/Latency,cluster..upstream_rq_time,true,Upstream request duration histogram for latency analysis,ms,histogram,envoy_cluster_upstream_rq_time,query-value|timeseries|top-list,average,none,context.envoy_cluster_name:20|context.service.name:20,context.env|context.team,context.envoy_cluster_name|context.service.name,Latency histogram not exported in pipeline,histogram support and percentile behavior must be validated in dashboard builder,replacement_for=*time_sum|*time_count,https://www.envoyproxy.io/docs/envoy/latest/configuration/upstream/cluster_manager/cluster_stats -Performance/Latency,cluster..upstream_cx_connect_ms,true,Upstream connection establishment duration histogram,ms,histogram,envoy_cluster_upstream_cx_connect_ms,timeseries|top-list,average,none,context.envoy_cluster_name:20|context.service.name:20,context.env|context.team,context.envoy_cluster_name|context.service.name,Connect latency histogram not exported in pipeline,cluster cardinality can spike with dynamic routing,stat=cluster..upstream_cx_connect_ms|prom=envoy_cluster_upstream_cx_connect_ms,https://www.envoyproxy.io/docs/envoy/latest/configuration/upstream/cluster_manager/cluster_stats -Availability/Health,cluster..membership_healthy,true,Healthy hosts in upstream cluster,{host},gauge,envoy_cluster_membership_healthy,query-value|timeseries,max,none,context.envoy_cluster_name:20|context.service.name:20,context.env|context.team,context.envoy_cluster_name|context.service.name,Host health telemetry unavailable,Safe,stat=cluster..membership_healthy,https://www.envoyproxy.io/docs/envoy/latest/configuration/upstream/cluster_manager/cluster_stats -Availability/Health,cluster..membership_total,true,Total hosts in upstream cluster including unhealthy,{host},gauge,envoy_cluster_membership_total,query-value|timeseries,max,none,context.envoy_cluster_name:20|context.service.name:20,context.env|context.team,context.envoy_cluster_name|context.service.name,Cluster membership telemetry unavailable,Safe,stat=cluster..membership_total,https://www.envoyproxy.io/docs/envoy/latest/configuration/upstream/cluster_manager/cluster_stats -Availability/Health,cluster..membership_degraded,true,Hosts marked degraded in upstream cluster,{host},gauge,envoy_cluster_membership_degraded,query-value|timeseries,max,none,context.envoy_cluster_name:20|context.service.name:20,context.env|context.team,context.envoy_cluster_name|context.service.name,No degraded members or feature not active,Safe,stat=cluster..membership_degraded,https://www.envoyproxy.io/docs/envoy/latest/configuration/upstream/cluster_manager/cluster_stats -Capacity/Saturation,cluster..circuit_breakers.default.cx_open,true,Circuit breaker open state for connection limit,1,gauge,envoy_cluster_circuit_breakers_default_cx_open,query-value|timeseries,max,none,context.envoy_cluster_name:20|context.service.name:20,context.env|context.team,context.envoy_cluster_name|context.service.name,Circuit breaker telemetry missing or feature disabled,Safe,stat=cluster..circuit_breakers.default.cx_open,https://www.envoyproxy.io/docs/envoy/latest/configuration/upstream/cluster_manager/cluster_circuit_breakers -Capacity/Saturation,cluster..circuit_breakers.default.rq_open,true,Circuit breaker open state for request limit,1,gauge,envoy_cluster_circuit_breakers_default_rq_open,query-value|timeseries,max,none,context.envoy_cluster_name:20|context.service.name:20,context.env|context.team,context.envoy_cluster_name|context.service.name,Circuit breaker telemetry missing or feature disabled,Safe,stat=cluster..circuit_breakers.default.rq_open,https://www.envoyproxy.io/docs/envoy/latest/configuration/upstream/cluster_manager/cluster_circuit_breakers -Capacity/Saturation,cluster..circuit_breakers.default.rq_pending_open,true,Circuit breaker pending request queue open state,1,gauge,envoy_cluster_circuit_breakers_default_rq_pending_open,query-value|timeseries,max,none,context.envoy_cluster_name:20|context.service.name:20,context.env|context.team,context.envoy_cluster_name|context.service.name,Pending breaker signal not emitted in all deployments,Safe,stat=cluster..circuit_breakers.default.rq_pending_open,https://www.envoyproxy.io/docs/envoy/latest/configuration/upstream/cluster_manager/cluster_circuit_breakers -Capacity/Saturation,cluster..circuit_breakers.default.rq_retry_open,true,Circuit breaker retry budget open state,1,gauge,envoy_cluster_circuit_breakers_default_rq_retry_open,query-value|timeseries,max,none,context.envoy_cluster_name:20|context.service.name:20,context.env|context.team,context.envoy_cluster_name|context.service.name,Retry breaker signal not emitted in all deployments,Safe,stat=cluster..circuit_breakers.default.rq_retry_open,https://www.envoyproxy.io/docs/envoy/latest/configuration/upstream/cluster_manager/cluster_circuit_breakers -Capacity/Saturation,cluster..upstream_rq_active,true,Current active upstream requests,{request},gauge,envoy_cluster_upstream_rq_active,query-value|timeseries,max,none,context.envoy_cluster_name:20|context.service.name:20,context.env|context.team,context.envoy_cluster_name|context.service.name,No active upstream requests at sample time,Safe,stat=cluster..upstream_rq_active,https://www.envoyproxy.io/docs/envoy/latest/configuration/upstream/cluster_manager/cluster_stats -Capacity/Saturation,cluster..upstream_rq_pending_total,true,Total pending upstream requests queue depth events,{request},counter,envoy_cluster_upstream_rq_pending_total,timeseries|top-list,sum,per-second,context.envoy_cluster_name:20|context.service.name:20,context.env|context.team,context.envoy_cluster_name|context.service.name,No pending request pressure observed,Safe,stat=cluster..upstream_rq_pending_total,https://www.envoyproxy.io/docs/envoy/latest/configuration/upstream/cluster_manager/cluster_stats -Capacity/Saturation,cluster..upstream_cx_pool_overflow,true,Upstream connection pool overflow events,{overflow},counter,envoy_cluster_upstream_cx_pool_overflow,timeseries|top-list,sum,per-second,context.envoy_cluster_name:20|context.service.name:20,context.env|context.team,context.envoy_cluster_name|context.service.name,No connection pool overflow observed,Safe,stat=cluster..upstream_cx_pool_overflow,https://www.envoyproxy.io/docs/envoy/latest/configuration/upstream/cluster_manager/cluster_stats -Throughput/Usage,http..downstream_rq_total,true,Total downstream HTTP requests handled by HCM,{request},counter,envoy_http_downstream_rq_total,query-value|timeseries|top-list,sum,per-second,context.envoy_http_conn_manager_prefix:10|context.service.name:20,context.env|context.team,context.envoy_http_conn_manager_prefix|context.service.name,HCM stats absent or non HTTP path,http prefix cardinality can rise with dynamic config,stat=http..downstream_rq_total,https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_conn_man/stats -Capacity/Saturation,http..downstream_cx_active,true,Current active downstream HTTP connections by HCM,{connection},gauge,envoy_http_downstream_cx_active,query-value|timeseries,max,none,context.envoy_http_conn_manager_prefix:10|context.service.name:20,context.env|context.team,context.envoy_http_conn_manager_prefix|context.service.name,No active downstream HTTP connections or HCM stats absent,http conn manager prefix may be high cardinality in dynamic configs,stat=http..downstream_cx_active|prom=envoy_http_downstream_cx_active,https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_conn_man/stats -Throughput/Usage,http..downstream_cx_rx_bytes_total,true,Total bytes received on downstream HTTP connections,By,counter,envoy_http_downstream_cx_rx_bytes_total,timeseries|query-value,sum,per-second,context.envoy_http_conn_manager_prefix:10|context.service.name:20,context.env|context.team,context.envoy_http_conn_manager_prefix|context.service.name,No downstream traffic or HCM stats absent,http conn manager prefix may be high cardinality in dynamic configs,stat=http..downstream_cx_rx_bytes_total|prom=envoy_http_downstream_cx_rx_bytes_total,https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_conn_man/stats -Throughput/Usage,http..downstream_cx_tx_bytes_total,true,Total bytes transmitted on downstream HTTP connections,By,counter,envoy_http_downstream_cx_tx_bytes_total,timeseries|query-value,sum,per-second,context.envoy_http_conn_manager_prefix:10|context.service.name:20,context.env|context.team,context.envoy_http_conn_manager_prefix|context.service.name,No downstream traffic or HCM stats absent,http conn manager prefix may be high cardinality in dynamic configs,stat=http..downstream_cx_tx_bytes_total|prom=envoy_http_downstream_cx_tx_bytes_total,https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_conn_man/stats -Errors/Failures,http..downstream_rq_5xx,false,Legacy per-class downstream metric not present in this Tsuga shape,{request},counter,envoy_http_downstream_rq_5xx,timeseries|query-value,sum,per-second,context.envoy_http_conn_manager_prefix:10|context.service.name:20,context.env|context.team,context.envoy_http_conn_manager_prefix|context.service.name,Use envoy_http_downstream_rq_xx filtered by response class 5xx,Safe,legacy=envoy_http_downstream_rq_5xx|replacement=envoy_http_downstream_rq_xx,https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_conn_man/stats -Errors/Failures,http..downstream_rq_4xx,false,Legacy per-class downstream metric not present in this Tsuga shape,{request},counter,envoy_http_downstream_rq_4xx,timeseries|query-value,sum,per-second,context.envoy_http_conn_manager_prefix:10|context.service.name:20,context.env|context.team,context.envoy_http_conn_manager_prefix|context.service.name,Use envoy_http_downstream_rq_xx filtered by response class 4xx,Safe,legacy=envoy_http_downstream_rq_4xx|replacement=envoy_http_downstream_rq_xx,https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_conn_man/stats -Errors/Failures,http..downstream_rq_xx,true,Downstream HTTP requests by response code class via context.envoy_response_code_class,{request},counter,envoy_http_downstream_rq_xx,timeseries|query-value|top-list,sum,per-second,context.envoy_http_conn_manager_prefix:10|context.envoy_response_code_class:5,context.env|context.team,context.envoy_http_conn_manager_prefix|context.envoy_response_code_class|context.service.name,Response class split unavailable if tag dropped,class split requires filter handling not separate metric names,replacement_for=*downstream_rq_4xx|*downstream_rq_5xx,https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_conn_man/stats -Performance/Latency,http..downstream_rq_time,true,Downstream HTTP request duration histogram,ms,histogram,envoy_http_downstream_rq_time,timeseries|top-list,average,none,context.envoy_http_conn_manager_prefix:10|context.service.name:20,context.env|context.team,context.envoy_http_conn_manager_prefix|context.service.name,Histogram not exported or filtered out,histogram support in builder must be validated,stat=http..downstream_rq_time,https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_conn_man/stats -Cost,NOT_APPLICABLE,false,Envoy metrics do not include direct billing data and cost requires external cloud billing join,Unknown,Unknown,Unknown,query-value,average,none,,context.env|context.team,context.service.name,Cost analysis requires infrastructure billing datasets,Safe,proxy=memory|proxy=request_volume,https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/observability/statistics -Capacity/Saturation,cluster..lb_healthy_panic,true,Load balancer panic mode where healthy host percentage dropped below panic threshold,1,counter,envoy_cluster_lb_healthy_panic,query-value|timeseries|top-list,sum,per-second,context.envoy_cluster_name:20|context.service.name:20,context.env|context.team,context.envoy_cluster_name|context.service.name,No panic events observed,Safe,stat=cluster..lb_healthy_panic,https://www.envoyproxy.io/docs/envoy/latest/configuration/upstream/cluster_manager/cluster_stats -Availability/Health,cluster..assignment_stale,true,Cluster assignment marked stale by control-plane update path,{event},counter,envoy_cluster_assignment_stale,query-value|timeseries|top-list,sum,per-second,context.envoy_cluster_name:20|context.service.name:20,context.env|context.team,context.envoy_cluster_name|context.service.name,No stale assignment events,Safe,stat=cluster..assignment_stale,https://www.envoyproxy.io/docs/envoy/latest/configuration/upstream/cluster_manager/cluster_stats -Availability/Health,cluster..assignment_timeout_received,true,Timed-out cluster assignment update events received,{timeout},counter,envoy_cluster_assignment_timeout_received,query-value|timeseries|top-list,sum,per-second,context.envoy_cluster_name:20|context.service.name:20,context.env|context.team,context.envoy_cluster_name|context.service.name,No assignment timeout events,Safe,stat=cluster..assignment_timeout_received,https://www.envoyproxy.io/docs/envoy/latest/configuration/upstream/cluster_manager/cluster_stats -Errors/Failures,cluster..upstream_rq_retry_limit_exceeded,true,Upstream requests where retry limit was exceeded,{retry},counter,envoy_cluster_upstream_rq_retry_limit_exceeded,query-value|timeseries|top-list,sum,per-second,context.envoy_cluster_name:20|context.service.name:20,context.env|context.team,context.envoy_cluster_name|context.service.name,No retry limit exceeded events,Safe,stat=cluster..upstream_rq_retry_limit_exceeded,https://www.envoyproxy.io/docs/envoy/latest/configuration/upstream/cluster_manager/cluster_stats -Errors/Failures,cluster..upstream_rq_retry_overflow,true,Retry attempts dropped due to overflow conditions,{overflow},counter,envoy_cluster_upstream_rq_retry_overflow,query-value|timeseries|top-list,sum,per-second,context.envoy_cluster_name:20|context.service.name:20,context.env|context.team,context.envoy_cluster_name|context.service.name,No retry overflow events,Safe,stat=cluster..upstream_rq_retry_overflow,https://www.envoyproxy.io/docs/envoy/latest/configuration/upstream/cluster_manager/cluster_stats -Errors/Failures,cluster..upstream_rq_retry_backoff_ratelimited,true,Retries rate-limited by backoff policy,{event},counter,envoy_cluster_upstream_rq_retry_backoff_ratelimited,query-value|timeseries|top-list,sum,per-second,context.envoy_cluster_name:20|context.service.name:20,context.env|context.team,context.envoy_cluster_name|context.service.name,No retry backoff rate-limit events,Safe,stat=cluster..upstream_rq_retry_backoff_ratelimited,https://www.envoyproxy.io/docs/envoy/latest/configuration/upstream/cluster_manager/cluster_stats -Latency/Performance,cluster..upstream_cx_connect_attempts_exceeded,true,Connection attempt limit exceeded before successful upstream connection,{event},counter,envoy_cluster_upstream_cx_connect_attempts_exceeded,query-value|timeseries|top-list,sum,per-second,context.envoy_cluster_name:20|context.service.name:20,context.env|context.team,context.envoy_cluster_name|context.service.name,No connect-attempt exceed events,Safe,stat=cluster..upstream_cx_connect_attempts_exceeded,https://www.envoyproxy.io/docs/envoy/latest/configuration/upstream/cluster_manager/cluster_stats -Errors/Failures,http..downstream_rq_idle_timeout,true,Downstream requests terminated due to idle timeout,{timeout},counter,envoy_http_downstream_rq_idle_timeout,query-value|timeseries|top-list,sum,per-second,context.envoy_http_conn_manager_prefix:10|context.service.name:20,context.env|context.team,context.envoy_http_conn_manager_prefix|context.service.name,No downstream idle timeout events,http conn manager prefix may be high cardinality in dynamic configs,stat=http..downstream_rq_idle_timeout,https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_conn_man/stats -Errors/Failures,http..downstream_rq_header_timeout,true,Downstream requests timed out waiting for headers,{timeout},counter,envoy_http_downstream_rq_header_timeout,query-value|timeseries|top-list,sum,per-second,context.envoy_http_conn_manager_prefix:10|context.service.name:20,context.env|context.team,context.envoy_http_conn_manager_prefix|context.service.name,No downstream header timeout events,http conn manager prefix may be high cardinality in dynamic configs,stat=http..downstream_rq_header_timeout,https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_conn_man/stats -Errors/Failures,http..downstream_rq_timeout,true,Downstream requests timed out after upstream attempt,{timeout},counter,envoy_http_downstream_rq_timeout,query-value|timeseries|top-list,sum,per-second,context.envoy_http_conn_manager_prefix:10|context.service.name:20,context.env|context.team,context.envoy_http_conn_manager_prefix|context.service.name,No downstream timeout events,http conn manager prefix may be high cardinality in dynamic configs,stat=http..downstream_rq_timeout,https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_conn_man/stats -Errors/Failures,http..downstream_rq_rx_reset,true,Downstream requests reset on receive path,{reset},counter,envoy_http_downstream_rq_rx_reset,query-value|timeseries|top-list,sum,per-second,context.envoy_http_conn_manager_prefix:10|context.service.name:20,context.env|context.team,context.envoy_http_conn_manager_prefix|context.service.name,No downstream request resets observed,http conn manager prefix may be high cardinality in dynamic configs,stat=http..downstream_rq_rx_reset,https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_conn_man/stats -Errors/Failures,http..downstream_rq_tx_reset,true,Downstream requests reset on transmit path,{reset},counter,envoy_http_downstream_rq_tx_reset,query-value|timeseries|top-list,sum,per-second,context.envoy_http_conn_manager_prefix:10|context.service.name:20,context.env|context.team,context.envoy_http_conn_manager_prefix|context.service.name,No downstream request resets observed,http conn manager prefix may be high cardinality in dynamic configs,stat=http..downstream_rq_tx_reset,https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_conn_man/stats -Resource Utilization,runtime.load_error,true,Runtime configuration load errors,{event},counter,envoy_runtime_load_error,query-value|timeseries|top-list,sum,per-second,context.service.name:20|context.scope:20,context.env|context.team,context.service.name|context.scope,No runtime load error events observed,Safe,stat=runtime.load_error,https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime -Resource Utilization,server.main_thread_watchdog_miss,true,Main thread watchdog miss events,{event},counter,envoy_server_main_thread_watchdog_miss,query-value|timeseries|top-list,sum,per-second,context.service.name:20|context.scope:20,context.env|context.team,context.service.name|context.scope,No main-thread watchdog miss events,Safe,stat=server.main_thread_watchdog_miss,https://www.envoyproxy.io/docs/envoy/latest/operations/admin -Resource Utilization,server.worker_watchdog_miss,true,Worker thread watchdog miss events,{event},counter,envoy_server_worker_watchdog_miss,query-value|timeseries|top-list,sum,per-second,context.service.name:20|context.scope:20,context.env|context.team,context.service.name|context.scope,No worker watchdog miss events,Safe,stat=server.worker_watchdog_miss,https://www.envoyproxy.io/docs/envoy/latest/operations/admin diff --git a/plugins/tsuga/skills/knowledge-technology/references/envoy/queries.md b/plugins/tsuga/skills/knowledge-technology/references/envoy/queries.md deleted file mode 100644 index caca9db..0000000 --- a/plugins/tsuga/skills/knowledge-technology/references/envoy/queries.md +++ /dev/null @@ -1,63 +0,0 @@ -# Envoy - -L7 proxy, usually sidecar or edge. Metrics are per-listener and per-cluster. - -## Incident shapes - -- **Upstream connect failures** — `upstream_cx_connect_fail` / `connect_timeout` rise → upstream or network -- **Upstream 5xx** — `upstream_rq_5xx` rises → upstream errors bubble -- **Pool overflow** — `upstream_rq_pending_overflow > 0` → Envoy dropped due to connection-pool saturation -- **Per-try timeouts** — `upstream_rq_per_try_timeout` rises → individual attempts timing out -- **Listener admission reject** — `downstream_cx_overflow` / `overload_reject` → admission rejecting -- **Memory pressure** — `server.memory_allocated` growing = leak or large buffers - -## Key metrics - -| Metric | Unit | Signal | -|---|---|---| -| `server.live` | 0/1 | Envoy health | -| `server.uptime` | seconds | Reset on restart | -| `server.memory_allocated` / `memory_heap_size` | bytes | Memory posture | -| `server.concurrency` | count | Worker threads | -| `listener..downstream_cx_total` / `active` | count | Downstream connections | -| `listener..downstream_cx_overflow` | count | Admission rejection | -| `listener..downstream_cx_overload_reject` | count | Overload-manager rejections | -| `cluster..upstream_cx_active` | count | Upstream connections | -| `cluster..upstream_cx_connect_fail` | count | Upstream connect failures | -| `cluster..upstream_cx_connect_timeout` | count | Upstream connect timeouts | -| `cluster..upstream_rq_total` | count | Upstream requests | -| `cluster..upstream_rq_2xx` / `4xx` / `5xx` | count | Status buckets | -| `cluster..upstream_rq_timeout` | count | Request timeouts | -| `cluster..upstream_rq_pending_overflow` | count | Pool saturation drops | -| `cluster..upstream_rq_retry` / `retry_success` | count | Retry state | -| `cluster..upstream_rq_per_try_timeout` | count | Per-attempt timeouts | - -## Derived signals - -- `upstream_rq_5xx / upstream_rq_total` — upstream error rate. Baseline < 0.01. -- `upstream_rq_retry / upstream_rq_total` — retry rate. > 0.05 sustained = flaky upstream. -- `upstream_rq_retry_success / upstream_rq_retry` — retry effectiveness. Low = retries not helping. -- `upstream_rq_pending_overflow / upstream_rq_total` — pool saturation. Any sustained = pool too small or upstream slow. - -## Log patterns - -Envoy access-log response flags: - -- `UH` — no healthy upstream -- `UF` — upstream connection failure -- `UT` — upstream request timeout -- `UC` — upstream connection terminated before response -- `UR` — upstream remote reset -- `URX` — retries exhausted -- `LR` — local connection reset -- `DC` — downstream client disconnected -- `OM` — overload manager active -- `RL` — rate limit service active - -## Gotchas - -- Metric names embed dynamic cluster / listener names. Configure labelling for filtering. -- Stats sink is async; metrics can lag a few seconds. -- `retry_success` counts in `rq_total` already (double-count risk in naive ratios). -- `downstream_cx_overflow=0` doesn't prove no admission problem; slow-upstream backpressure shows elsewhere. -- Connection pooling is per-cluster-per-worker-thread. Per-worker saturation can be invisible in aggregates. diff --git a/plugins/tsuga/skills/knowledge-technology/references/gcp-pubsub/README.md b/plugins/tsuga/skills/knowledge-technology/references/gcp-pubsub/README.md deleted file mode 100644 index 09f58fa..0000000 --- a/plugins/tsuga/skills/knowledge-technology/references/gcp-pubsub/README.md +++ /dev/null @@ -1,713 +0,0 @@ -# Google Cloud Pub/Sub Integration Context Bundle - -**Technology:** Google Cloud Pub/Sub -**Deployment:** Managed (GCP) -**Environment:** prod -**Persona:** SRE Dev and ops -**Telemetry preference:** mixed -**Integration scope:** core service only -**Primary use-case:** reliability and performance - -## How to use this bundle -- **07** (`07_google-cloud-pub-sub_dashboard_plan.yaml`) — section structure, widget specs, and coverage map for building dashboards -- **05** (`05_google-cloud-pub-sub_metric_inventory.csv`) — source of truth for all metrics, types, units, aggregations, and group-bys -- **09** (`09_google-cloud-pub-sub_section_notes_and_playbooks.md`) — paste-ready note content for every section, plus triage chains and playbooks -- **06** (`06_google-cloud-pub-sub_derived_signals.csv`) — formula widgets: inputs, aliases, formulas, output units - -## Confirmed Tsuga prefixes -- `pubsub.googleapis.com/*` — **CONFIRMED** via Tsuga API discovery. 43 metrics found matching prefix. 32 confirmed with data, 4 listed but not found in current environment (push, dead letter, health score). Metric names use GCP dot-separated format (e.g., `pubsub.googleapis.com/topic/send_request_count`). - -## Discovery status -Discovery: completed (Stage 2, 2026-02-10) -- 43 metrics enumerated via `/v1/metrics/names-and-types` -- 17 counter metrics confirmed as delta temporality via `/v1/metrics/metadata` -- Context fields confirmed via `/v1/metrics/attributes` on 15 representative metrics -- See `12_google-cloud-pub-sub_discovery_reconciliation.md` for full details - -## Bundle files - -| # | Filename | Purpose | -|---|---|---| -| 00 | `00_google-cloud-pub-sub_cover.md` | This file. Metadata, navigation, top sources. | -| 01 | `01_google-cloud-pub-sub_executive_overview.md` | What Pub/Sub is, what "good" looks like, top incident shapes. | -| 02 | `02_google-cloud-pub-sub_key_concepts.md` | Glossary (24 terms), concept map, entities, Tsuga field mapping. | -| 03 | `03_google-cloud-pub-sub_golden_signals.md` | Traffic/Errors/Latency/Saturation mapped to Pub/Sub. | -| 04 | `04_google-cloud-pub-sub_telemetry_sources.md` | Source matrix, optional features, "no data" interpretation. | -| 05 | `05_google-cloud-pub-sub_metric_inventory.csv` | 37 metrics: names, types, units, agg, post_fn, group_by. (32 confirmed + 4 unconfirmed + 1 N/A) | -| 06 | `06_google-cloud-pub-sub_derived_signals.csv` | 9 derived signals with formulas and input specs. | -| 07 | `07_google-cloud-pub-sub_dashboard_plan.yaml` | 7 sections, 2 dashboards, ~40 widgets, coverage map. | -| 09 | `09_google-cloud-pub-sub_section_notes_and_playbooks.md` | Section notes, 21 triage chains, 7 playbooks. | -| 10 | `10_google-cloud-pub-sub_caveats_footguns.md` | 24 caveats tagged to sections. | -| 11 | `11_google-cloud-pub-sub_unknowns_verify_next.yaml` | 7 unknowns — all RESOLVED via Stage 2 discovery. | -| 12 | `12_google-cloud-pub-sub_discovery_reconciliation.md` | Stage 2 reconciliation report. | - -## Top sources - -1. [GCP Pub/Sub Metrics Reference](https://cloud.google.com/monitoring/api/metrics_gcp#gcp-pubsub) — Authoritative list of all pubsub.googleapis.com/* metrics with types, units, and labels. -2. [Pub/Sub Overview](https://cloud.google.com/pubsub/docs/overview) — Architecture, topic/subscription model, delivery modes. -3. [Pub/Sub Monitoring Guide](https://cloud.google.com/pubsub/docs/monitoring) — How to monitor Pub/Sub with Cloud Monitoring, key metrics to watch. -4. [Subscription Properties](https://cloud.google.com/pubsub/docs/subscription-properties) — Ack deadline, DLT, ordering, exactly-once, retention, retry policy. -5. [Topic Troubleshooting](https://cloud.google.com/pubsub/docs/topic-troubleshooting) — Publish errors, latency, schema issues, SMT failures. -6. [Push Troubleshooting](https://cloud.google.com/pubsub/docs/push-troubleshooting) — Push endpoint failures, response classes, push backoff. -7. [Pull Troubleshooting](https://cloud.google.com/pubsub/docs/pull-troubleshooting) — Streaming pull issues, backlog growth, ack deadline expiry. -8. [Metrics Autoscaling Best Practices](https://cloud.google.com/pubsub/docs/metrics-autoscaling-best-practices) — Using backlog metrics for autoscaling decisions. -9. [Push Subscriptions](https://cloud.google.com/pubsub/docs/push) — Push delivery mechanics, authentication, backoff behavior. -10. [Pub/Sub Architecture](https://cloud.google.com/pubsub/docs/overview) — Two-plane architecture, message lifecycle, regional storage. - - ---- - -# Google Cloud Pub/Sub - Executive Overview - -## What it is -Google Cloud Pub/Sub is a fully managed, serverless messaging service that decouples event producers from consumers. It handles hundreds of millions of messages per second with at-least-once (or exactly-once) delivery, automatic scaling, and global reach across all GCP regions. Topics receive published messages; subscriptions deliver them to consumers via pull, push, BigQuery export, or Cloud Storage export. - -## What "good" looks like -- Publish latency (p99) under 50ms; send request error rate near zero. -- Subscription backlog (`num_undelivered_messages`) stable or draining; `oldest_unacked_message_age` below the ack deadline. -- Push endpoints returning 2xx; no sustained push backoff. -- Dead-letter message count near zero (messages are being processed successfully). -- Streaming pull connections stable; no excessive `mod_ack_deadline` activity. - -## Paging intent (high-level) -Page on: subscription backlog age growing unbounded, sustained publish errors, dead-letter escalation, push endpoint failures. - -## Top 3 incident shapes - -1. **Backlog growth / consumer lag** - `oldest_unacked_message_age` climbing, `num_undelivered_messages` rising. Start with: Subscription Health section. -2. **Publish failures / elevated latency** - `send_request_count` errors spike, `send_request_latencies` p95+ degrades. Start with: Publishing Performance section. -3. **Push endpoint failures** - `push_request_count` non-ack responses, push backoff engaged. Start with: Push Delivery section. - ---- - -### Confirmed by sources -- Architecture and data flow: [Pub/Sub Architecture](https://cloud.google.com/pubsub/docs/overview) -- Metrics reference: [GCP Pub/Sub Metrics](https://cloud.google.com/monitoring/api/metrics_gcp#gcp-pubsub) -- Troubleshooting guides: [Topic Troubleshooting](https://cloud.google.com/pubsub/docs/topic-troubleshooting), [Pull Troubleshooting](https://cloud.google.com/pubsub/docs/pull-troubleshooting), [Push Troubleshooting](https://cloud.google.com/pubsub/docs/push-troubleshooting) - -### Best-practice inference -- Latency thresholds (p99 < 50ms) based on GCP SLA and common production baselines. -- Dead-letter near zero as health indicator is standard messaging practice. - - ---- - -# Google Cloud Pub/Sub - Key Concepts - -## Glossary (>= 20 terms) - -| Term | Definition | Operational meaning | Dashboard section affected | -|---|---|---|---| -| **Topic** | Named resource to which publishers send messages. | Primary publish-side entity. Group metrics by topic_id. | Publishing Performance | -| **Subscription** | Named resource representing the stream of messages from a topic to a subscriber. | Primary consume-side entity. Most operational metrics are per-subscription. | Subscription Health, Push Delivery, Pull Delivery | -| **Publisher** | Client application that creates and sends messages to a topic. | Source of publish load. Monitor send_request_count and latency. | Publishing Performance | -| **Subscriber** | Client application that receives messages from a subscription. | Consumer of messages. Health reflected in ack rates and backlog. | Subscription Health | -| **Message** | Data + optional attributes published to a topic. Max 10 MB. | Unit of work. Size affects byte_cost and throughput metrics. | Publishing Performance, Throughput | -| **Acknowledge (Ack)** | Subscriber confirms message processing. Removes from backlog. | Core delivery guarantee. Failed acks cause redelivery. | Subscription Health | -| **Ack Deadline** | Time a subscriber has to ack before redelivery. Default 10s, max 600s. | Too short = spurious redelivery; too long = slow failure detection. | Subscription Health | -| **Backlog** | Undelivered messages waiting in a subscription. | Primary health signal. Growing backlog = consumer can't keep up. | Subscription Health | -| **Pull Delivery** | Subscriber actively requests messages via Pull or StreamingPull RPC. | Most common pattern for server-side consumers. | Pull Delivery | -| **Push Delivery** | Pub/Sub sends messages to a subscriber's HTTPS endpoint. | Used for serverless (Cloud Run, Cloud Functions). | Push Delivery | -| **Streaming Pull** | Long-lived bidirectional gRPC stream for low-latency pull delivery. | Preferred pull method. Connection stability matters. | Pull Delivery | -| **Dead Letter Topic (DLT)** | Topic where unprocessable messages are forwarded after max delivery attempts. | Safety net. Rising DLT count = processing failures. | Dead Letter & Errors | -| **Message Ordering** | Delivers messages with same ordering key in publish order. | Adds head-of-line blocking risk per ordering key. | Subscription Health | -| **Exactly-Once Delivery** | Guarantees each message delivered and acked exactly once (pull only). | Adds overhead; uses acknowledgment IDs for dedup. | Subscription Health | -| **Retained Acked Messages** | Acknowledged messages kept for replay via Seek. | Storage cost; enables replay/reprocessing. | Subscription Health | -| **Seek** | Rewind or fast-forward subscription to a timestamp or snapshot. | Operational tool for replaying or skipping messages. | Subscription Health | -| **Snapshot** | Point-in-time capture of subscription backlog state. | Used for seek operations. Has its own metrics. | Snapshots | -| **Flow Control** | Client-side mechanism to limit outstanding messages. | Prevents OOM on subscriber. Affects throughput. | Pull Delivery | -| **Push Backoff** | Exponential delay Pub/Sub applies when push endpoint returns errors. 100ms-60s. | Protects failing endpoints but delays delivery. | Push Delivery | -| **Subscription Filter** | Attribute-based filter; non-matching messages auto-acked. | Reduces processing load but still incurs publish cost. | Subscription Health | -| **Schema** | Protobuf or Avro schema associated with a topic for message validation. | Publish rejects on schema mismatch (INVALID_ARGUMENT). | Publishing Performance | -| **Single Message Transform (SMT)** | Lightweight message transformation before delivery. | Failures reject entire publish batch. | Publishing Performance | -| **BigQuery Subscription** | Writes messages directly to BigQuery table. | No subscriber code needed. Has unique delivery metrics. | Subscription Health | -| **Cloud Storage Subscription** | Writes messages to GCS as batched files. | Batch-oriented delivery. Has file-specific metrics. | Subscription Health | - -## Concept Map (>= 25 lines) - -``` -Publisher -> publishes to -> Topic (topic is the entry point for all messages) -Topic -> fans out to -> Subscription(s) (1:N fan-out; each subscription gets all messages) -Subscription -> delivers to -> Subscriber (via pull, streaming pull, push, BigQuery, or GCS) -Subscription -> has -> Backlog (undelivered messages waiting for ack) -Subscription -> has -> Ack Deadline (time before message redelivery) -Subscription -> optionally has -> Dead Letter Topic (unprocessable messages forwarded here) -Subscription -> optionally has -> Filter (auto-acks non-matching messages) -Subscription -> optionally has -> Retry Policy (exponential backoff for nacks) -Subscription -> optionally has -> Message Ordering (ordered delivery per ordering key) -Subscription -> optionally has -> Exactly-Once Delivery (dedup for pull subscriptions) -Subscription -> can be -> Pull, Push, BigQuery, or Cloud Storage type -Pull Subscription -> uses -> StreamingPull RPC (preferred, long-lived gRPC stream) -Pull Subscription -> uses -> Pull RPC (legacy, polling-based) -Push Subscription -> sends to -> HTTPS Endpoint (must respond with 2xx to ack) -Push Subscription -> subject to -> Push Backoff (exponential delay on errors) -Topic -> optionally has -> Schema (Protobuf or Avro validation) -Topic -> optionally has -> Message Retention (retains messages 10min-31 days) -Topic -> optionally has -> SMT (transforms messages before delivery) -Message -> has -> Data + Attributes (attributes used for filtering/routing) -Message -> has -> Ordering Key (optional; enables ordered delivery) -Dead Letter Topic -> is a -> regular Topic (can have its own subscriptions for analysis) -Dead Letter Topic -> tracks -> max delivery attempts (5-100, default 5) -Backlog -> measured by -> num_undelivered_messages + oldest_unacked_message_age -Snapshot -> captures -> Subscription backlog state (for seek/replay) -Seek -> rewinds/fast-forwards -> Subscription position (timestamp or snapshot) -Topic metrics -> scoped by -> project_id, topic_id -Subscription metrics -> scoped by -> project_id, subscription_id -Push metrics -> additionally scoped by -> response_code, response_class -``` - -## Entities and Dimensions (>= 12) - -| Dimension | Description | Why useful | Cardinality risk | Safe top-N | Do NOT group-by | -|---|---|---|---|---|---| -| `project_id` | GCP project containing the resource | Multi-project visibility | Low (bounded by org) | 10 | | -| `topic_id` | Topic name | Per-topic publish metrics | Medium (user-defined) | 20 | | -| `subscription_id` | Subscription name | Per-subscription consume metrics | Medium (user-defined) | 20 | | -| `response_code` | HTTP/gRPC response code on requests | Error breakdown (200, 400, 403, 429, 500, etc.) | Low (~20 codes) | 10 | | -| `response_class` | Response class (ack, deadline_exceeded, remote_server_4xx, etc.) | Push health categorization | Low (~8 classes) | 10 | | -| `delivery_type` | Pull, push, BigQuery, Cloud Storage | Subscription delivery mode | Very low (4 values) | 4 | | -| `ordering_key` | Message ordering key | Head-of-line blocking diagnosis | HIGH | | Yes - unbounded user-defined values | -| `message_id` | Unique message identifier | | EXTREME | | Yes - unique per message | -| `region` | GCP region where messages are stored | Regional health analysis | Low (~30 regions) | 10 | | -| `ack_type` | Type of ack operation | Ack behavior analysis | Very low | 5 | | -| `subtype` | Specific operation subtype (e.g., `throttled`, `sent`, `filtered`) | Fine-grained throughput analysis | Low | 5 | | -| `cloud_account_id` | GCP project number (numeric) | Cross-project correlation | Low | 10 | | -| `dead_letter_topic_id` | Dead letter destination topic | Dead letter routing | Low | 10 | | - -## Tsuga Field Mapping Table - -| Vendor / GCP Dimension | Recommended Tsuga context.* key | Must-exist vs Optional | Notes | -|---|---|---|---| -| `project_id` | `context.cloud.account.id` or `context.project_id` | Must-exist | Primary ownership boundary | -| `topic_id` | `context.topic_id` | Must-exist (topic metrics) | Core entity for publish metrics | -| `subscription_id` | `context.subscription_id` | Must-exist (subscription metrics) | Core entity for consume metrics | -| `response_code` | `context.response_code` | Optional | Present on request-count metrics | -| `response_class` | `context.response_class` | Optional | Present on push metrics | -| `delivery_type` | `context.delivery_type` | Optional | Differentiates pull/push/BQ/GCS | -| `region` | `context.cloud.region` | Optional | Regional breakdown | -| `subtype` | `context.subtype` | Optional | Operation subtypes | -| `env` | `context.env` | Must-exist (Tsuga convention) | Unknown if populated for GCP integrations | -| `team` | `context.team` | Must-exist (Tsuga convention) | Unknown if populated for GCP integrations | - -**Unknown:** Exact Tsuga context field names for GCP integration dimensions. The GCP integration may map `project_id` -> `context.project_id` or `context.cloud.account.id`. Stage 2 discovery will resolve. Added to 11_unknowns. - ---- - -### Confirmed by sources -- Topic/Subscription model: [Pub/Sub Overview](https://cloud.google.com/pubsub/docs/overview) -- Subscription properties (ack deadline, DLT, filters, ordering, exactly-once): [Subscription Properties](https://cloud.google.com/pubsub/docs/subscription-properties) -- Push delivery and backoff: [Push Subscriptions](https://cloud.google.com/pubsub/docs/push) -- Metric dimensions (response_code, response_class): [Pub/Sub Metrics](https://cloud.google.com/monitoring/api/metrics_gcp#gcp-pubsub) -- Schemas and SMT: [Topic Troubleshooting](https://cloud.google.com/pubsub/docs/topic-troubleshooting) - -### Best-practice inference -- Tsuga context.* field mapping inferred from common GCP integration patterns. -- Cardinality assessments inferred from typical production environments. -- `context.env` and `context.team` as must-exist fields is Tsuga convention, not GCP-specific. - - ---- - -# Google Cloud Pub/Sub - Golden Signals - -## Traffic - -**What it means for Pub/Sub:** -Message throughput - both publish-side (messages/bytes into topics) and consume-side (messages delivered to subscribers). Traffic is the fundamental indicator of system utilization. - -**Typical causes when it degrades:** -- Publisher application failures or deployments reducing publish rate -- Subscriber scaling down or crashing, reducing consumption rate -- Quota exhaustion throttling publish or subscribe operations -- Network issues between publishers/subscribers and GCP endpoints - -**Best telemetry sources:** -- `topic/send_message_operation_count` (publish throughput) -- `topic/send_byte_count` (publish bytes) -- `subscription/sent_message_count` (delivery throughput) -- `subscription/byte_cost` (delivery bytes) -- `subscription/pull_message_operation_count` (pull operations) -- `subscription/streaming_pull_message_operation_count` (streaming pull operations) - -**What people page on:** -- Sudden drop in publish rate with no planned maintenance -- Delivery rate dropping while backlog grows (consumers failing) -- Unexpected zero-traffic periods on critical topics - -**Section questions:** -1. Is the expected volume of messages being published? (Publishing Performance) -2. Are subscribers consuming messages at the expected rate? (Subscription Health) -3. How does traffic distribute across topics and subscriptions? (Throughput) - ---- - -## Errors - -**What it means for Pub/Sub:** -Failed operations - publish rejections, delivery failures, dead-letter escalations, expired ack deadlines, and push endpoint errors. - -**Typical causes when it degrades:** -- Schema validation failures rejecting publishes (INVALID_ARGUMENT) -- Push endpoint returning 4xx/5xx or timing out -- Subscriber nacking messages or letting ack deadlines expire -- Quota or permission errors (RESOURCE_EXHAUSTED, PERMISSION_DENIED) -- KMS key issues causing FAILED_PRECONDITION on encrypted topics -- Dead letter threshold reached on unprocessable messages - -**Best telemetry sources:** -- `topic/send_request_count` grouped by `response_code` (publish errors) -- `subscription/push_request_count` grouped by `response_class` (push errors) -- `subscription/dead_letter_message_count` (unprocessable messages) -- `subscription/expired_ack_deadlines_count` (subscriber too slow or crashed) -- `subscription/exactly_once_warning_count` (exactly-once delivery issues) - -**What people page on:** -- Sustained publish error rate above baseline -- Dead letter message count rising (messages cannot be processed) -- Push endpoint returning non-ack responses persistently -- Expired ack deadlines accelerating (subscriber health issue) - -**Section questions:** -1. Are publishes succeeding without errors? (Publishing Performance) -2. Are push endpoints healthy and responding correctly? (Push Delivery) -3. Are messages being dead-lettered or failing delivery? (Dead Letter & Errors) - ---- - -## Latency - -**What it means for Pub/Sub:** -Time from publish to subscriber delivery, and the age of the oldest unprocessed message. For push subscriptions, also the endpoint response time. - -**Typical causes when it degrades:** -- Consumer processing slower than publish rate (backlog age grows) -- Push endpoint slow to respond (high push_request_latencies) -- Publisher-side latency from client config issues (batch settings, network) -- Flow control throttling on subscriber side -- Message ordering causing head-of-line blocking -- Exactly-once delivery adding acknowledgment overhead - -**Best telemetry sources:** -- `topic/send_request_latencies` (publish-side latency distribution) -- `subscription/oldest_unacked_message_age` (end-to-end delivery lag) -- `subscription/push_request_latencies` (push endpoint response time) -- `subscription/delivery_latency_health_score` (composite health 0-1) - -**What people page on:** -- `oldest_unacked_message_age` exceeding SLO threshold (minutes to hours) -- Publish latency p95+ sustained above 100ms -- Push request latency exceeding ack deadline (causes redelivery) -- Delivery latency health score dropping below 0.5 - -**Section questions:** -1. Is publish latency within acceptable bounds? (Publishing Performance) -2. How old is the oldest unprocessed message per subscription? (Subscription Health) -3. Are push endpoints responding within the ack deadline? (Push Delivery) - ---- - -## Saturation - -**What it means for Pub/Sub:** -How close the system is to operational limits - backlog size, quota utilization, and resource constraints on the subscriber side. - -**Typical causes when it degrades:** -- Subscriber throughput < publish throughput (backlog accumulation) -- Approaching Pub/Sub quotas (publish rate, subscription count, message size) -- Retained acknowledged messages consuming storage -- Too many outstanding messages per subscriber (flow control limits) - -**Best telemetry sources:** -- `subscription/num_undelivered_messages` (backlog depth) -- `subscription/backlog_bytes` (backlog size) -- `subscription/num_outstanding_messages` (in-flight to subscriber) -- `subscription/num_retained_acked_messages` (retained for replay) -- `subscription/retained_acked_bytes` (retained storage) -- `topic/num_unacked_messages_by_region` (regional backlog) - -**What people page on:** -- Backlog growing unbounded (consumer not keeping up) -- Backlog bytes approaching storage cost thresholds -- Outstanding messages near flow control limits - -**Section questions:** -1. Is the subscription backlog stable or growing? (Subscription Health) -2. How much storage is consumed by retained messages? (Capacity & Retention) -3. Are any subscriptions approaching operational limits? (Capacity & Retention) - ---- - -### Confirmed by sources -- Metrics and their meaning: [GCP Pub/Sub Metrics](https://cloud.google.com/monitoring/api/metrics_gcp#gcp-pubsub) -- Monitoring best practices: [Monitor Pub/Sub](https://cloud.google.com/pubsub/docs/monitoring) -- Push troubleshooting and response classes: [Push Troubleshooting](https://cloud.google.com/pubsub/docs/push-troubleshooting) -- Autoscaling signals (backlog): [Metrics Autoscaling Best Practices](https://cloud.google.com/pubsub/docs/metrics-autoscaling-best-practices) - -### Best-practice inference -- Latency thresholds (p95 < 100ms publish) based on production experience. -- Delivery latency health score < 0.5 as page-worthy is inferred (score is 0-1). -- Flow control saturation signals inferred from client library documentation. - - ---- - -# Google Cloud Pub/Sub Section Notes & Playbooks - ---- - -## Part 1: Overview Mission Note - -**Google Cloud Pub/Sub** -Fully managed messaging service for event-driven architectures. -GCP integration, all topics and subscriptions. - -[Pub/Sub Docs](https://cloud.google.com/pubsub/docs/overview) | [Monitoring Guide](https://cloud.google.com/pubsub/docs/monitoring) | [Deep Dive Dashboard](#) - ---- - -## Part 2: Section Explanation Notes - -### # Publishing - Are messages being published successfully? - -### So what? -**Healthy:** Publish rate matches expected application throughput, error rate near zero, p95 latency under 50ms. -**Concerning:** Error rate rising AND latency spiking = likely quota exhaustion or backend issue. Error rate rising but latency normal = schema validation or permission failures (check response_code breakdown). -**Gotcha:** `send_request_latencies` is in **microseconds**, not milliseconds. A value of 50,000 = 50ms, which is fine. - -### Now what? -- Check **Publish Requests by Response Code** for error type breakdown -- If RESOURCE_EXHAUSTED: check quota dashboard and consider requesting increase -- If INVALID_ARGUMENT: check schema configuration and recent topic changes - ---- - -### # Subscription Health - Are subscribers keeping up with message delivery? - -### So what? -**Healthy:** Backlog near zero or stable, oldest unacked message age under 60s, delivery health score near 1.0. -**Concerning:** Backlog growing AND oldest message age climbing = consumers falling behind. Backlog stable but oldest age high = some messages stuck (possibly ordering key head-of-line blocking or poison messages). -**Gotcha:** `num_undelivered_messages` is per-subscription, not per-topic. Two subscriptions on the same topic can have wildly different backlogs. Don't average `oldest_unacked_message_age`; use max to find the worst subscription. - -### Now what? -- Check **Top Subscriptions by Backlog** to find the lagging subscription -- Check **Expired Ack Deadlines** to see if subscriber is crashing mid-processing -- If backlog is growing: verify subscriber instances are running and scaled appropriately - ---- - -### # Push Delivery - Are push endpoints healthy and responding? - -### So what? -**Healthy:** Push success rate >99%, push latency p95 under the ack deadline, response_class is overwhelmingly "ack". -**Concerning:** Success rate dropping = endpoint issues. `deadline_exceeded` responses = endpoint too slow. `remote_server_5xx` = endpoint errors. `unreachable` = connectivity failure or bad URL. -**Gotcha:** Push backoff (100ms-60s exponential) kicks in when endpoints return errors. During backoff, delivery rate drops dramatically even after the endpoint recovers. Recovery is gradual. - -### Now what? -- Check **Push Requests by Response Class** for the error type -- If deadline_exceeded: verify endpoint can respond within ack deadline; consider increasing deadline -- If unreachable: verify endpoint URL and network connectivity - ---- - -### # Pull Delivery - Are pull and streaming pull subscribers active? - -### So what? -**Healthy:** Pull/streaming pull request rate stable, response codes predominantly success. -**Concerning:** Pull request rate dropping to zero = subscriber disconnected. Streaming pull errors increasing = connection instability (network issues, subscriber crashes). -**Gotcha:** Streaming pull connections can show occasional UNAVAILABLE or DEADLINE_EXCEEDED errors during normal operation. The client library handles reconnection automatically. Concern only when errors dominate and backlog grows. - -### Now what? -- Check **Streaming Pull Responses by Code** for error patterns -- If subscriber appears disconnected: check application logs and deployment status -- If errors rising: check network connectivity between subscriber and Pub/Sub endpoint - ---- - -### # Errors & Dead Letters - Are messages failing delivery permanently? - -### So what? -**Healthy:** Dead letter rate zero, expired ack deadlines near zero, no exactly-once warnings. -**Concerning:** Dead letter rate > 0 sustained = messages permanently failing. Expired ack deadlines climbing = subscriber can't process fast enough or is crashing. Exactly-once warnings = potential duplicate delivery. -**Gotcha:** `dead_letter_message_count` requires DLT to be configured. Without it, unprocessable messages are redelivered forever (no metric signal). A small number of dead letters during deployments is expected. - -### Now what? -- Check **Top Subscriptions by Dead Letters** to identify affected subscription -- Inspect dead letter topic for message patterns (malformed data, missing fields) -- If expired ack deadlines high: consider increasing ack deadline or optimizing processing - ---- - -### # Capacity & Retention - How much storage is consumed and is backlog bounded? - -### So what? -**Healthy:** Backlog bytes stable or decreasing, retained storage within budget, drain time under 10 minutes. -**Concerning:** Backlog bytes growing continuously = storage cost rising and consumer falling behind. Retained acked bytes growing = historical storage accumulating (charges after 24h). -**Gotcha:** Retained acked messages incur additional storage charges after 24 hours. If you enabled "retain acknowledged messages" for replay capability, monitor the storage cost. - -### Now what? -- Check **Top Subscriptions by Backlog Bytes** for the largest consumers -- If drain time is high: scale up subscribers or investigate processing bottleneck -- If retained storage is growing: review retention policy and disable if replay not needed - ---- - -### # Throughput - How are operations distributed across the system? - -### So what? -**Healthy:** Ack rate tracks delivery rate, ModAck ratio under 50%, byte cost proportional to message volume. -**Concerning:** ModAck ratio very high (>80%) = subscribers consistently needing more time. Low ack rate relative to sent rate = redelivery loop (messages sent, not acked, resent). -**Gotcha:** ModAck (modify ack deadline) is normal and healthy. Client libraries extend deadlines automatically for messages being processed. A 10-30% ratio is typical. - -### Now what? -- Check **ModAck to Sent Ratio** for trend -- If ModAck ratio climbing: increase ack deadline at the subscription level -- If ack rate << sent rate: investigate subscriber health (crashing, slow processing, nacking) - ---- - -### # Snapshots - What is the state of subscription snapshots? - -### So what? -**Healthy:** Snapshot message counts stable, oldest message age within retention period. -**Concerning:** Snapshot size growing large = potential seek operations will replay many messages. Oldest message age approaching retention limit = snapshot may become invalid. -**Gotcha:** Snapshots expire when the oldest message in the snapshot exceeds the message retention duration (default 7 days). An expired snapshot cannot be used for Seek. - -### Now what? -- If snapshot is large and approaching age limit: decide whether to refresh or delete -- Snapshots are operational tools; they should not accumulate indefinitely - ---- - -## Part 3: Cause-Effect Triage Chains (>= 20) - -1. If **Backlog Messages (count)** is rising steadily -> check **Delivery Rate (msg/s)** -> if delivery rate is dropping, check subscriber health -> likely cause: subscriber pods crashing or scaled down -> scale up subscribers. (Confirmed) -2. If **Oldest Unacked Message Age (s)** is climbing -> check **Top Subscriptions by Backlog** -> identify the lagging subscription -> check **Expired Ack Deadlines (msg/s)** -> likely cause: subscriber too slow or crashing. (Confirmed) -3. If **Publish Error Rate (%)** spikes -> check **Publish Requests by Response Code** -> if RESOURCE_EXHAUSTED -> likely cause: quota limit hit -> request quota increase. (Confirmed) -4. If **Publish Error Rate (%)** shows INVALID_ARGUMENT -> check recent topic schema changes -> likely cause: schema mismatch between publisher and topic -> update publisher or schema. (Confirmed) -5. If **Push Success Rate (%)** drops below 99% -> check **Push Requests by Response Class** -> if deadline_exceeded -> check **Push Latency p95** -> likely cause: slow endpoint -> optimize endpoint or increase ack deadline. (Confirmed) -6. If **Push Requests by Response Class** shows `unreachable` -> check endpoint URL and network -> likely cause: endpoint down or misconfigured URL -> fix endpoint availability. (Confirmed) -7. If **Push Requests by Response Class** shows `remote_server_5xx` -> check endpoint application logs -> likely cause: endpoint application error -> fix application. (Confirmed) -8. If **Dead Letter Rate (msg/s)** > 0 sustained -> check **Top Subscriptions by Dead Letters** -> inspect dead letter topic messages -> likely cause: poison messages or downstream failure -> fix processing logic or downstream service. (Confirmed) -9. If **Expired Ack Deadlines (msg/s)** is climbing -> check subscriber CPU/memory -> check **ModAck Rate (msg/s)** -> if ModAck also high, subscriber is trying but too slow -> likely cause: processing bottleneck -> optimize processing or increase ack deadline. (Mixed) -10. If **Delivery Health Score** drops below 0.5 -> check **Oldest Unacked Message Age** and **Backlog Messages** -> identify contributing subscription -> likely cause: systemic delivery issues -> investigate subscriber and push endpoint health. (Confirmed) -11. If **Streaming Pull Responses by Code** shows persistent errors -> check network connectivity -> check subscriber application logs -> likely cause: network issues or subscriber misconfiguration -> fix connectivity. (Inference) -12. If **Publish Rate (msg/s)** drops to zero unexpectedly -> check publisher application health -> check GCP service status -> likely cause: publisher crash or GCP outage -> restart publisher or wait for GCP recovery. (Inference) -13. If **Backlog Messages** is zero but **Oldest Unacked Message Age** is non-zero -> check for stuck messages -> likely cause: ordering key head-of-line blocking or flow control -> investigate ordering key distribution or flow control settings. (Inference) -14. If **Backlog Size (bytes)** is growing but **Backlog Messages** is stable -> likely cause: message sizes increasing -> check **Publish Throughput (KB/s)** trend -> investigate why messages are larger. (Inference) -15. If **Total Retained Storage (MB)** is growing continuously -> check **Retained Acked Messages Over Time** -> likely cause: retain-acked-messages enabled without cleanup -> review retention policy. (Inference) -16. If **Est. Backlog Drain Time (s)** is increasing -> check **Ack Rate (msg/s)** trend -> if ack rate declining, subscriber throughput is dropping -> likely cause: subscriber degradation -> investigate subscriber health. (Inference) -17. If **ModAck to Sent Ratio (%)** exceeds 80% -> check message processing time -> likely cause: ack deadline too short for processing duration -> increase ack deadline. (Inference) -18. If **Pull Request Rate (req/s)** is high but **Backlog Messages** is growing -> check response codes on pull requests -> if errors present -> likely cause: pull errors preventing message delivery -> investigate error type. (Inference) -19. If **Publish Latency p95 (ms)** spikes -> check publisher client config -> check GCP region health -> likely cause: client-side batching issues or GCP regional degradation -> adjust batch settings or failover region. (Mixed) -20. If **Exactly-Once Warnings Over Time** is non-zero -> check subscriber ack patterns -> likely cause: slow acking causing redelivery attempts before ack processed -> optimize subscriber ack timing. (Inference) -21. If **Byte Cost Over Time** spikes -> check **Top Topics by Publish Rate** -> likely cause: burst traffic or new publisher -> verify traffic is expected. (Inference) - ---- - -## Part 4: Operational Playbooks (6-10) - -### Playbook 1: Growing Subscription Backlog -**Trigger:** **Backlog Messages (count)** trending upward, **Oldest Unacked Message Age (s)** > 300s -**Decision rule:** If backlog growth rate is positive for > 5 minutes, investigate subscriber capacity. -**Steps:** -1. Check **Top Subscriptions by Backlog** to identify affected subscriptions -2. Check **Delivery Rate (msg/s)** to confirm consumption rate has dropped -3. Check **Expired Ack Deadlines (msg/s)** for subscriber failure signals -4. Check **Ack Rate Over Time by Subscription** for the affected subscription -5. Check subscriber application logs and deployment status -6. If subscriber healthy but slow: check **ModAck to Sent Ratio** for processing time issues -7. Scale up subscribers or increase processing capacity -**Likely causes:** -- Subscriber deployment/rollback reduced instance count -- Downstream dependency failure causing slow processing -- Message volume spike exceeding subscriber capacity -- Subscriber crash loop -**Next actions:** -- Scale subscriber instances -- Check downstream service health -- If ordering enabled: check for stuck ordering keys -- Consider temporarily pausing non-critical subscriptions -**Label:** Confirmed - -### Playbook 2: Publish Failures -**Trigger:** **Publish Error Rate (%)** > 0.1%, **Publish Requests by Response Code** shows non-success codes -**Decision rule:** Any sustained publish error rate warrants immediate investigation as it means data loss risk. -**Steps:** -1. Check **Publish Requests by Response Code** for error type breakdown -2. If RESOURCE_EXHAUSTED: check GCP quotas dashboard -3. If INVALID_ARGUMENT: check topic schema configuration and recent changes -4. If FAILED_PRECONDITION: check KMS key status for encrypted topics -5. Check **Top Topics by Publish Rate** to identify affected topics -6. Check publisher application logs for client-side errors -**Likely causes:** -- Quota exhaustion (publish rate, message size, topic count) -- Schema validation failure after schema update -- KMS key disabled or inaccessible -- Permission changes (IAM) -**Next actions:** -- Request quota increase if needed -- Roll back schema changes -- Re-enable KMS key -- Verify IAM permissions -**Label:** Confirmed - -### Playbook 3: Push Endpoint Degradation -**Trigger:** **Push Success Rate (%)** < 99%, **Push Requests by Response Class** shows non-ack responses -**Decision rule:** Push success rate below 99% for > 2 minutes indicates endpoint health issue. -**Steps:** -1. Check **Push Requests by Response Class** for dominant error type -2. Check **Push Latency p95** to see if endpoint is slow (deadline_exceeded) -3. Check **Top Subscriptions by Push Errors** to identify affected subscriptions -4. Check **Backlog Messages** to assess impact on delivery -5. Verify endpoint availability and application health -6. Check **Oldest Unacked Message Age** to assess delivery lag -**Likely causes:** -- Endpoint application errors (5xx) -- Endpoint timeout (processing slower than ack deadline) -- Endpoint unreachable (network, DNS, URL misconfiguration) -- Authentication/permission failure (4xx) -**Next actions:** -- Fix endpoint application errors -- Increase ack deadline if endpoint needs more processing time -- Verify endpoint URL and network path -- Check endpoint authentication configuration -**Label:** Confirmed - -### Playbook 4: Dead Letter Escalation -**Trigger:** **Dead Letter Rate (msg/s)** > 0 sustained for > 5 minutes -**Decision rule:** Any sustained dead lettering means messages are permanently failing and need investigation. -**Steps:** -1. Check **Top Subscriptions by Dead Letters** to identify affected subscriptions -2. Check **Expired Ack Deadlines (msg/s)** on same subscriptions -3. Check subscription's max delivery attempts configuration -4. Inspect messages in the dead letter topic for patterns -5. Check subscriber application logs for processing errors -**Likely causes:** -- Poison messages (malformed data, unexpected format) -- Downstream service failure causing consistent processing errors -- Bug in subscriber processing logic -- Max delivery attempts too low for transient errors -**Next actions:** -- Inspect dead letter messages for common patterns -- Fix subscriber processing logic or downstream dependency -- Consider increasing max delivery attempts for transient errors -- Set up alerting on dead letter topic -**Label:** Mixed - -### Playbook 5: High Publish Latency -**Trigger:** **Publish Latency p95 (ms)** > 100ms sustained -**Decision rule:** p95 publish latency above 100ms for > 5 minutes indicates publisher or service issue. -**Steps:** -1. Check **Publish Latency Over Time** for pattern (gradual vs sudden) -2. Check **Publish Rate (msg/s)** for correlation with load changes -3. Check **Publish Requests by Response Code** for errors (errors inflate latency) -4. Check **Top Topics by Publish Rate** to see if specific topics are affected -5. Check publisher client configuration (batch settings, retry config) -**Likely causes:** -- Client-side batching configuration issues -- Publisher creating new clients per request instead of reusing -- GCP regional degradation -- High message sizes increasing network transfer time -**Next actions:** -- Review and optimize publisher batch settings -- Ensure single publisher client per topic per application -- Consider message compression for large payloads -- Check GCP status page for regional issues -**Label:** Mixed - -### Playbook 6: Exactly-Once Delivery Issues -**Trigger:** **Exactly-Once Warnings Over Time** count rising -**Decision rule:** Investigate if warnings are sustained; occasional warnings during subscriber restarts are normal. -**Steps:** -1. Check which subscriptions have exactly-once enabled -2. Check **Expired Ack Deadlines** on those subscriptions -3. Check **Oldest Unacked Message Age** for the affected subscriptions -4. Check subscriber ack patterns and processing time -5. Verify subscriber is using the correct client library version for exactly-once support -**Likely causes:** -- Subscriber acking too slowly (ack deadline expiring before ack processed) -- Network issues causing ack delivery delays -- Client library version doesn't fully support exactly-once semantics -**Next actions:** -- Increase ack deadline -- Upgrade client library -- Verify network connectivity -- Consider if exactly-once is truly needed (at-least-once + idempotent processing may suffice) -**Label:** Inference - -### Playbook 7: Storage Cost Growth -**Trigger:** **Total Retained Storage (MB)** or **Backlog Size (bytes)** growing continuously -**Decision rule:** If storage growth is unbounded over 24 hours, investigate cause and cost impact. -**Steps:** -1. Check **Top Subscriptions by Backlog Bytes** for largest consumers -2. Check **Retained Acked Messages Over Time** for retained storage growth -3. Check **Est. Backlog Drain Time** for affected subscriptions -4. Check subscription retention policy configuration -5. Check if any subscriptions are abandoned (no active subscribers) -6. Estimate cost impact based on GCP pricing -**Likely causes:** -- Subscriber not consuming (abandoned subscription) -- Consumer slower than publisher (growing backlog) -- Retained acked messages accumulating without cleanup -- Message retention period too long for use case -**Next actions:** -- Delete abandoned subscriptions -- Scale up consumers for active subscriptions -- Reduce retention period if full retention not needed -- Disable retain-acked-messages if replay not required -**Label:** Inference - - ---- - -# Google Cloud Pub/Sub - Caveats & Footguns - -## High-cardinality dimensions to avoid - -- **[publishing-performance, subscription-health]** Do NOT group-by `ordering_key`. Ordering keys are user-defined and can have unbounded cardinality (one per user, per session, etc.). Use `topic_id` or `subscription_id` instead. (Inference) -- **[publishing-performance]** Do NOT group-by `message_id`. Every message has a unique ID. Grouping by it produces one series per message with zero analytical value. ([Pub/Sub Docs](https://cloud.google.com/pubsub/docs/overview)) -- **[subscription-health, pull-delivery]** Combining `subscription_id` and `response_code` group-bys on timeseries can produce high series counts in environments with many subscriptions. Use top-list instead, or group by only one dimension. (Inference) -- **[publishing-performance]** `topic_id` cardinality depends on the application. Some teams create topics dynamically (per-tenant, per-event-type). If topic count exceeds 100, use top-N=10 and rely on top-list for the full picture. (Inference) - -## Misleading metrics and wrong aggregations - -- **[subscription-health]** `num_undelivered_messages` is NOT the same as "messages in the topic." It counts only messages that have not been delivered to THIS subscription. Different subscriptions on the same topic can have different backlog sizes. Do not sum across subscriptions to get "total topic backlog." ([Pub/Sub Monitoring](https://cloud.google.com/pubsub/docs/monitoring)) -- **[subscription-health]** `oldest_unacked_message_age` can spike briefly during Seek operations (rewinding sets the oldest message to the seek target). This is intentional, not an incident. ([Pub/Sub Docs](https://cloud.google.com/pubsub/docs/subscription-properties)) -- **[subscription-health]** Do NOT average `oldest_unacked_message_age` across subscriptions. Use `max` to surface the worst-case subscription. Averaging hides the lagging subscription behind healthy ones. (Inference) -- **[throughput-operations]** `byte_cost` does NOT equal actual bytes transferred. It includes operation overhead (attributes, headers) and is used for billing purposes. Do not use it as a bandwidth metric; use `send_byte_count` for actual data volume. ([GCP Pub/Sub Metrics](https://cloud.google.com/monitoring/api/metrics_gcp#gcp-pubsub)) -- **[subscription-health]** `sent_message_count` counts messages SENT to subscribers, not messages successfully processed. A message sent but not acked will be resent and counted again. High sent_message_count with low ack_message_count = redelivery storm. ([Pub/Sub Monitoring](https://cloud.google.com/pubsub/docs/monitoring)) -- **[push-delivery]** Do NOT use `push_request_latencies` percentile as "message processing time." It measures the time Pub/Sub waits for the endpoint response, which includes network latency + endpoint processing. The metric is in **microseconds**, not milliseconds. ([Push Troubleshooting](https://cloud.google.com/pubsub/docs/push-troubleshooting)) - -## Unit pitfalls - -- **[publishing-performance]** `send_request_latencies` is in **microseconds** (us), not milliseconds. Displaying as ms without conversion makes latency appear 1000x lower than reality. ([GCP Pub/Sub Metrics](https://cloud.google.com/monitoring/api/metrics_gcp#gcp-pubsub)) -- **[push-delivery]** `push_request_latencies` is also in **microseconds** (us). Same pitfall as above. ([GCP Pub/Sub Metrics](https://cloud.google.com/monitoring/api/metrics_gcp#gcp-pubsub)) -- **[subscription-health]** `oldest_unacked_message_age` is in **seconds**. For display, convert to minutes or hours for ages > 300s to improve readability. (Inference) -- **[capacity-retention]** `backlog_bytes` and `retained_acked_bytes` are in raw bytes. Use data normalizer (B -> KiB/MiB/GiB) for readability. (Inference) - -## Sampling/temporality pitfalls - -- **[publishing-performance, subscription-health, dead-letter-errors, pull-delivery, push-delivery]** Counter temporality (delta vs cumulative) depends on the Tsuga GCP integration. GCP Cloud Monitoring counters are delta by default (report increments per sampling interval). However, the Tsuga integration may transform them. Stage 2 discovery will confirm whether `per-second` or `rate` is correct. Both options listed in 05. (Inference) -- **[subscription-health]** GCP metrics have a minimum granularity of **60 seconds**. Short-lived backlog spikes (< 1 min) may not appear in the data. Do not expect sub-minute resolution. ([GCP Cloud Monitoring Docs](https://cloud.google.com/monitoring/api/v3/metrics-details)) -- **[publishing-performance]** Publish metrics (`send_request_count`, `send_byte_count`) may show **1-3 minute delay** from real time. Do not page on "zero publishes" unless the gap exceeds 5 minutes. ([GCP Pub/Sub Monitoring](https://cloud.google.com/pubsub/docs/monitoring)) - -## "This looks bad but isn't" - -- **[subscription-health]** `oldest_unacked_message_age` jumping from 0 to a few seconds during normal operation is expected. It reflects the processing window between delivery and ack. Concern only if it grows continuously. (Inference) -- **[subscription-health]** `num_outstanding_messages` spiking during a burst is normal. It means the subscriber has received messages and is processing them. Concern only if it stays high AND backlog grows (subscriber holding messages but not acking). (Inference) -- **[dead-letter-errors]** A small number of dead-lettered messages during deployments or schema changes is expected. The dead letter mechanism is working as designed. Concern only if the rate is sustained or increasing. (Inference) -- **[pull-delivery]** `streaming_pull_response_count` with occasional error codes (UNAVAILABLE, DEADLINE_EXCEEDED) is normal. The client library reconnects automatically. Concern only if error responses dominate and backlog grows. ([Pull Troubleshooting](https://cloud.google.com/pubsub/docs/pull-troubleshooting)) -- **[throughput-operations]** `mod_ack_deadline_message_count` > 0 is expected and healthy. The client library automatically extends ack deadlines for messages being processed. It becomes concerning only if the ratio to sent messages is very high (>80%). (Inference) - -## Optional-feature traps (metrics absent unless X enabled) - -- **[dead-letter-errors]** `dead_letter_message_count` requires a dead letter topic to be configured on the subscription. Without DLT, this metric has no data, but messages that exhaust max delivery attempts are simply redelivered forever. ([Subscription Properties](https://cloud.google.com/pubsub/docs/subscription-properties)) -- **[dead-letter-errors]** `exactly_once_warning_count` requires exactly-once delivery to be enabled on the subscription. Without it, this metric has no data. Exactly-once is only available for pull subscriptions. ([Subscription Properties](https://cloud.google.com/pubsub/docs/subscription-properties)) -- **[push-delivery]** All `push_*` metrics require at least one push subscription. If the environment uses only pull subscriptions, the entire push delivery section will be empty. (Inference) -- **[publishing-performance]** `message_transform_latencies` requires Single Message Transforms (SMT) to be configured on at least one topic. Without SMT, this metric has no data. ([Topic Troubleshooting](https://cloud.google.com/pubsub/docs/topic-troubleshooting)) -- **[snapshots]** All `snapshot/*` metrics require snapshots to exist. Snapshots are optional and relatively uncommon. The entire snapshots section may be empty. ([GCP Pub/Sub Metrics](https://cloud.google.com/monitoring/api/metrics_gcp#gcp-pubsub)) -- **[capacity-retention]** `retained_acked_bytes` and `num_retained_acked_messages` require "retain acknowledged messages" to be enabled on the subscription. Without it, these metrics are zero. ([Subscription Properties](https://cloud.google.com/pubsub/docs/subscription-properties)) - - ---- - diff --git a/plugins/tsuga/skills/knowledge-technology/references/gcp-pubsub/metrics.csv b/plugins/tsuga/skills/knowledge-technology/references/gcp-pubsub/metrics.csv deleted file mode 100644 index 8a7be9a..0000000 --- a/plugins/tsuga/skills/knowledge-technology/references/gcp-pubsub/metrics.csv +++ /dev/null @@ -1,38 +0,0 @@ -theme,name,confirmed,definition,unit,type,tsuga_metric_name,widget_suggestions,aggregation,post_function,group_by,required_context_fields,optional_context_fields,absence_means,cardinality_footguns,variants,citations -Throughput/Usage,pubsub.googleapis.com/topic/send_message_operation_count,true,Number of publish message operations per topic. Core publish throughput signal. DEPRECATED but present in Tsuga.,{operation},counter,pubsub.googleapis.com/topic/send_message_operation_count,timeseries|query-value|top-list,sum,per-second,context.topic_id:20,context.env|context.team,context.cloud.account.id|context.cloud.region|context.project_id|context.response_code,No messages being published or GCP integration not configured,Safe,gcp=pubsub.googleapis.com/topic/send_message_operation_count,https://cloud.google.com/monitoring/api/metrics_gcp#gcp-pubsub -Errors/Failures,pubsub.googleapis.com/topic/send_request_count,true,"Number of publish requests grouped by response_code and response_class. Key error signal for publishing.",{request},counter,pubsub.googleapis.com/topic/send_request_count,timeseries|query-value|top-list,sum,per-second,context.topic_id:20|context.response_code:10,context.env|context.team,context.cloud.account.id|context.cloud.region|context.project_id|context.response_class,No publish requests or integration not configured,Safe,gcp=pubsub.googleapis.com/topic/send_request_count,https://cloud.google.com/monitoring/api/metrics_gcp#gcp-pubsub -Performance/Latency,pubsub.googleapis.com/topic/send_request_latencies,true,Distribution of publish request latencies in microseconds.,us,histogram,pubsub.googleapis.com/topic/send_request_latencies,timeseries|query-value,percentile(p95),none,context.topic_id:20,context.env|context.team,context.cloud.account.id|context.cloud.region|context.project_id|context.response_code|context.has_ordering|context.schema_type,No publish requests,Safe,gcp=pubsub.googleapis.com/topic/send_request_latencies,https://cloud.google.com/monitoring/api/metrics_gcp#gcp-pubsub -Throughput/Usage,pubsub.googleapis.com/topic/message_sizes,true,Distribution of publish message sizes in bytes.,By,histogram,pubsub.googleapis.com/topic/message_sizes,timeseries|query-value,percentile(p95),none,context.topic_id:20,context.env|context.team,context.cloud.account.id|context.cloud.region|context.project_id,No messages published,Safe,gcp=pubsub.googleapis.com/topic/message_sizes,https://cloud.google.com/monitoring/api/metrics_gcp#gcp-pubsub -Throughput/Usage,pubsub.googleapis.com/topic/byte_cost,true,"Cost of operations measured in bytes. Includes publish and transform costs.",By,counter,pubsub.googleapis.com/topic/byte_cost,timeseries|query-value,sum,per-second,context.topic_id:20|context.operation_type:5,context.env|context.team,context.cloud.account.id|context.cloud.region|context.project_id|context.response_code,No publish operations,Safe,gcp=pubsub.googleapis.com/topic/byte_cost,https://cloud.google.com/monitoring/api/metrics_gcp#gcp-pubsub -Throughput/Usage,pubsub.googleapis.com/topic/config_updates_count,true,Count of configuration changes on a topic.,{operation},counter,pubsub.googleapis.com/topic/config_updates_count,timeseries,sum,per-second,context.topic_id:20,context.env|context.team,context.cloud.account.id|context.project_id,No config changes (normal in steady state),Safe,gcp=pubsub.googleapis.com/topic/config_updates_count,https://cloud.google.com/monitoring/api/metrics_gcp#gcp-pubsub -Capacity/Saturation,pubsub.googleapis.com/topic/num_unacked_messages_by_region,true,Number of unacknowledged messages across all subscriptions for a topic by region.,{message},gauge,pubsub.googleapis.com/topic/num_unacked_messages_by_region,timeseries|query-value,sum,none,context.topic_id:20|context.cloud.region:10,context.env|context.team,context.cloud.account.id|context.project_id,All messages acked or no subscriptions exist,Safe,gcp=pubsub.googleapis.com/topic/num_unacked_messages_by_region,https://cloud.google.com/monitoring/api/metrics_gcp#gcp-pubsub -Performance/Latency,pubsub.googleapis.com/topic/oldest_unacked_message_age_by_region,true,Age of oldest unacknowledged message in a topic by region in seconds.,s,gauge,pubsub.googleapis.com/topic/oldest_unacked_message_age_by_region,timeseries|query-value,max,none,context.topic_id:20|context.cloud.region:10,context.env|context.team,context.cloud.account.id|context.project_id,All messages acked,Safe,gcp=pubsub.googleapis.com/topic/oldest_unacked_message_age_by_region,https://cloud.google.com/monitoring/api/metrics_gcp#gcp-pubsub -Capacity/Saturation,pubsub.googleapis.com/topic/num_retained_acked_messages_by_region,true,Number of acknowledged messages retained in a topic by region.,{message},gauge,pubsub.googleapis.com/topic/num_retained_acked_messages_by_region,timeseries|query-value,sum,none,context.topic_id:20|context.cloud.region:10,context.env|context.team,context.cloud.account.id|context.project_id,Retention not enabled or messages expired,Safe,gcp=pubsub.googleapis.com/topic/num_retained_acked_messages_by_region,https://cloud.google.com/monitoring/api/metrics_gcp#gcp-pubsub -Capacity/Saturation,pubsub.googleapis.com/topic/unacked_bytes_by_region,true,Total bytes of unacknowledged messages in a topic by region.,By,gauge,pubsub.googleapis.com/topic/unacked_bytes_by_region,timeseries|query-value,sum,none,context.topic_id:20|context.cloud.region:10,context.env|context.team,context.cloud.account.id|context.project_id,All messages acked,Safe,gcp=pubsub.googleapis.com/topic/unacked_bytes_by_region,https://cloud.google.com/monitoring/api/metrics_gcp#gcp-pubsub -Capacity/Saturation,pubsub.googleapis.com/topic/num_retained_messages,true,Number of messages retained in a topic.,{message},gauge,pubsub.googleapis.com/topic/num_retained_messages,timeseries|query-value,sum,none,context.topic_id:20,context.env|context.team,context.cloud.account.id|context.cloud.region|context.project_id,Topic retention not configured,Safe,gcp=pubsub.googleapis.com/topic/num_retained_messages,https://cloud.google.com/monitoring/api/metrics_gcp#gcp-pubsub -Capacity/Saturation,pubsub.googleapis.com/topic/retained_bytes,true,Total byte size of messages retained in a topic.,By,gauge,pubsub.googleapis.com/topic/retained_bytes,timeseries|query-value,sum,none,context.topic_id:20,context.env|context.team,context.cloud.account.id|context.cloud.region|context.project_id,Topic retention not configured,Safe,gcp=pubsub.googleapis.com/topic/retained_bytes,https://cloud.google.com/monitoring/api/metrics_gcp#gcp-pubsub -Performance/Latency,pubsub.googleapis.com/topic/oldest_retained_message_age,true,Age of oldest message retained in a topic in seconds.,s,gauge,pubsub.googleapis.com/topic/oldest_retained_message_age,timeseries|query-value,max,none,context.topic_id:20,context.env|context.team,context.cloud.account.id|context.cloud.region|context.project_id,Topic retention not configured,Safe,gcp=pubsub.googleapis.com/topic/oldest_retained_message_age,https://cloud.google.com/monitoring/api/metrics_gcp#gcp-pubsub -Throughput/Usage,pubsub.googleapis.com/subscription/sent_message_count,true,Number of messages sent to subscribers. Core consumption throughput.,{message},counter,pubsub.googleapis.com/subscription/sent_message_count,timeseries|query-value|top-list,sum,per-second,context.subscription_id:20|context.delivery_type:5,context.env|context.team,context.cloud.account.id|context.cloud.region|context.project_id,No delivery activity or subscriber not connected,Safe,gcp=pubsub.googleapis.com/subscription/sent_message_count,https://cloud.google.com/monitoring/api/metrics_gcp#gcp-pubsub -Throughput/Usage,pubsub.googleapis.com/subscription/byte_cost,true,Cost of subscription operations measured in bytes.,By,counter,pubsub.googleapis.com/subscription/byte_cost,timeseries|query-value,sum,per-second,context.subscription_id:20,context.env|context.team,context.cloud.account.id|context.cloud.region|context.project_id,No subscription operations,Safe,gcp=pubsub.googleapis.com/subscription/byte_cost,https://cloud.google.com/monitoring/api/metrics_gcp#gcp-pubsub -Throughput/Usage,pubsub.googleapis.com/subscription/pull_request_count,true,Number of pull requests with response_code and response_class labels.,{request},counter,pubsub.googleapis.com/subscription/pull_request_count,timeseries|query-value,sum,per-second,context.subscription_id:20|context.response_code:10,context.env|context.team,context.cloud.account.id|context.cloud.region|context.project_id|context.response_class,No pull subscribers active,Safe,gcp=pubsub.googleapis.com/subscription/pull_request_count,https://cloud.google.com/monitoring/api/metrics_gcp#gcp-pubsub -Throughput/Usage,pubsub.googleapis.com/subscription/pull_message_operation_count,true,Number of pull message operations. DEPRECATED but present in Tsuga.,{operation},counter,pubsub.googleapis.com/subscription/pull_message_operation_count,timeseries|query-value,sum,per-second,context.subscription_id:20,context.env|context.team,context.cloud.account.id|context.cloud.region|context.project_id,No pull subscribers or empty responses,Safe,gcp=pubsub.googleapis.com/subscription/pull_message_operation_count,https://cloud.google.com/monitoring/api/metrics_gcp#gcp-pubsub -Throughput/Usage,pubsub.googleapis.com/subscription/streaming_pull_response_count,true,Number of streaming pull responses with response_code and response_class.,{response},counter,pubsub.googleapis.com/subscription/streaming_pull_response_count,timeseries|query-value,sum,per-second,context.subscription_id:20|context.response_code:10,context.env|context.team,context.cloud.account.id|context.cloud.region|context.project_id|context.response_class,No streaming pull subscribers active,Safe,gcp=pubsub.googleapis.com/subscription/streaming_pull_response_count,https://cloud.google.com/monitoring/api/metrics_gcp#gcp-pubsub -Throughput/Usage,pubsub.googleapis.com/subscription/streaming_pull_message_operation_count,true,Number of streaming pull message operations. DEPRECATED but present in Tsuga.,{operation},counter,pubsub.googleapis.com/subscription/streaming_pull_message_operation_count,timeseries|query-value,sum,per-second,context.subscription_id:20,context.env|context.team,context.cloud.account.id|context.cloud.region|context.project_id,No streaming pull subscribers or no messages,Safe,gcp=pubsub.googleapis.com/subscription/streaming_pull_message_operation_count,https://cloud.google.com/monitoring/api/metrics_gcp#gcp-pubsub -Throughput/Usage,pubsub.googleapis.com/subscription/ack_message_count,true,Number of messages acknowledged by subscribers.,{message},counter,pubsub.googleapis.com/subscription/ack_message_count,timeseries|query-value,sum,per-second,context.subscription_id:20|context.delivery_type:5,context.env|context.team,context.cloud.account.id|context.cloud.region|context.project_id,No acks (subscriber not processing or not connected),Safe,gcp=pubsub.googleapis.com/subscription/ack_message_count,https://cloud.google.com/monitoring/api/metrics_gcp#gcp-pubsub -Throughput/Usage,pubsub.googleapis.com/subscription/config_updates_count,true,Count of configuration changes on a subscription.,{operation},counter,pubsub.googleapis.com/subscription/config_updates_count,timeseries,sum,per-second,context.subscription_id:20,context.env|context.team,context.cloud.account.id|context.project_id,No config changes (normal in steady state),Safe,gcp=pubsub.googleapis.com/subscription/config_updates_count,https://cloud.google.com/monitoring/api/metrics_gcp#gcp-pubsub -Capacity/Saturation,pubsub.googleapis.com/subscription/num_undelivered_messages,true,"Number of undelivered messages (backlog). THE primary subscription health gauge.",{message},gauge,pubsub.googleapis.com/subscription/num_undelivered_messages,timeseries|query-value|top-list,sum,none,context.subscription_id:20,context.env|context.team,context.cloud.account.id|context.cloud.region|context.project_id,Backlog empty - healthy or no messages published,Safe,gcp=pubsub.googleapis.com/subscription/num_undelivered_messages,https://cloud.google.com/monitoring/api/metrics_gcp#gcp-pubsub -Performance/Latency,pubsub.googleapis.com/subscription/oldest_unacked_message_age,true,"Age of oldest unacknowledged message in seconds. Primary delivery lag indicator.",s,gauge,pubsub.googleapis.com/subscription/oldest_unacked_message_age,timeseries|query-value|top-list,max,none,context.subscription_id:20,context.env|context.team,context.cloud.account.id|context.cloud.region|context.project_id,No unacked messages - fully caught up,Safe,gcp=pubsub.googleapis.com/subscription/oldest_unacked_message_age,https://cloud.google.com/monitoring/api/metrics_gcp#gcp-pubsub -Capacity/Saturation,pubsub.googleapis.com/subscription/backlog_bytes,true,Total size of undelivered messages in bytes.,By,gauge,pubsub.googleapis.com/subscription/backlog_bytes,timeseries|query-value|top-list,sum,none,context.subscription_id:20,context.env|context.team,context.cloud.account.id|context.cloud.region|context.project_id,Backlog empty,Safe,gcp=pubsub.googleapis.com/subscription/backlog_bytes,https://cloud.google.com/monitoring/api/metrics_gcp#gcp-pubsub -Capacity/Saturation,pubsub.googleapis.com/subscription/num_retained_acked_messages,true,Number of acknowledged messages retained for replay via Seek.,{message},gauge,pubsub.googleapis.com/subscription/num_retained_acked_messages,timeseries|query-value,sum,none,context.subscription_id:20,context.env|context.team,context.cloud.account.id|context.cloud.region|context.project_id,Retained acked messages not enabled or all expired,Safe,gcp=pubsub.googleapis.com/subscription/num_retained_acked_messages,https://cloud.google.com/monitoring/api/metrics_gcp#gcp-pubsub -Capacity/Saturation,pubsub.googleapis.com/subscription/retained_acked_bytes,true,Total size of retained acknowledged messages in bytes.,By,gauge,pubsub.googleapis.com/subscription/retained_acked_bytes,timeseries|query-value,sum,none,context.subscription_id:20,context.env|context.team,context.cloud.account.id|context.cloud.region|context.project_id,Retained acked messages not enabled,Safe,gcp=pubsub.googleapis.com/subscription/retained_acked_bytes,https://cloud.google.com/monitoring/api/metrics_gcp#gcp-pubsub -Errors/Failures,pubsub.googleapis.com/subscription/expired_ack_deadlines_count,true,Number of messages with expired ack deadlines (redelivered).,{message},counter,pubsub.googleapis.com/subscription/expired_ack_deadlines_count,timeseries|query-value,sum,per-second,context.subscription_id:20|context.delivery_type:5,context.env|context.team,context.cloud.account.id|context.cloud.region|context.project_id,All messages acked within deadline - healthy,Safe,gcp=pubsub.googleapis.com/subscription/expired_ack_deadlines_count,https://cloud.google.com/monitoring/api/metrics_gcp#gcp-pubsub -Errors/Failures,pubsub.googleapis.com/subscription/exactly_once_warning_count,true,Number of exactly-once delivery warnings. Present only when exactly-once is enabled.,{warning},counter,pubsub.googleapis.com/subscription/exactly_once_warning_count,timeseries|query-value,sum,per-second,context.subscription_id:20,context.env|context.team,context.cloud.account.id|context.cloud.region|context.project_id,Exactly-once delivery not enabled or no warnings,Safe,gcp=pubsub.googleapis.com/subscription/exactly_once_warning_count,https://cloud.google.com/monitoring/api/metrics_gcp#gcp-pubsub -Performance/Latency,pubsub.googleapis.com/subscription/ack_latencies,true,Distribution of ack latencies in milliseconds.,ms,histogram,pubsub.googleapis.com/subscription/ack_latencies,timeseries|query-value,percentile(p95),none,context.subscription_id:20|context.delivery_type:5,context.env|context.team,context.cloud.account.id|context.cloud.region|context.project_id,No ack activity,Safe,gcp=pubsub.googleapis.com/subscription/ack_latencies,https://cloud.google.com/monitoring/api/metrics_gcp#gcp-pubsub -Availability/Health,pubsub.googleapis.com/subscription/open_streaming_pulls,true,Number of open StreamingPull streams per subscription.,{connection},gauge,pubsub.googleapis.com/subscription/open_streaming_pulls,timeseries|query-value,sum,none,context.subscription_id:20|context.status:5,context.env|context.team,context.cloud.account.id|context.cloud.region|context.project_id,No streaming pull subscribers connected,Safe,gcp=pubsub.googleapis.com/subscription/open_streaming_pulls,https://cloud.google.com/monitoring/api/metrics_gcp#gcp-pubsub -Throughput/Usage,pubsub.googleapis.com/subscription/pull_ack_request_count,true,Number of pull ack requests with response_code and response_class.,{request},counter,pubsub.googleapis.com/subscription/pull_ack_request_count,timeseries,sum,per-second,context.subscription_id:20|context.response_code:10,context.env|context.team,context.cloud.account.id|context.cloud.region|context.project_id|context.response_class,No pull ack requests,Safe,gcp=pubsub.googleapis.com/subscription/pull_ack_request_count,https://cloud.google.com/monitoring/api/metrics_gcp#gcp-pubsub -Throughput/Usage,pubsub.googleapis.com/snapshot/config_updates_count,true,Count of configuration changes on snapshots.,{operation},counter,pubsub.googleapis.com/snapshot/config_updates_count,timeseries,sum,per-second,,context.env|context.team,context.cloud.account.id|context.project_id,No snapshot config changes,Safe,gcp=pubsub.googleapis.com/snapshot/config_updates_count,https://cloud.google.com/monitoring/api/metrics_gcp#gcp-pubsub -Errors/Failures,pubsub.googleapis.com/subscription/push_request_count,false,"Number of push requests. Key push health signal. NOT FOUND in current Tsuga instance (no push subscriptions).",{request},counter,pubsub.googleapis.com/subscription/push_request_count,timeseries|query-value|top-list,sum,per-second,context.subscription_id:20|context.response_class:10,context.env|context.team,context.cloud.account.id|context.response_code,No push subscriptions configured,Safe,gcp=pubsub.googleapis.com/subscription/push_request_count,https://cloud.google.com/monitoring/api/metrics_gcp#gcp-pubsub -Performance/Latency,pubsub.googleapis.com/subscription/push_request_latencies,false,"Distribution of push request latencies in microseconds. NOT FOUND in current Tsuga instance (no push subscriptions).",us,histogram,pubsub.googleapis.com/subscription/push_request_latencies,timeseries|query-value,percentile(p95),none,context.subscription_id:20,context.env|context.team,context.cloud.account.id,No push subscriptions configured,Safe,gcp=pubsub.googleapis.com/subscription/push_request_latencies,https://cloud.google.com/monitoring/api/metrics_gcp#gcp-pubsub -Errors/Failures,pubsub.googleapis.com/subscription/dead_letter_message_count,false,"Number of messages forwarded to dead letter topic. NOT FOUND in current Tsuga instance (no DLT configured).",{message},counter,pubsub.googleapis.com/subscription/dead_letter_message_count,timeseries|query-value|top-list,sum,per-second,context.subscription_id:20,context.env|context.team,context.cloud.account.id,No dead letter topic configured,Safe,gcp=pubsub.googleapis.com/subscription/dead_letter_message_count,https://cloud.google.com/monitoring/api/metrics_gcp#gcp-pubsub -Availability/Health,pubsub.googleapis.com/subscription/delivery_latency_health_score,false,"Composite health score (0-1). NOT FOUND in current Tsuga instance.",1,gauge,pubsub.googleapis.com/subscription/delivery_latency_health_score,query-value|timeseries,average,none,context.subscription_id:20,context.env|context.team,context.cloud.account.id,Metric not available in this Tsuga instance,Safe,gcp=pubsub.googleapis.com/subscription/delivery_latency_health_score,https://cloud.google.com/pubsub/docs/push-troubleshooting -Cost,NOT_APPLICABLE,false,"Pub/Sub does not expose direct cost metrics. Proxy: topic/byte_cost and subscription/byte_cost correlate with data processing costs.",,,,,,,,,,,, diff --git a/plugins/tsuga/skills/knowledge-technology/references/gcp-pubsub/queries.md b/plugins/tsuga/skills/knowledge-technology/references/gcp-pubsub/queries.md deleted file mode 100644 index 1261637..0000000 --- a/plugins/tsuga/skills/knowledge-technology/references/gcp-pubsub/queries.md +++ /dev/null @@ -1,55 +0,0 @@ -# Google Cloud Pub/Sub - -Managed pub/sub. Healthy: ack ≈ publish, bounded oldest-unacked-age, no undelivered backlog. - -## Incident shapes - -- **Subscription backlog growth** — `num_undelivered_messages` climbs → consumer slower than producer -- **Oldest-unacked-age rising linearly** — consumer stuck (sawtooth is normal processing lag) -- **Consumer stall** — `ack_message_count` drops, `sent_message_count` flat → consumer alive but not acking -- **Push-subscription failures** — push endpoint returning non-200 → undelivered climbs -- **Publish latency / throttling** — `send_request_latencies` p99 spikes → quota pressure -- **Retention runway** — `backlog_bytes` growing near retention limit → data-loss risk - -## Key metrics - -Prefixed `pubsub.googleapis.com/`: - -| Metric | Unit | Signal | -|---|---|---| -| `topic/send_message_operation_count` | count | Publishes | -| `topic/send_request_latencies` | ms | Publish latency | -| `topic/message_sizes` | bytes | Message size distribution | -| `topic/oldest_unacked_message_age_by_region` | s | Topic-level backlog age | -| `subscription/sent_message_count` | count | Delivered to subscribers | -| `subscription/ack_message_count` | count | Acks received | -| `subscription/pull_request_count` | count | Pull calls | -| `subscription/streaming_pull_response_count` | count | Streaming pull | -| `subscription/num_undelivered_messages` | count | Backlog size | -| `subscription/oldest_unacked_message_age` | s | Oldest pending-ack age | -| `subscription/backlog_bytes` | bytes | Backlog volume | - -## Derived signals - -- `ack_message_count / sent_message_count` — processing ratio. < 1.0 sustained = consumer stuck. -- First-derivative of `num_undelivered_messages` — backlog trajectory. -- `retention_duration_seconds - oldest_unacked_message_age` — retention runway until message loss. - -## Log patterns - -Consumer-side + GCP audit logs: - -- `DEADLINE_EXCEEDED` — ack deadline too short or consumer slow -- `RESOURCE_EXHAUSTED` — subscription throughput or quota hit -- `INVALID_ARGUMENT` — malformed ack ID; message redelivered -- `PERMISSION_DENIED` — IAM misconfig -- Audit log: `google.pubsub.v1.Publisher.Publish` — producer activity - -## Gotchas - -- Pulled-not-acked messages are redelivered after ack deadline (default 10s). Long processing must extend via `modifyAckDeadline`. -- `num_undelivered_messages` excludes leased-not-acked. True backlog = undelivered + in-flight. -- Retention defaults 7d (max 31d). Approaching without acking = data loss. -- Ordering keys serialize delivery per key; one slow key stalls its entire stream without affecting aggregates. -- Push subscriptions: HTTP 200 from endpoint acks the message even if the endpoint had side-effect errors internally. -- Dead-letter topic is per-subscription and opt-in. Subs without DLT can drop messages after max-delivery-attempts. diff --git a/plugins/tsuga/skills/knowledge-technology/references/gcp-storage/README.md b/plugins/tsuga/skills/knowledge-technology/references/gcp-storage/README.md deleted file mode 100644 index bfd9e57..0000000 --- a/plugins/tsuga/skills/knowledge-technology/references/gcp-storage/README.md +++ /dev/null @@ -1,712 +0,0 @@ -# Google Cloud Storage Integration Context Bundle - -**Technology:** Google Cloud Storage (GCS) -**Deployment:** Managed (GCP) -**Environment:** prod -**Persona:** SRE Dev and ops -**Telemetry preference:** mixed -**Integration scope:** core service only -**Primary use-case:** reliability and performance - ---- - -## How to use this bundle - -- **Dashboard plan:** Start with `07_google-cloud-storage_dashboard_plan.yaml` for section structure, widget specs, and coverage map. -- **Metric truth:** Use `05_google-cloud-storage_metric_inventory.csv` for exact metric names, units, aggregations, and post-functions. -- **Notes and playbooks:** Use `09_google-cloud-storage_section_notes_and_playbooks.md` for all note widget prose and operational runbooks. - ---- - -## Confirmed Tsuga prefixes - -- `storage.googleapis.com/*` — **CONFIRMED** (11/21 metrics present in Tsuga; 10 missing due to feature gating: replication, authn, authz, autoclass, v2/deleted_bytes) -- Naming convention: **original GCP slash notation** (e.g., `storage.googleapis.com/api/request_count`) - -Discovery: performed 2026-02-10. 11 metrics confirmed, 4 unexpected (anywhere_cache/*, v2/total_byte_seconds, v2/total_count), 10 missing (feature-gated). - ---- - -## Bundle files - -| # | Filename | Purpose | -|---|---|---| -| 00 | `00_google-cloud-storage_cover.md` | This file — metadata, navigation, and Stage 2 handoff contract | -| 01 | `01_google-cloud-storage_executive_overview.md` | What GCS is, what "good" looks like, top incident shapes | -| 02 | `02_google-cloud-storage_key_concepts.md` | Glossary (22 terms), concept map (29 lines), entities/dimensions (13), Tsuga field mapping | -| 03 | `03_google-cloud-storage_golden_signals.md` | Traffic/Errors/Latency/Saturation mapped to GCS metrics and section questions | -| 04 | `04_google-cloud-storage_telemetry_sources.md` | Source matrix, feature gating table, sampling intervals | -| 05 | `05_google-cloud-storage_metric_inventory.csv` | 21 metrics (11 confirmed, 10 missing): names, types, units, Tsuga mapping, aggregations, group-bys | -| 06 | `06_google-cloud-storage_derived_signals.csv` | 14 derived signals with formulas, inputs, and interpretation | -| 07 | `07_google-cloud-storage_dashboard_plan.yaml` | 7 sections, 2 dashboards (overview + deep dive), 39 widgets, coverage map | -| 09 | `09_google-cloud-storage_section_notes_and_playbooks.md` | Mission note, 7 section notes, 22 triage chains, 7 playbooks | -| 10 | `10_google-cloud-storage_caveats_footguns.md` | 22 caveats tagged by section across 6 categories | -| 11 | `11_google-cloud-storage_unknowns_verify_next.yaml` | 3 remaining unknowns | -| 12 | `12_google-cloud-storage_discovery_reconciliation.md` | Discovery & reconciliation report | - ---- - -## Top sources - -1. [Google Cloud metrics reference (P-Z)](https://docs.cloud.google.com/monitoring/api/metrics_gcp_p_z) — Canonical metric definitions for all `storage.googleapis.com/*` metrics including types, units, and labels. -2. [Cloud Storage monitoring overview](https://docs.cloud.google.com/storage/docs/monitoring) — Official monitoring guidance including metric update frequencies and dashboard recommendations. -3. [Data storage SLI metrics](https://docs.cloud.google.com/stackdriver/docs/solutions/slo-monitoring/sli-metrics/data-storage-metrics) — Confirms absence of server-side latency metric; defines SLI-eligible metrics. -4. [Request rate and access distribution guidelines](https://docs.cloud.google.com/storage/docs/request-rate) — Per-bucket rate limits, auto-scaling behavior, sequential naming anti-pattern. -5. [Bandwidth usage overview](https://docs.cloud.google.com/storage/docs/bandwidth-usage) — Bandwidth quotas (200 Gbps default), quota metrics, CDN exemptions. -6. [Cloud Storage consistency](https://docs.cloud.google.com/storage/docs/consistency) — Strong consistency model, IAM eventual consistency (~1 min), HMAC key propagation. -7. [Managing turbo replication](https://docs.cloud.google.com/storage/docs/managing-turbo-replication) — RPO monitoring, turbo_max_delay metric, multi-hour reporting delay. -8. [Using uniform bucket-level access](https://docs.cloud.google.com/storage/docs/using-uniform-bucket-level-access) — ACL metrics, migration guidance, access_id cardinality warning. -9. [Cloud Storage troubleshooting](https://docs.cloud.google.com/storage/docs/troubleshooting) — Retry amplification, soft-delete retention, common error patterns. -10. [Cloud Storage quotas and limits](https://docs.cloud.google.com/storage/quotas) — Rate limits, object size limits, bucket creation limits. - - ---- - -# Google Cloud Storage — Executive Overview - -## What it is -Google Cloud Storage (GCS) is GCP's fully managed object storage service. It stores blobs (files, media, backups, logs, data lake objects) in **buckets** across single-region, dual-region, and multi-region locations. Applications interact via JSON/gRPC APIs; GCS auto-scales reads and writes without provisioning. - -## What "good" looks like -- `api/request_count` returns mostly `OK` responses; 5xx and `RESOURCE_EXHAUSTED` codes are near zero. -- `network/sent_bytes_count` and `network/received_bytes_count` track steady, predictable bandwidth aligned with application traffic patterns. -- `storage/total_bytes` grows linearly in line with data retention policy; no unexpected jumps (orphaned multipart uploads, versioning accumulation, soft-delete retention). -- Replication-enabled buckets show `replication/meeting_rpo = 1` consistently. - -## Paging intent (symptom-level) -- Sustained 5xx error rate or `RESOURCE_EXHAUSTED` throttling on production buckets. -- Replication RPO violations on dual/multi-region buckets with turbo replication SLAs. -- Bandwidth quota usage approaching limits (>80% of project quota). - -## Top 3 incident shapes and first dashboard section -| Incident shape | First section to check | -|---|---| -| Elevated API error rate (5xx spike or permission denied wave) | **Errors & Reliability** — break down by `response_code` and `method` | -| Unexpected storage growth (cost spike, quota risk) | **Storage Capacity & Growth** — check `total_bytes` by `storage_class` and object count trends | -| Bandwidth throttling / quota exhaustion | **Network & Bandwidth** — compare egress rate to quota limits | - ---- - -### Confirmed by sources -- GCS returns gRPC status codes (`OK`, `PERMISSION_DENIED`, `RESOURCE_EXHAUSTED`, etc.) as the `response_code` label on `api/request_count` ([Google Cloud metrics reference](https://docs.cloud.google.com/monitoring/api/metrics_gcp_p_z)). -- GCS has no server-side latency metric; latency must be measured client-side ([Data storage SLI metrics](https://docs.cloud.google.com/stackdriver/docs/solutions/slo-monitoring/sli-metrics/data-storage-metrics)). -- Storage metrics (`total_bytes`, `object_count`) update once per 24 hours ([Cloud Storage monitoring](https://docs.cloud.google.com/storage/docs/monitoring)). - -### Best-practice inference -- Bandwidth quota saturation at 80% as a leading indicator is an operational best practice, not a documented GCS threshold. -- Linear storage growth as "normal" assumes well-configured lifecycle rules; without them, growth can be super-linear due to versioning or soft-delete. - - ---- - -# Google Cloud Storage — Key Concepts - -## Glossary (>= 20 terms) - -| Term | Definition | Operational meaning | Dashboard section affected | -|---|---|---|---| -| **Bucket** | Top-level container for objects. Globally unique name, tied to a project and location. | Primary grouping dimension (`bucket_name`). All metrics are scoped per bucket. | All sections | -| **Object** | An immutable blob stored in a bucket, identified by name (key). | The unit of storage. Object count and size drive capacity metrics. | Storage Capacity & Growth | -| **Storage class** | Pricing/access tier: Standard, Nearline, Coldline, Archive. | Determines at-rest cost and retrieval fees. Group `total_bytes` by `storage_class` to see distribution. | Storage Capacity & Growth, Cost | -| **Location type** | Single-region, dual-region, or multi-region. Determines redundancy and latency. | Dual/multi-region buckets have replication metrics; single-region do not. | Replication & Durability | -| **Turbo replication** | Dual-region feature guaranteeing 15-minute RPO for cross-region sync. | Monitored via `turbo_max_delay` and `meeting_rpo`. Only exists on dual-region buckets with turbo enabled. | Replication & Durability | -| **RPO (Recovery Point Objective)** | Maximum acceptable data loss window during failover. | `meeting_rpo = 0` means the bucket has violated its replication target. | Replication & Durability | -| **API method** | The operation type: `ReadObject`, `WriteObject`, `ListObjects`, `DeleteObject`, `InsertObject`, `ComposeObject`, `CopyObject`, etc. | Label on `api/request_count`. Group by `method` to understand traffic profile. | Traffic & Request Volume | -| **Response code** | gRPC-style status code: `OK`, `NOT_FOUND`, `PERMISSION_DENIED`, `RESOURCE_EXHAUSTED`, `INTERNAL`, `UNAVAILABLE`, etc. | Label on `api/request_count`. Filter for non-OK to find errors. | Errors & Reliability | -| **Lifecycle rule** | Automated policy to delete objects or transition storage classes based on age, creation date, or custom time. | Missing or misconfigured rules cause unexpected storage growth. | Storage Capacity & Growth | -| **Object versioning** | Bucket setting that retains noncurrent versions of objects on overwrite/delete. | Noncurrent versions accumulate storage cost. Check `object_count` growth trends. | Storage Capacity & Growth | -| **Soft delete** | Default 90-day retention of deleted objects (recoverable). | Soft-deleted objects still appear in `storage/v2/total_bytes` with `type=soft-deleted`. | Storage Capacity & Growth | -| **Uniform bucket-level access** | Disables per-object ACLs; all access controlled via IAM only. | ACL metrics (`authz/*`) should be zero after migration. Non-zero = legacy access paths remain. | Access & Security | -| **IAM** | Identity and Access Management. Roles control who can read/write/admin buckets and objects. | `PERMISSION_DENIED` responses indicate IAM misconfigurations. | Errors & Reliability, Access & Security | -| **ACL** | Access Control List (legacy). Per-object or per-bucket access grants. | `authz/acl_operations_count` tracks ACL mutations; goal is to migrate to uniform access. | Access & Security | -| **Autoclass** | Automatic storage class transitions based on access patterns. | Tracked via `autoclass/transition_operation_count` and `transitioned_bytes_count`. Only present on autoclass-enabled buckets. | Cost & Storage Class | -| **Bandwidth quota** | Per-project, per-region limit on egress throughput (default 200 Gbps). | `quota/*/usage` vs `quota/*/limit` shows saturation. `RESOURCE_EXHAUSTED` = quota hit. | Network & Bandwidth | -| **Anywhere Cache** | Zone-level cache for hot object reads. Reduces latency and egress. | Tracked via `anywhere_cache/request_count` with `cache_hit` label. Only exists if enabled. | Network & Bandwidth | -| **Multipart upload** | Splitting large objects into parts for parallel upload. | Abandoned multipart uploads consume storage until cleaned up by lifecycle rules. | Storage Capacity & Growth | -| **Compose** | Server-side concatenation of up to 32 objects into one. | Counted as a Class A operation in `api/request_count` with method `ComposeObject`. | Traffic & Request Volume | -| **HMAC key** | Hash-based authentication key for S3-compatible access to GCS. | HMAC key state changes take up to 3 minutes to propagate (eventual consistency). | Access & Security | -| **Request rate scaling** | GCS auto-scales per-bucket from ~5K reads/s and ~1K writes/s baseline. | Scaling takes minutes. Ramp up gradually (double every 20 min). Fast ramp = `RESOURCE_EXHAUSTED`. | Traffic & Request Volume, Errors & Reliability | -| **Early deletion fee** | Charge for deleting objects before their storage class minimum duration (30/90/365 days). | Not directly metricked; inferred from `autoclass/transition_operation_count` and object age. | Cost & Storage Class | - ---- - -## Concept Map (>= 25 lines) - -``` -Bucket -> contains -> Objects (all storage metrics scoped to bucket) -Bucket -> has -> Location type (determines replication metrics availability) -Bucket -> has -> Default storage class (affects new object placement) -Bucket -> may enable -> Object versioning (causes noncurrent version accumulation) -Bucket -> may enable -> Lifecycle rules (automates deletion and class transitions) -Bucket -> may enable -> Turbo replication (dual-region only; adds RPO monitoring) -Bucket -> may enable -> Autoclass (automatic storage class transitions) -Bucket -> may enable -> Anywhere Cache (zone-level cache for hot reads) -Bucket -> may enable -> Soft delete (default 90-day retention of deleted objects) -Bucket -> enforces -> Access control (IAM or legacy ACLs) -Object -> belongs to -> Storage class (Standard, Nearline, Coldline, Archive) -Object -> has -> Size in bytes (contributes to storage/total_bytes) -Object -> may have -> Noncurrent versions (if versioning enabled) -API request -> targets -> Bucket + Object (method + response_code labels) -API request -> counted by -> api/request_count metric -API request -> may fail with -> Response code (OK, NOT_FOUND, PERMISSION_DENIED, etc.) -API request -> transfers -> Network bytes (sent_bytes_count / received_bytes_count) -Network egress -> limited by -> Bandwidth quota (per-project, per-region) -Bandwidth quota -> triggers -> RESOURCE_EXHAUSTED when exceeded -Storage class -> determines -> At-rest cost + retrieval fees -Lifecycle rule -> transitions -> Objects between storage classes -Lifecycle rule -> deletes -> Objects after retention period -Autoclass -> transitions -> Objects based on access patterns (measured by transitions metric) -Replication -> syncs -> Objects across regions (dual/multi-region buckets) -Replication -> monitored by -> meeting_rpo + turbo_max_delay metrics -Anywhere Cache -> caches -> Hot reads in specific zones (cache_hit label) -Authentication -> tracked by -> authn/authentication_count metric -ACL operations -> tracked by -> authz/acl_operations_count metric -Quota usage -> approaching limit -> Leading indicator of throttling -``` - ---- - -## Entities and Dimensions (>= 12) - -| Dimension | Source label | Tsuga field (recommended) | Why useful | Cardinality risk | Safe Top-N | Do NOT group-by | -|---|---|---|---|---|---|---| -| Project ID | `project_id` (resource label) | `context.cloud.account.id` | Scope to a single GCP project | Low (bounded by org) | 10 | — | -| Bucket name | `bucket_name` (resource label) | `context.gcp.bucket_name` | Primary entity for all GCS metrics | Medium — can be high in multi-project orgs | 25 | — | -| Location | `location` (resource label) | `context.cloud.region` | Region/multi-region grouping for latency and cost analysis | Low (bounded by GCP locations) | 10 | — | -| API method | `method` (metric label) | `context.method` | Distinguish read/write/list/delete patterns | Low (~20 distinct methods) | 15 | — | -| Response code | `response_code` (metric label) | `context.response_code` | Error breakdown; filter for non-OK | Low (~15 status codes) | 10 | — | -| Storage class | `storage_class` (metric label) | `context.storage_class` | Cost and capacity breakdown by tier | Very low (4 classes) | 5 | — | -| Authentication method | `authentication_method` (metric label) | `context.authn_method` | Security audit of auth patterns | Low | 5 | — | -| Access ID | `access_id` (metric label on authn) | `context.access_id` | Identify specific service accounts | **HIGH** — unbounded | — | Do NOT group-by without filter; use top-list only | -| ACL operation | `acl_operation` (metric label) | `context.acl_operation` | Track ACL usage for uniform-access migration | Very low | 5 | — | -| Cache hit | `cache_hit` (metric label, anywhere_cache) | `context.cache_hit` | Cache effectiveness breakdown | Very low (true/false) | 2 | — | -| Zone (anywhere cache) | `zone` (metric label, anywhere_cache) | `context.zone` | Per-zone cache performance | Low | 10 | — | -| RPO status | `rpo_status` (metric label, replication) | `context.rpo_status` | Distinguish met vs missed replications | Very low | 2 | — | -| Type (v2 storage) | `type` (metric label, storage/v2) | `context.storage_type` | Distinguish live vs soft-deleted objects | Very low | 3 | — | - ---- - -## Tsuga Field Mapping Table (REQUIRED) - -| Vendor/exporter dimension | Recommended `context.*` key | Must-exist | Notes | -|---|---|---|---| -| `project_id` | `context.cloud.account.id` | Yes | Standard GCP project mapping | -| `bucket_name` | `context.gcp.bucket_name` | Yes | Primary entity for GCS | -| `location` | `context.cloud.region` | Yes | Region or multi-region identifier | -| `method` | `context.method` | Yes | API method (ReadObject, WriteObject, etc.) | -| `response_code` | `context.response_code` | Yes | gRPC status code | -| `storage_class` | `context.storage_class` | Optional | Present only on storage metrics | -| `authentication_method` | `context.authn_method` | Optional | Present only on authn metrics | -| `access_id` | `context.access_id` | Optional | Present only on authn metrics; high cardinality | -| `acl_operation` | `context.acl_operation` | Optional | Present only on authz metrics | -| `cache_hit` | `context.cache_hit` | Optional | Present only on anywhere_cache metrics | -| `zone` | `context.zone` | Optional | Present only on anywhere_cache metrics | -| `rpo_status` | `context.rpo_status` | Optional | Present only on replication metrics | -| `type` | `context.storage_type` | Optional | Present only on storage/v2 metrics (live, soft-deleted, etc.) | -| `context.env` | `context.env` | **Unknown** | Standard Tsuga convention — must verify in Stage 2 | -| `context.team` | `context.team` | **Unknown** | Standard Tsuga convention — must verify in Stage 2 | - ---- - -### Confirmed by sources -- Resource labels (`project_id`, `bucket_name`, `location`) confirmed in [Google Cloud metrics reference](https://docs.cloud.google.com/monitoring/api/metrics_gcp_p_z). -- Metric labels (`method`, `response_code`, `storage_class`) confirmed in [Cloud Storage monitoring](https://docs.cloud.google.com/storage/docs/monitoring). -- ~20 distinct API methods listed in [Cloud Storage JSON API reference](https://cloud.google.com/storage/docs/json_api/v1). -- `access_id` label high cardinality risk noted in [Authentication metrics docs](https://docs.cloud.google.com/storage/docs/using-uniform-bucket-level-access). - -### Best-practice inference -- Tsuga `context.*` field names are inferred from common conventions (`context.cloud.account.id`, `context.cloud.region`). Exact field names must be verified in Stage 2 discovery. -- `context.env` and `context.team` are assumed Tsuga standard fields but may not be present on GCP cloud metrics (these are typically applied to agent-collected metrics, not cloud integrations). - - ---- - -# Google Cloud Storage — Golden Signals - -## Traffic - -### What it means for GCS -Request volume and data transfer rates across all bucket operations. GCS traffic is measured by API call count (`api/request_count`) and network bytes (`network/sent_bytes_count`, `network/received_bytes_count`). Traffic patterns reveal whether applications are reading, writing, listing, or deleting objects, and at what intensity. - -### Typical causes when it degrades -- Application deployment changes (new batch jobs, migration scripts) -- Misconfigured retry loops amplifying requests (429 -> retry -> 429 loop) -- Upstream traffic shift (CDN cache purge driving reads back to origin) -- List operations increasing due to growing object counts or directory-style enumeration - -### Best telemetry sources -- `api/request_count` grouped by `method` — primary traffic signal -- `network/sent_bytes_count` — egress bandwidth (cost driver) -- `network/received_bytes_count` — ingress bandwidth -- `anywhere_cache/request_count` — cache-layer traffic (if enabled) - -### What people page on -- Unexpected drop in write traffic (data pipeline failure) -- Sustained spike in read traffic beyond capacity (auto-scaling lag) -- Request rate approaching per-bucket scaling limits (precursor to throttling) - -### Section questions -1. **Is request volume within expected bounds?** → Traffic & Request Volume section -2. **Which operations dominate traffic?** → Traffic & Request Volume section (method breakdown) -3. **How much data is flowing in and out?** → Network & Bandwidth section - ---- - -## Errors - -### What it means for GCS -Failed API requests indicated by non-`OK` response codes. GCS errors fall into client errors (caller's fault: `PERMISSION_DENIED`, `NOT_FOUND`, `INVALID_ARGUMENT`) and server errors (GCS's fault: `INTERNAL`, `UNAVAILABLE`, `DEADLINE_EXCEEDED`). Throttling (`RESOURCE_EXHAUSTED`) sits between — it's GCS enforcing limits but the caller can mitigate. - -### Typical causes when it degrades -- IAM policy changes causing `PERMISSION_DENIED` wave (propagation delay ~1 min) -- Bucket rate limit exceeded (`RESOURCE_EXHAUSTED` / 429) -- GCS regional incident (`INTERNAL`, `UNAVAILABLE`) -- Application bugs accessing non-existent objects (`NOT_FOUND`) -- Token expiration or rotation (`UNAUTHENTICATED`) - -### Best telemetry sources -- `api/request_count` filtered by `response_code != OK` — primary error signal -- `api/request_count` where `response_code` in (`INTERNAL`, `UNAVAILABLE`, `DEADLINE_EXCEEDED`) — server-side errors -- `api/request_count` where `response_code = RESOURCE_EXHAUSTED` — throttling -- `api/request_count` where `response_code = PERMISSION_DENIED` — access control issues - -### What people page on -- Server-error rate (5xx equivalent) rising above baseline for production buckets -- Throttling rate sustaining beyond transient burst -- Sudden `PERMISSION_DENIED` spike affecting production service accounts - -### Section questions -1. **Are API requests succeeding?** → Errors & Reliability section -2. **Is GCS throttling our requests?** → Errors & Reliability section (RESOURCE_EXHAUSTED breakdown) -3. **Are there access control problems?** → Errors & Reliability section (PERMISSION_DENIED trend) - ---- - -## Latency - -### What it means for GCS -Time to complete API operations. **Critical caveat: GCS provides no server-side latency metric.** Latency can only be measured client-side — via gRPC client metrics (`client/grpc/client/attempt/duration`) or application-instrumented spans. Replication delay (`turbo_max_delay`) is the closest server-side "latency" signal, measuring how long until writes propagate across regions. - -### Typical causes when it degrades -- GCS auto-scaling lag (bucket newly receiving high traffic) -- Sequential key naming causing hotspots on a single backend -- Geographic distance between client and bucket region -- Large object sizes increasing transfer time -- Client-side resource constraints (CPU, memory, network) - -### Best telemetry sources -- `client/grpc/client/attempt/duration` — per-attempt latency (gRPC clients only) -- `client/grpc/client/call/duration` — end-to-end call latency including retries (gRPC clients only) -- `replication/turbo_max_delay` — max age of unsynced objects (dual-region turbo only) -- Application-instrumented spans (not available in GCS server-side metrics) - -### What people page on -- Client-observed p99 latency exceeding SLO for critical paths -- Replication delay exceeding 15-minute RPO on turbo-enabled buckets -- Sustained latency degradation correlated with `RESOURCE_EXHAUSTED` responses - -### Section questions -1. **Is replication keeping up?** → Replication & Durability section -2. **Are there signs of backend throttling affecting perceived latency?** → Errors & Reliability section (RESOURCE_EXHAUSTED as proxy) - ---- - -## Saturation - -### What it means for GCS -Resource utilization approaching hard limits. For a managed service like GCS, saturation manifests as bandwidth quota usage, per-bucket request rate limits, storage capacity growth, and cache pressure (if Anywhere Cache is enabled). - -### Typical causes when it degrades -- Bandwidth quota approaching project limit (200 Gbps default) -- Per-bucket request rate exceeding auto-scale capacity -- Storage growing faster than lifecycle rules can clean up (versioning, soft-delete accumulation) -- Anywhere Cache eviction rate rising (cache undersized for working set) -- Object count growing into millions (slows listing operations) - -### Best telemetry sources -- `quota/*/usage` vs `quota/*/limit` — bandwidth quota saturation -- `quota/*/exceeded` — any non-zero value = quota breach -- `api/request_count` where `response_code = RESOURCE_EXHAUSTED` — rate limit saturation -- `storage/total_bytes` by `storage_class` — capacity trends (daily granularity) -- `storage/object_count` — object count trends (daily granularity) -- `anywhere_cache_metering/eviction_byte_count` — cache pressure (if enabled) - -### What people page on -- Bandwidth quota usage >80% of limit (leading indicator) -- `RESOURCE_EXHAUSTED` error count sustaining beyond transient burst -- Storage cost spike from unexpected `total_bytes` growth -- Object count crossing millions (listing performance degradation) - -### Section questions -1. **Is storage growing as expected?** → Storage Capacity & Growth section -2. **Are we approaching bandwidth or request rate limits?** → Network & Bandwidth section -3. **Is the cache keeping up with demand?** → Network & Bandwidth section (if Anywhere Cache enabled) - ---- - -### Confirmed by sources -- GCS does not expose a server-side latency metric ([Data storage SLI metrics](https://docs.cloud.google.com/stackdriver/docs/solutions/slo-monitoring/sli-metrics/data-storage-metrics)). -- gRPC client-side metrics (`client/grpc/client/attempt/duration`, `client/grpc/client/call/duration`) are the only latency signals ([gRPC client-side metrics](https://docs.cloud.google.com/storage/docs/client-side-metrics)). -- Default bandwidth quota is 200 Gbps per project per region ([Bandwidth usage overview](https://docs.cloud.google.com/storage/docs/bandwidth-usage)). -- Per-bucket baseline: ~5,000 reads/s, ~1,000 writes/s; auto-scales over minutes ([Request rate guidelines](https://docs.cloud.google.com/storage/docs/request-rate)). -- `RESOURCE_EXHAUSTED` response code indicates rate limit or quota exceeded ([Cloud Storage error responses](https://cloud.google.com/storage/docs/json_api/v1/status-codes)). -- Replication metrics (`meeting_rpo`, `turbo_max_delay`) only exist for dual/multi-region buckets with turbo replication ([Managing turbo replication](https://docs.cloud.google.com/storage/docs/managing-turbo-replication)). - -### Best-practice inference -- Using bandwidth quota at >80% as a leading indicator is operational best practice, not a documented GCS threshold. -- Listing performance degradation at millions of objects is widely reported but not formally documented with a specific threshold. -- Cache eviction rate as a saturation signal is inferred from general caching principles applied to Anywhere Cache. - - ---- - -# Google Cloud Storage — Section Notes & Playbooks - ---- - -## Part 1: Overview Mission Note - -**Google Cloud Storage** -Fully managed object storage for blobs, backups, logs, and data lake objects across GCP regions. - -**Scope:** All GCS buckets reporting via the GCP Cloud Storage integration. Covers API traffic, errors, network bandwidth, storage capacity, replication health, and access patterns. Does not cover client-side latency (requires gRPC client instrumentation) or Anywhere Cache. - -**Links:** -- [Cloud Storage monitoring overview](https://docs.cloud.google.com/storage/docs/monitoring) -- [Request rate and access guidelines](https://docs.cloud.google.com/storage/docs/request-rate) -- [GCS Deep Dive dashboard](#) *(link to deep dive)* - ---- - -## Part 2: Section Explanation Notes - -### # Are GCS request rates and traffic patterns within expected bounds? - -#### So what? -Healthy GCS traffic shows a **stable request rate** with a predictable read/write mix matching your application pattern. A sudden spike in `ReadObject` may mean a CDN cache purge is sending reads back to origin. A spike in `ListObjects` may indicate a misconfigured batch job enumerating large buckets. **Watch out:** client libraries automatically retry 429 and 5xx errors — the server-side `Total Request Rate` may be higher than your application intends due to **retry amplification**. - -#### Now what? -Check **Request Rate by Method** to identify which operation type is driving the change → check **Top Buckets by Request Rate** to isolate the bucket → correlate with recent deployments or upstream traffic shifts. - ---- - -### # Are API requests succeeding and is GCS throttling us? - -#### So what? -GCS errors use gRPC status codes, not HTTP codes. **Server errors** (`INTERNAL`, `UNAVAILABLE`, `DEADLINE_EXCEEDED`) indicate GCS-side issues and are the only errors that count against the GCS SLA. **Client errors** (`PERMISSION_DENIED`, `NOT_FOUND`) are caller-side issues. **Throttling** (`RESOURCE_EXHAUSTED`) means the bucket is hitting per-bucket rate limits (~5K reads/s, ~1K writes/s baseline) or project bandwidth quotas. **Watch out:** a brief `PERMISSION_DENIED` spike after IAM policy changes is normal — IAM grants are eventually consistent (~1 min propagation). - -#### Now what? -Check **Error Rate (%)** to assess severity → break down via **Errors by Response Code** to classify server vs client vs throttling → check **Top Buckets by Error Rate** to isolate the affected bucket → if throttling, reduce request rate or ramp up gradually (double every 20 min). - ---- - -### # How much data is flowing in and out, and are we approaching bandwidth limits? - -#### So what? -Egress bandwidth is the primary **cost driver** for GCS — ingress is free. Sustained high egress may approach the project bandwidth quota (200 Gbps default per region). **Watch out:** bandwidth metrics are in **bytes/s** but quota limits are in **bits/s** — multiply by 8 when comparing. A sudden egress spike on a bucket that normally has low traffic may indicate data exfiltration or a misconfigured data pipeline. - -#### Now what? -Check **Egress Bandwidth (MB/s)** for overall rate → check **Top Buckets by Egress** to find the source → check **Egress by Method** to understand if it's `ReadObject` (normal reads) or `CopyObject`/`RewriteObject` (cross-bucket transfer) → compare with quota limits if approaching saturation. - ---- - -### # How much data is stored and is storage growing as expected? - -#### So what? -Storage metrics update **once per 24 hours** — don't panic over sudden step-changes, they're just the daily refresh catching up. Healthy storage shows linear growth matching your data retention policy. **Super-linear growth** usually means: (1) object versioning accumulating noncurrent versions, (2) lifecycle rules not firing (misconfigured conditions), (3) soft-delete retention preserving "deleted" objects for up to 90 days, or (4) abandoned multipart uploads. **Watch out:** `total_bytes` (v1) excludes soft-deleted objects — actual billed storage may be higher. - -#### Now what? -Check **Storage by Class** to see tier distribution → check **Top Buckets by Storage** to find the largest consumers → check **Object Count** trends for unexpected growth → check **Deleted Bytes per Day** to verify lifecycle/cleanup is working → if growth is unexpected, verify lifecycle rules and object versioning settings. - ---- - -### # Is cross-region replication healthy and meeting RPO targets? - -#### So what? -Replication metrics **only exist for dual/multi-region buckets with turbo replication**. No data = feature not enabled, not a failure. `meeting_rpo = 1` means all objects are replicating within the 15-minute RPO target. `turbo_max_delay` shows the worst-case: the oldest unsynced object's age in seconds (threshold: 900s = 15 min). **Watch out:** replication metrics have a **multi-hour reporting delay** — an RPO violation happening right now may not appear for hours. Use `api/request_count` error patterns as a faster leading indicator of regional issues. - -#### Now what? -Check **Meeting RPO** status → if 0, check **Max Replication Delay** to see how far behind → check **Missed RPO Minutes (last 30d)** for historical pattern → if delay is rising, check for high write throughput or large object uploads that may be overwhelming replication capacity. - ---- - -### # Are authentication patterns normal and is ACL migration progressing? - -#### So what? -Authentication metrics show who is accessing GCS and how. The goal for ACL metrics is **zero** — any non-zero `ACL Operations Rate` means legacy ACL-based access is still in use, blocking migration to uniform bucket-level access (the recommended state). **Watch out:** `access_id` is a high-cardinality label (one per service account). Never group by `access_id` without filtering to a specific bucket first. Use `authn_method` for broad auth pattern analysis. - -#### Now what? -Check **Auth Requests by Method** for expected auth patterns → check **ACL Operations Rate** — if non-zero, identify which buckets still use ACLs → check **ACL-Based Object Access** — if non-zero, those objects are being accessed via ACL grants and need IAM migration before enabling uniform access. - ---- - -### # Are objects in the right storage classes and is Autoclass optimizing costs? - -#### So what? -Autoclass metrics only appear for buckets with Autoclass enabled. **High transition counts right after enabling Autoclass are normal** — it's reclassifying existing objects based on historical access. Counts should stabilize within days. Sustained high transitions after the initial period may indicate **thrashing** — objects being accessed just enough to trigger upward transitions, then cooling off and transitioning back down. **Watch out:** small objects (<1 MB) cost more to transition (Class A operation fees) than they save in storage, so high transition counts on small-object buckets may actually increase costs. - -#### Now what? -Check **Autoclass Transitions Over Time** for stabilization pattern → check **Autoclass Transitioned Bytes** to see if transitions involve large or small objects → cross-reference with **Storage by Class** in the Capacity section to verify objects are landing in expected tiers. - ---- - -## Part 3: Cause-Effect Triage Chains (>= 20 chains) - -1. If **Error Rate (%)** rises above 1% → check **Errors by Response Code** to classify error type → if mostly `INTERNAL`/`UNAVAILABLE`, likely GCS regional incident → check GCS status page. (Confirmed) - -2. If **Server Error Rate (%)** spikes → check **Errors by Response Code** for `INTERNAL` vs `UNAVAILABLE` → check **Top Buckets by Error Rate** to see if it's one bucket or all → likely regional GCS issue → check GCP status dashboard. (Confirmed) - -3. If **Throttle Rate (%)** rises above 0 → check **Top Buckets by Request Rate** for the hottest bucket → check **Request Rate by Method** for the dominant operation → likely per-bucket rate limit hit → reduce request rate or ramp gradually. (Confirmed) - -4. If **Total Request Rate** drops suddenly → check **Error Rate (%)** for correlated error spike → check **Errors by Response Code** for `UNAVAILABLE` → if errors are zero, upstream application stopped making requests → check application health. (Mixed) - -5. If **Egress Bandwidth (MB/s)** spikes → check **Top Buckets by Egress** to identify source → check **Egress by Method** for operation type → if `ReadObject` spike, likely CDN cache purge or batch job → verify with application team. (Inference) - -6. If **Egress Bandwidth (MB/s)** approaches project quota → check quota metrics if available → reduce read-heavy workloads or request quota increase → implement client-side caching. (Inference) - -7. If **Ingress Bandwidth (MB/s)** drops to zero → check **Total Request Rate** for `WriteObject`/`InsertObject` methods → likely upstream data pipeline failure → check ingestion application health. (Mixed) - -8. If **Total Storage (GiB)** grows faster than expected → check **Storage by Class** for which tier is growing → check **Top Buckets by Storage** for the growing bucket → check **Object Count** trends → likely versioning accumulation or lifecycle rules not firing. (Mixed) - -9. If **Total Objects (count)** exceeds millions → listing operations will degrade → check **Request Rate by Method** for `ListObjects` latency symptoms (high count) → consider restructuring object naming or using prefix filters. (Inference) - -10. If **Deleted Bytes per Day** is zero on buckets with lifecycle rules → lifecycle rules may be misconfigured → check rule conditions (age, creation date, storage class) match actual objects. (Inference) - -11. If **Meeting RPO** drops to 0 → check **Max Replication Delay (s)** for current lag → check **Missed RPO Minutes (last 30d)** for historical context → if delay > 900s, turbo replication is actively failing → check for high write throughput. (Confirmed) - -12. If **Max Replication Delay (s)** exceeds 900 → RPO is being violated → check **Total Request Rate** for write volume spike → check for large object uploads → if sustained, may need to reduce write rate to allow replication to catch up. (Confirmed) - -13. If **Errors by Response Code** shows spike in `PERMISSION_DENIED` → check if IAM policy recently changed → if yes, wait ~1 minute for propagation → if persists, check IAM bindings for affected service accounts. (Confirmed) - -14. If **Errors by Response Code** shows spike in `NOT_FOUND` → check recent deployments for changed object key patterns → check if a cache layer was flushed → usually a client-side bug, not a GCS issue. (Inference) - -15. If **Auth Requests by Method** shows unexpected authentication method → security concern → check for unauthorized HMAC key usage or unexpected OAuth clients → review IAM audit logs. (Inference) - -16. If **ACL Operations Rate** is non-zero → legacy ACL access is active → check **ACL-Based Object Access** for which buckets → plan migration to uniform bucket-level access. (Confirmed) - -17. If **ACL-Based Object Access** is non-zero → objects are being accessed via ACL grants → identify dependent applications → update them to use IAM-based access → then enable uniform bucket-level access. (Confirmed) - -18. If **Autoclass Transitions** remain high after initial stabilization → possible thrashing → check **Autoclass Transitioned Bytes** for object sizes → if small objects, Autoclass may not be cost-effective → consider manual storage class assignment. (Inference) - -19. If **Request Rate by Method** shows `ListObjects` dominating → possible inefficient enumeration pattern → check application code for recursive listing → consider using prefix/delimiter parameters or hierarchical namespace buckets for better performance. (Inference) - -20. If **Errors by Response Code** shows `RESOURCE_EXHAUSTED` AND **Egress Bandwidth** is near quota → project bandwidth quota hit → request quota increase via Cloud Console → implement backoff for 429 errors → consider CDN for read-heavy workloads. (Confirmed) - -21. If **Total Storage (GiB)** grows but **Deleted Bytes per Day** is also high → storage is churning (high write + delete rate) → check **Storage by Class** for class distribution → verify this is expected behavior (log rotation, temp file patterns). (Inference) - -22. If **Net Bandwidth (MB/s)** shows asymmetric pattern (high egress, low ingress) → read-heavy workload → candidate for Anywhere Cache or CDN to reduce origin egress and costs. (Inference) - ---- - -## Part 4: Operational Playbooks (6-10 playbooks) - -### Playbook 1: Elevated Server Error Rate -**Trigger:** **Server Error Rate (%)** > 0.1% sustained for 5+ minutes -**Decision rule:** If errors are bucket-specific, it's likely a hotspot or misconfig. If errors are project-wide, it's likely a GCS regional incident. -**Steps:** -1. Check **Server Error Rate (%)** to confirm the spike is sustained, not a single data point -2. Check **Errors by Response Code** to identify the specific error code (`INTERNAL` vs `UNAVAILABLE` vs `DEADLINE_EXCEEDED`) -3. Check **Top Buckets by Error Rate** to determine if errors are isolated to one bucket or widespread -4. Check **Request Rate by Method** to see if a specific operation type correlates with errors -5. If bucket-specific: check **Top Buckets by Request Rate** to see if the bucket is being overloaded -6. If project-wide: check [GCP Status Dashboard](https://status.cloud.google.com/) for known incidents -**Likely causes:** -- GCS regional incident or degradation -- Bucket rate limit exceeded (triggering `UNAVAILABLE` under load) -- Backend rebalancing after sudden traffic increase -**Next actions:** -- If GCS incident: wait for resolution, implement client-side retries with exponential backoff -- If overloaded bucket: reduce request rate, distribute across multiple buckets -- If new traffic pattern: ramp up gradually (double every 20 minutes) -**Label:** Mixed - -### Playbook 2: Request Throttling (RESOURCE_EXHAUSTED) -**Trigger:** **Throttle Rate (%)** > 0 sustained for 2+ minutes -**Decision rule:** If per-bucket rate limit, reduce per-bucket request rate. If bandwidth quota, request quota increase. -**Steps:** -1. Check **Throttle Rate (%)** to quantify the impact -2. Check **Top Buckets by Request Rate** to identify the throttled bucket -3. Check **Request Rate by Method** to identify the dominant operation (reads vs writes) -4. Check **Egress Bandwidth (MB/s)** to see if bandwidth quota is also a factor -5. Check **Errors by Response Code** to confirm `RESOURCE_EXHAUSTED` is the error code -**Likely causes:** -- Per-bucket request rate exceeded (~5K reads/s, ~1K writes/s baseline) -- Project bandwidth quota exceeded (200 Gbps default) -- Sudden traffic ramp without gradual warm-up -- Sequential key naming causing backend hotspot -**Next actions:** -- Implement exponential backoff for 429 responses -- Ramp up traffic gradually (double every 20 minutes) -- Randomize object name prefixes to avoid hotspots -- Request bandwidth quota increase via Cloud Console if needed -**Label:** Confirmed - -### Playbook 3: Unexpected Storage Growth -**Trigger:** **Total Storage (GiB)** daily growth exceeds expected baseline by 2x+ -**Decision rule:** If growth is from noncurrent versions, review versioning policy. If from a specific bucket, check lifecycle rules. -**Steps:** -1. Check **Total Storage (GiB)** to confirm the growth magnitude -2. Check **Storage by Class** to identify which storage class is growing -3. Check **Top Buckets by Storage** to isolate the growing bucket -4. Check **Top Buckets by Object Count** to see if object count is also growing disproportionately -5. Check **Deleted Bytes per Day** to verify lifecycle cleanup is working -6. If soft-delete suspected: check v2/total_bytes with type label (not in default dashboard — may need ad-hoc query) -**Likely causes:** -- Object versioning accumulating noncurrent versions without lifecycle cleanup -- Soft-delete retention (default 90 days) preserving "deleted" objects -- Abandoned multipart uploads consuming storage -- Lifecycle rules misconfigured (wrong age/class conditions) -- Unexpected data pipeline producing more data than planned -**Next actions:** -- Verify lifecycle rules are correctly configured on the growing bucket -- Set `AbortIncompleteMultipartUpload` lifecycle rule if missing -- Reduce soft-delete retention period if 90 days is excessive -- Add noncurrent version deletion lifecycle rule -**Label:** Mixed - -### Playbook 4: Replication RPO Violation -**Trigger:** **Meeting RPO** = 0 AND **Max Replication Delay (s)** > 900 -**Decision rule:** If delay is growing, replication is falling further behind. If delay is stable near threshold, it may self-recover. -**Steps:** -1. Check **Meeting RPO** to confirm violation -2. Check **Max Replication Delay (s)** for current lag value -3. Check **Replication Delay Over Time** for trend (growing vs stabilizing) -4. Check **Missed RPO Minutes (last 30d)** for historical frequency -5. Check **Total Request Rate** and **Ingress Bandwidth** to see if high write volume is overwhelming replication -**Likely causes:** -- Sustained high write throughput exceeding replication bandwidth -- Very large object uploads taking longer to replicate -- GCS cross-region replication infrastructure degradation -**Next actions:** -- If write-driven: reduce write rate temporarily to allow replication to catch up -- If large objects: consider smaller chunk sizes for write-heavy workflows -- If GCS-side: check GCP status page for replication service issues -- Review RPO SLA requirements — brief violations may be acceptable -**Label:** Confirmed - -### Playbook 5: Permission Denied Wave -**Trigger:** **Errors by Response Code** shows spike in `PERMISSION_DENIED` -**Decision rule:** If correlated with a recent IAM change, wait for propagation. If persistent, investigate IAM bindings. -**Steps:** -1. Check **Errors by Response Code** to confirm `PERMISSION_DENIED` is the dominant error -2. Check **Top Buckets by Error Rate** to identify affected buckets -3. Check **Errors by Method** to see which operations are failing -4. Check Cloud Audit Logs for recent IAM policy changes (not in this dashboard — requires log viewer) -5. Check **Auth Requests by Method** for unexpected authentication patterns -**Likely causes:** -- Recent IAM policy change with ~1 minute propagation delay -- Service account key rotation leaving a gap -- Uniform bucket-level access migration breaking ACL-dependent workflows -- Cross-project access misconfiguration -**Next actions:** -- If recent IAM change: wait 1-2 minutes for propagation, then recheck -- If persistent: verify IAM bindings on affected bucket using `gcloud storage buckets get-iam-policy` -- If service account issue: check key validity and rotation status -- Check if uniform bucket-level access was recently enabled on a bucket with ACL-dependent clients -**Label:** Mixed - -### Playbook 6: Bandwidth Quota Approaching Limit -**Trigger:** **Egress Bandwidth (MB/s)** sustained high, approaching known project quota -**Decision rule:** If a single bucket drives most egress, optimize that bucket. If distributed, request quota increase. -**Steps:** -1. Check **Egress Bandwidth (MB/s)** to quantify current rate -2. Check **Top Buckets by Egress** to identify the top egress sources -3. Check **Egress by Method** to understand the operation pattern -4. Check **Request Rate by Method** for `ReadObject` volume -5. Check for `RESOURCE_EXHAUSTED` in **Errors by Response Code** (appears when quota is actually hit) -**Likely causes:** -- High-traffic read workload (analytics, data export, media serving) -- CDN cache purge driving reads back to GCS origin -- Data migration or cross-region copy job -- Batch processing reading large datasets -**Next actions:** -- Request bandwidth quota increase via Cloud Console (up to 1 Tbps self-service) -- Implement Cloud CDN or Media CDN for cacheable content -- Enable Anywhere Cache for zone-local hot reads -- Stagger batch workloads across time to reduce peak bandwidth -**Label:** Mixed - -### Playbook 7: ACL Migration Assessment -**Trigger:** **ACL Operations Rate** or **ACL-Based Object Access** is non-zero -**Decision rule:** If ACL activity is declining, migration is progressing. If stable or growing, legacy workflows are still active. -**Steps:** -1. Check **ACL Operations Rate** to see current ACL activity level -2. Check **ACL-Based Object Access** to see if objects are being accessed via ACLs -3. Check **Auth Requests by Method** to understand the auth landscape -4. Identify buckets still using ACLs (not in this dashboard — requires per-bucket check) -5. Plan IAM-based access replacement for each ACL-dependent workflow -**Likely causes:** -- Legacy applications still using ACL-based access patterns -- Third-party integrations that set object ACLs -- Workflows that haven't been migrated to IAM after initial setup -**Next actions:** -- Audit which buckets have ACL activity using `authz/acl_operations_count` grouped by bucket -- For each bucket, identify clients using ACLs -- Migrate clients to IAM-based access -- Enable uniform bucket-level access on migrated buckets (irreversible after 90 days) -**Label:** Confirmed - - ---- - -# Google Cloud Storage — Caveats & Footguns - -## High-cardinality dimensions to avoid - -- **[traffic-request-volume, errors-reliability]** `access_id` on `authn/authentication_count` is unbounded (one per service account/HMAC key). Never group-by without filtering to a specific bucket first. Use `authn_method` instead for auth pattern analysis. (Confirmed: [Uniform bucket-level access docs](https://docs.cloud.google.com/storage/docs/using-uniform-bucket-level-access)) - -- **[traffic-request-volume, storage-capacity-growth]** `bucket_name` can be high-cardinality in large orgs with hundreds of buckets. Always use top-N limits (25 max) when grouping by bucket. Pre-filter by project/region in global filters. (Inference) - -## Misleading metrics and wrong aggregations - -- **[storage-capacity-growth]** `storage/total_bytes` and `storage/object_count` update **once per 24 hours**. Do NOT use `average` aggregation over short windows — the value will appear flat with step-changes at the daily refresh. Use `max` aggregation and view at 1d+ granularity. (Confirmed: [Cloud Storage monitoring](https://docs.cloud.google.com/storage/docs/monitoring)) - -- **[storage-capacity-growth]** `storage/total_bytes` (v1) excludes soft-deleted objects. If soft-delete retention is enabled (default: 90 days), actual billed storage is higher. Use `storage/v2/total_bytes` with `type` label to see the full picture. (Confirmed: [Cloud Storage troubleshooting](https://docs.cloud.google.com/storage/docs/troubleshooting)) - -- **[errors-reliability]** `NOT_FOUND` responses are client errors, not GCS failures. High `NOT_FOUND` rates usually mean application bugs (wrong object keys) or race conditions, not storage reliability issues. Do not include them in "server error rate" calculations. (Inference) - -- **[errors-reliability]** `PERMISSION_DENIED` after an IAM policy change may persist for ~1 minute due to eventual consistency of IAM grants. A brief spike after policy changes is normal and not an outage. (Confirmed: [Cloud Storage consistency](https://docs.cloud.google.com/storage/docs/consistency)) - -- **[traffic-request-volume]** `api/request_count` counts server-side requests. Client libraries automatically retry 408, 429, and 5xx errors with exponential backoff. The client sees fewer errors than `api/request_count` shows because retries are transparent. Conversely, total request count may be higher than the application intends due to retry amplification. (Confirmed: [Cloud Storage troubleshooting](https://docs.cloud.google.com/storage/docs/troubleshooting)) - -## Unit pitfalls - -- **[network-bandwidth]** `network/sent_bytes_count` and `network/received_bytes_count` are in **bytes**, not bits. Bandwidth quotas (`quota/*`) are in **bits/s**. When comparing egress rate to quota limits, convert: bytes/s * 8 = bits/s. (Confirmed: [Bandwidth usage overview](https://docs.cloud.google.com/storage/docs/bandwidth-usage)) - -- **[storage-capacity-growth]** `storage/total_byte_seconds` is in `By.s` (byte-seconds) — a billing unit, not a storage size. Do NOT display this as "bytes" or use it for capacity dashboards. Use `total_bytes` for capacity. (Confirmed: [Google Cloud metrics reference](https://docs.cloud.google.com/monitoring/api/metrics_gcp_p_z)) - -- **[replication-durability]** `replication/turbo_max_delay` is in seconds, but the RPO target is 15 minutes (900 seconds). Display the threshold line at 900s when building timeseries widgets to make RPO violations visually obvious. (Confirmed: [Managing turbo replication](https://docs.cloud.google.com/storage/docs/managing-turbo-replication)) - -## Sampling/temporality pitfalls - -- **[traffic-request-volume, errors-reliability, network-bandwidth]** `api/request_count`, `network/sent_bytes_count`, and `network/received_bytes_count` are **DELTA** counters (GCP Cloud Monitoring type: `DELTA`). In Tsuga, use `per-second` (not `rate`) for delta counters. If the Tsuga integration reports them as cumulative, use `rate` instead. **Stage 2 discovery must confirm temporality.** (Confirmed: [Google Cloud metrics reference](https://docs.cloud.google.com/monitoring/api/metrics_gcp_p_z)) - -- **[storage-capacity-growth]** Storage metrics are **daily gauges**. Setting a dashboard time window shorter than 24 hours may show empty data or a single flat point. Use 7d+ windows for storage trends. (Confirmed: [Cloud Storage monitoring](https://docs.cloud.google.com/storage/docs/monitoring)) - -- **[replication-durability]** Replication metrics have a **multi-hour reporting delay**. An RPO violation happening now may not show in metrics for hours. Do not use these for real-time incident detection — use `api/request_count` error patterns as a leading indicator instead. (Confirmed: [Managing turbo replication](https://docs.cloud.google.com/storage/docs/managing-turbo-replication)) - -## "This looks bad but isn't" - -- **[errors-reliability]** A spike in `NOT_FOUND` responses after a deployment is normal if the new code references objects that haven't been created yet, or if a cache layer was flushed. Check if the `NOT_FOUND` rate returns to baseline within minutes. (Inference) - -- **[replication-durability]** `meeting_rpo = 0` with `turbo_max_delay < 900s` can happen briefly during metric calculation. The metrics update at different intervals. Only escalate if `turbo_max_delay` exceeds 900s for sustained periods. (Inference) - -- **[cost-storage-class]** High Autoclass transition counts immediately after enabling Autoclass are expected — it's reclassifying existing objects based on historical access patterns. Counts should stabilize within days. (Confirmed: [Autoclass docs](https://docs.google.com/storage/docs/autoclass)) - -- **[storage-capacity-growth]** `object_count` appearing to jump suddenly is usually the daily refresh catching up with a day's worth of writes, not a sudden burst. Correlate with `api/request_count` method=InsertObject to confirm actual write rate. (Inference) - -## Optional-feature traps (metrics absent unless X enabled) - -- **[replication-durability]** Replication metrics (`meeting_rpo`, `turbo_max_delay`, `missing_rpo_minutes_last_30d`) are **only present** for dual/multi-region buckets with turbo replication. For single-region buckets, these metrics will never appear. Dashboard must show a gating note, not an error state. (Confirmed: [Managing turbo replication](https://docs.cloud.google.com/storage/docs/managing-turbo-replication)) - -- **[cost-storage-class]** Autoclass metrics (`transition_operation_count`, `transitioned_bytes_count`) are **only present** when Autoclass is enabled on a bucket. Absence = feature off, not a problem. (Confirmed: [Autoclass docs](https://docs.cloud.google.com/storage/docs/autoclass)) - -- **[access-security]** ACL metrics (`authz/*`) are **only present** for buckets that have NOT enabled uniform bucket-level access. If all buckets use uniform access (the recommended state), these metrics will be absent. Absence is the goal. (Confirmed: [Uniform bucket-level access](https://docs.cloud.google.com/storage/docs/using-uniform-bucket-level-access)) - -- **[access-security]** `authn/authentication_count` requires the GCP integration to collect `authn/*` metrics. Not all integration configurations include this metric family by default — it may need explicit opt-in. (Inference) - -- **[traffic-request-volume, errors-reliability]** The `response_code` label uses gRPC status codes (`OK`, `PERMISSION_DENIED`, `RESOURCE_EXHAUSTED`), NOT HTTP status codes (200, 403, 429). Do not build filters using HTTP numeric codes. (Confirmed: [Google Cloud metrics reference](https://docs.cloud.google.com/monitoring/api/metrics_gcp_p_z)) - - ---- - diff --git a/plugins/tsuga/skills/knowledge-technology/references/gcp-storage/metrics.csv b/plugins/tsuga/skills/knowledge-technology/references/gcp-storage/metrics.csv deleted file mode 100644 index 07647ec..0000000 --- a/plugins/tsuga/skills/knowledge-technology/references/gcp-storage/metrics.csv +++ /dev/null @@ -1,22 +0,0 @@ -theme,name,confirmed,definition,unit,type,tsuga_metric_name,widget_suggestions,aggregation,post_function,group_by,required_context_fields,optional_context_fields,absence_means,cardinality_footguns,variants,citations -Throughput/Usage,storage.googleapis.com/api/request_count,true,Number of API calls per bucket. Primary traffic signal for all GCS operations. Labeled by method and response_code.,1,counter,storage.googleapis.com/api/request_count,timeseries|query-value|top-list,sum,per-second,context.bucket_name:25|context.method:15|context.response_code:10,context.cloud.account.id|context.cloud.region|context.env|context.team,context.method|context.response_code|context.location|context.project_id,Bucket has no API traffic in the window,Safe — method (~20) and response_code (~15) are bounded,gcp=storage.googleapis.com/api/request_count,https://docs.cloud.google.com/monitoring/api/metrics_gcp_p_z -Throughput/Usage,storage.googleapis.com/network/sent_bytes_count,true,Bytes sent (egress) over the network per bucket. Primary bandwidth and cost signal. Labeled by method and response_code.,By,counter,storage.googleapis.com/network/sent_bytes_count,timeseries|query-value,sum,per-second,context.bucket_name:25|context.method:15,context.cloud.account.id|context.cloud.region|context.env|context.team,context.response_code|context.location,No egress traffic from this bucket,Safe,gcp=storage.googleapis.com/network/sent_bytes_count,https://docs.cloud.google.com/monitoring/api/metrics_gcp_p_z -Throughput/Usage,storage.googleapis.com/network/received_bytes_count,true,Bytes received (ingress) over the network per bucket. Measures upload/write bandwidth.,By,counter,storage.googleapis.com/network/received_bytes_count,timeseries|query-value,sum,per-second,context.bucket_name:25|context.method:15,context.cloud.account.id|context.cloud.region|context.env|context.team,context.response_code|context.location,No ingress traffic to this bucket,Safe,gcp=storage.googleapis.com/network/received_bytes_count,https://docs.cloud.google.com/monitoring/api/metrics_gcp_p_z -Capacity/Saturation,storage.googleapis.com/storage/total_bytes,true,"Total size of all objects in the bucket grouped by storage class. Updated once per 24 hours. Primary capacity metric.",By,gauge,storage.googleapis.com/storage/total_bytes,timeseries|query-value|top-list,max,none,context.bucket_name:25|context.storage_class:5,context.cloud.account.id|context.cloud.region|context.env|context.team,context.location,Bucket is empty or metric not yet populated (first 24h),Safe,gcp=storage.googleapis.com/storage/total_bytes,https://docs.cloud.google.com/monitoring/api/metrics_gcp_p_z -Capacity/Saturation,storage.googleapis.com/storage/object_count,true,"Number of objects per bucket by storage class. Updated once per 24 hours.",1,gauge,storage.googleapis.com/storage/object_count,timeseries|query-value|top-list,max,none,context.bucket_name:25|context.storage_class:5,context.cloud.account.id|context.cloud.region|context.env|context.team,context.location,Bucket is empty,Safe,gcp=storage.googleapis.com/storage/object_count,https://docs.cloud.google.com/monitoring/api/metrics_gcp_p_z -Capacity/Saturation,storage.googleapis.com/storage/total_byte_seconds,true,"Daily storage consumed in byte-seconds. Used for billing calculations. Reported as gauge in Tsuga (GCP type: cumulative).",By.s,gauge,storage.googleapis.com/storage/total_byte_seconds,timeseries,max,none,context.bucket_name:25|context.storage_class:5,context.cloud.account.id|context.cloud.region|context.env|context.team,context.location,Bucket is empty,Safe,gcp=storage.googleapis.com/storage/total_byte_seconds,https://docs.cloud.google.com/monitoring/api/metrics_gcp_p_z -Errors/Failures,storage.googleapis.com/api/request_count,true,"Same metric as traffic but filtered for error response codes. Server errors: INTERNAL|UNAVAILABLE|DEADLINE_EXCEEDED. Client errors: PERMISSION_DENIED|NOT_FOUND|INVALID_ARGUMENT. Throttling: RESOURCE_EXHAUSTED.",1,counter,storage.googleapis.com/api/request_count,timeseries|query-value|top-list,sum,per-second,context.response_code:10|context.bucket_name:25|context.method:15,context.cloud.account.id|context.cloud.region|context.env|context.team,,No errors — healthy,Safe,gcp=storage.googleapis.com/api/request_count,https://docs.cloud.google.com/monitoring/api/metrics_gcp_p_z -Availability/Health,storage.googleapis.com/replication/meeting_rpo,false,"Whether the bucket is meeting its Recovery Point Objective. 1 = meeting 0 = not meeting. Only exists for dual/multi-region buckets with turbo replication. NOT FOUND in Tsuga — feature not enabled in this account.",1,gauge,storage.googleapis.com/replication/meeting_rpo,query-value|timeseries,min,none,,context.cloud.account.id|context.cloud.region,context.bucket_name,Feature not enabled or bucket is single-region — NOT an outage,Safe,gcp=storage.googleapis.com/replication/meeting_rpo,https://docs.cloud.google.com/storage/docs/managing-turbo-replication -Availability/Health,storage.googleapis.com/replication/turbo_max_delay,false,"Maximum age in seconds of an unsynced object. NOT FOUND in Tsuga — turbo replication not enabled in this account.",s,gauge,storage.googleapis.com/replication/turbo_max_delay,timeseries|query-value,max,none,,context.cloud.account.id|context.cloud.region,context.bucket_name,Turbo replication not enabled — NOT a problem,Safe,gcp=storage.googleapis.com/replication/turbo_max_delay,https://docs.cloud.google.com/storage/docs/managing-turbo-replication -Availability/Health,storage.googleapis.com/replication/missing_rpo_minutes_last_30d,false,"Number of minutes in last 30 days where RPO was missed. NOT FOUND in Tsuga — turbo replication not enabled.",1,gauge,storage.googleapis.com/replication/missing_rpo_minutes_last_30d,query-value,max,none,,context.cloud.account.id|context.cloud.region,context.bucket_name,Turbo replication not enabled,Safe,gcp=storage.googleapis.com/replication/missing_rpo_minutes_last_30d,https://docs.cloud.google.com/storage/docs/managing-turbo-replication -Throughput/Usage,storage.googleapis.com/authn/authentication_count,false,"Number of authenticated API requests. NOT FOUND in Tsuga — metric not collected in this account.",1,counter,storage.googleapis.com/authn/authentication_count,timeseries|top-list,sum,per-second,context.bucket_name:25,context.cloud.account.id|context.cloud.region,,Not collected by this GCP integration,Safe,gcp=storage.googleapis.com/authn/authentication_count,https://docs.cloud.google.com/storage/docs/using-uniform-bucket-level-access -Throughput/Usage,storage.googleapis.com/authz/acl_operations_count,false,"Number of ACL operations. NOT FOUND in Tsuga — metric not collected or uniform access enabled.",1,counter,storage.googleapis.com/authz/acl_operations_count,timeseries|query-value,sum,per-second,context.bucket_name:25,context.cloud.account.id|context.cloud.region,,Not collected or uniform bucket-level access enabled (desired state),Safe,gcp=storage.googleapis.com/authz/acl_operations_count,https://docs.cloud.google.com/storage/docs/using-uniform-bucket-level-access -Throughput/Usage,storage.googleapis.com/authz/acl_based_object_access_count,false,"Number of object accesses via ACL. NOT FOUND in Tsuga — not collected or uniform access enabled.",1,counter,storage.googleapis.com/authz/acl_based_object_access_count,timeseries|query-value,sum,per-second,context.bucket_name:25,context.cloud.account.id|context.cloud.region,,Not collected or uniform bucket-level access enabled (desired state),Safe,gcp=storage.googleapis.com/authz/acl_based_object_access_count,https://docs.cloud.google.com/storage/docs/using-uniform-bucket-level-access -Cost,storage.googleapis.com/autoclass/transition_operation_count,false,"Number of storage class transitions by Autoclass. NOT FOUND in Tsuga — Autoclass not enabled.",1,counter,storage.googleapis.com/autoclass/transition_operation_count,timeseries|query-value,sum,per-second,context.bucket_name:25,context.cloud.account.id|context.cloud.region,,Autoclass not enabled on this bucket — NOT an error,Safe,gcp=storage.googleapis.com/autoclass/transition_operation_count,https://docs.cloud.google.com/storage/docs/autoclass -Cost,storage.googleapis.com/autoclass/transitioned_bytes_count,false,"Total bytes transitioned by Autoclass. NOT FOUND in Tsuga — Autoclass not enabled.",By,counter,storage.googleapis.com/autoclass/transitioned_bytes_count,timeseries|query-value,sum,per-second,context.bucket_name:25,context.cloud.account.id|context.cloud.region,,Autoclass not enabled or no transitions needed,Safe,gcp=storage.googleapis.com/autoclass/transitioned_bytes_count,https://docs.cloud.google.com/storage/docs/autoclass -Capacity/Saturation,storage.googleapis.com/storage/v2/total_bytes,true,"Total size of all objects and multipart uploads. V2 adds type label to distinguish live vs soft-deleted. Daily update.",By,gauge,storage.googleapis.com/storage/v2/total_bytes,timeseries|query-value,max,none,context.bucket_name:25|context.storage_class:5|context.type:3,context.cloud.account.id|context.cloud.region|context.env|context.team,context.location,Bucket is empty,Safe,gcp=storage.googleapis.com/storage/v2/total_bytes,https://docs.cloud.google.com/monitoring/api/metrics_gcp_p_z -Capacity/Saturation,storage.googleapis.com/storage/v2/deleted_bytes,false,"Bytes deleted per day by storage class. NOT FOUND in Tsuga — may not be collected.",By,counter,storage.googleapis.com/storage/v2/deleted_bytes,timeseries|query-value,sum,per-second,context.bucket_name:25|context.storage_class:5,context.cloud.account.id|context.cloud.region,,No deletions in the period or metric not collected,Safe,gcp=storage.googleapis.com/storage/v2/deleted_bytes,https://docs.cloud.google.com/monitoring/api/metrics_gcp_p_z -Throughput/Usage,storage.googleapis.com/anywhere_cache/request_count,true,"Number of Anywhere Cache API calls by method and cache hit status. UNEXPECTED — found in Tsuga but was excluded from core scope.",1,counter,storage.googleapis.com/anywhere_cache/request_count,timeseries|query-value,sum,per-second,context.bucket_name:25|context.method:15|context.anywhere_cache_hit:2|context.anywhere_cache_zone:10,context.cloud.account.id|context.cloud.region|context.env|context.team,context.response_code,Anywhere Cache not configured for this bucket/zone,Safe,gcp=storage.googleapis.com/anywhere_cache/request_count,https://docs.cloud.google.com/storage/docs/anywhere-cache -Throughput/Usage,storage.googleapis.com/anywhere_cache/sent_bytes_count,true,"Bytes sent from Anywhere Cache by method and hit status.",By,counter,storage.googleapis.com/anywhere_cache/sent_bytes_count,timeseries|query-value,sum,per-second,context.bucket_name:25|context.method:15|context.anywhere_cache_hit:2|context.anywhere_cache_zone:10,context.cloud.account.id|context.cloud.region|context.env|context.team,context.response_code,Anywhere Cache not configured,Safe,gcp=storage.googleapis.com/anywhere_cache/sent_bytes_count,https://docs.cloud.google.com/storage/docs/anywhere-cache -Capacity/Saturation,storage.googleapis.com/storage/v2/total_byte_seconds,true,"V2 daily storage in byte-seconds with type label. Reported as gauge in Tsuga. UNEXPECTED — found in Tsuga.",By.s,gauge,storage.googleapis.com/storage/v2/total_byte_seconds,timeseries,max,none,context.bucket_name:25|context.storage_class:5|context.type:3,context.cloud.account.id|context.cloud.region|context.env|context.team,,Bucket is empty,Safe,gcp=storage.googleapis.com/storage/v2/total_byte_seconds,https://docs.cloud.google.com/monitoring/api/metrics_gcp_p_z -Capacity/Saturation,storage.googleapis.com/storage/v2/total_count,true,"V2 object count with type label (live vs soft-deleted). UNEXPECTED — found in Tsuga.",1,gauge,storage.googleapis.com/storage/v2/total_count,timeseries|query-value|top-list,max,none,context.bucket_name:25|context.storage_class:5|context.type:3,context.cloud.account.id|context.cloud.region|context.env|context.team,,Bucket is empty,Safe,gcp=storage.googleapis.com/storage/v2/total_count,https://docs.cloud.google.com/monitoring/api/metrics_gcp_p_z diff --git a/plugins/tsuga/skills/knowledge-technology/references/gcp-storage/queries.md b/plugins/tsuga/skills/knowledge-technology/references/gcp-storage/queries.md deleted file mode 100644 index 2761299..0000000 --- a/plugins/tsuga/skills/knowledge-technology/references/gcp-storage/queries.md +++ /dev/null @@ -1,60 +0,0 @@ -# Google Cloud Storage - -Object storage. Metrics cover API calls, throughput, stored volume, replication, access. Healthy: low error rate, bounded request latency, RPO met, no unauthorized access shifts. - -## Incident shapes - -- **API error spike** — `api/request_count{response_code=~"5xx"}` climbs -- **Rate limiting / 429** — `api/request_count{response_code="429"}` rises → hot-partition throttle -- **Egress cost runaway** — `network/sent_bytes_count` spikes → unusual download pattern -- **Replication RPO drift** — `replication/meeting_rpo < 100%` → regional replication lagging -- **Auth anomalies** — `authn/authentication_count` pattern shift → credential rotation / IAM change - -## Key metrics - -Prefixed `storage.googleapis.com/`: - -| Metric | Unit | Signal | -|---|---|---| -| `api/request_count` | count | API calls by method, response_code | -| `network/sent_bytes_count` | bytes | Egress | -| `network/received_bytes_count` | bytes | Ingress | -| `storage/total_bytes` | bytes | Bucket footprint | -| `storage/object_count` | count | Number of objects | -| `storage/total_byte_seconds` | byte-seconds | Storage billing unit | -| `replication/meeting_rpo` | % | Percent of objects meeting RPO | -| `replication/turbo_max_delay` | s | Turbo replication max delay | -| `replication/missing_rpo_minutes_last_30d` | minutes | Accumulated RPO-miss | -| `authn/authentication_count` | count | Auth attempts | -| `authz/acl_operations_count` | count | ACL operations | -| `authz/acl_based_object_access_count` | count | ACL-governed accesses | -| `autoclass/transition_operation_count` | count | Autoclass tier transitions | -| `autoclass/transitioned_bytes_count` | bytes | Data transitioned | -| `anywhere_cache/request_count` | count | Anywhere cache requests | - -## Derived signals - -- `request_count{5xx} / request_count` — error rate. Baseline < 0.01. -- `request_count{429} / request_count` — throttle rate. Any sustained positive = hot partition. -- Derivative of `network/sent_bytes_count` — egress trend. Spike = investigation target. -- Derivative of `storage/total_bytes` — growth rate. Rapid growth with stable usage = orphaned writes / versions. - -## Log patterns - -GCS Data Access audit logs (must be enabled): - -- `storage.googleapis.com/GetObject` / `PutObject` / `DeleteObject` — per-op events -- `protoPayload.status.code=7` — PERMISSION_DENIED -- `protoPayload.status.code=13` — INTERNAL -- `protoPayload.status.code=8` — RESOURCE_EXHAUSTED (rate limit / quota) -- `principalEmail` + `callerIp` — who and where -- ListObjects spike from a new principal = cost-runaway or enumeration signal - -## Gotchas - -- Metric collection lag 1-5 min; sub-minute events don't show. -- `api/request_count` excludes frontend-rejected requests (DDoS protection). Totals can undercount real traffic. -- Rate limiting is per-object / per-prefix. Hot key throttles one pattern while bucket looks fine. Audit logs show per-op detail. -- Object versioning isn't visible in `object_count`; versioned buckets can have many hidden historical versions driving storage cost. -- Autoclass can move objects to colder tiers; retrieval from Coldline/Archive is minutes, not seconds. Latency spike on a mature bucket = autoclass. -- Uniform bucket-level access and object ACLs interact unpredictably; `acl_based_object_access_count > 0` on uniform-access = misconfiguration. diff --git a/plugins/tsuga/skills/knowledge-technology/references/haproxy/README.md b/plugins/tsuga/skills/knowledge-technology/references/haproxy/README.md deleted file mode 100644 index 2b7cf90..0000000 --- a/plugins/tsuga/skills/knowledge-technology/references/haproxy/README.md +++ /dev/null @@ -1,214 +0,0 @@ -# HAProxy Integration Context Bundle - -## Metadata -**Technology:** HAProxy -**Deployment:** self-hosted -**Environment:** prod -**Persona:** SRE Dev and ops -**Telemetry preference:** mixed -**Integration scope:** core service only -**Primary use-case:** reliability and performance - -## How to use this bundle -- Use `01_haproxy_metrics.csv` as the source of truth for metric names, units, temporality assumptions, and safe query behavior. -- Use `02_haproxy_dashboard_plan.yaml` as the implementation blueprint for sections, widgets, derived signals, notes, triage chains, and playbooks. -- Use `03_haproxy_state.yaml` as the machine-readable bundle state (stage outcomes, corrections, access-method audit, and unresolved unknowns). -- Use `04_haproxy_memory.md` for consolidated Stage 2/3 memory (reconciliation decisions, access-method history, and created dashboard outcomes). -- Stage 2 will reconcile all metric names, context fields, and temporality against live Tsuga data and patch this bundle. - -## What it is and what "good" looks like - -### Confirmed by sources -- HAProxy is a multi-threaded, event-driven proxy that exposes frontend, backend, listener, and server traffic/health counters through `show stat` and Prometheus interfaces. [S1][S2][S3][S5] -- The native Prometheus service is enabled via `http-request use-service prometheus-exporter` and can export scope-filtered metrics; high-cardinality configurations can be expensive to scrape. [S2][S5][S6] -- HAProxy runtime stats include queue depth (`qcur`), active sessions (`scur`), byte counters (`bin`,`bout`), request/response errors (`ereq`,`eresp`), status code classes (`hrsp_5xx`), and backend/server health indicators. [S1][S3] -- The OpenTelemetry HAProxy receiver maps these runtime fields into `haproxy.*` metrics and marks many core counters as cumulative sums. [S7][S8] -- "Good" operational state is stable request/session throughput, low queue pressure, low connect/request/response error growth, and healthy backend/server availability with no sustained downtime growth. [S1][S3][S7] -- Paging intent in dashboard form: rapidly distinguish traffic surge, upstream dependency degradation, and HAProxy saturation before user-visible outage widens. - -### Best-practice inference -- Incident shape 1: **Backend degradation**. Signals: error counters rise (`connections.errors`, `responses.errors`), backend downtime/check-fail metrics rise, request throughput remains non-zero. Start in `errors-failures`. -- Incident shape 2: **Saturation under load**. Signals: queued requests rise, session counts push toward limits, connection rates spike. Start in `saturation-capacity`. -- Incident shape 3: **Latency creep before failures**. Signals: session/connect/queue/response average times drift up while hard errors remain moderate. Start in `latency-performance`. -- Dashboard success criteria for on-call: one glance for health posture, one click for per-proxy/per-service attribution, and explicit missing-data handling when optional metrics are disabled. - -## Key concepts - -### Glossary - -| Term | Definition | Operational meaning | Dashboard section | -|---|---|---|---| -| Frontend | HAProxy entrypoint that accepts client traffic | First place to confirm inbound load and denied sessions | traffic-throughput | -| Backend | HAProxy upstream pool definition | Tracks upstream availability, queueing, and failures | availability-health | -| Server | Individual backend target | Isolates failing instances within a backend | availability-health | -| Listener | Bound socket handling accepted connections | Identifies edge pressure and connection handling issues | traffic-throughput | -| Session | HAProxy end-to-end transaction lifecycle | Used for active load and saturation posture | saturation-capacity | -| Queue (`qcur`) | Requests waiting for backend assignment | Early warning of capacity shortfall | saturation-capacity | -| Active sessions (`scur`) | Current in-flight sessions | Core concurrency signal | saturation-capacity | -| Session rate (`rate`) | Sessions per second over recent second | Burst and trend signal | traffic-throughput | -| Request rate (`req_rate`) | HTTP requests per second over recent second | Throughput and surge detection | traffic-throughput | -| Connect errors (`econ`) | Failed attempts to connect to backend | Dependency/network failure indicator | errors-failures | -| Request errors (`ereq`) | Request parsing/processing errors | Client/input or proxy-path issues | errors-failures | -| Response errors (`eresp`) | Response path errors including server aborts | Upstream failure or midstream abort indicator | errors-failures | -| Denied requests (`dreq`) | Security/ACL-denied requests | Policy-driven rejects vs availability failures | errors-failures | -| Denied responses (`dresp`) | Security/ACL-denied responses | Policy post-processing effects | errors-failures | -| Redispatch (`wredis`) | Request re-routed to alternate server | Often indicates unstable upstream targets | errors-failures | -| Retry (`wretr`) | Backend connection retry event | Retry amplification can hide/worsen incidents | errors-failures | -| Check failures (`chkfail`) | Failed health checks while server considered up | Predictive signal for backend deterioration | availability-health | -| Downtime | Cumulative downtime seconds | Indicates sustained unavailability over time | availability-health | -| Active/backup servers (`act`,`bck`) | Count of active or backup backend servers | Capacity envelope and failover posture | availability-health | -| Session limit (`slim`) | Configured max sessions | Reference denominator for utilization | saturation-capacity | -| Average queue/connect/response/session time | Rolling latency-like gauges over recent requests | Triage whether pain is wait, connect, or serve time | latency-performance | -| `status_code` attribute | OTel dimension for 1xx/2xx/3xx/4xx/5xx/other | Enables error-rate formulas without separate metrics | errors-failures | -| `haproxy.proxy_name` attribute | OTel proxy-level identity | Primary bounded group-by for ownership/blast radius | all | -| `haproxy.service_name` attribute | OTel service role (FRONTEND/BACKEND/listener/server name) | Split by traffic role and pipeline stage | all | - -[S1][S3][S7][S8] - -### Entities and dimensions - -| Entity/Dimension | Why useful | Cardinality risk | Safe top-N | Do NOT group-by guidance | -|---|---|---|---|---| -| `context.env` | Environment boundary for prod/staging | Low | 5 | Always keep as global filter, not deep chart split | -| `context.team` | Ownership routing during incidents | Low | 10 | Avoid as first diagnostic axis for technical root cause | -| `context.proxy` | Stable frontend/backend/listener/server identity in discovered Tsuga data | Medium | 20 | Do not combine with high-card path/client labels | -| `context.service.name` | Service-level ownership and blast-radius split | Medium | 20 | Avoid mixing with raw server address in overview KPIs | -| `context.server.address` | Socket/server endpoint attribution | Medium | 20 | Can explode with ephemeral addresses in dynamic configs | -| `context.service.name` | Organization service boundary | Medium | 20 | Do not pair with pod UID in same chart | -| `context.scope.name` | Fallback stable scope when proxy_name missing | Medium | 20 | Use as fallback, not alongside proxy_name unless needed | -| `context.code` | HTTP class/code slices on response totals | Low | 6 | Keep bounded to class-level, avoid raw status if absent | -| `context.k8s.cluster.name` | Multi-cluster blast radius | Low-Medium | 10 | Skip for non-k8s deployments | -| `context.k8s.namespace.name` | Tenant/workload partitioning | Medium | 20 | Avoid in top KPI widgets | -| `context.host` | Host-level hotspot detection | Medium-High | 20 | Prefer scope/service for default views | -| `context.cloud.region` | Regional failure domain | Low | 12 | Only if consistently enriched | -| `context.cloud.account.id` | Multi-account segmentation | Medium | 10 | Avoid if single-account deployment | - -### Tsuga field mapping - -| Vendor/exporter dimension | Recommended context.* key | Must-exist vs optional | -|---|---|---| -| `proxy_name` / `pxname` | `context.proxy` | Optional (confirmed present) | -| `svname` (server/listener/backend target) | `context.service.name` | Optional (confirmed present) | -| `addr` | `context.server.address` | Optional (confirmed present) | -| Response code class/code labels | `context.code` | Optional (confirmed on `*_http_responses_total`) | -| Frontend/backend/server role identity | `context.service.name` | Optional | -| Runtime socket node identity | `context.scope.name` | Optional fallback | -| Service tag from collector enrichment | `context.service.name` | Optional | -| Environment tag (org standard) | `context.env` | Must-exist | -| Team tag (org standard) | `context.team` | Must-exist | -| Kubernetes cluster enrichment | `context.k8s.cluster.name` | Optional | -| Kubernetes namespace enrichment | `context.k8s.namespace.name` | Optional | -| Cloud region enrichment | `context.cloud.region` | Optional | -| Cloud account enrichment | `context.cloud.account.id` | Optional | - -### Confirmed by sources -- `proxy_name`, `service_name`, and `addr` are emitted as HAProxy receiver resource attributes. [S7] -- `status_code` attribute is documented on `haproxy.requests.total`. [S7] - -### Best-practice inference -- Stage 2 discovery confirmed Tsuga key shape uses `context.proxy`, `context.service.name`, `context.server.address`, and `context.code` (on response metrics), not `context.haproxy.*`. -- `context.scope.name` remains a practical fallback when proxy/service labels are sparse. - -## Golden signals - -### Confirmed by sources -| Signal | What it means for HAProxy | Typical degradation causes | Best telemetry sources | What people page on | Section questions | -|---|---|---|---|---|---| -| Traffic | Incoming and forwarded load level at frontend/listener/backend surfaces | Traffic spikes, reconnect storms, uneven proxy distribution | `haproxy.sessions.rate`, `haproxy.requests.rate`, `haproxy.connections.rate`, `haproxy.*.total` [S3][S7][S8] | Sustained throughput jump with concurrent queue growth | Is ingress load normal? Which proxy/service absorbs most traffic? | -| Errors | Failures across connect/request/response/security paths | Upstream dependency failure, ACL/policy rejects, malformed request bursts | `connections.errors`, `requests.errors`, `responses.errors`, denied/retry/redispatch counters [S1][S3][S7] | Error-rate acceleration or retries climbing while throughput stays high | Are failures upstream-connect, request parsing, or response-path dominant? | -| Latency | Time spent waiting/connecting/responding over rolling windows | Backend slowness, network degradation, queue buildup | `sessions.average`, `requests.average_time`, `connections.average_time`, `responses.average_time` [S3][S7] | Session/response time drift with stable load | Is delay queueing, connect, or server response time? | -| Saturation | Concurrency and queue pressure vs configured limits | Session limit pressure, backend capacity collapse, retry amplification | `sessions.count`, `sessions.limit`, `requests.queued`, `active/backup`, session limit stats [S1][S3][S7] | Queue growth and utilization high enough to threaten drops/timeouts | Are we running out of session/queue capacity? Is failover pool sufficient? | - -### Best-practice inference -- For HAProxy, connect/request/response error families are often more operationally useful than generic CPU/memory because they expose dependency path failure directly. -- Queue and session-limit ratios should be treated as first-class saturation KPIs, not secondary charts. - -## Telemetry sources - -### Confirmed by sources -| Source type | How collected | What it provides | Pros/cons | Common pitfalls | -|---|---|---|---|---| -| Runtime API `show stat` / stats socket | CLI/socket polling (`stats socket`, `show stat`, CSV/typed/json) | Canonical frontend/backend/server/listener counters and state | Most complete low-level data; requires socket access and secure permissions | Misreading cumulative counters as instant rates; stats socket not exposed [S1][S2] | -| Built-in Prometheus exporter (PROMEX) | `http-request use-service prometheus-exporter` and scrape `/metrics` | Native Prometheus metric families (`haproxy_frontend_*`, `haproxy_backend_*`, etc.) | No sidecar needed; broad ecosystem compatibility; can filter scopes/metrics | High cost on huge configs; more verbose/slower than CSV; optional extra counters can explode cardinality [S2][S5][S6] | -| Prometheus retired external exporter | Scrapes `?stats;csv` and re-exports | Compatibility path for legacy deployments | Useful in old setups | Officially retired; migrate to built-in exporter [S9] | -| OpenTelemetry HAProxy receiver | Polls socket or HTTP stats endpoint | Normalized `haproxy.*` metrics with documented attributes and optional metric toggles | Easy OTel pipeline alignment; explicit metric catalog | Many metrics marked development/beta; optional metrics disabled by default unless enabled [S7][S8] | -| HAProxy stats dashboard/UI | `stats enable` + `stats uri` | Human-readable point-in-time health status | Quick operator visibility | Not machine-optimized; requires auth/hardening | [S2][S10] | - -### Best-practice inference -- "No data" in optional metrics usually means disabled metric family or disabled HAProxy feature, not necessarily healthy zero. -- In mixed telemetry deployments, prefer one canonical family (`haproxy_*` OTel or `haproxy_frontend_*` PROMEX) per widget to avoid double-counting. - -## Caveats and footguns -- **[traffic-throughput]** `sessions.rate` and `requests.rate` are already per-second gauges; applying rate/per-second again corrupts meaning. (S7) -- **[errors-failures]** `responses.errors` includes server abort behavior; do not interpret as only proxy-generated response faults. (S7) -- **[errors-failures]** Retry and redispatch growth can mask backend instability while user-facing errors still look moderate. (S1, S7) -- **[saturation-capacity]** Queue growth (`qcur`) with flat throughput is a hard saturation warning, not normal burst noise. (S1) -- **[saturation-capacity]** `sessions.count` without `sessions.limit` loses utilization context and causes false calm. (S7) -- **[availability-health]** `downtime` is cumulative and monotonic; it should be shown as rate/increase for recent incidents, not absolute value only. (S7) -- **[availability-health]** `failed_checks` only counts failed checks while server is up; it can miss complete-down periods if interpreted alone. (S7) -- **[latency-performance]** Average time metrics (`ttime`,`qtime`,`ctime`,`rtime`) are rolling averages over recent requests, not percentile latency. (S7) -- **[latency-performance]** Comparing latency averages across proxies with very different traffic mix can mislead triage. (Inference) -- **[errors-failures]** Denied requests/responses may represent policy intent, not outage; pair with config change context before paging. (S1, S2) -- **[traffic-throughput]** Promex can export extra counters and protocol-specific families; enabling all can create heavy scrape payloads. (S5) -- **[traffic-throughput, secondary-signals]** QUIC/H2/H1 metric families are optional and often absent unless those protocol paths are active. (S5, S6) -- **[runtime-operations]** On large configurations, `/metrics` generation can be significantly slower and more verbose than CSV stats. (S5) -- **[runtime-operations]** Missing stats socket or auth misconfiguration causes telemetry blind spots that look like "all zeros" in dashboards. (S2, S9) -- **[availability-health]** Active/backup server counts can change from config or maintenance actions, not only hard failures. (S1) -- **[saturation-capacity]** Connection and session limits are configuration-driven; a low configured limit can make moderate traffic look pathological. (S2) -- **[errors-failures]** Status-class splits from `requests.total` require status_code attribute propagation; without it, class widgets must be gated. (S7) -- **[traffic-throughput]** Grouping by raw address can explode in dynamic service discovery or ephemeral listener setups. (Inference) -- **[runtime-operations]** Mixing OTel `haproxy.*` and Promex `haproxy_frontend_*` in one KPI can double-count the same underlying traffic. (Inference) -- **[latency-performance, saturation-capacity]** Rising queue time with stable connect time usually points to backend processing saturation, not network path failure. (Inference) -- **[availability-health]** "No data" for optional metrics may simply mean those receiver metrics are disabled by default. (S7) -- **[errors-failures]** `connections.errors` and `requests.errors` trace different failure stages; merging them without label context hides root cause. (S1, S7) -- **[traffic-throughput]** Scope/metrics query-string filters in Promex can silently hide expected signals if scrape URI differs across environments. (S5) - -## Confirmed Tsuga prefixes -- `haproxy_*` — **CONFIRMED** (225 metrics discovered in Tsuga over 24h window; unioned across paged scans to avoid non-deterministic catalog ordering). -- `haproxy_frontend_*` — **CONFIRMED** (33 metrics discovered). -- `haproxy_backend_*` — **CONFIRMED** (55 metrics discovered). -- `haproxy_server_*` — **CONFIRMED** (58 metrics discovered). - -## Discovery status -Discovery: completed in Stage 2. -- `METRICS_FOUND`: 225 HAProxy-prefixed metrics from a 1014-metric catalog (24h window ending 2026-02-17). -- Stage 1 OTel-style names (`haproxy_requests_total`, `haproxy_sessions_count`, etc.) were largely absent; Tsuga uses PROMEX-style families (`haproxy_frontend_*`, `haproxy_backend_*`, `haproxy_server_*`). -- Context keys were corrected to discovered fields: `context.proxy`, `context.service.name`, `context.server.address`, `context.scope.name`, and `context.code` (response metrics). -- Counter temporality for sampled HAProxy counters is `delta`, so dashboard queries were corrected to `sum + per-second`. -- Scalar data spot-checks via MCP `aggregate-scalar` were inconclusive due repeated internal server errors; catalog + metadata evidence was used. - -## Top sources -1. https://docs.haproxy.org/2.8/management.html - Why: canonical architecture and `show stat` CSV field semantics (`qcur`, `scur`, `stot`, `bin`, `bout`, `ereq`, `eresp`, `hrsp_5xx`). -2. https://docs.haproxy.org/2.8/configuration.html - Why: authoritative configuration of stats socket and native `prometheus-exporter` service hook. -3. https://raw.githubusercontent.com/open-telemetry/opentelemetry-collector-contrib/main/receiver/haproxyreceiver/documentation.md - Why: exact OTel HAProxy metric catalog, units, temporality, and attributes. -4. https://raw.githubusercontent.com/open-telemetry/opentelemetry-collector-contrib/main/receiver/haproxyreceiver/README.md - Why: receiver collection model (socket/HTTP endpoint), defaults, and metric enablement behavior. -5. https://raw.githubusercontent.com/haproxy/haproxy/master/addons/promex/README - Why: built-in Prometheus exporter behavior, filtering controls, performance caveats, and exported metric families. -6. https://raw.githubusercontent.com/prometheus/haproxy_exporter/master/README.md - Why: migration guidance and legacy exporter constraints for environments not yet on native exporter. -7. https://www.haproxy.com/documentation/haproxy-configuration-tutorials/alerts-and-monitoring/prometheus/ - Why: official HAProxy tutorial for Prometheus endpoint setup and modern metric families. -8. https://www.haproxy.com/documentation/haproxy-runtime-api/reference/show-stat/ - Why: runtime command reference for traffic statistics retrieval and formats. -9. https://www.haproxy.com/documentation/haproxy-runtime-api/reference/show-info/ - Why: runtime process-level info surface for operational context and runtime posture. -10. https://www.haproxy.com/documentation/haproxy-configuration-tutorials/reliability/health-checks/ - Why: health-check behavior context that influences check-failure, downtime, and backend availability interpretation. - ---- - -**Citation key** -- [S1] https://docs.haproxy.org/2.8/management.html -- [S2] https://docs.haproxy.org/2.8/configuration.html -- [S3] https://www.haproxy.com/documentation/haproxy-runtime-api/reference/show-stat/ -- [S4] https://www.haproxy.com/documentation/haproxy-runtime-api/reference/show-info/ -- [S5] https://raw.githubusercontent.com/haproxy/haproxy/master/addons/promex/README -- [S6] https://www.haproxy.com/documentation/haproxy-configuration-tutorials/alerts-and-monitoring/prometheus/ -- [S7] https://raw.githubusercontent.com/open-telemetry/opentelemetry-collector-contrib/main/receiver/haproxyreceiver/documentation.md -- [S8] https://raw.githubusercontent.com/open-telemetry/opentelemetry-collector-contrib/main/receiver/haproxyreceiver/README.md -- [S9] https://raw.githubusercontent.com/prometheus/haproxy_exporter/master/README.md -- [S10] https://www.haproxy.com/documentation/haproxy-configuration-tutorials/alerts-and-monitoring/statistics/ diff --git a/plugins/tsuga/skills/knowledge-technology/references/haproxy/metrics.csv b/plugins/tsuga/skills/knowledge-technology/references/haproxy/metrics.csv deleted file mode 100644 index 3e3d263..0000000 --- a/plugins/tsuga/skills/knowledge-technology/references/haproxy/metrics.csv +++ /dev/null @@ -1,34 +0,0 @@ -theme,name,confirmed,definition,unit,type,tsuga_metric_name,widget_suggestions,aggregation,post_function,group_by,required_context_fields,optional_context_fields,absence_means,cardinality_footguns,variants,citations -Availability/Health,haproxy_backend_active_servers,true,Number of active servers (backend) or active flag (server scope),{servers},gauge,haproxy_backend_active_servers,query-value|timeseries|max,max,none,context.proxy:20|context.service.name:20,context.env|context.team,context.proxy|context.service.name,Optional metric disabled or role not exporting active state,Scope differences require careful aggregation,otel=haproxy.active|raw=act,https://raw.githubusercontent.com/open-telemetry/opentelemetry-collector-contrib/main/receiver/haproxyreceiver/documentation.md -Availability/Health,haproxy_backend_backup_servers,true,Number of backup servers or backup flag at server scope,{servers},gauge,haproxy_backend_backup_servers,query-value|timeseries|max,max,none,context.proxy:20|context.service.name:20,context.env|context.team,context.proxy|context.service.name,Optional metric disabled or no backup pool configured,Backup count alone does not prove failover readiness,otel=haproxy.backup|raw=bck,https://raw.githubusercontent.com/open-telemetry/opentelemetry-collector-contrib/main/receiver/haproxyreceiver/documentation.md -Availability/Health,haproxy_backend_client_aborts_total,true,Client-aborted data transfers often tied to client timeout behavior,{cancellations},counter,haproxy_backend_client_aborts_total,timeseries|query-value|top-list,sum,per-second,context.proxy:20|context.service.name:20,context.env|context.team,context.proxy|context.service.name,Optional metric disabled or no client aborts,Can spike from client deploys independent of backend health,otel=haproxy.clients.canceled|raw=cli_abrt,https://raw.githubusercontent.com/open-telemetry/opentelemetry-collector-contrib/main/receiver/haproxyreceiver/documentation.md -Availability/Health,haproxy_backend_downtime_seconds_total,true,Cumulative downtime seconds for backend or server,s,counter,haproxy_backend_downtime_seconds_total,timeseries|query-value|top-list,sum,per-second,context.proxy:20|context.service.name:20,context.env|context.team,context.proxy|context.service.name,No downtime accumulation in selected window,Cumulative metric should be viewed as increase/rate not raw absolute only,otel=haproxy.downtime|raw=downtime,https://raw.githubusercontent.com/open-telemetry/opentelemetry-collector-contrib/main/receiver/haproxyreceiver/documentation.md -Availability/Health,haproxy_backend_status,true,PROMEX backend status gauge (up/down semantics by exporter mapping),1,gauge,haproxy_backend_status,query-value|timeseries|max,max,none,context.proxy:20|context.service.name:20,context.env|context.team,context.proxy|context.service.name,PROMEX not enabled or backend status filtered out,Status encoding must be verified before thresholds,raw=haproxy_backend_status,https://raw.githubusercontent.com/haproxy/haproxy/master/addons/promex/README -Availability/Health,haproxy_server_check_failures_total,true,Failed health checks while server reported up,{checks},counter,haproxy_server_check_failures_total,timeseries|query-value|top-list,sum,per-second,context.proxy:20|context.service.name:20,context.env|context.team,context.proxy|context.service.name,No recent failed checks or optional metric disabled,Misses complete-down states if interpreted alone,otel=haproxy.failed_checks|raw=chkfail,https://raw.githubusercontent.com/open-telemetry/opentelemetry-collector-contrib/main/receiver/haproxyreceiver/documentation.md -Availability/Health,haproxy_server_status,true,PROMEX server status gauge per server target,1,gauge,haproxy_server_status,timeseries|top-list|max,max,none,context.proxy:20|context.service.name:20,context.env|context.team,context.proxy|context.service.name|context.server.address,Server status metric absent or scope filtered,Server labels can become high-cardinality with templates,raw=haproxy_server_status,https://raw.githubusercontent.com/haproxy/haproxy/master/addons/promex/README -Capacity/Saturation,haproxy_backend_current_queue,true,Current queued requests without server assignment indicating pressure,{requests},gauge,haproxy_backend_current_queue,timeseries|query-value|top-list,max,none,context.proxy:20|context.service.name:20,context.env|context.team,context.proxy|context.service.name,No queue pressure or metric disabled,Interpretation depends on receiver temporality mapping; verify in Stage 2,otel=haproxy.requests.queued|raw=qcur,https://raw.githubusercontent.com/open-telemetry/opentelemetry-collector-contrib/main/receiver/haproxyreceiver/documentation.md -Capacity/Saturation,haproxy_backend_weight,true,Effective backend/server weight for load-balancing decisions,1,gauge,haproxy_backend_weight,timeseries|query-value|top-list,max,none,context.proxy:20|context.service.name:20,context.env|context.team,context.proxy|context.service.name,Optional metric disabled or static config not exposing weights,Weight interpretation differs by algorithm and role,otel=haproxy.weight|raw=weight,https://raw.githubusercontent.com/open-telemetry/opentelemetry-collector-contrib/main/receiver/haproxyreceiver/documentation.md -Capacity/Saturation,haproxy_frontend_connections_total,true,Cumulative frontend connections accepted,{connections},counter,haproxy_frontend_connections_total,timeseries|query-value|top-list,sum,per-second,context.proxy:20|context.service.name:20,context.env|context.team,context.proxy|context.service.name,Optional metric disabled or no frontend traffic,Can be redundant with sessions if mixed families are plotted together,otel=haproxy.connections.total|raw=conn_tot,https://raw.githubusercontent.com/open-telemetry/opentelemetry-collector-contrib/main/receiver/haproxyreceiver/documentation.md -Capacity/Saturation,haproxy_frontend_current_sessions,true,Current in-flight sessions and main concurrency pressure signal,{sessions},gauge,haproxy_frontend_current_sessions,timeseries|query-value|max,max,none,context.proxy:20|context.service.name:20,context.env|context.team,context.proxy|context.service.name,No active sessions or telemetry path missing,Proxy name churn can create noisy top-lists,otel=haproxy.sessions.count|raw=scur,https://raw.githubusercontent.com/open-telemetry/opentelemetry-collector-contrib/main/receiver/haproxyreceiver/documentation.md -Capacity/Saturation,haproxy_frontend_limit_sessions,true,Configured session limit used as denominator for utilization,{sessions},gauge,haproxy_frontend_limit_sessions,query-value|timeseries,max,none,context.proxy:20|context.service.name:20,context.env|context.team,context.proxy|context.service.name,Limit not exported by current role/receiver config,Limit differs by frontend/backend and can mislead global averages,otel=haproxy.sessions.limit|raw=slim,https://raw.githubusercontent.com/open-telemetry/opentelemetry-collector-contrib/main/receiver/haproxyreceiver/documentation.md -Capacity/Saturation,haproxy_process_current_connections,true,PROMEX process-level current connection count for global HAProxy pressure,{connections},gauge,haproxy_process_current_connections,query-value|timeseries,max,none,context.scope.name:20|context.service.name:20,context.env|context.team,context.scope.name|context.service.name,Global process metric unavailable or scope filtered,Process metric may not map cleanly to per-backend incidents,raw=haproxy_process_current_connections,https://raw.githubusercontent.com/haproxy/haproxy/master/addons/promex/README -Errors/Failures,haproxy_backend_connection_errors_total,true,Connection attempts that failed while reaching backend servers,{errors},counter,haproxy_backend_connection_errors_total,timeseries|query-value|top-list,sum,per-second,context.proxy:20|context.service.name:20,context.env|context.team,context.proxy|context.service.name,Backend connect path currently healthy or metric disabled,Needs pairing with timeout/retry signals for diagnosis,otel=haproxy.connections.errors|raw=econ,https://raw.githubusercontent.com/open-telemetry/opentelemetry-collector-contrib/main/receiver/haproxyreceiver/documentation.md -Errors/Failures,haproxy_backend_http_responses_total,true,Backend HTTP responses by code class; used for 5xx ratio and response distribution.,{responses},counter,haproxy_backend_http_responses_total,timeseries|query-value|top-list,sum,per-second,context.proxy:20|context.code:6,context.env|context.team,context.proxy|context.service.name|context.code,Backend response-class metrics are missing or exporter scope excludes response labels.,Use response code classes not raw status labels for stable Top-N.,raw=haproxy_backend_http_responses_total,https://raw.githubusercontent.com/haproxy/haproxy/master/addons/promex/README -Errors/Failures,haproxy_backend_redispatch_warnings_total,true,Requests redispatched to alternate servers after initial selection,{requests},counter,haproxy_backend_redispatch_warnings_total,timeseries|query-value|top-list,sum,per-second,context.proxy:20|context.service.name:20,context.env|context.team,context.proxy|context.service.name,No redispatch events in window,High redispatch with retries indicates unstable pool,otel=haproxy.requests.redispatched|raw=wredis,https://raw.githubusercontent.com/open-telemetry/opentelemetry-collector-contrib/main/receiver/haproxyreceiver/documentation.md -Errors/Failures,haproxy_backend_response_errors_total,true,Cumulative response-path errors including aborted server responses,{errors},counter,haproxy_backend_response_errors_total,timeseries|query-value|top-list,sum,per-second,context.proxy:20|context.service.name:20,context.env|context.team,context.proxy|context.service.name,No response-path errors in window,May include server abort semantics not only proxy faults,otel=haproxy.responses.errors|raw=eresp,https://raw.githubusercontent.com/open-telemetry/opentelemetry-collector-contrib/main/receiver/haproxyreceiver/documentation.md -Errors/Failures,haproxy_backend_retry_warnings_total,true,Connection retries while reaching backend servers,{retries},counter,haproxy_backend_retry_warnings_total,timeseries|query-value|top-list,sum,per-second,context.proxy:20|context.service.name:20,context.env|context.team,context.proxy|context.service.name,No retries observed or retry logic disabled,Retry spikes can amplify backend load,otel=haproxy.connections.retries|raw=wretr,https://raw.githubusercontent.com/open-telemetry/opentelemetry-collector-contrib/main/receiver/haproxyreceiver/documentation.md -Errors/Failures,haproxy_frontend_request_errors_total,true,Cumulative request-path errors including malformed/processing failures,{errors},counter,haproxy_frontend_request_errors_total,timeseries|query-value|top-list,sum,per-second,context.proxy:20|context.service.name:20,context.env|context.team,context.proxy|context.service.name,No request-path errors in window,Can reflect client issues rather than backend outage,otel=haproxy.requests.errors|raw=ereq,https://raw.githubusercontent.com/open-telemetry/opentelemetry-collector-contrib/main/receiver/haproxyreceiver/documentation.md -Errors/Failures,haproxy_frontend_requests_denied_total,true,Requests denied by policy/security logic,{requests},counter,haproxy_frontend_requests_denied_total,timeseries|query-value|top-list,sum,per-second,context.proxy:20|context.service.name:20,context.env|context.team,context.proxy|context.service.name,No deny events or policy path disabled,Can be expected policy behavior not outage,otel=haproxy.requests.denied|raw=dreq,https://raw.githubusercontent.com/open-telemetry/opentelemetry-collector-contrib/main/receiver/haproxyreceiver/documentation.md -Errors/Failures,haproxy_frontend_responses_denied_total,true,Responses denied by policy/security logic,{responses},counter,haproxy_frontend_responses_denied_total,timeseries|query-value|top-list,sum,per-second,context.proxy:20|context.service.name:20,context.env|context.team,context.proxy|context.service.name,No deny events or policy path disabled,Interpret with config change context to avoid false incidents,otel=haproxy.responses.denied|raw=dresp,https://raw.githubusercontent.com/open-telemetry/opentelemetry-collector-contrib/main/receiver/haproxyreceiver/documentation.md -Performance/Latency,haproxy_backend_connect_time_average_seconds,true,Average connect time over recent sample window,s,gauge,haproxy_backend_connect_time_average_seconds,timeseries|query-value|top-list,max,none,context.proxy:20|context.service.name:20,context.env|context.team,context.proxy|context.service.name,Optional metric disabled or no connect attempts,Interpret with connection errors/timeouts to isolate path issue,otel=haproxy.connections.average_time|raw=ctime,https://raw.githubusercontent.com/open-telemetry/opentelemetry-collector-contrib/main/receiver/haproxyreceiver/documentation.md -Performance/Latency,haproxy_backend_queue_time_average_seconds,true,Average queue time over recent sample window,s,gauge,haproxy_backend_queue_time_average_seconds,timeseries|query-value|top-list,max,none,context.proxy:20|context.service.name:20,context.env|context.team,context.proxy|context.service.name,Optional metric disabled or no sampled requests,Queue average should be correlated with queued requests for interpretation,otel=haproxy.requests.average_time|raw=qtime,https://raw.githubusercontent.com/open-telemetry/opentelemetry-collector-contrib/main/receiver/haproxyreceiver/documentation.md -Performance/Latency,haproxy_backend_response_time_average_seconds,true,Average response time over recent sample window,s,gauge,haproxy_backend_response_time_average_seconds,timeseries|query-value|top-list,max,none,context.proxy:20|context.service.name:20,context.env|context.team,context.proxy|context.service.name,Optional metric disabled or no sampled responses,Average can hide long-tail spikes under mixed traffic,otel=haproxy.responses.average_time|raw=rtime,https://raw.githubusercontent.com/open-telemetry/opentelemetry-collector-contrib/main/receiver/haproxyreceiver/documentation.md -Performance/Latency,haproxy_backend_total_time_average_seconds,true,Average total session time over recent sample window,s,gauge,haproxy_backend_total_time_average_seconds,timeseries|query-value|max,max,none,context.proxy:20|context.service.name:20,context.env|context.team,context.proxy|context.service.name,No sampled sessions in interval,Average is not percentile and can mask tails,otel=haproxy.sessions.average|raw=ttime,https://raw.githubusercontent.com/open-telemetry/opentelemetry-collector-contrib/main/receiver/haproxyreceiver/documentation.md -Throughput/Usage,haproxy_backend_http_requests_total,true,PROMEX backend HTTP request volume by backend label,{requests},counter,haproxy_backend_http_requests_total,timeseries|query-value|top-list,sum,per-second,context.proxy:20|context.service.name:20,context.env|context.team,context.proxy|context.service.name|context.server.address,PROMEX not enabled or backend scope filtered out,Cardinality rises with many dynamic backends,raw=haproxy_backend_http_requests_total,https://raw.githubusercontent.com/haproxy/haproxy/master/addons/promex/README -Throughput/Usage,haproxy_backend_loadbalanced_total,true,Number of times a server was selected for sessions or redispatch,{selections},counter,haproxy_backend_loadbalanced_total,timeseries|query-value|top-list,sum,per-second,context.proxy:20|context.service.name:20,context.env|context.team,context.proxy|context.service.name,No selection events in selected window,Can overlap with request totals depending on protocol mix,otel=haproxy.server_selected.total|raw=lbtot,https://raw.githubusercontent.com/open-telemetry/opentelemetry-collector-contrib/main/receiver/haproxyreceiver/documentation.md -Throughput/Usage,haproxy_frontend_bytes_in_total,true,Inbound bytes received by HAProxy,By,counter,haproxy_frontend_bytes_in_total,timeseries|query-value|top-list,sum,per-second,context.proxy:20|context.service.name:20,context.env|context.team,context.proxy|context.service.name,No inbound traffic or metric disabled,Can double-count with multiple telemetry families,otel=haproxy.bytes.input|raw=bin,https://raw.githubusercontent.com/open-telemetry/opentelemetry-collector-contrib/main/receiver/haproxyreceiver/documentation.md -Throughput/Usage,haproxy_frontend_bytes_out_total,true,Outbound bytes sent by HAProxy,By,counter,haproxy_frontend_bytes_out_total,timeseries|query-value|top-list,sum,per-second,context.proxy:20|context.service.name:20,context.env|context.team,context.proxy|context.service.name,No outbound traffic or metric disabled,Byte counters can spike on retransmit-heavy conditions,otel=haproxy.bytes.output|raw=bout,https://raw.githubusercontent.com/open-telemetry/opentelemetry-collector-contrib/main/receiver/haproxyreceiver/documentation.md -Throughput/Usage,haproxy_frontend_current_session_rate,true,Sessions per second gauge used for ingress load and burst detection,{sessions/s},gauge,haproxy_frontend_current_session_rate,timeseries|query-value|max,max,none,context.proxy:20|context.service.name:20,context.env|context.team,context.proxy|context.service.name,No session-producing traffic or metric disabled,High-card grouping by address/service combinations can clutter charts,otel=haproxy.sessions.rate|raw=rate,https://raw.githubusercontent.com/open-telemetry/opentelemetry-collector-contrib/main/receiver/haproxyreceiver/documentation.md -Throughput/Usage,haproxy_frontend_http_requests_rate_max,true,HTTP requests per second over recent elapsed second gauge,{requests/s},gauge,haproxy_frontend_http_requests_rate_max,timeseries|query-value,max,none,context.proxy:20|context.service.name:20,context.env|context.team,context.proxy|context.service.name,Frontend traffic not present or metric disabled,Grouping by raw address may explode,otel=haproxy.requests.rate|raw=req_rate,https://raw.githubusercontent.com/open-telemetry/opentelemetry-collector-contrib/main/receiver/haproxyreceiver/documentation.md -Throughput/Usage,haproxy_frontend_http_requests_total,true,Total HTTP requests received and response-class split foundation for traffic and error rates,{requests},counter,haproxy_frontend_http_requests_total,timeseries|query-value|top-list,sum,per-second,context.proxy:20|context.service.name:20,context.env|context.team,context.proxy|context.service.name,Frontend request totals unavailable due exporter scope or ingest outage,Use proxy-level grouping; avoid address-level fanout in overview,otel=haproxy.requests.total|raw=req_tot/hrsp_*,https://raw.githubusercontent.com/open-telemetry/opentelemetry-collector-contrib/main/receiver/haproxyreceiver/documentation.md -Cost,NOT_APPLICABLE,false,HAProxy telemetry does not expose direct cloud billing; cost dashboards require external billing datasets and can use throughput plus session utilization as proxy drivers,Unknown,Unknown,Unknown,query-value,average,none,,context.env|context.team,context.service.name,Cost metrics require separate billing/export pipelines,Safe,proxy=haproxy.requests.total|proxy=haproxy.sessions.count,https://docs.haproxy.org/2.8/management.html diff --git a/plugins/tsuga/skills/knowledge-technology/references/haproxy/queries.md b/plugins/tsuga/skills/knowledge-technology/references/haproxy/queries.md deleted file mode 100644 index 7be4281..0000000 --- a/plugins/tsuga/skills/knowledge-technology/references/haproxy/queries.md +++ /dev/null @@ -1,59 +0,0 @@ -# HAProxy - -L4/L7 load balancer. Healthy: backends active, zero failed health checks, bounded response time, low 5xx rate, queue depth ≈ 0. - -## Incident shapes - -- **Backend unhealthy** — active_servers drops, check_failures rise → backend sick, not HAProxy -- **Queue backpressure** — `current_queue > 0` sustained → backend can't keep up -- **Backend errors** — `response_errors_total` / `connection_errors_total` rise → backend misbehaving -- **Session saturation** — `current_sessions / limit_sessions` → 1.0 → new connections rejected -- **Redispatches** — retry on different backend after first failed → backend flakiness - -## Key metrics - -| Metric | Unit | Signal | -|---|---|---| -| `haproxy_backend_active_servers` | count | Drop = health-check failures | -| `haproxy_backend_backup_servers` | count | Standby in use | -| `haproxy_backend_status` | 0/1 | 1=UP, 0=DOWN (all servers failed) | -| `haproxy_server_status` | 0/1 | Per-server | -| `haproxy_server_check_failures_total` | count | Health-check failures | -| `haproxy_backend_current_queue` | count | Requests waiting for backend slot | -| `haproxy_backend_downtime_seconds_total` | seconds | Cumulative downtime | -| `haproxy_frontend_connections_total` | count | New-connection rate | -| `haproxy_frontend_current_sessions` / `limit_sessions` | count | Sessions + ceiling | -| `haproxy_frontend_request_errors_total` | count | Frontend errors | -| `haproxy_frontend_requests_denied_total` | count | ACL denies | -| `haproxy_backend_connection_errors_total` | count | Connect to backend failures | -| `haproxy_backend_response_errors_total` | count | Backend response protocol errors | -| `haproxy_backend_retry_warnings_total` | count | Same-server retries | -| `haproxy_backend_redispatch_warnings_total` | count | Cross-server retries | -| `haproxy_backend_http_responses_total{code=5xx}` | count | Backend 5xx | -| `haproxy_backend_connect_time_average_seconds` | s | Backend connect latency | -| `haproxy_backend_queue_time_average_seconds` | s | Queue wait | -| `haproxy_backend_response_time_average_seconds` | s | Backend response | - -## Derived signals - -- `current_sessions / limit_sessions` — session utilization. > 0.85 sustained = capacity headroom running out. -- `retry + redispatch` rate — retry pressure. Any sustained = backend flakiness. -- `connect_time + queue_time + response_time` split — locates the dominant latency phase. - -## Log patterns - -- `SC--` — session closed before request (client abort) -- `SD--` — session closed by server with data -- `sH--` / `SH--` — server reset / aborted -- `NOSRV` — no server available -- `Server X is DOWN` / `UP` — health transitions -- `Backend X has no server available` — all backends failed -- HTTP codes in access logs: 502 / 503 / 504 distinguish connect / no-server / timeout - -## Gotchas - -- Access-log timers are in ms typically. `-1` means the phase didn't complete. -- `backend_status = 0` = ALL servers failed, not just one. Use `server_status` per-server. -- Session limits are per-process. Multi-process (nbthread) shares per-thread; true utilization needs config. -- `redispatch_warnings` counts attempts, not distinct failed requests. High rate ≠ many requests failed. -- Weighted backends skew traffic; per-server error analysis without knowing weights can mislead. diff --git a/plugins/tsuga/skills/knowledge-technology/references/istio/README.md b/plugins/tsuga/skills/knowledge-technology/references/istio/README.md deleted file mode 100644 index 5ac707b..0000000 --- a/plugins/tsuga/skills/knowledge-technology/references/istio/README.md +++ /dev/null @@ -1,282 +0,0 @@ -# Istio Integration Context Bundle - -## Metadata -**Technology:** Istio -**Deployment:** self-hosted -**Environment:** prod -**Persona:** SRE Dev and ops -**Telemetry preference:** mixed -**Integration scope:** core service only -**Primary use-case:** reliability and performance - -## How to use this bundle -- Use `01_istio_metrics.csv` as the source of truth for metric names, units, temporality assumptions, and safe query behavior. -- Use `02_istio_dashboard_plan.yaml` for the dashboard section layout, widgets, derived signals, explanation notes, triage chains, and playbooks. -- Use `03_istio_state.yaml` for machine-readable stage status, assumptions, and explicit unknowns that Stage 2 must verify. -- Use `04_istio_memory.md` for the human-readable handoff summary and tradeoffs made in Stage 1. -- Stage 2 will create `05_istio_metric_catalog.csv` as the discovered Tsuga metric catalog used for reconciliation and coverage checks. -- Stage 4 should read `## Log intelligence (Stage 4 handoff)` in this file and `03_istio_state.yaml` `log_intel` before creating log routes. - -## What it is and what "good" looks like - -### Confirmed by sources -- Istio is a service mesh that layers traffic management, security, and observability over application workloads, typically through Envoy sidecars or gateways in Kubernetes. [S1][S2][S3] -- Istio ships a standard set of service metrics for HTTP, gRPC, and TCP traffic, including request totals, duration histograms, byte histograms, gRPC message counters, and TCP connection and byte counters. [S1][S4][S5] -- Istio telemetry is shaped heavily by labels such as `reporter`, source and destination workload/service labels, canonical service labels, response code fields, and connection security policy. These dimensions make dashboards useful, but also create real cardinality risk. [S1][S4] -- Good mesh health means traffic is flowing, error rates are low, latency remains stable, mTLS posture matches policy, and no workload or namespace is disproportionately failing or saturating. [S1][S2][S6] -- Access logging is optional rather than universal. When enabled, Envoy access logs are emitted to standard output and can be collected from proxy containers. [S7] -- First-response dashboard routing for common incidents: - - Upstream service failure or policy breakage: start in `errors-failures`. - - Latency regression without obvious errors: start in `latency-performance`. - - Mesh rollout, certificate, or peer-identity concern: start in `security-identity`. -- High-level paging intent for dashboards: surface user-visible request failures, identify whether blast radius is local or mesh-wide, and separate real dependency trouble from telemetry gaps or policy-induced drops. [S1][S6][S8] - -### Best-practice inference -- For this integration, the most decision-useful surface is the data plane service telemetry (`istio_*`) rather than raw Envoy internals. Envoy internals are better handled by the separate Envoy integration when deeper proxy debugging is needed. -- Ambient mode, waypoint-specific behavior, and `istiod` control-plane health are important, but they should be treated as gated or follow-on coverage until Stage 2 confirms which metrics and context fields actually exist in Tsuga. -- The top three incident shapes likely to matter most in a general Istio dashboard are: - 1. **Destination service degradation**: request rate stays normal while 5xx or gRPC failures rise. Start in `errors-failures`. - 2. **Latency-led mesh degradation**: p95 duration rises before success rate collapses. Start in `latency-performance`. - 3. **Policy or identity regression**: mTLS mode flips, requests disappear, or one namespace loses traffic after rollout. Start in `security-identity`. - -## Key concepts - -### Glossary -| Term | Definition | Operational meaning | Dashboard section | -|---|---|---|---| -| Service mesh | Infrastructure layer that handles service-to-service networking concerns | Changes in the mesh can impact many workloads at once | availability-health | -| Sidecar proxy | Envoy proxy injected next to an application pod | Source of most request-level telemetry in sidecar mode | traffic-volume | -| Gateway | Envoy-based ingress or egress entrypoint managed by Istio | Gateway failures can look mesh-wide even when backend apps are healthy | availability-health | -| Telemetry API | Istio API used to configure metrics, logs, and traces | Metric presence and labels can change through policy, not just code | security-identity | -| Reporter | Label indicating whether the metric came from source or destination perspective | Double counting risk if source and destination are mixed carelessly | traffic-volume | -| Canonical service | Stable workload identity used for service ownership views | Better grouping key than raw pod or revision labels during rollouts | traffic-volume | -| Canonical revision | Revision identity for a canonical service | Helps isolate bad rollouts without grouping by pod name | availability-health | -| Source workload | Workload originating a request | Useful for blaming noisy callers or bad client retries | traffic-volume | -| Destination workload | Workload receiving a request | Primary dimension for service health triage | errors-failures | -| Response code | HTTP response status | Fast first-pass view of failure classes | errors-failures | -| gRPC response status | gRPC-specific response classification | Needed when HTTP code alone is not enough | errors-failures | -| Connection security policy | Label showing whether the request used mTLS | Critical for validating mesh security posture and policy drift | security-identity | -| Request duration histogram | Histogram of request latency in milliseconds | Best base for p95 latency and tail-latency dashboards | latency-performance | -| Request bytes | Histogram of inbound request size | Helps explain large payload latency or resource pressure | traffic-volume | -| Response bytes | Histogram of outbound response size | Useful for spotting payload blowups and egress-heavy paths | traffic-volume | -| TCP reporting duration | Interval used for active TCP metric reporting | Low traffic TCP services can appear bursty or delayed | saturation-capacity | -| Workload identity | Combination of namespace, workload, service account, and policy identity | Identity drift often explains selective failures | security-identity | -| Peer exchange | Istio mechanism for sharing metadata across TCP connections | Missing metadata reduces dimension richness on TCP dashboards | security-identity | -| Namespace | Kubernetes tenancy and operational boundary | Safe first drilldown for ownership and blast radius | availability-health | -| Revision | Istio control-plane or workload revision marker | Useful for rollout correlation and canary validation | availability-health | -| mTLS | Mutual TLS between workloads | A posture change may be either expected hardening or accidental breakage | security-identity | -| Destination service | Service FQDN or logical destination label | Best high-level grouping key for service health | errors-failures | -| Source principal | Identity of the calling workload | Valuable for security investigations but sometimes absent | security-identity | -| Ambient mode | Istio dataplane mode without per-pod sidecars | Metric shapes and dimensions may differ from sidecar assumptions | availability-health | -| Waypoint proxy | Envoy proxy handling traffic for ambient workloads | Adds another ownership and routing layer to inspect | latency-performance | - -[S1][S2][S3][S4][S5][S6][S7] - -### Concept Map -```text -Client workload -> sends request to -> source Envoy proxy (why: source-side telemetry can record intent before destination handling) -Source Envoy proxy -> applies -> routing and policy decisions (why: retries, routing rules, and auth can change outcomes before app code) -Source Envoy proxy -> forwards to -> destination Envoy proxy or gateway (why: mesh hop where security and policy are enforced) -Destination Envoy proxy -> forwards to -> destination workload (why: last mesh hop before application handling) -Request -> increments -> istio_requests_total (why: baseline request traffic and denominator for many ratios) -Request -> contributes to -> istio_request_duration_milliseconds (why: tail latency is often the first user-visible symptom) -Request payload -> contributes to -> istio_request_bytes (why: larger payloads can explain latency and resource pressure) -Response payload -> contributes to -> istio_response_bytes (why: payload explosions can signal misuse or oversized responses) -gRPC call -> increments -> istio_request_messages_total and istio_response_messages_total (why: message-heavy paths can regress without HTTP error spikes) -TCP flow -> increments -> istio_tcp_connections_opened_total (why: connection churn exposes retry storms or connection instability) -TCP flow -> increments -> istio_tcp_connections_closed_total (why: close/open imbalance can reveal unhealthy churn) -TCP flow -> contributes to -> istio_tcp_sent_bytes_total and istio_tcp_received_bytes_total (why: non-HTTP traffic still needs throughput and saturation views) -Telemetry labels -> include -> reporter (why: source and destination perspectives must not be naively summed together) -Telemetry labels -> include -> source and destination workload/service fields (why: service ownership and blast radius depend on them) -Telemetry labels -> include -> canonical service labels (why: stable grouping during rollouts is more important than pod-level granularity) -Telemetry labels -> include -> connection_security_policy (why: mTLS posture is an operational and security signal) -Telemetry API -> can suppress or add -> metric tags (why: dashboards must gate on missing dimensions rather than assuming universal presence) -Access logging -> emits -> Envoy access logs to stdout when enabled (why: logs become the fastest evidence path for policy or routing failures) -Request failures -> first appear in -> response code and grpc status dimensions (why: classify app errors vs transport or policy errors quickly) -Rising latency -> often precedes -> rising errors (why: slow upstreams degrade before they fail hard) -Namespace ownership -> maps to -> context.team and service dimensions (why: triage should land with the right team quickly) -Control-plane rollout -> changes -> telemetry shape and mesh behavior (why: revision-aware views help distinguish rollout regressions from app regressions) -Ambient mode -> changes -> dataplane topology and labels (why: sidecar-era assumptions may not hold) -Gateway traffic -> can concentrate -> external user impact (why: ingress issues can look broader than single-service failures) -Missing telemetry -> may mean -> policy/config disabled metrics rather than healthy zero (why: absence is not evidence of health) -``` - -### Entities and dimensions -| Dimension | Why useful | Cardinality risk | Safe top-N suggestion | -|---|---|---|---| -| `context.env` | Environment split for prod vs non-prod | Low | 5 | -| `context.team` | Ownership boundary for triage | Low | 20 | -| `context.destination_service_namespace` | Best first-pass blast-radius grouping | Medium | 20 | -| `context.service.name` | Stable service filter if Tsuga enriches it | Medium | 20 | -| `context.destination_service` | Best destination health view | Medium | 20 | -| `context.destination_workload` | Useful for rollout or hot workload isolation | Medium | 15 | -| `context.source_workload` | Useful for noisy caller identification | Medium | 15 | -| `context.destination_canonical_service` | Safer grouping during revisions and rollouts | Medium | 15 | -| `context.source_canonical_service` | Safer caller grouping during revisions | Medium | 15 | -| `context.response_code` | Good for breakdowns, not dashboard filter default | Low | 10 | -| `context.grpc_response_status` | Required for gRPC-only triage | Low | 10 | -| `context.connection_security_policy` | Essential for security posture KPIs | Very low | 3 | -| `context.reporter` | Needed to prevent double counting | Very low | 2 | -| `context.k8s.cluster.name` | Multi-cluster view when present | Low | 10 | -| `context.k8s.pod.name` | Deep forensic drilldown only | High | 10 | -| `context.net.peer.ip` | Useful for security investigations only | Very high | do NOT group-by on dashboards | - -**Do NOT group-by:** raw request path, pod UID, peer IP, source principal, or full destination FQDN without a bounded top-N and a strong reason. These are high-cardinality traps for mesh telemetry. [S1][S4][S6] - -### Tsuga field mapping - -### Confirmed by sources -The org-confirmed Tsuga context fields from `.env` are `context.env` and `context.team`. - -### Confirmed in Tsuga (Stage 2 MCP discovery) -The live Istio metrics in Tsuga use plain keys such as `context.reporter`, `context.destination_service`, `context.destination_service_namespace`, `context.response_code`, `context.grpc_response_status`, and `context.connection_security_policy`, rather than the `context.istio.*` names originally inferred in Stage 1. - -### Best-practice inference -| Vendor/exporter dimension | Recommended context.* key | Must-exist vs optional | -|---|---|---| -| `reporter` | `context.reporter` | must-exist | -| `source_workload` | `context.source_workload` | optional | -| `source_workload_namespace` | `context.destination_service_namespace` or `context.source_workload_namespace` | optional | -| `source_canonical_service` | `context.source_canonical_service` | optional | -| `source_canonical_revision` | `context.source_canonical_revision` | optional | -| `destination_service` | `context.destination_service` | must-exist | -| `destination_service_namespace` | `context.destination_service_namespace` | optional | -| `destination_workload` | `context.destination_workload` | optional | -| `destination_workload_namespace` | `context.destination_workload_namespace` | optional | -| `destination_canonical_service` | `context.destination_canonical_service` | optional | -| `destination_canonical_revision` | `context.destination_canonical_revision` | optional | -| `response_code` | `context.response_code` | optional | -| `grpc_response_status` | `context.grpc_response_status` | optional | -| `connection_security_policy` | `context.connection_security_policy` | optional | -| `request_protocol` | `context.request_protocol` | optional | -| `source_principal` | `context.source_principal` | optional | -| `destination_principal` | `context.destination_principal` | optional | -| `source_cluster` | `context.k8s.cluster.name` or `context.source_cluster` | optional | -| `destination_cluster` | `context.destination_cluster` | optional | - -## Golden signals - -### Confirmed by sources -| Signal | What it means for Istio | Best telemetry | Typical causes when degraded | What people page on | Section questions | -|---|---|---|---|---|---| -| Traffic | Request and message flow across the mesh | `istio_requests_total`, `istio_request_messages_total`, `istio_response_messages_total`, size histograms [S1][S4][S5] | Traffic drains, route misconfig, gateway outage, partial namespace isolation | A service or namespace suddenly loses expected traffic or one destination dominates traffic unexpectedly | Is traffic present? Which destination services or namespaces changed most? | -| Errors | Failed HTTP/gRPC outcomes and abnormal service-path behavior | `istio_requests_total` with response labels and gRPC status labels [S1][S4][S5] | Upstream app failures, authz/authn policy breakage, TLS mismatch, retry storms | Sustained 5xx rise, gRPC failure concentration, one workload or namespace going red | Are failures broad or localized? Are they HTTP or gRPC led? | -| Latency | Time requests spend completing through the mesh | `istio_request_duration_milliseconds` [S1][S4] | Slow upstreams, oversized payloads, gateway bottlenecks, retry inflation | p95 or p99 latency rising across key services while throughput remains normal | Is tail latency broad or concentrated? Did payload size or protocol mix change? | -| Saturation | Pressure in connection churn, byte throughput, or heavy payload paths | TCP connection and byte counters plus request/response size histograms [S1][S4][S5] | Connection churn, large payloads, inefficient retries, long-lived TCP imbalance | Sudden churn, one service dominating mesh bytes, latency rising with stable request rate | Is one service over-consuming the mesh? Is connection churn preceding user-visible errors? | - -### Best-practice inference -- Treat `reporter=destination` as the default surface for service health KPIs so request totals are not double counted. -- Use request-size and response-size trends as explanatory signals rather than primary paging KPIs. -- Security posture belongs adjacent to golden signals for Istio because a mesh can be "up" while identity or mTLS posture is silently wrong. - -## Telemetry sources - -### Confirmed by sources -| Source type | How collected | What it provides | Pros/cons | Common pitfalls | -|---|---|---|---|---| -| Istio standard service metrics | Exported from Istio-managed proxies and scraped by Prometheus-compatible systems | HTTP, gRPC, and TCP service telemetry with mesh labels | High operational value, standardized names; can still be label-heavy | Mixing source and destination reporters, assuming all labels exist everywhere [S1][S4] | -| Telemetry API overrides | Istio Telemetry resources change metric tags and providers | Metric/tag customization and selective telemetry shaping | Powerful for tailoring dashboards; makes metric shape mutable | Dashboards can silently break if tags are removed or renamed [S2][S6] | -| Prometheus addon / Prometheus scraping | Queries the scraped `istio_*` series | Ground truth examples for actual metric names and labels | Easiest way to verify names; common reference path | Scrape gaps or secure-scrape config issues look like missing service health [S8][S9] | -| Envoy access logs | Envoy proxy stdout when access logging is enabled | Per-request evidence for routing, status, latency, and policy debugging | Excellent incident evidence; complements metrics | Disabled by default in many deployments; mixed formats if customized [S7] | - -### Best-practice inference -- "No data" for `istio_*` often means telemetry policy, scraping, or workload injection problems before it means healthy zero. -- gRPC message counters and some labels are protocol-specific; their absence can be expected on non-gRPC traffic. -- Ambient-mode deployments may expose different operational pivot points than sidecar mode, so Stage 2 should verify which dimensions actually appear in Tsuga. - -## Log intelligence (Stage 4 handoff) - -### Confirmed by sources - -| Source | Access path | Typical format | Structured vs unstructured | Evidence | -|---|---|---|---|---| -| Sidecar or gateway Envoy access logs | Proxy container stdout via `kubectl logs` or centralized log collection | Envoy text access log line | Unstructured text by default | Istio access-log task says Envoy proxies print access information to standard output and shows the default format [S7] | -| Customized Envoy access logs | Telemetry API or mesh config | Custom text or JSON depending on operator configuration | Structured or unstructured | Istio access-log task documents Telemetry API and MeshConfig based customization [S7] | - -Known log formats: -- **Envoy default access log** - - Sample line: `[2020-04-03T04:57:50.690Z] "GET /productpage HTTP/1.1" 200 - "-" "-" 0 612 90 89 "10.244.0.8" "curl/7.64.1" "6fcb..." "productpage:9080" "127.0.0.1:9080" outbound|9080||productpage.default.svc.cluster.local 127.0.0.1:57620 10.96.55.58:9080 10.244.0.11:9080 inbound|9080|| 10.244.0.11:41890 10.244.0.11:9080 10.244.0.8:0 outbound_.9080_._.productpage.default.svc.cluster.local default` - - Delimiter and shape notes: whitespace-delimited line with quoted request and user-agent fields, then upstream cluster and address tokens. - - Timestamp pattern: bracketed ISO-8601 at line start. - - Quoting behavior: request line, referer, user agent, request id, authority, and upstream host fields are quoted. - - Optional fields: response flags, route name, and cluster tokens vary by configuration. - -Candidate query filters for Stage 4: -- Precise: `context.service.name:istio-proxy AND context.log_name:access` - - Rationale: likely to isolate centralized access logs if the collector preserves service and log stream names. - - Risk: may miss logs if the org uses a different `context.service.name` mapping. -- Fallback: `message:*HTTP/1.* AND (message:*outbound|* OR message:*inbound|*)` - - Rationale: catches default Envoy access-log lines even when enrichment is sparse. - - Risk: text-shape filter can match non-Istio Envoy logs or partially customized formats. - -Attribute mapping hints: - -| Raw field | Suggested Tsuga key | Confidence | Notes | -|---|---|---|---| -| timestamp | `timestamp` | High | Parse bracketed ISO timestamp directly | -| method | `http.method` | High | Taken from quoted request segment | -| path | `http.url` | High | Preserve raw path first; URL parsing can happen later | -| protocol | `http.version` | High | Usually `HTTP/1.1` or `HTTP/2` | -| status | `http.status_code` | High | Numeric coercion required | -| duration | `http.latency` | Medium | Verify unit in emitted format before normalization | -| request id | `http.request_id` | Medium | Present in default format but may be empty | -| authority | `server.address` | Medium | Quoted authority can be host or host:port | -| upstream cluster | `istio.upstream_cluster` | High | Useful for route and service attribution | -| response flags | `envoy.response_flags` | High | Keep raw flags rather than over-normalizing | - -Parsing risks: -- Customized Telemetry API access-log formats can invalidate default-token Grok patterns. -- Envoy default format contains quoted segments and bare tokens intermixed, so naive split-on-space parsing breaks. -- Some deployments emit JSON access logs instead of text. -- Duration unit is easy to mis-assume; Stage 4 should verify against current format before setting a normalizer. -- Ingress and sidecar access logs may share shape but differ operationally. - -### Best-practice inference -- If logs are centralized with Kubernetes enrichment, expect namespace and pod dimensions to be easier to rely on than Istio-specific semantic keys. -- Keep a fallback parser path for host-only vs host:port authority tokens. -- Treat access logs as optional evidence, not guaranteed coverage. - -## Caveats and footguns -- **[traffic-volume]** Do not sum source and destination reporters together for the same KPI. Pick one default reporter, usually destination, or the dashboard will double count. [S1][S4] -- **[errors-failures]** `istio_requests_total` can show failures caused by policy, upstream app errors, or gateway problems; response code alone is not enough for blame. [S1][S6] -- **[latency-performance]** `istio_request_duration_milliseconds` is a histogram. Use percentiles or carefully chosen summary views, not naive arithmetic across percentiles. [S1] -- **[latency-performance]** Request duration unit is milliseconds, not seconds. Normalizer mistakes can silently produce misleading charts. [S1] -- **[traffic-volume, saturation-capacity]** Request and response byte histograms explain payload shape, but they are not always the best primary "throughput" KPI. [S1][S4] -- **[security-identity]** `connection_security_policy` can be absent if telemetry customization removed it or if the integration path does not preserve the label. (Inference) -- **[security-identity]** A drop in mTLS-labeled traffic can be either a regression or an intentional plaintext exception. Pair security dashboards with current policy context. (Inference) -- **[availability-health]** Missing mesh metrics may indicate workloads are not injected or not scraped, not that the service is healthy. [S8][S9] -- **[availability-health]** Gateway-heavy traffic can dominate user-visible behavior; service-level dashboards can look healthy while ingress is broken. (Inference) -- **[traffic-volume]** `source_workload` and `destination_workload` are useful but can churn during rollouts; canonical service is usually safer for high-level comparisons. [S4] -- **[traffic-volume, errors-failures]** Full destination service FQDNs can be noisy in legends. Prefer canonical service or shorter service labels when available. (Inference) -- **[errors-failures]** gRPC failures may not map cleanly to HTTP 5xx expectations. Keep a grpc-status breakdown when the protocol mix is meaningful. [S1][S4] -- **[saturation-capacity]** TCP metrics are reported on a timer for active connections and at connection end, so low-volume services can look jagged. [S5] -- **[saturation-capacity]** Opened vs closed TCP connections over a short window is a churn indicator, not a literal concurrency gauge. (Inference) -- **[latency-performance, saturation-capacity]** Large request or response sizes can drive latency without error spikes. Keep payload views near latency widgets. (Inference) -- **[availability-health, security-identity]** Telemetry API changes can remove labels that dashboards depend on without changing metric names. [S2][S6] -- **[security-identity]** Ambient mode and waypoint proxies may change the practical meaning of sidecar-era ownership fields. (Inference) -- **[availability-health]** `reporter` is low cardinality and safe; pod names and principals are not. Avoid pod-level timeseries in the overview. (Inference) -- **[errors-failures, traffic-volume]** Query-value widgets should stay aggregated and ungroupped. Use top-lists for "who is impacted" instead of forcing QV semantics. (Inference from Tsuga rules) -- **[traffic-volume, errors-failures, latency-performance]** If the org normalizes Prometheus names to dotted Tsuga names in Stage 2, patch all references consistently; partial renames will break formulas. (Inference) -- **[availability-health, security-identity]** Access logs may be disabled even when metrics work. Stage 4 must not assume logs exist just because metrics do. [S7] - -## Confirmed Tsuga prefixes -- `istio_*` — **CONFIRMED** (7 live metrics found in Tsuga via MCP on April 2, 2026: `istio_build`, request, duration, size, and gRPC message families) -- `pilot_*` — **CONFIRMED** (28 live metrics found in Tsuga via MCP on April 2, 2026; control-plane coverage is real and can support secondary diagnostics) - -## Discovery status -- Discovery: completed in Stage 2 via Tsuga MCP plus catalog bootstrap. -- Confirmed live `istio_*` metrics: `istio_build`, `istio_requests_total`, `istio_request_duration_milliseconds`, `istio_request_bytes`, `istio_response_bytes`, `istio_request_messages_total`, `istio_response_messages_total`. -- Confirmed live `pilot_*` metrics: 28, including push, convergence, queue, endpoint-readiness, and routing-conflict families. -- Notable gap: the documented Istio TCP metric families are not currently present in Tsuga, so the Stage 2 baseline was re-centered on request-path and pilot control-plane metrics. - -## Top sources -- [S1] https://istio.io/latest/docs/reference/config/metrics/ — canonical list of Istio standard metrics and label semantics. -- [S2] https://istio.io/latest/docs/reference/config/telemetry/ — Telemetry API reference explaining how metrics and logs can be altered. -- [S3] https://istio.io/latest/docs/overview/what-is-istio/ — concise product-level explanation of Istio’s operational role. -- [S4] https://istio.io/latest/docs/tasks/observability/metrics/querying-metrics/ — shows real Prometheus queries and example `istio_requests_total` usage. -- [S5] https://istio.io/latest/docs/tasks/observability/metrics/tcp-metrics/ — confirms TCP metric names and reporting behavior. -- [S6] https://istio.io/latest/docs/tasks/observability/metrics/customize-metrics/ — confirms that operators can alter metric shape and labels. -- [S7] https://istio.io/latest/docs/tasks/observability/logs/access-log/ — canonical Istio access-log behavior and default format guidance. -- [S8] https://istio.io/latest/docs/tasks/observability/metrics/ — observability task index anchoring the supported metrics workflows. -- [S9] https://istio.io/latest/docs/tasks/observability/metrics/secure-metrics/ — secure scraping path and reasons mesh metrics can disappear operationally. -- [S10] https://istio.io/latest/docs/ops/deployment/performance-and-scalability/ — deployment-level performance guidance used to shape saturation and blast-radius assumptions. diff --git a/plugins/tsuga/skills/knowledge-technology/references/istio/metrics.csv b/plugins/tsuga/skills/knowledge-technology/references/istio/metrics.csv deleted file mode 100644 index f2e90f4..0000000 --- a/plugins/tsuga/skills/knowledge-technology/references/istio/metrics.csv +++ /dev/null @@ -1,14 +0,0 @@ -theme,name,confirmed,definition,unit,type,tsuga_metric_name,widget_suggestions,aggregation,post_function,group_by,required_context_fields,optional_context_fields,absence_means,cardinality_footguns,variants,citations -Throughput/Usage,istio_requests_total,true,Total service requests seen by Istio and the main denominator for request-based health views,{request},counter,istio_requests_total,query-value|timeseries|top-list,sum,rate,context.destination_service:20|context.destination_canonical_service:15|context.destination_service_namespace:20,context.env|context.team,context.reporter|context.destination_service|context.response_code|context.connection_security_policy,No request traffic or request telemetry missing,double-count risk if source and destination reporter are mixed,prom=istio_requests_total,https://istio.io/latest/docs/reference/config/metrics/|https://istio.io/latest/docs/tasks/observability/metrics/querying-metrics/ -Latency/Performance,istio_request_duration_milliseconds,true,Request latency histogram in milliseconds for mesh traffic used for p95 and mean latency views,ms,histogram,istio_request_duration_milliseconds,query-value|timeseries|top-list,percentile(p95),none,context.destination_service:20|context.destination_service_namespace:20,context.env|context.team,context.reporter|context.destination_service|context.request_protocol,Latency histogram not scraped or service not receiving traffic,do not group by destination workload and destination service together on overview charts,prom=istio_request_duration_milliseconds,https://istio.io/latest/docs/reference/config/metrics/ -Throughput/Usage,istio_request_bytes,true,Histogram of request payload sizes in bytes used to explain traffic mix and latency changes,By,histogram,istio_request_bytes,query-value|timeseries|top-list,average,none,context.destination_service:20|context.destination_service_namespace:20,context.env|context.team,context.reporter|context.request_protocol,Request-size telemetry missing or protocol mix not represented,high-cardinality service plus app breakdowns can clutter charts,prom=istio_request_bytes,https://istio.io/latest/docs/reference/config/metrics/ -Throughput/Usage,istio_response_bytes,true,Histogram of response payload sizes in bytes used to identify heavy responders and egress-heavy services,By,histogram,istio_response_bytes,query-value|timeseries|top-list,average,none,context.destination_service:20|context.destination_service_namespace:20,context.env|context.team,context.reporter|context.request_protocol,Response-size telemetry missing or protocol mix not represented,avoid layering response size with response code and workload in the same chart,prom=istio_response_bytes,https://istio.io/latest/docs/reference/config/metrics/ -Throughput/Usage,istio_request_messages_total,true,Total gRPC request messages sent from the client side of a gRPC exchange,{message},counter,istio_request_messages_total,query-value|timeseries|top-list,sum,rate,context.destination_service:20|context.destination_service_namespace:20,context.env|context.team,context.reporter|context.destination_service,Either no gRPC traffic or gRPC-specific metric absent on this path,group only by bounded destination service or namespace keys,prom=istio_request_messages_total,https://istio.io/latest/docs/reference/config/telemetry/ -Throughput/Usage,istio_response_messages_total,true,Total gRPC response messages sent from the server side of a gRPC exchange,{message},counter,istio_response_messages_total,query-value|timeseries|top-list,sum,rate,context.destination_service:20|context.destination_service_namespace:20,context.env|context.team,context.reporter|context.destination_service,Either no gRPC traffic or gRPC-specific metric absent on this path,group only by bounded destination service or namespace keys,prom=istio_response_messages_total,https://istio.io/latest/docs/reference/config/telemetry/ -Availability/Health,pilot_endpoint_not_ready,true,Number of endpoints known to Pilot that are not ready and may reduce destination reachability,{endpoint},gauge,pilot_endpoint_not_ready,query-value|timeseries,max,none,,context.env|context.team,context.service.name|context.telemetry_source,No current endpoint-readiness issue or pilot telemetry absent,Safe,prom=pilot_endpoint_not_ready,https://istio.io/latest/docs/ops/deployment/performance-and-scalability/ -Availability/Health,pilot_dns_cluster_without_endpoints,true,Count of DNS-discovered clusters with no endpoints available to Pilot,{cluster},gauge,pilot_dns_cluster_without_endpoints,query-value|timeseries,max,none,,context.env|context.team,context.service.name|context.telemetry_source,No currently empty DNS-backed cluster set or no pilot telemetry,Safe,prom=pilot_dns_cluster_without_endpoints,https://istio.io/latest/docs/ops/deployment/performance-and-scalability/ -Capacity/Saturation,pilot_xds_pushes,true,Total XDS pushes emitted by Pilot and a useful measure of control-plane churn,{push},counter,pilot_xds_pushes,query-value|timeseries|bar,sum,rate,context.type:10,context.env|context.team,context.type|context.service.name,No recent push activity or pilot telemetry missing,type dimension can vary by push family but remains bounded,prom=pilot_xds_pushes,https://istio.io/latest/docs/ops/deployment/performance-and-scalability/ -Capacity/Saturation,pilot_push_triggers,true,Total push-trigger events recorded by Pilot showing why control-plane recalculations are happening,{trigger},counter,pilot_push_triggers,query-value|timeseries|bar,sum,rate,context.type:10,context.env|context.team,context.type|context.service.name,No recent trigger activity or pilot telemetry missing,Safe,prom=pilot_push_triggers,https://istio.io/latest/docs/ops/deployment/performance-and-scalability/ -Latency/Performance,pilot_proxy_convergence_time,true,Histogram of how long proxies take to converge on a pushed configuration,ms,histogram,pilot_proxy_convergence_time,query-value|timeseries,percentile(p95),none,,context.env|context.team,context.service.name|context.telemetry_source,No recent convergence samples or no pilot telemetry,Safe,prom=pilot_proxy_convergence_time,https://istio.io/latest/docs/ops/deployment/performance-and-scalability/ -Latency/Performance,pilot_proxy_queue_time,true,Histogram of how long proxy update work waits in Pilot before processing,ms,histogram,pilot_proxy_queue_time,query-value|timeseries,percentile(p95),none,,context.env|context.team,context.service.name|context.telemetry_source,No recent queue-time samples or no pilot telemetry,Safe,prom=pilot_proxy_queue_time,https://istio.io/latest/docs/ops/deployment/performance-and-scalability/ -Cost,NOT_APPLICABLE,true,Direct cost metrics are not part of Istio standard telemetry so request and payload metrics remain the best proxy signals for mesh cost pressure,Unknown,Unknown,NOT_APPLICABLE,query-value|timeseries,Unknown,none,,context.env|context.team,context.destination_service|context.destination_service_namespace,No direct cost series should be expected,Safe,proxy=istio_request_bytes|istio_response_bytes|istio_requests_total,https://istio.io/latest/docs/reference/config/metrics/ diff --git a/plugins/tsuga/skills/knowledge-technology/references/istio/queries.md b/plugins/tsuga/skills/knowledge-technology/references/istio/queries.md deleted file mode 100644 index 6ac5c38..0000000 --- a/plugins/tsuga/skills/knowledge-technology/references/istio/queries.md +++ /dev/null @@ -1,54 +0,0 @@ -# Istio - -Service mesh: Istiod control plane + Envoy sidecar data plane. Data-plane metrics are generated by the sidecars. Healthy: low error rate, bounded request duration, fast config push, no endpoint gaps. - -Also load `envoy.md` for data-plane detail. - -## Incident shapes - -- **Inter-service 5xx spike** — `istio_requests_total{response_code="5xx"}` rises → usually target workload, not mesh -- **Config push lag** — `pilot_proxy_convergence_time` p99 spikes → sidecars slow to apply; stale routing -- **Endpoint discovery gap** — `pilot_endpoint_not_ready` / `dns_cluster_without_endpoints` nonzero → service has no healthy pods -- **Request duration regression** — `istio_request_duration_milliseconds` p95 climbs → downstream slow or mesh overhead -- **Sidecar down / unready** — traffic bypasses mesh or fails entirely - -## Key metrics - -| Metric | Unit | Signal | -|---|---|---| -| `istio_requests_total` | count | Per source/destination/response_code | -| `istio_request_duration_milliseconds` | ms | Latency histogram | -| `istio_request_bytes` / `response_bytes` | bytes | Payload sizes | -| `istio_request_messages_total` / `response_messages_total` | count | gRPC streaming | -| `pilot_endpoint_not_ready` | count | Endpoints not-ready at push time | -| `pilot_dns_cluster_without_endpoints` | count | Clusters with no resolved endpoints | -| `pilot_xds_pushes` | count | Config pushes | -| `pilot_push_triggers` | count | Push-trigger events | -| `pilot_proxy_convergence_time` | ms | Sidecar convergence latency | -| `pilot_proxy_queue_time` | ms | Queue before push | - -## Derived signals - -- Error rate by (source_workload, destination_workload) — localizes caller/callee pair. -- `p99 / p50` latency per destination — widening gap = long-tail issues. -- `convergence_time` p99 — high = control plane overloaded, stale config risk. -- `1 - (dns_cluster_without_endpoints / total_clusters)` — endpoint health fraction. - -## Log patterns - -Sidecar access-log flags — see `envoy.md` (`UH`, `NR`, `UC`, `UT`, `DC`). - -Istiod: - -- `Handled resource ... push` — config push events -- `push context pre-compute timed out` — control-plane overload -- `endpoint not ready` — discovery lagging -- mTLS STRICT mode denials — auth rejections - -## Gotchas - -- `istio_requests_total` cardinality is high (source × destination × namespace × response_code). Aggregate carefully. -- `reporter` label distinguishes sidecar: `source` = caller's sidecar, `destination` = callee's. 5xx with `reporter=source` only = target never received. -- mTLS PERMISSIVE → STRICT transitions can silently break non-mesh clients. Check `response_flags` for `AuthenticationPolicyRejected`. -- Gateway / VirtualService changes apply asynchronously; "new config not working" is often a convergence-time issue. -- Sidecar injection is namespace-controlled. Workloads without a sidecar don't appear in `istio_*` metrics — silently outside the mesh. diff --git a/plugins/tsuga/skills/knowledge-technology/references/jvm/README.md b/plugins/tsuga/skills/knowledge-technology/references/jvm/README.md deleted file mode 100644 index c288bbf..0000000 --- a/plugins/tsuga/skills/knowledge-technology/references/jvm/README.md +++ /dev/null @@ -1,410 +0,0 @@ -# JVM metrics Integration Context Bundle - -## Metadata - -- **Technology:** JVM metrics (OpenTelemetry `jvm.*` semantic conventions) -- **Deployment:** self-hosted (runtime embedded in Java applications) -- **Environment:** prod -- **Persona:** SRE Dev and ops -- **Telemetry preference:** mixed (OTel Java agent primary; JMX exporter fallback) -- **Integration scope:** core service only -- **Primary use-case:** reliability and performance -- **Confirmed namespace:** `jvm.` (Stage 2 verified — 12 metrics confirmed, 8 Development-status metrics absent) - ---- - -## How to use this bundle - -- `01_jvm-metrics_metrics.csv` — Source of truth for all metrics: names, types, units, safe aggregations, post-functions, and group-by dimensions. -- `02_jvm-metrics_dashboard_plan.yaml` — Dashboard structure: sections, widgets, derived signals (formulas), explanation notes, triage chains, and playbooks. -- `03_jvm-metrics_state.yaml` — Machine-readable stage status, assumptions, unknowns (especially OTel attribute field names in Tsuga), and open items Stage 2 must resolve. -- `04_jvm-metrics_memory.md` — Human-readable narrative of Stage 1 decisions, tradeoffs, and Stage 2 priorities. -- `05_jvm-metrics_metric_catalog.csv` — Stage 2 generated catalog: 12 confirmed metrics, pruned attribute keys, curated descriptions. -- Stage 4 should read the "Log intelligence (Stage 4 handoff)" section below and `03_jvm-metrics_state.yaml → log_intel` before creating log routes. - ---- - -## What it is and what "good" looks like - -The JVM (Java Virtual Machine) is the managed runtime environment that executes Java, Kotlin, Scala, and Clojure applications. Unlike service-level metrics (HTTP request rate, DB query latency), JVM metrics describe the **runtime health of the process itself**: memory pressure, garbage collection behavior, CPU time consumed, and thread utilization. Every Java service running on a JVM emits these metrics. - -**What "good" looks like:** -- Heap utilization: 40–70% in steady state with a visible saw-tooth GC pattern (allocated then freed) -- Post-GC live set: stable over time; growing baseline = active memory leak -- GC pause p99: < 100 ms for G1/ZGC/Shenandoah minor GC; zero or near-zero major (full) GC events -- Blocked threads: < 2% of total threads; no sustained climbing trend -- JVM CPU utilization: < 80% sustained; spikes during GC are expected -- Metaspace: stable after JVM warmup (first 2–5 minutes); not approaching MaxMetaspaceSize limit - -**Top 3 incident shapes:** -1. **Memory Leak / Heap Exhaustion** — Heap utilization climbs gradually, post-GC live set grows between restarts, GC rate increases but frees less each cycle → first section: `heap-memory` + `gc-pressure` -2. **GC Storm / Stop-the-World Freeze** — GC pause p99 spikes, major GC events appear, application latency jumps, CPU spikes during GC cycles → first section: `gc-pressure` + `cpu-utilization` -3. **Thread Contention / Deadlock** — Blocked thread count grows, runnable count flat, CPU low (threads waiting, not working), application throughput drops → first section: `threads` - -**Confirmed by sources:** OTel semantic conventions define the exact metric names, types, and attributes. JVM MXBean API defines the pool name values. [https://opentelemetry.io/docs/specs/semconv/runtime/jvm-metrics/] - -**Best-practice inference:** The "good" thresholds above are inferred from operational experience with G1 GC. ZGC and Shenandoah have sub-millisecond stop-the-world targets; adjust thresholds for those collectors accordingly. - ---- - -## Key concepts - -### Glossary - -| Term | Definition | Operational meaning | Dashboard section | -|---|---|---|---| -| Heap | JVM-managed memory region for object allocation; bounded by -Xmx | Primary memory resource; exhaustion = OOM | heap-memory | -| Non-Heap | JVM memory outside the heap: Metaspace, CodeCache, Compressed Class Space | Grows with class loading and JIT compilation | non-heap-buffers | -| GC (Garbage Collection) | Automatic reclamation of unreachable heap objects | Pauses application threads (STW) or runs concurrently | gc-pressure | -| Minor GC | GC event collecting the young generation (Eden + Survivor spaces) | Frequent, short; expected multiple times per minute | gc-pressure | -| Major GC / Full GC | GC event collecting the old generation (and sometimes the whole heap) | Infrequent, long; even one per hour for G1 is a warning sign | gc-pressure | -| Stop-the-World (STW) | GC phase during which all application threads are paused | Directly adds latency to all in-flight requests | gc-pressure | -| Eden Space | Young generation area where new objects are first allocated | High throughput here is normal; rapid exhaustion = high allocation rate | heap-memory | -| Survivor Space | Young generation holding objects that survived at least one minor GC | Growing survivors = objects living longer than expected | heap-memory | -| Old Gen | Long-lived objects promoted from young generation | Growing Old Gen after GC = live set growth or leak | heap-memory | -| Metaspace | Non-heap region holding class metadata (replaces PermGen since Java 8) | Unlimited by default; climbing = classloader leak | non-heap-buffers | -| CodeCache | Non-heap region holding JIT-compiled native code | Exhaustion disables JIT, causing severe CPU regression | non-heap-buffers | -| UpDownCounter | OTel instrument type representing a current value that can go up or down | Treated as gauge in Tsuga; do NOT apply per-second post-function | heap-memory, threads | -| Live Set | The actual amount of memory in use by live objects, measured post-GC | Best predictor of heap headroom; use jvm.memory.used_after_last_gc | heap-memory | -| G1 GC | Region-based garbage collector (default since Java 9); targets pause time goals | Separate minor (Young Gen) and major (Old Gen) GC events | gc-pressure | -| ZGC | Ultra-low-latency GC (Java 11+); sub-millisecond pauses for terabyte heaps | "Pause" metrics show very low values; "cycle" metrics show total concurrent work | gc-pressure | -| Shenandoah | Low-latency GC (Red Hat); concurrent evacuation | Similar to ZGC in pause profile; different pool names | gc-pressure | -| Direct Buffer | NIO ByteBuffer allocated outside heap via allocateDirect() | Off-heap; NOT visible in heap metrics; leaks cause native OOM | non-heap-buffers | -| Mapped Buffer | Memory-mapped files via FileChannel.map() | Used by RocksDB, Lucene, Chronicle Map; large allocations are expected | non-heap-buffers | -| Thread State | Java thread lifecycle state: runnable, blocked, waiting, timed_waiting, new, terminated | Blocked = contention; waiting = idle pool threads (normal) | threads | -| Platform Thread | OS-managed Java thread (1:1 with OS thread); counted by jvm.thread.count | Java 21+ virtual threads are NOT counted here | threads | -| Virtual Thread | Lightweight thread (Project Loom, Java 21+); NOT measured by jvm.thread.count | If your service uses virtual threads, jvm.thread.count is misleading | threads | -| Allocation Rate | Rate at which new objects are created in Eden Space | High allocation rate = frequent minor GC; not measured directly in OTel jvm.* | heap-memory | - -### Concept Map - -``` -Client requests → Application threads (runnable) → allocate objects → Eden Space (heap) -Eden Space → fills up → triggers Minor GC → evacuates to Survivor Space or Old Gen -Survivor Space → fills beyond threshold → promotes to Old Gen -Old Gen → fills up → triggers Major GC → stop-the-world pause → application latency spikes -Major GC → if heap 100% full → OutOfMemoryError (heap) → JVM crash or OOM kill - -JVM process → executes bytecode → JIT compiler → compiles hot paths → stores in CodeCache -CodeCache → fills up → JIT disabled → CPU spike (interpreted execution is 10-100x slower) - -Class loader → loads new class definitions → stores metadata in Metaspace -Metaspace → grows with class count → if MaxMetaspaceSize hit → OutOfMemoryError (Metaspace) -Classloader leak → orphaned classloaders not GC'd → Metaspace climbs after each redeploy - -Application threads → request JVM CPU time → jvm.cpu.recent_utilization rises -GC threads → also consume CPU → GC overhead = cpu time during STW + concurrent GC -GC overhead > 98% (GCOverheadLimitExceeded) → JVM throws OOM as safety valve - -Netty/gRPC/Kafka → allocate direct ByteBuffers → off-heap (not in jvm.memory.used[heap]) -Direct buffer leak → jvm.buffer.memory.used[direct] climbs → native OOM → crash without heap warning - -Thread pool → creates platform threads → monitored by jvm.thread.count -Lock contention → threads block on synchronized → jvm.thread.count[blocked] rises -Blocked threads → CPU drops → throughput collapses → latency climbs -Thread deadlock → blocked count never recovers → requires thread dump analysis - -service.name (OTel resource) → maps to context.service.name in Tsuga → primary group-by -deployment.environment → maps to context.env → global filter -k8s.namespace.name → maps to context.k8s.namespace.name → secondary filter -jvm.memory.pool.name → metric attribute → context.jvm.memory.pool.name in Tsuga (CONFIRMED Stage 2) -``` - -### Entities and dimensions - -| Dimension | Why useful | Cardinality risk | Safe top-N | Do NOT group-by | -|---|---|---|---|---| -| `context.service.name` | Primary entity — one JVM per service instance | Medium (10s–100s of services) | top 10 | — | -| `context.env` | Separates prod vs staging JVM behavior | Low (2–5 values) | — (filter, not group-by) | — | -| `context.team` | Ownership routing | Low (10–30 teams) | top 10 | — | -| `context.k8s.cluster.name` | JVM behavior varies by cluster (node sizes, limits) | Low (2–10 clusters) | top 5 | — | -| `context.k8s.namespace.name` | Workload isolation boundary | Medium (10s–100s) | top 10 | — | -| `context.k8s.pod.name` | Individual pod JVM (noisy neighbors, OOM kills) | HIGH (1000s) | top 10 only in deep dive | Do NOT use in overview | -| `jvm.memory.pool.name` | Memory pool breakdown (Old Gen, Metaspace, etc.) | Low (5–8 values per GC) | all (< 10 values) | — | -| `jvm.memory.type` | Heap vs non-heap separation | Very low (2 values) | all | — | -| `jvm.gc.action` | Minor vs major GC distinction | Very low (2 values) | all | — | -| `jvm.gc.name` | Collector-specific GC event name | Low (2–3 values per JVM) | all | — | -| `jvm.gc.cause` | Why GC was triggered (Development status) | Low (3–10 values) | top 5 | — | -| `jvm.thread.state` | Thread lifecycle state | Very low (6 values) | all | — | -| `jvm.thread.daemon` | Daemon vs user thread split | Very low (2 values) | all | — | -| `jvm.buffer.pool.name` | Direct vs mapped buffer separation | Very low (2 values) | all | — | -| `process.pid` | Individual JVM process identity | HIGH (varies per pod restart) | — | Do NOT group-by — use service.name | - -**Note:** `jvm.memory.pool.name`, `jvm.gc.action`, `jvm.gc.name`, `jvm.thread.state`, `jvm.buffer.pool.name` are OTel metric-level attributes, NOT OTel resource attributes. **Stage 2 confirmed** their Tsuga field names are prefixed with `context.jvm.` (e.g., `context.jvm.memory.pool.name`). All 02 dashboard plan filters and group-by fields have been updated accordingly. - -### Tsuga field mapping - -**OTel resource attributes → context.\* fields (used in global filters and group-by):** - -| OTel resource attribute | Recommended context.\* key | Must-exist vs optional | -|---|---|---| -| `service.name` | `context.service.name` | Must-exist — primary JVM identity | -| `deployment.environment` | `context.env` | Must-exist (confirmed in .env) | -| `service.namespace` | `context.team` | Optional (confirmed in .env) | -| `k8s.cluster.name` | `context.k8s.cluster.name` | Optional (k8s deployments) | -| `k8s.namespace.name` | `context.k8s.namespace.name` | Optional (k8s deployments) | -| `k8s.pod.name` | `context.k8s.pod.name` | Optional (high cardinality — use carefully) | -| `k8s.node.name` | `context.k8s.node.name` | Optional (k8s deployments) | -| `host.name` | `context.host.name` | Optional (non-k8s deployments) | -| `process.pid` | Unknown — likely `context.process.pid` | Do NOT group-by (high cardinality) | - -**OTel metric-level attributes → metric dimension filters (CONFIRMED Stage 2):** - -| OTel metric attribute | OTel metric(s) | Tsuga field (confirmed) | Notes | -|---|---|---|---| -| `jvm.memory.pool.name` | jvm.memory.* | `context.jvm.memory.pool.name` | GC-specific values (G1 Eden Space, Metaspace, etc.) — pool names not yet enumerated | -| `jvm.memory.type` | jvm.memory.* | `context.jvm.memory.type` | `heap` or `non_heap` | -| `jvm.gc.action` | jvm.gc.duration | `context.jvm.gc.action` | `end of minor GC` or `end of major GC` | -| `jvm.gc.name` | jvm.gc.duration | `context.jvm.gc.name` | GC-specific (G1 Young Generation, etc.) | -| `jvm.gc.cause` | jvm.gc.duration | Not confirmed (Development status — metric absent) | Not enumerated | -| `jvm.thread.state` | jvm.thread.count | `context.jvm.thread.state` | runnable, blocked, waiting, timed_waiting, new, terminated | -| `jvm.thread.daemon` | jvm.thread.count | `context.jvm.thread.daemon` | `true` or `false` | -| `jvm.buffer.pool.name` | jvm.buffer.* | `context.jvm.buffer.pool.name` | `direct` or `mapped` (Development-status metrics — absent in this env) | - -**Confirmed Stage 2:** All metric attribute field names confirmed via `tsuga_build_metric_catalog.py` discovery. `context.service.name` confirmed present on all 12 metrics. `context.k8s.*` fields confirmed present. All 02 dashboard plan filters/group-by fields updated to use `context.jvm.*` prefix. - ---- - -## Golden signals - -### Traffic (proxy: active thread count + class loading rate) - -JVM metrics do not directly measure application traffic (HTTP requests, DB queries — those are application-layer). The best traffic proxies are: -- **Runnable thread count** (`jvm.thread.count[runnable]`): threads actively doing work scale with load -- **Class load rate** (`jvm.class.loaded` rate): spikes during deployments or hot-reload events - -**Typical causes of degradation:** thread pool exhaustion under load, class loading storms (dynamic proxies, CGLIB), blocked threads reducing effective throughput. - -**What people escalate on:** "API is timing out but CPU looks normal" = likely blocked threads stealing capacity from runnable threads. - -**Section questions this answers:** Threads by State, Total Threads trend. - -**Confirmed by sources:** Thread state semantics from JVM ThreadMXBean [https://docs.oracle.com/en/java/docs/api/java.management/java/lang/management/ThreadMXBean.html] -**Best-practice inference:** Using runnable thread count as a traffic proxy is an industry convention for JVM observability. - -### Errors (heap OOM, metaspace OOM, GC overhead exceeded) - -JVM "errors" are resource exhaustion events, not application errors: -- **Heap OOM**: `jvm.memory.used[heap]` ≈ `jvm.memory.limit[heap]` AND GC rate maxing out with no relief -- **Metaspace OOM**: `jvm.memory.used[Metaspace]` ≈ `jvm.memory.committed[Metaspace]` climbing without bound -- **GC Overhead Limit Exceeded**: JVM safety valve — throws OOM when > 98% of CPU time is spent in GC; hard to detect before the crash - -**Typical causes:** Memory leaks (retained references), classloader leaks (redeploy without full GC), direct buffer leaks (Netty), unbounded cache growth. - -**What people escalate on:** "JVM OOM killed" alerts from k8s; these dashboards help identify which exhaustion type caused the kill. - -**Section questions this answers:** Heap Utilization (%), Old Gen Pressure (%), Metaspace Pressure (%), GC Pause p95. - -**Confirmed by sources:** Java OOM error types [https://docs.oracle.com/en/java/docs/api/java.base/java/lang/OutOfMemoryError.html] -**Best-practice inference:** GC overhead threshold of 98% is a JVM default; actual OOM timing depends on heap size and allocation rate. - -### Latency (GC pause duration) - -GC stop-the-world pauses are direct application latency additions. During a major GC pause, ALL application threads are frozen. A 500ms full GC = 500ms added to every in-flight request's latency. - -- **Primary signal:** `jvm.gc.duration` percentile(p95/p99) filtered to `jvm.gc.action=end of major GC` -- **Secondary:** p95 of ALL GC pauses (including minor) for latency baseline - -**What people escalate on:** p99 latency SLO breach coinciding with GC pause spikes (correlation usually visible with 1–2min time alignment). - -**Section questions this answers:** GC Pause p95, Major GC Time Ratio (%), GC Duration by Collector. - -**Confirmed by sources:** OTel jvm.gc.duration semantics [https://opentelemetry.io/docs/specs/semconv/runtime/jvm-metrics/] -**Best-practice inference:** The correlation between GC pauses and application latency SLOs is documented in G1 GC tuning guides [https://www.oracle.com/technical-resources/articles/java/g1gc.html] - -### Saturation (memory %, threads, CPU, file descriptors) - -JVM saturation is multi-dimensional: -- **Heap saturation**: Heap Utilization (%) approaching 85–90% after GC = imminent OOM -- **Thread saturation**: Blocked Thread Ratio (%) > 5% sustained = lock contention -- **CPU saturation**: JVM CPU Utilization > 80% sustained (especially if driven by GC) -- **File descriptor saturation**: File Descriptor Utilization (%) > 80% = risk of "Too many open files" -- **Direct buffer saturation**: Direct Buffer Utilization (%) > 80% = risk of native OOM - -**Typical causes:** Undersized heap, insufficient thread pools, CPU-intensive computation without parallelism limits, FD leaks in frameworks. - -**What people escalate on:** k8s OOM kills (heap or native), application 503s from thread exhaustion. - -**Section questions this answers:** All sections — saturation is cross-cutting. - -**Confirmed by sources:** JVM tuning literature; OTel jvm.cpu.recent_utilization semantics. -**Best-practice inference:** Thresholds are industry conventions; tune per workload. - ---- - -## Telemetry sources - -| Source type | How collected | What it provides | Pros | Cons | Common pitfalls | -|---|---|---|---|---|---| -| OTel Java Agent (automatic) | Attach `-javaagent:opentelemetry-javaagent.jar` | All `jvm.*` semconv metrics auto-collected from MXBeans | Zero code change; covers full jvm.* namespace; actively maintained | Requires agent attachment; small overhead (~1% CPU) | Agent version matters: `jvm.*` namespace only available in agent >= 2.0 (semconv 1.21+). Older agents emit `process.runtime.jvm.*` — check version! | -| OTel Java SDK (manual) | Instrument code with OTel SDK; call JvmObservabilityInstrumenter | Same metrics as agent but with manual control | Fine-grained control | Developer effort; easy to miss metrics | Development-stability metrics (buffer, file_descriptor) may not be enabled by default in SDK | -| JMX Exporter (Prometheus) | Run as javaagent or separate process; scrape /metrics | JVM MXBean metrics but with Prometheus naming | Widely used; good k8s support | Different metric names (`process_runtime_jvm_*`); requires prometheus_jmx_exporter | Metric names differ from OTel semconv — dashboards built for `jvm.*` will not find JMX exporter data | -| OTel JMX Metrics Receiver | OTEL Collector JMX receiver; scrapes JVM MXBeans | `jvm.*` namespace compliant | Works with OTel Collector pipelines | Requires JMX port exposure; additional config | JMX port exposure is a security concern in some environments | -| Spring Boot Actuator + Micrometer | Spring apps; exposes /actuator/metrics | JVM metrics via Micrometer (mapped to OTel on export) | Native to Spring; no agent needed | Micrometer ↔ OTel name mapping may introduce drift | Some Micrometer metric names differ from OTel semconv; verify `jvm.*` naming in your OTel exporter | -| "No data" diagnosis | — | — | — | — | Check: (1) agent version (need >= 2.0), (2) OTEL_METRICS_EXPORTER env var set, (3) correct namespace in collector routing, (4) Development-status metrics may be disabled by feature flag | - -**Confirmed by sources:** Agent capabilities [https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/instrumentation/runtime-telemetry/runtime-telemetry-java8/library/README.md] -**Best-practice inference:** JMX Exporter as fallback, feature flag status for Development metrics. - ---- - -## Log intelligence (Stage 4 handoff) - -### Confirmed by sources - -The JVM itself produces several distinct log streams. Java application logs are app-specific (format defined by the application), but JVM-generated logs have known formats. - -**Log sources matrix:** - -| Source | Access path | Typical format | Structured vs unstructured | Evidence | -|---|---|---|---|---| -| GC Logs | `-Xlog:gc*:file=gc.log` JVM flag | GC-specific structured text | Semi-structured (GC-specific fields) | [https://openjdk.org/jeps/271] | -| JVM crash logs | `hs_err_pid.log` in working dir | Fixed-section free text | Unstructured | OpenJDK crash report format | -| Application logs | Application-defined (log4j, logback, slf4j) | JSON, logfmt, or plain text | Varies by framework config | — | -| k8s stdout/stderr | Pod stdout captured by container runtime | Application-dependent | Depends on logger config | — | - -**Known log formats:** - -**GC Log (Unified Logging, Java 9+, -Xlog:gc*):** -``` -[2024-01-15T10:23:45.123+0000][gc] GC(42) Pause Young (Normal) (G1 Evacuation Pause) 512M->256M(2048M) 45.234ms -[2024-01-15T10:23:50.456+0000][gc,heap] GC(42) Heap before GC invocations=42 (full 0): -[2024-01-15T10:24:00.789+0000][gc] GC(43) Pause Full (System.gc()) 1024M->512M(2048M) 1234.567ms -``` -- Timestamp: ISO 8601 with timezone offset -- GC number in parentheses: `GC(N)` -- Action: `Pause Young`, `Pause Full`, `Pause Remark`, etc. -- Memory: `before->after(max)` in MB -- Duration: milliseconds at end - -**Application Log (Logback JSON layout, common Spring Boot):** -```json -{"timestamp":"2024-01-15T10:23:45.123Z","level":"ERROR","logger":"c.e.MyService","message":"Connection pool exhausted","service":"my-service","trace":"abc123"} -``` - -### Best-practice inference - -Most Java services in k8s log to stdout as JSON (Spring Boot + Logback JSON encoder is the de facto standard). GC logs are usually written to a separate file and may not be in the k8s log stream unless explicitly redirected. - -**Candidate query filters for Stage 4:** -1. **Precise:** `context.service.name: AND source:stdout` — captures application logs for a specific service; rationale: JVM app logs go to stdout in k8s -2. **Broader fallback:** `context.k8s.namespace.name:` — captures all services in a namespace; risk: high volume, may include non-Java services - -**Attribute mapping hints:** - -| Raw field (logback JSON) | Suggested Tsuga key | Confidence | Notes | -|---|---|---|---| -| `timestamp` | `timestamp` | High | ISO 8601; parse as timestamp | -| `level` | `log.level` | High | Standard log level field | -| `logger` | `log.logger` | Medium | Logger class name | -| `message` | `message` | High | Log message body | -| `service` | `context.service.name` | Medium | May be `service.name` in OTel-enriched logs | -| `trace` | `trace.id` | Medium | Trace correlation ID | -| `thread` | `thread.name` | Medium | Thread name (logback default) | - -**Parsing risks:** -- Multiline: GC logs span multiple lines per GC event; application stack traces are multiline -- Format variability: Different Java frameworks use different JSON keys (`@timestamp` vs `timestamp`, `msg` vs `message`) -- No canonical log format: JVM metrics use OTel but logs use application-specific frameworks -- GC log rotation: GC log files are rotated separately from application logs; may not appear in k8s log stream - ---- - -## Caveats and footguns - -- **[heap-memory]** `jvm.memory.used` aggregated across ALL pools includes both heap and non-heap. Always filter by `jvm.memory.type:heap` for heap-specific widgets; without this filter, values are inflated. (Confirmed — OTel semconv attribute definition) - -- **[heap-memory]** `jvm.memory.limit` is the JVM-configured max heap (`-Xmx`), not total host memory. Heap Utilization (%) must compare `used[heap]` to `limit[heap]`, not to available RAM. (Confirmed — OTel semconv) - -- **[heap-memory]** Instantaneous `jvm.memory.used[heap]` oscillates naturally in a saw-tooth pattern between GC cycles. Point-in-time snapshots are noisy. Use a 5-minute average or look at the trend. The peak of each saw-tooth cycle should be your alert baseline. (Best-practice inference) - -- **[heap-memory]** `jvm.memory.used_after_last_gc` is the ground truth for live object size. Use it for leak detection, not `jvm.memory.used`. A service where `used_after_last_gc` grows monotonically between restarts has a memory leak. (Confirmed — OTel semconv: "Measure of memory used, after the most recent garbage collection event on this pool") - -- **[heap-memory]** Memory pool names are GC-specific and differ between collectors (G1: `G1 Eden Space`, `G1 Old Gen`; Parallel: `PS Eden Space`, `PS Old Gen`; ZGC: `ZHeap`). Widgets that filter by a specific pool name will show no data for services using a different GC. Gate accordingly or group-by pool name. (Confirmed — JVM MXBean API docs) - -- **[heap-memory, gc-pressure]** `jvm.memory.*` instruments are UpDownCounter (semantically a gauge). Do NOT apply `per-second` or `rate` post-functions to these metrics — they will produce nonsensical values. Use `sum` or `average` aggregation with `none` post-function. (Confirmed — OTel semconv instrument type) - -- **[gc-pressure]** `jvm.gc.duration` is a histogram. Percentile aggregations (`percentile(p95)`) require Tsuga histogram support. Verify in Stage 2 whether this metric appears with histogram bucket data or only as a scalar sum/count. (Best-practice inference — Tsuga-specific) - -- **[gc-pressure]** Minor GC (young generation) pauses are expected and frequent (every 1–30 seconds under normal load for G1). Do NOT alert on minor GC frequency alone. Alert on MAJOR GC pause duration and rate. (Confirmed — G1 GC tuning guide) - -- **[gc-pressure]** For ZGC and Shenandoah, "major GC" events (`jvm.gc.action=end of major GC`) represent concurrent cycles, not stop-the-world freezes. Their "major GC" pause times target < 1ms. Apply different alert thresholds for these collectors vs G1. (Confirmed — ZGC and Shenandoah documentation) - -- **[gc-pressure]** `jvm.gc.cause` attribute has Development stability status and may not be emitted by default in all OTel agent versions. Widgets that filter or group by GC cause must be gated on this attribute's availability. (Confirmed — OTel semconv stability marker) - -- **[gc-pressure]** A single G1 Full GC event (not G1 Young Generation — those are normal) is almost always a signal worth investigating. Even 1 per hour is unusual in healthy production systems. (Best-practice inference) - -- **[threads]** `jvm.thread.count` measures **platform threads only**. Java 21+ applications using virtual threads (Project Loom) will appear to have very low thread counts. If the service uses virtual threads, this metric significantly underrepresents concurrency. (Confirmed — OTel semconv: "Number of executing platform threads") - -- **[threads]** `waiting` thread state is normal for idle thread pool workers. A spike in `waiting` alone is not concerning. Only alert when `waiting` increases while `runnable` decreases — that indicates threads are waiting for work that isn't arriving (possible upstream starvation). (Best-practice inference) - -- **[threads]** Thread state is a snapshot taken at the OTel scrape interval (default 1 minute). Blocked threads that resolve within one scrape interval will not appear. Short-lived contention spikes are invisible unless scrape interval is reduced. (Best-practice inference) - -- **[cpu-utilization]** `jvm.cpu.recent_utilization` is a ratio from 0.0 to 1.0 relative to all available CPUs. A value of 1.0 on a 4-CPU pod means 400% in top(1) notation. Always display this with `context.jvm.cpu.count` for interpretation. (Confirmed — OTel semconv: "Recent CPU utilization for the process ... 1.0 means maximum usage of all CPUs") - -- **[cpu-utilization]** High `jvm.cpu.recent_utilization` during high GC rate = GC consuming CPU. High CPU during low GC rate = computation work. These require different remediation. Look at GC rate in tandem. (Best-practice inference) - -- **[cpu-utilization]** `jvm.system.cpu.utilization` and `jvm.system.cpu.load_1m` have Development stability status and may not be available in all OTel agent versions (requires the runtime-telemetry-java17 module). Gate these widgets. (Confirmed — OTel semconv stability marker) - -- **[non-heap-buffers]** Direct buffer memory (`jvm.buffer.memory.used[direct]`) is **entirely off-heap** and does NOT appear in `jvm.memory.used[heap]`. A JVM can look perfectly healthy on heap metrics while native memory is close to OOM. Always monitor direct buffers for Netty, gRPC, Kafka client, and NIO-heavy services. (Confirmed — JVM BufferPoolMXBean docs) - -- **[non-heap-buffers]** Metaspace has no hard limit by default (`-XX:MaxMetaspaceSize` not set). `jvm.memory.limit[Metaspace]` may be 0 or very large. Use `jvm.memory.committed[Metaspace]` as the denominator for Metaspace pressure calculation. (Confirmed — Java 8+ Metaspace behavior) - -- **[non-heap-buffers]** `jvm.buffer.*` metrics have Development stability status and may not be enabled by default. Check OTel agent configuration for `otel.instrumentation.jvm.enabled` or equivalent feature flags. (Confirmed — OTel semconv stability marker) - -- **[class-loading]** `jvm.class.count` has Development stability status in OTel semconv but is **confirmed present** in this Tsuga environment (Stage 2 verified). Used directly in dashboards. If absent in other environments, fall back to tracking `jvm.class.loaded` minus `jvm.class.unloaded` as a class count proxy. (Stage 2 confirmed — present in 05 catalog) - -- **[class-loading]** Class count grows during JVM startup and stabilizes after warmup (typically 2–5 minutes). A class count that keeps growing after warmup = classloader leak. Each redeploy that doesn't restart the JVM compounds the leak. (Best-practice inference) - ---- - -## Confirmed Tsuga prefixes - -- `jvm.` — **CONFIRMED** (Stage 2 verified via `tsuga_build_metric_catalog.py`; 12 metrics present; 8 Development-status metrics absent — see 05 catalog) - ---- - -## Discovery status - -Discovery: **Stage 2 complete** (2026-02-24). Access method: Tsuga CLI v1.0.2 via `tsuga_search_metrics.py` + `tsuga_build_metric_catalog.py`. - -**Results:** -- Prefix `jvm.`: **12 metrics confirmed** -- Metrics absent (all Development-status): `jvm.memory.init`, `jvm.system.cpu.utilization`, `jvm.system.cpu.load_1m`, `jvm.buffer.memory.used`, `jvm.buffer.memory.limit`, `jvm.buffer.count`, `jvm.file_descriptor.count`, `jvm.file_descriptor.limit` -- OTel metric attribute field naming confirmed: `jvm.*` attributes accessed as `context.jvm.*` in Tsuga -- `context.service.name` confirmed on all 12 metrics -- Histogram support for `jvm.gc.duration`: inconclusive (spot-check API returned 500 errors; handled gracefully in Stage 3) -- GC pool names (u02): not yet enumerated — Old Gen Pressure widget filter uses `context.jvm.memory.pool.name:G1 Old Gen` (G1 assumed); validate in Tsuga dashboards - -## Dashboard status - -**Stage 3 complete** (2026-02-24). - -| Dashboard | Tsuga ID | Widgets | -|---|---|---| -| [Integration] JVM metrics - Overview | `f0hj-26a5t-72pw` | 45 | -| [Integration] JVM metrics - Deep Dive | `58c5-6jb6q-mw3j` | 51 | - ---- - -## Top sources - -1. **https://opentelemetry.io/docs/specs/semconv/runtime/jvm-metrics/** — OTel canonical reference for all `jvm.*` metric names, types, units, attributes, and stability status. Primary source for all metric definitions in this bundle. - -2. **https://opentelemetry.io/docs/specs/semconv/attributes-registry/jvm/** — OTel attribute registry for jvm.* attribute names and values (pool names, thread states, GC actions). Primary source for all dimension values. - -3. **https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/instrumentation/runtime-telemetry/runtime-telemetry-java8/library/README.md** — Java 8+ OTel runtime telemetry library; defines which metrics are collected and their MXBean sources. - -4. **https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/instrumentation/runtime-telemetry/runtime-telemetry-java17/library/README.md** — Java 17+ telemetry library; defines Development-status JFR-based metrics (cpu.context_switch, memory.allocation, network.io). - -5. **https://docs.oracle.com/en/java/docs/api/java.management/java/lang/management/MemoryPoolMXBean.html** — JVM MemoryPoolMXBean API; canonical source for pool name values and their semantics. - -6. **https://docs.oracle.com/en/java/docs/api/java.management/java/lang/management/GarbageCollectorMXBean.html** — JVM GarbageCollectorMXBean API; GC notification info for GC name/action values. - -7. **https://www.oracle.com/technical-resources/articles/java/g1gc.html** — Oracle G1 GC tuning guide; defines minor vs major GC semantics, pause time goals, and operational thresholds. - -8. **https://openjdk.org/jeps/318** — ZGC design JEP; explains ultra-low-latency GC architecture and why ZGC "major GC" events are concurrent, not stop-the-world. - -9. **https://openjdk.org/jeps/271** — Unified JVM logging (JEP 271); defines GC log format used in Java 9+ (-Xlog:gc* flag). - -10. **https://grafana.com/grafana/dashboards/18812-jvm-overview-opentelemetry/** — Grafana Labs JVM Overview OTel dashboard; reference for industry-standard widget groupings and derived signals for jvm.* metrics. diff --git a/plugins/tsuga/skills/knowledge-technology/references/jvm/metrics.csv b/plugins/tsuga/skills/knowledge-technology/references/jvm/metrics.csv deleted file mode 100644 index ecd1915..0000000 --- a/plugins/tsuga/skills/knowledge-technology/references/jvm/metrics.csv +++ /dev/null @@ -1,22 +0,0 @@ -theme,name,confirmed,definition,unit,type,tsuga_metric_name,widget_suggestions,aggregation,post_function,group_by,required_context_fields,optional_context_fields,absence_means,cardinality_footguns,variants,citations -Capacity/Saturation,jvm.memory.used,true,Measure of memory currently used by the JVM across memory pools. Must filter by context.jvm.memory.type to separate heap from non-heap. Primary signal for heap pressure and leak detection. UpDownCounter — treat as gauge.,By,gauge,jvm.memory.used,timeseries|query-value|bar|top-list,sum,none,context.jvm.memory.pool.name:10|context.service.name:10,context.env|context.service.name,context.k8s.cluster.name|context.k8s.namespace.name|context.team,OTel agent not attached or metrics export disabled; verify agent attachment and OTEL_METRICS_EXPORTER setting,"context.jvm.memory.pool.name values differ by GC (G1: G1 Old Gen vs Parallel: PS Old Gen); do not group cross-GC without awareness. Avoid context.k8s.pod.name (high cardinality).",otel=jvm.memory.used|legacy=process.runtime.jvm.memory.usage,https://opentelemetry.io/docs/specs/semconv/runtime/jvm-metrics/ -Capacity/Saturation,jvm.memory.committed,true,Measure of memory committed (allocated from OS and available for JVM use) in each pool. Used as denominator for non-heap pressure when jvm.memory.limit is 0 (Metaspace without MaxMetaspaceSize). UpDownCounter — treat as gauge.,By,gauge,jvm.memory.committed,timeseries|query-value,sum,none,context.jvm.memory.pool.name:10|context.service.name:10,context.env|context.service.name,context.k8s.cluster.name|context.k8s.namespace.name,OTel agent not attached or collection disabled,Same GC-specific pool name cardinality as jvm.memory.used. Do not group by pod name.,otel=jvm.memory.committed|legacy=process.runtime.jvm.memory.committed,https://opentelemetry.io/docs/specs/semconv/runtime/jvm-metrics/ -Capacity/Saturation,jvm.memory.limit,true,Measure of maximum memory obtainable for each pool. For heap pools: equals -Xmx; for Metaspace without -XX:MaxMetaspaceSize: may be -1 or very large. UpDownCounter — treat as gauge.,By,gauge,jvm.memory.limit,query-value,sum,none,context.jvm.memory.pool.name:10|context.service.name:10,context.env|context.service.name,context.k8s.cluster.name,OTel agent not attached,"Metaspace limit may be -1 (unlimited); use committed as denominator for Metaspace pressure instead. GC-specific pool names vary.",otel=jvm.memory.limit|legacy=process.runtime.jvm.memory.limit,https://opentelemetry.io/docs/specs/semconv/runtime/jvm-metrics/ -Capacity/Saturation,jvm.memory.used_after_last_gc,true,Measure of memory used after the most recent GC event on each pool. This is the live set size — the baseline below which the heap cannot shrink. Best metric for memory leak detection. UpDownCounter — treat as gauge.,By,gauge,jvm.memory.used_after_last_gc,timeseries|query-value,sum,none,context.jvm.memory.pool.name:10|context.service.name:10,context.env|context.service.name,context.k8s.cluster.name,OTel agent not attached or GC not yet run; appears after first GC event,"Only available on heap pools that track collection usage. Non-heap pools may not emit this metric. GC-specific pool names vary.",otel=jvm.memory.used_after_last_gc,https://opentelemetry.io/docs/specs/semconv/runtime/jvm-metrics/ -Capacity/Saturation,jvm.memory.init,false,Initial memory that was requested from the OS at JVM startup for each pool. Informational; rarely changes after startup. Development stability status in some OTel versions. UpDownCounter — treat as gauge.,By,gauge,jvm.memory.init,query-value,sum,none,context.service.name:10,context.env|context.service.name,,OTel agent not attached or Development feature disabled,Safe — low cardinality.,otel=jvm.memory.init|legacy=process.runtime.jvm.memory.init,https://opentelemetry.io/docs/specs/semconv/runtime/jvm-metrics/ -Performance/Latency,jvm.gc.duration,true,"Duration of JVM garbage collection actions recorded as a histogram. Use percentile(p95/p99) for pause latency; use count with per-second for GC rate; use sum with per-second for GC CPU overhead. Filter context.jvm.gc.action to separate minor (end of minor GC) from major (end of major GC) events. CRITICAL: Histogram support in Tsuga must be verified in Stage 2.",s,histogram,jvm.gc.duration,timeseries|query-value|bar|top-list,percentile(p95)|count,none|per-second,context.jvm.gc.name:10|context.jvm.gc.action:2|context.service.name:10,context.env|context.service.name,context.k8s.cluster.name|context.k8s.namespace.name|context.team,"Histogram metric — verify that Tsuga exposes percentile aggregations for this metric. If not available as histogram, only sum/count may work.","context.jvm.gc.name values differ by GC implementation. jvm.gc.cause is Development status and may not be available. Do not group by pod name.",otel=jvm.gc.duration|legacy=process.runtime.jvm.gc.duration,https://opentelemetry.io/docs/specs/semconv/runtime/jvm-metrics/ -Capacity/Saturation,jvm.thread.count,true,"Number of currently executing platform threads. Includes runnable, blocked, waiting, timed_waiting threads. Does NOT count virtual threads (Java 21+ Project Loom). Use context.jvm.thread.state attribute to filter by state. UpDownCounter — treat as gauge.",{thread},gauge,jvm.thread.count,timeseries|query-value|bar,sum,none,context.jvm.thread.state:6|context.jvm.thread.daemon:2|context.service.name:10,context.env|context.service.name,context.k8s.cluster.name|context.k8s.namespace.name,OTel agent not attached; OR service uses only virtual threads (Java 21+) and platform thread count is near-zero by design,"context.jvm.thread.state has only 6 values (safe). context.jvm.thread.daemon has 2 values (safe). Do not group by pod name.",otel=jvm.thread.count|legacy=process.runtime.jvm.threads.count,https://opentelemetry.io/docs/specs/semconv/runtime/jvm-metrics/ -Throughput/Usage,jvm.class.loaded,true,Total number of classes that have been loaded since JVM startup. Monotonically increasing counter. Use per-second rate for class load rate. Class loading spikes correlate with deployments and dynamic proxy generation.,{class},counter,jvm.class.loaded,timeseries|query-value,sum,per-second,context.service.name:10,context.env|context.service.name,context.k8s.cluster.name,OTel agent not attached,Safe — no high-cardinality attributes.,otel=jvm.class.loaded|legacy=process.runtime.jvm.classes.loaded,https://opentelemetry.io/docs/specs/semconv/runtime/jvm-metrics/ -Throughput/Usage,jvm.class.unloaded,true,Total number of classes unloaded since JVM startup. Monotonically increasing counter. In most production JVMs with G1 and fixed classpath: near-zero. Rising value with OSGi or dynamic proxies is normal; rising value without dynamic loading = potential classloader leak being partially recovered.,{class},counter,jvm.class.unloaded,timeseries|query-value,sum,per-second,context.service.name:10,context.env|context.service.name,context.k8s.cluster.name,OTel agent not attached,Safe — no high-cardinality attributes.,otel=jvm.class.unloaded|legacy=process.runtime.jvm.classes.unloaded,https://opentelemetry.io/docs/specs/semconv/runtime/jvm-metrics/ -Capacity/Saturation,jvm.class.count,true,Current number of classes loaded in the JVM (loaded minus unloaded). UpDownCounter — treat as gauge. Development stability status — may not be available in all OTel agent versions. Gate on availability.,{class},gauge,jvm.class.count,query-value|timeseries,sum,none,context.service.name:10,context.env|context.service.name,context.k8s.cluster.name,OTel agent not attached OR Development feature not enabled (gate this widget),Safe — no high-cardinality attributes.,otel=jvm.class.count,https://opentelemetry.io/docs/specs/semconv/runtime/jvm-metrics/ -Throughput/Usage,jvm.cpu.time,true,CPU time consumed by the JVM process as reported by the OS (monotonically increasing counter). Use per-second rate to get current CPU time per second. More precise than utilization ratio for CPU throttling analysis in k8s cgroups.,s,counter,jvm.cpu.time,timeseries,sum,per-second,context.service.name:10,context.env|context.service.name,context.k8s.cluster.name,OTel agent not attached,Safe — no high-cardinality attributes.,otel=jvm.cpu.time,https://opentelemetry.io/docs/specs/semconv/runtime/jvm-metrics/ -Capacity/Saturation,jvm.cpu.count,true,"Number of processors (CPUs/vCPUs) available to the JVM. Stable gauge — typically constant unless container CPU limits change. Informational for interpreting jvm.cpu.recent_utilization (1.0 = all CPUs saturated).",{cpu},gauge,jvm.cpu.count,query-value,sum,none,context.service.name:10,context.env|context.service.name,,OTel agent not attached,Safe — typically 1–32 values; low cardinality.,otel=jvm.cpu.count,https://opentelemetry.io/docs/specs/semconv/runtime/jvm-metrics/ -Capacity/Saturation,jvm.cpu.recent_utilization,true,"Recent CPU utilization ratio for the JVM process. Range: 0.0–1.0; 1.0 = all available CPUs at 100%. Combines JIT compilation, GC threads, and application thread CPU. Gauge — use as-is with average aggregation.",1,gauge,jvm.cpu.recent_utilization,timeseries|query-value,average,none,context.service.name:10,context.env|context.service.name,context.k8s.cluster.name,OTel agent not attached,"Safe — single float value per series. Display as percent (multiply by 100 or use percent normalizer). Value 1.0 = saturation, not 100% of 1 CPU.",otel=jvm.cpu.recent_utilization,https://opentelemetry.io/docs/specs/semconv/runtime/jvm-metrics/ -Capacity/Saturation,jvm.system.cpu.utilization,false,Recent CPU utilization for the whole host system (not just JVM). Range: 0.0–1.0. Development stability status — requires runtime-telemetry-java17 module (Java 17+ OTel instrumentation). Gate on availability. Use alongside jvm.cpu.recent_utilization to compute JVM CPU share.,1,gauge,jvm.system.cpu.utilization,timeseries|query-value,average,none,context.service.name:10,context.env|context.service.name,context.k8s.cluster.name,Development feature not enabled; OTel agent < 2.0 or Java 17+ module not loaded (gate this widget),Safe.,otel=jvm.system.cpu.utilization,https://opentelemetry.io/docs/specs/semconv/runtime/jvm-metrics/ -Capacity/Saturation,jvm.system.cpu.load_1m,false,System CPU load average over the last 1 minute (Unix load average). Development stability status. Platform-specific: not available on Windows. Useful for distinguishing JVM saturation from host saturation. Number of threads in run queue.,{run_queue_item},gauge,jvm.system.cpu.load_1m,timeseries,average,none,context.service.name:10,context.env|context.service.name,,Development feature not enabled OR Windows host (not available on Windows),Safe.,otel=jvm.system.cpu.load_1m,https://opentelemetry.io/docs/specs/semconv/runtime/jvm-metrics/ -Capacity/Saturation,jvm.buffer.memory.used,false,Memory used by NIO buffer pools (direct and mapped). Off-heap memory — does NOT appear in jvm.memory.used[heap]. Development stability status. Critical for Netty/gRPC/Kafka services. UpDownCounter — treat as gauge.,By,gauge,jvm.buffer.memory.used,timeseries|query-value,sum,none,context.jvm.buffer.pool.name:2|context.service.name:10,context.env|context.service.name,context.k8s.cluster.name,Development feature not enabled; verify jvm.buffer.* availability in Stage 2 (gate this section if missing),"context.jvm.buffer.pool.name has only 2 values (direct and mapped). Safe. Do not use pod name group-by.",otel=jvm.buffer.memory.used|legacy=process.runtime.jvm.buffer.usage,https://opentelemetry.io/docs/specs/semconv/runtime/jvm-metrics/ -Capacity/Saturation,jvm.buffer.memory.limit,false,Total capacity of NIO buffer pools. Denominator for direct/mapped buffer utilization formula. Development stability status. UpDownCounter — treat as gauge.,By,gauge,jvm.buffer.memory.limit,query-value,sum,none,context.jvm.buffer.pool.name:2|context.service.name:10,context.env|context.service.name,context.k8s.cluster.name,Development feature not enabled (gate with jvm.buffer.memory.used),Same as jvm.buffer.memory.used — 2 pool name values.,otel=jvm.buffer.memory.limit|legacy=process.runtime.jvm.buffer.limit,https://opentelemetry.io/docs/specs/semconv/runtime/jvm-metrics/ -Throughput/Usage,jvm.buffer.count,false,Number of buffers in each NIO buffer pool. Development stability status. Useful for detecting buffer count growing while memory is stable (small many-buffer leak pattern). UpDownCounter — treat as gauge.,{buffer},gauge,jvm.buffer.count,timeseries|bar,sum,none,context.jvm.buffer.pool.name:2|context.service.name:10,context.env|context.service.name,,Development feature not enabled,Safe — 2 pool name values.,otel=jvm.buffer.count|legacy=process.runtime.jvm.buffer.count,https://opentelemetry.io/docs/specs/semconv/runtime/jvm-metrics/ -Capacity/Saturation,jvm.file_descriptor.count,false,Number of open file descriptors as reported by the JVM. Includes sockets and file handles. Development stability status. Critical for services with many connections — approaching the OS fd limit causes java.io.IOException: Too many open files. UpDownCounter — treat as gauge.,{file_descriptor},gauge,jvm.file_descriptor.count,timeseries|query-value,sum,none,context.service.name:10,context.env|context.service.name,context.k8s.cluster.name,Development feature not enabled; OTel agent version too old (gate this widget),Safe.,otel=jvm.file_descriptor.count,https://opentelemetry.io/docs/specs/semconv/runtime/jvm-metrics/ -Capacity/Saturation,jvm.file_descriptor.limit,false,Maximum number of open file descriptors the JVM process may have (OS-level limit). Set via ulimit -n. Development stability status. Denominator for File Descriptor Utilization formula. UpDownCounter — treat as gauge.,{file_descriptor},gauge,jvm.file_descriptor.limit,query-value,sum,none,context.service.name:10,context.env|context.service.name,,Development feature not enabled (gate with jvm.file_descriptor.count),Safe.,otel=jvm.file_descriptor.limit,https://opentelemetry.io/docs/specs/semconv/runtime/jvm-metrics/ -Cost,NOT_APPLICABLE,false,JVM runtime metrics do not have direct cost dimensions. Cost monitoring for Java services is better tracked at the infrastructure level (host CPU hours and memory allocation) or via application-level metrics (requests processed per unit of compute). Proxy metrics for cost efficiency: jvm.cpu.recent_utilization (CPU waste) and jvm.memory.used vs jvm.memory.limit gap (over-provisioned heap).,,,,,,,,,,,,, diff --git a/plugins/tsuga/skills/knowledge-technology/references/jvm/queries.md b/plugins/tsuga/skills/knowledge-technology/references/jvm/queries.md deleted file mode 100644 index cfa806b..0000000 --- a/plugins/tsuga/skills/knowledge-technology/references/jvm/queries.md +++ /dev/null @@ -1,60 +0,0 @@ -# JVM - -Java / Kotlin / Scala / Clojure runtime. Healthy: memory growing and collecting cleanly, brief GC pauses, stable threads, reasonable CPU, FD headroom. - -## Incident shapes - -- **Memory leak / OOM** — `memory.used` grows + `memory.used_after_last_gc` also grows → GC can't reclaim -- **GC pause storms** — `gc.duration` p99 spikes → app pauses, requests stall -- **Thread leak** — `thread.count` grows unboundedly → pool / executor not shutting down -- **Class-load thrash** — `class.loaded - class.unloaded` grows → dynamic-class libs leaking -- **CPU saturation** — `cpu.recent_utilization ≈ 1` sustained → GC overhead, hot loop, or real work -- **Direct memory exhaustion** — `buffer.memory.used` near limit → off-heap leak (Netty, NIO) -- **FD exhaustion** — `file_descriptor.count / limit` near 1 → socket/file leak - -## Key metrics - -| Metric | Unit | Signal | -|---|---|---| -| `jvm.memory.used` (pool) | bytes | Heap / non-heap per pool | -| `jvm.memory.committed` (pool) | bytes | Committed from OS | -| `jvm.memory.limit` (pool) | bytes | Max heap / non-heap | -| `jvm.memory.used_after_last_gc` (pool) | bytes | Post-GC survivors — growth = true leak | -| `jvm.gc.duration` (action, gc) | ms | GC pause duration | -| `jvm.thread.count` | count | Live threads; growth = leak | -| `jvm.class.loaded` / `unloaded` / `count` | count | Class load state | -| `jvm.cpu.time` | ns | Cumulative JVM CPU | -| `jvm.cpu.recent_utilization` | ratio | Recent JVM CPU | -| `jvm.system.cpu.utilization` | ratio | Host CPU | -| `jvm.system.cpu.load_1m` | load | Host load | -| `jvm.buffer.memory.used` / `limit` | bytes | Direct / mapped buffers | -| `jvm.buffer.count` | count | Buffer allocations | -| `jvm.file_descriptor.count` / `limit` | count | FD state | - -## Derived signals - -- `memory.used{heap} / memory.limit{heap}` — heap utilization. Near 1 sustained = GC overhead, OOM risk. -- Trend of `memory.used_after_last_gc` — leak indicator. Monotonic = leak. -- GC time / wallclock — GC overhead. > 0.1 sustained = GC starving the app. -- `thread.count` slope under steady workload — positive = leak. -- `1 - file_descriptor.count / file_descriptor.limit` — FD headroom. < 0.1 = accept errors soon. - -## Log patterns - -- `java.lang.OutOfMemoryError: Java heap space` — heap OOM -- `java.lang.OutOfMemoryError: Direct buffer memory` — off-heap OOM -- `java.lang.OutOfMemoryError: GC overhead limit exceeded` — GC > 98% CPU recovering < 2% heap -- `java.lang.OutOfMemoryError: unable to create native thread` — thread/OS limit -- `Full GC (Allocation Failure)` — full GC from heap pressure -- `Pause Young (G1 Evacuation Pause)` with high ms — long G1 pause -- `java.net.SocketException: Too many open files` — FD exhaustion -- `java.lang.StackOverflowError` — deep recursion - -## Gotchas - -- `memory.used` oscillates with GC; `used_after_last_gc` is the trending metric (removes the sawtooth). -- G1, ZGC, Shenandoah have very different pause profiles. Normal G1 (occasional 100ms) is abnormal on ZGC (sub-10ms). -- "Memory used" covers heap + managed non-heap. Direct / mapped buffers are in `buffer.memory.used` and leak independently. -- Pooled thread leaks may not show in `thread.count`; they show in CPU, scheduling latency, or stack dumps. -- Class-load growth with flat unload count = permgen/metaspace leak (dynamic-class libs like CGLib). -- K8s memory limits apply at the OS level. JVM `-Xmx` > pod memory limit → OOM-killed with no Java-side OOM. diff --git a/plugins/tsuga/skills/knowledge-technology/references/kafka/README.md b/plugins/tsuga/skills/knowledge-technology/references/kafka/README.md deleted file mode 100644 index 5977a1b..0000000 --- a/plugins/tsuga/skills/knowledge-technology/references/kafka/README.md +++ /dev/null @@ -1,272 +0,0 @@ -# kafka Integration Context Bundle - -## Metadata -**Technology:** kafka -**Deployment:** self-hosted -**Environment:** prod -**Persona:** SRE Dev and ops -**Telemetry preference:** mixed -**Integration scope:** core service only -**Primary use-case:** reliability and performance - -## How to use this bundle -- Use `01_kafka_metrics.csv` as the source of truth for metric semantics, units, and provisional counter math. -- Use `02_kafka_dashboard_plan.yaml` for dashboard structure: sections, widgets, derived signals, gating, and triage content. -- Use `03_kafka_state.yaml` for machine-readable status, assumptions, discovery unknowns, and Stage 2 priorities. -- Use `04_kafka_memory.md` for the human-readable handoff narrative and rationale. -- Stage 2 will create `05_kafka_metric_catalog.csv` as the discovered Tsuga inventory used for reconciliation and deep-dive coverage checks. -- Stage 4 should read this file's `Log intelligence (Stage 4 handoff)` section and `03_kafka_state.yaml` `log_intel` block before designing route processors. - -## What it is and what "good" looks like - -### Confirmed by sources -- Kafka is a distributed log platform where producers write records to topic partitions and consumers read by offset; broker health depends on partition leadership, replication, and request handling ([Kafka design docs](https://kafka.apache.org/documentation/#design), [Kafka monitoring docs](https://kafka.apache.org/36/documentation/#monitoring)). -- For this environment, live Tsuga metrics confirm rich `kafka.consumer.*` client telemetry plus broker and partition health metrics (`kafka.partition.*`, `kafka.request.*`, `kafka.controller.*`) and workload counters (`kafka_workload_*`) from recent ingestion scans (Tsuga metric search, 2026-02-27). -- "Good" means consumer lag stays bounded, consume throughput tracks produce throughput, rebalance churn remains low, request failures remain low versus total request volume, and partition replication risk (`underReplicated`/`offline`) remains near zero. -- Incident shape 1: consumer lag growth while consume rate drops; start with `consumer-lag-and-progress`. -- Incident shape 2: rebalance/auth churn causing unstable consumption; start with `rebalance-and-coordination`. -- Incident shape 3: broker/partition health regression (offline or under-replicated partitions); start with `broker-and-partition-health`. - -### Best-practice inference -- For Kafka operations, consumer lag is usually a higher-value reliability signal than CPU because lag is directly tied to delay and possible retention-window risk. -- In mixed telemetry setups, client-side consumer metrics can be available even when some broker-side dimensions are sparse; section gating should prevent false confidence. -- First responder flow should be: verify lag and throughput balance, then test coordination stability, then confirm broker replication safety. - -## Key concepts - -### Glossary - -| Term | Definition | Operational meaning | Dashboard section | -|---|---|---|---| -| Broker | Kafka server node handling partitions and requests | Broker availability and request behavior gate platform reliability | broker-and-partition-health | -| Topic | Named stream of records | Ownership and workload boundaries are usually topic-scoped | request-path-and-workload | -| Partition | Ordered append-only shard of a topic | Lag and replica safety are partition-level phenomena | broker-and-partition-health | -| Leader partition | Replica currently serving reads/writes | Leader instability drives request failures and lag spikes | broker-and-partition-health | -| ISR (in-sync replicas) | Replica set considered caught up to leader | Shrinking ISR raises data-loss/failover risk | broker-and-partition-health | -| Under-replicated partition | Partition with ISR smaller than replication factor | Primary replication-risk signal for paging | broker-and-partition-health | -| Offline partition | Partition lacking an available leader | Direct availability incident indicator | broker-and-partition-health | -| Controller | Broker managing cluster metadata transitions | Controller churn often correlates with instability | broker-and-partition-health | -| Consumer group | Coordinated set of consumers sharing partitions | Group churn and lag determine processing timeliness | consumer-lag-and-progress | -| Consumer lag | Offset difference between latest and committed/processed records | Core user-visible delay signal | consumer-lag-and-progress | -| Rebalance | Partition ownership redistribution among group members | Frequent failed rebalances reduce effective throughput | rebalance-and-coordination | -| Commit | Consumer checkpoint write of processed offset | Commit latency/failure affects replay and duplicate risk | rebalance-and-coordination | -| Fetch | Consumer poll operation for records | Fetch latency/size indicates broker-network and batching health | throughput-and-fetch | -| Throttle | Broker-imposed backpressure delay | Sustained throttling means contention or quota pressure | throughput-and-fetch | -| Request path | End-to-end Kafka protocol request handling | Request failures and latency show broker/client health | request-path-and-workload | -| Produce rate | Records/messages entering Kafka | Compared with consume rate to detect backlog trajectory | request-path-and-workload | -| Consume rate | Records/messages read from Kafka | Falling consume rate with steady produce rate predicts lag growth | consumer-lag-and-progress | -| Retention window | Time/size policy for topic data retention | High lag near retention bounds risks data unavailability for consumers | consumer-lag-and-progress | -| Flush | Log flush behavior to durable storage | Flush latency spikes can precede broker request latency increases | broker-and-partition-health | -| Client ID | Kafka client identity tag | Useful bounded breakdown for noisy or lagging consumers | rebalance-and-coordination | -| Service instance | Runtime instance emitting telemetry | Fast blast-radius pivot when one deployment regresses | availability-health | -| Node ID | Broker numeric identity in client metrics | Useful for routing failures to broker owners | broker-and-partition-health | - -### Concept Map - -```text -Producer -> writes records to -> Topic partition (why: establishes ingress workload) -Topic partition -> assigns one -> Leader replica (why: serial write/read authority) -Leader replica -> replicates to -> ISR followers (why: durability and failover safety) -ISR shrink -> increases -> under-replicated partitions (why: resilience degradation) -Leader unavailable -> creates -> offline partition (why: direct read/write outage) -Consumer group -> owns -> partition assignments (why: controls parallel processing) -Rebalance -> reassigns -> partitions between consumers (why: keeps group membership coherent) -Failed rebalance -> delays -> steady consumption (why: assignment churn stalls progress) -Commit operation -> advances -> group offset (why: confirms processed progress) -Commit slowdown -> increases -> duplicate/replay exposure (why: checkpoint staleness) -Fetch request -> returns -> record batches (why: governs consumer throughput) -Fetch throttle -> reduces -> effective consume rate (why: quota/backpressure control) -Produce rate > consume rate -> grows -> lag (why: backlog accumulation) -Lag growth -> increases -> end-to-end message delay (why: users see stale processing) -Lag near retention boundary -> risks -> missed processing window (why: old data may expire) -Request failures -> correlate with -> broker/controller instability (why: control/data plane faults) -Controller activity changes -> coincide with -> leadership movement (why: metadata plane churn) -Flush latency increase -> raises -> request latency tail (why: storage pressure propagation) -Client-id skew -> exposes -> bad actor consumers (why: one client can dominate failures) -Service instance dimension -> maps -> deploy rollout impact (why: regressions often version-scoped) -Environment/team filters -> reduce -> incident blast radius ambiguity (why: faster ownership routing) -Partition health + lag + throughput -> form -> triage chain backbone (why: covers correctness and timeliness) -``` - -### Entities and dimensions - -| Entity/Dimension | Why useful | Cardinality risk | Safe top-N | Do NOT group-by guidance | -|---|---|---|---|---| -| `context.env` | Environment scoping | Low | 5 | Keep as dashboard global filter | -| `context.team` | Ownership scoping | Low | 10 | Keep as dashboard global filter | -| `context.service.name` | Service boundary for producer/consumer apps | Medium | 15 | Avoid combining with pod and client-id in same chart | -| `context.service.instance.id` | Pinpoint bad instances quickly | High | 12 | Do not use unbounded in overview | -| `context.client-id` | Consumer/client hot-spot detection | Medium | 12 | Avoid stacking with pod + instance | -| `context.k8s.cluster.name` | Cluster blast radius split | Medium | 8 | Use with env filter | -| `context.k8s.namespace.name` | Namespace ownership split | Medium | 12 | Avoid together with pod and client-id | -| `context.k8s.pod.name` | Pod-level regressions after rollout | High | 10 | Deep dive only | -| `context.host.name` | Host/node affinity checks | Medium | 10 | Prefer service instance when possible | -| `context.node-id` | Broker-specific request path diagnosis | Medium | 10 | Verify availability in Stage 2 before enforcing | -| `context.topic` | Topic-level lag and throughput ownership | High | 15 | Never leave unbounded | -| `context.status` | Success/failure slicing for request path | Low | 6 | Use only where metric supports status | -| `context.scope` | Instrumentation fallback dimension | Low | 10 | Use only as fallback dimension | - -### Tsuga field mapping - -#### Confirmed by sources -| Vendor/exporter dimension | Recommended context.* key | Must-exist vs optional | -|---|---|---| -| Environment | `context.env` | Must-exist | -| Team | `context.team` | Must-exist | -| Kafka consumer client id | `context.client-id` | Optional but present on many `kafka.consumer.*` metrics (Tsuga metric scan) | -| Service identity | `context.service.name` | Optional but present on many `kafka.consumer.*` and workload metrics (Tsuga metric scan) | -| Service instance | `context.service.instance.id` | Optional but present on many `kafka.consumer.*` metrics (Tsuga metric scan) | -| Topic (workload producer/consumer counters) | `context.topic` | Optional (confirmed on `kafka_workload_*`) | -| Broker node id (some request-size metrics) | `context.node-id` | Optional (observed on selected consumer request metrics) | - -#### Best-practice inference -| Vendor/exporter dimension | Recommended context.* key | Must-exist vs optional | -|---|---|---| -| Consumer group id | `context.consumer.group` | Optional; Unknown in current scan | -| Partition id | `context.partition` | Optional; confirmed on consumer lag metrics in Stage 2 discovery | -| Broker id (broker-side namespace) | `context.kafka.broker.id` | Optional; Unknown in current scan | -| Topic (broker-side namespaces) | `context.topic` | Optional; confirmed on consumer lag and workload metrics in Stage 2 discovery | - -## Golden signals - -### Confirmed by sources -| Signal | Meaning for kafka | Typical degradations | Best telemetry sources | What people page on | Section questions | -|---|---|---|---|---|---| -| Traffic | Balance of produce/consume and request throughput | Producer surge, consumer slowdown, throttling | `kafka_workload_messages_produced_total`, `kafka_workload_messages_consumed_total`, `kafka.request.count`, `kafka.consumer.records_consumed_total` | Produced keeps rising while consumed stalls and lag grows | Are we keeping up with incoming workload? | -| Errors | Request and auth/coordination failure pressure | Broker errors, auth drift, rebalance failures | `kafka.request.failed`, `kafka.consumer.failed_authentication_total`, `kafka.consumer.failed_rebalance_total` | Error ratio climbs or auth failures spike after rollout | Are failures transient noise or systemic? | -| Latency | Commit/fetch/request tails that drive processing delay | Storage pressure, throttling, broker contention | `kafka.consumer.fetch_latency_max`, `kafka.consumer.commit_latency_max`, `kafka.request.time.99p`, `kafka.logs.flush.time.99p` | P99 latency spikes sustained across multiple clients/groups | Is delay from client coordination or broker path? | -| Saturation | Replication and partition health risk surfaces | Under-replicated or offline partitions, controller churn | `kafka.partition.underReplicated`, `kafka.partition.offline`, `kafka.controller.active.count`, `kafka.partition.count` | Non-zero offline partitions or rising under-replication | Are we at risk of availability or durability events? | - -### Best-practice inference -- Lag trend is usually the best user-impact proxy, while request counters are leading indicators for platform stress. -- Rebalance failure share is a practical early warning for consumer instability even before large lag appears. -- Combining partition health and request failure ratio reduces false positives from short-lived deploy turbulence. - -## Telemetry sources - -### Confirmed by sources -| Source type | How collected | What it provides | Pros/cons | Common pitfalls | -|---|---|---|---|---| -| Kafka broker JMX / broker monitoring | Kafka broker exposes operational metrics via JMX integrations ([Kafka monitoring docs](https://kafka.apache.org/36/documentation/#monitoring), [Confluent Kafka monitoring](https://docs.confluent.io/platform/current/kafka/monitoring.html)) | Partition health, request stats, controller metrics | Rich broker-level state / requires broker-side collection and naming normalization | Missing broker metrics often means collection gap, not healthy-zero | -| Kafka client metrics (producer/consumer) via OTel Java instrumentation | OTel Java kafka-clients common instrumentation exports standardized `kafka.consumer.*` / `kafka.producer.*` metrics ([OTel Java kafka metrics README](https://raw.githubusercontent.com/open-telemetry/opentelemetry-java-instrumentation/main/instrumentation/kafka/kafka-clients/kafka-clients-common/library/README.md)) | Lag, rebalance, commit/fetch/request behavior on clients | Strong app-facing signal / may miss broker-only failures | -| OTel Collector Kafka metrics receiver | Collector integration maps Kafka JMX broker metrics to semantic names ([OTel kafkametricsreceiver metadata](https://raw.githubusercontent.com/open-telemetry/opentelemetry-collector-contrib/main/receiver/kafkametricsreceiver/metadata.yaml)) | Broker and partition metrics in OTel model | Good for fleet-standard ingestion / optional metrics may require config | -| Synthetic workload counters | Local workload emitter produces `kafka_workload_*` counters and latency histogram | Produce/consume balance and synthetic latency | Simple balance signal / can diverge from real production traffic | -| Kafka logs | Broker and controller logs from Kafka process/log4j stack ([Confluent platform logging](https://docs.confluent.io/platform/current/monitor/cp-logging.html), [Kafka ops docs](https://kafka.apache.org/36/documentation/#ops)) | Leader election, replication, controller, auth failures | High-fidelity incident context / format varies by deployment | - -### Best-practice inference -- "No data" for workload counters often means workload job paused or sandbox profile changed, not broker outage. -- Client-side metrics can remain healthy while broker replication deteriorates; keep broker section independent from client section. -- Counter temporality (delta vs cumulative) must be validated in Stage 2 before final post-function selection. - -## Log intelligence (Stage 4 handoff) - -### Confirmed by sources - -#### Log sources matrix -| Source | Access path | Typical format | Structured vs unstructured | Evidence | -|---|---|---|---|---| -| Broker server log | Kafka broker log file path configured via log4j | Timestamp + level + logger + message text | Unstructured text by default | [Confluent platform logging](https://docs.confluent.io/platform/current/monitor/cp-logging.html), [Kafka ops docs](https://kafka.apache.org/36/documentation/#ops) | -| Controller / broker process stdout | Container stdout/stderr (Kubernetes, Docker) | log4j text unless JSON layout configured | Usually unstructured | [Confluent platform logging](https://docs.confluent.io/platform/current/monitor/cp-logging.html) | -| Optional JSON log layout | log4j2 JSON template layout when enabled | JSON line/object logs | Structured JSON | [Confluent platform logging](https://docs.confluent.io/platform/current/monitor/cp-logging.html) | - -#### Known log formats -1. Broker text log (common default): - - Sample line: `[2026-02-27 00:15:12,771] INFO [ReplicaFetcherManager on broker 2] Removed fetcher for partitions Set(topicA-3) (kafka.server.ReplicaFetcherManager)` - - Delimiter/shape notes: bracketed timestamp, level token, free-form component section, trailing logger name in parentheses. - - Timestamp pattern: `YYYY-MM-DD HH:MM:SS,mmm`. - - Quoting behavior: quoted topic/group names appear inline without fixed JSON keys. - - Optional fields: broker id, partition list, correlation ids vary by logger. -2. Client/consumer log (application side): - - Sample line: `2026-02-27 00:16:03.412 WARN [Consumer clientId=orders-consumer, groupId=orders] Attempt to heartbeat failed since group is rebalancing` - - Delimiter/shape notes: timestamp + level + context block + message. - - Timestamp pattern: `YYYY-MM-DD HH:MM:SS.mmm`. - - Optional fields: `clientId`, `groupId`, `topic`, `partition` may be absent. - -#### Candidate query filters for Stage 4 -- Precise: `context.service.name:kafka AND (message:*ERROR* OR message:*WARN* OR message:*under-replicated* OR message:*election* OR message:*rebalance*)` - - Rationale: targets high-value reliability events. - - Risk: depends on consistent `context.service.name` mapping. -- Fallback: `(message:*kafka* OR message:*consumer*) AND (message:*error* OR message:*warn* OR message:*rebalance* OR message:*timeout*)` - - Rationale: broad capture when structured labels are sparse. - - Risk: noisier and may capture non-Kafka app logs. - -#### Attribute mapping hints -| Raw field | Suggested Tsuga key | Confidence | Notes | -|---|---|---|---| -| timestamp token | `timestamp` | High | Preserve timezone and normalize to UTC | -| log level | `severity_text` | High | Map WARN/ERROR/INFO consistently | -| logger name/class | `context.logger.name` | Medium | Useful for controller vs replica modules | -| broker id (`broker 2`) | `context.node-id` | Medium | Prefer numeric extraction | -| topic-partition token (`topicA-3`) | `context.topic` + `context.partition` | Medium | Requires regex split | -| client id (`clientId=...`) | `context.client-id` | High | Aligns with metric dimension | -| group id (`groupId=...`) | `context.consumer.group` | Medium | Verify availability and naming | -| message body | `message` | High | Keep full text for secondary parsing | - -#### Parsing risks -- Broker and app consumer logs can mix formats in the same stream. -- Partition notation (`topic-12`) is ambiguous if topic names contain dashes. -- Multiline Java stack traces require continuation handling. -- Some logs only embed structured fields in message text, not fixed keys. -- Locale/timezone differences across hosts can skew temporal correlation. - -### Best-practice inference -- Start with a split route: JSON parse branch first, then text grok fallback. -- Keep initial parser focused on WARN/ERROR/controller/replication patterns, then expand once false positives are controlled. -- Preserve raw message in all branches to allow iterative parser tuning. - -## Caveats and footguns -- **[availability-health]** `kafka.controller.active.count` can be non-intuitive in single-controller mode; interpret trend and correlated error signals, not raw value alone. (Inference) -- **[availability-health]** `kafka.brokers` may reflect registration state and can lag real network reachability during controller churn. (Inference) -- **[consumer-lag-and-progress]** Lag metrics can drop during rebalance pauses without true recovery; verify consume rate after rebalance settles. (Inference) -- **[consumer-lag-and-progress]** `records_lag_max` is a tail metric and may spike from one partition while average behavior remains stable. (Inference) -- **[consumer-lag-and-progress]** Lag alone does not indicate data loss risk unless compared to retention window. (Inference) -- **[throughput-and-fetch]** Fetch latency spikes without throughput drop may reflect temporary broker throttling, not sustained outage. (Inference) -- **[throughput-and-fetch]** `fetch_size_avg` can rise during healthy batching optimization; pair with `fetch_latency_max` and lag. (Inference) -- **[throughput-and-fetch]** Request size changes after producer config rollouts can alter throughput interpretation without reliability regression. (Inference) -- **[throughput-and-fetch]** Counter metrics must not be shown as raw cumulative values; use rate/per-second after temporality confirmation. (Inference) -- **[rebalance-and-coordination]** Rebalance frequency can increase during deploys and autoscaling events without lasting incident impact. (Inference) -- **[rebalance-and-coordination]** Failed authentication bursts are often configuration drift and can be localized to one client-id. (Inference) -- **[rebalance-and-coordination]** Commit latency max is a tail signal; use with commit total for stability context. (Inference) -- **[rebalance-and-coordination]** `last_poll_seconds_ago` spikes can indicate stalled consumers but can also appear during controlled maintenance windows. (Inference) -- **[broker-and-partition-health]** `underReplicated` and `offline` values are high-severity even when traffic appears normal. ([Kafka monitoring docs](https://kafka.apache.org/36/documentation/#monitoring)) -- **[broker-and-partition-health]** `partition.count` changes during topic lifecycle events can invalidate naive ratio baselines. (Inference) -- **[broker-and-partition-health]** Flush time percentile metrics may be sparse depending on collection config and scrape interval. (Inference) -- **[broker-and-partition-health]** `leaderElection.unclean.count` increments indicate riskier failover behavior and should be treated as exceptional. ([Kafka monitoring docs](https://kafka.apache.org/36/documentation/#monitoring)) -- **[request-path-and-workload]** Workload counters (`kafka_workload_*`) may represent test traffic, not all production traffic. (Inference) -- **[request-path-and-workload]** `request.failed` ratio can be skewed by transient retries at startup; inspect sustained windows. (Inference) -- **[request-path-and-workload]** Histogram latency (`kafka_workload_operation_latency_ms`) requires consistent percentile handling; avoid mixing with max-only gauges without labeling. (Inference) -- **[request-path-and-workload]** Topic-level grouping can explode cardinality; keep top-N bounded and use service/team filters first. (Inference) -- **[consumer-lag-and-progress, request-path-and-workload]** Produce-consume imbalance should be interpreted with lag direction, not in isolation, to avoid false alarms during planned backfills. (Inference) - -## Confirmed Tsuga prefixes -- `kafka.*` — **CONFIRMED** (119 metrics found in Tsuga scan over 24h, 2026-02-27) -- `kafka.consumer.*` — **CONFIRMED** (86 metrics found in Tsuga scan over 24h, 2026-02-27) -- `kafka.consumer_group.*` — **CONFIRMED** (5 metrics found in Tsuga scan over 24h, 2026-02-27) -- `kafka.partition.*` — **CONFIRMED** (7 metrics found in Tsuga scan over 24h, 2026-02-27) -- `kafka.request.*` — **CONFIRMED** (6 metrics found in Tsuga scan over 24h, 2026-02-27) -- `kafka.logs.flush.*` — **CONFIRMED** (3 metrics found in Tsuga scan over 24h, 2026-02-27) -- `kafka.controller.*` — **CONFIRMED** (1 metric found in Tsuga scan over 24h, 2026-02-27) -- `kafka.isr.*` — **CONFIRMED** (1 metric found in Tsuga scan over 24h, 2026-02-27) -- `kafka.leaderElection.*` — **CONFIRMED** (1 metric found in Tsuga scan over 24h, 2026-02-27) -- `kafka.topic.*` — **CONFIRMED** (1 metric found in Tsuga scan over 24h, 2026-02-27) -- `kafka_workload_*` — **CONFIRMED** (3 metrics found in Tsuga scan over 24h, 2026-02-27) - -## Discovery status -- Stage 2 reconciliation complete: 33/33 metrics from `01_kafka_metrics.csv` were confirmed in Tsuga with no missing rows. -- Unexpected discovered metrics: 86 additional Kafka metrics were cataloged in `05_kafka_metric_catalog.csv` and curated for later deep-dive enrichment decisions. -- Counter temporality was reconciled against live metadata: mixed delta and cumulative counters are now represented with exact post-function guidance in `01` and `02`. -- Context field registry was validated from live attributes and `05` was pruned to integration-general keys (`context.env`, `context.team`, `context.service.*`, `context.client-id`, `context.topic`, `context.partition`, `context.node-id`, `context.k8s.*`). -- Producer-side `kafka.producer.*` metrics were still not observed and remain an open verification item. - -## Top sources -- https://kafka.apache.org/documentation/#design — Kafka architecture model and core semantics used for concept map and glossary. -- https://kafka.apache.org/36/documentation/#monitoring — Official monitoring guidance and broker health signal semantics. -- https://kafka.apache.org/36/documentation/#ops — Operational practices and logging/cluster operation context. -- https://docs.confluent.io/platform/current/kafka/monitoring.html — Broker monitoring/JMX operational interpretations. -- https://docs.confluent.io/platform/current/monitor/monitor-consumer-lag.html — Consumer lag operational meaning and troubleshooting patterns. -- https://docs.confluent.io/platform/current/monitor/cp-logging.html — Kafka/Confluent logging configuration and collection context. -- https://raw.githubusercontent.com/open-telemetry/opentelemetry-java-instrumentation/main/instrumentation/kafka/kafka-clients/kafka-clients-common/library/README.md — Authoritative OTel kafka client metric names and definitions. -- https://raw.githubusercontent.com/open-telemetry/opentelemetry-collector-contrib/main/receiver/kafkametricsreceiver/metadata.yaml — OTel collector Kafka broker metric names and metadata mappings. -- https://pkg.go.dev/github.com/open-telemetry/opentelemetry-collector-contrib/receiver/kafkametricsreceiver — Receiver behavior and scope confirmation. -- https://kafka.apache.org/documentation/#consumerconfigs — Consumer-side behavior and config context used for lag/rebalance interpretation guardrails. -- Internal Tsuga metric search (`tools/tsuga_search_metrics.py`, 2026-02-27) — Live evidence for confirmed prefixes and available metric families in this environment. diff --git a/plugins/tsuga/skills/knowledge-technology/references/kafka/metrics.csv b/plugins/tsuga/skills/knowledge-technology/references/kafka/metrics.csv deleted file mode 100644 index 59385d3..0000000 --- a/plugins/tsuga/skills/knowledge-technology/references/kafka/metrics.csv +++ /dev/null @@ -1,35 +0,0 @@ -theme,name,confirmed,definition,unit,type,tsuga_metric_name,widget_suggestions,aggregation,post_function,group_by,required_context_fields,optional_context_fields,absence_means,cardinality_footguns,variants,citations -Availability/Health,kafka.brokers,true,Counts brokers currently visible in Kafka cluster telemetry and acts as a coarse availability baseline.,Unknown,counter,kafka.brokers,query-value|timeseries|top-list,max,none,context.service.name:10|context.scope:10,context.env|context.team,context.service.name|context.k8s.cluster.name|context.service.instance.id,Metric pipeline is disabled for this component or no relevant activity occurred in the window.,Use bounded top-N and avoid grouping by pod plus client-id together.,tsuga=kafka.brokers,internal-tsuga-metric-scan-2026-02-27|https://kafka.apache.org/36/documentation/#monitoring -Availability/Health,kafka.controller.active.count,true,In Kafka controller telemetry this cumulative counter grows with event activity for kafka.controller.active.count.,{count},counter,kafka.controller.active.count,query-value|timeseries|top-list,sum,rate,context.service.name:10|context.scope:10,context.env|context.team,context.service.name|context.k8s.cluster.name|context.service.instance.id,Metric pipeline is disabled for this component or no relevant activity occurred in the window.,Use bounded top-N and avoid grouping by pod plus client-id together.,tsuga=kafka.controller.active.count,internal-tsuga-metric-scan-2026-02-27|https://kafka.apache.org/36/documentation/#monitoring -Capacity/Saturation,kafka.partition.count,true,In Kafka partition telemetry this cumulative counter grows with event activity for kafka.partition.count.,{count},counter,kafka.partition.count,query-value|timeseries|top-list,max,none,context.service.name:10|context.scope:10,context.env|context.team,context.service.name|context.k8s.cluster.name|context.service.instance.id,Metric pipeline is disabled for this component or no relevant activity occurred in the window.,Use bounded top-N and avoid grouping by pod plus client-id together.,tsuga=kafka.partition.count,internal-tsuga-metric-scan-2026-02-27|https://kafka.apache.org/36/documentation/#monitoring -Capacity/Saturation,kafka.partition.underReplicated,true,Counts replication-risk events where partition ISR drops below replication factor and signals durability pressure.,Unknown,counter,kafka.partition.underReplicated,query-value|timeseries|top-list,sum,rate,context.service.name:10|context.scope:10,context.env|context.team,context.service.name|context.k8s.cluster.name|context.service.instance.id,Metric pipeline is disabled for this component or no relevant activity occurred in the window.,Use bounded top-N and avoid grouping by pod plus client-id together.,tsuga=kafka.partition.underReplicated,internal-tsuga-metric-scan-2026-02-27|https://kafka.apache.org/36/documentation/#monitoring -Availability/Health,kafka.partition.offline,true,Counts offline partition events that indicate leader unavailability and direct availability impact.,Unknown,counter,kafka.partition.offline,query-value|timeseries|top-list,sum,rate,context.service.name:10|context.scope:10,context.env|context.team,context.service.name|context.k8s.cluster.name|context.service.instance.id,Metric pipeline is disabled for this component or no relevant activity occurred in the window.,Use bounded top-N and avoid grouping by pod plus client-id together.,tsuga=kafka.partition.offline,internal-tsuga-metric-scan-2026-02-27|https://kafka.apache.org/36/documentation/#monitoring -Capacity/Saturation,kafka.isr.operation.count,true,In Kafka isr telemetry this cumulative counter grows with event activity for kafka.isr.operation.count.,1,counter,kafka.isr.operation.count,query-value|timeseries|top-list,sum,rate,context.service.name:10|context.scope:10,context.env|context.team,context.service.name|context.k8s.cluster.name|context.service.instance.id,Metric pipeline is disabled for this component or no relevant activity occurred in the window.,Use bounded top-N and avoid grouping by pod plus client-id together.,tsuga=kafka.isr.operation.count,internal-tsuga-metric-scan-2026-02-27|https://kafka.apache.org/36/documentation/#monitoring -Errors/Failures,kafka.leaderElection.unclean.count,true,In Kafka leaderElection telemetry this counter reports per-interval event volume for kafka.leaderElection.unclean.count.,{count},counter,kafka.leaderElection.unclean.count,query-value|timeseries|top-list,sum,per-second,context.service.name:10|context.scope:10,context.env|context.team,context.service.name|context.k8s.cluster.name|context.service.instance.id,Metric pipeline is disabled for this component or no relevant activity occurred in the window.,Use bounded top-N and avoid grouping by pod plus client-id together.,tsuga=kafka.leaderElection.unclean.count,internal-tsuga-metric-scan-2026-02-27|https://kafka.apache.org/36/documentation/#monitoring -Performance/Latency,kafka.logs.flush.time.99p,true,In Kafka logs telemetry this gauge reports the current observed level for kafka.logs.flush.time.99p.,ms,gauge,kafka.logs.flush.time.99p,query-value|timeseries|top-list,max,none,context.service.name:10|context.scope:10,context.env|context.team,context.service.name|context.k8s.cluster.name|context.service.instance.id,Metric pipeline is disabled for this component or no relevant activity occurred in the window.,Use bounded top-N and avoid grouping by pod plus client-id together.,tsuga=kafka.logs.flush.time.99p,internal-tsuga-metric-scan-2026-02-27|https://kafka.apache.org/36/documentation/#monitoring -Throughput/Usage,kafka.request.count,true,In Kafka request telemetry this counter reports per-interval event volume for kafka.request.count.,{count},counter,kafka.request.count,query-value|timeseries|top-list,sum,per-second,context.service.name:10|context.scope:10,context.env|context.team,context.service.name|context.k8s.cluster.name|context.service.instance.id,Metric pipeline is disabled for this component or no relevant activity occurred in the window.,Use bounded top-N and avoid grouping by pod plus client-id together.,tsuga=kafka.request.count,internal-tsuga-metric-scan-2026-02-27|https://kafka.apache.org/36/documentation/#monitoring -Errors/Failures,kafka.request.failed,true,Counts failed broker requests and is the numerator for request failure rate analysis.,Unknown,counter,kafka.request.failed,query-value|timeseries|top-list,sum,per-second,context.service.name:10|context.scope:10,context.env|context.team,context.service.name|context.k8s.cluster.name|context.service.instance.id,Metric pipeline is disabled for this component or no relevant activity occurred in the window.,Use bounded top-N and avoid grouping by pod plus client-id together.,tsuga=kafka.request.failed,internal-tsuga-metric-scan-2026-02-27|https://kafka.apache.org/36/documentation/#monitoring -Performance/Latency,kafka.request.time.99p,true,In Kafka request telemetry this gauge reports the current observed level for kafka.request.time.99p.,ms,gauge,kafka.request.time.99p,query-value|timeseries|top-list,max,none,context.service.name:10|context.scope:10,context.env|context.team,context.service.name|context.k8s.cluster.name|context.service.instance.id,Metric pipeline is disabled for this component or no relevant activity occurred in the window.,Use bounded top-N and avoid grouping by pod plus client-id together.,tsuga=kafka.request.time.99p,internal-tsuga-metric-scan-2026-02-27|https://kafka.apache.org/36/documentation/#monitoring -Performance/Latency,kafka.request.time.total,true,In Kafka request telemetry this counter reports per-interval event volume for kafka.request.time.total.,ms,counter,kafka.request.time.total,query-value|timeseries|top-list,sum,per-second,context.service.name:10|context.scope:10,context.env|context.team,context.service.name|context.k8s.cluster.name|context.service.instance.id,Metric pipeline is disabled for this component or no relevant activity occurred in the window.,Use bounded top-N and avoid grouping by pod plus client-id together.,tsuga=kafka.request.time.total,internal-tsuga-metric-scan-2026-02-27|https://kafka.apache.org/36/documentation/#monitoring -Capacity/Saturation,kafka.consumer_group.lag,true,In Kafka consumer_group telemetry this gauge reports the current observed level for kafka.consumer_group.lag.,{count},gauge,kafka.consumer_group.lag,query-value|timeseries|top-list,max,none,context.topic:12|context.partition:12,context.env|context.team,context.topic|context.partition|context.scope,Metric pipeline is disabled for this component or no relevant activity occurred in the window.,Use bounded top-N and avoid grouping by pod plus client-id together.,tsuga=kafka.consumer_group.lag,internal-tsuga-metric-scan-2026-02-27|https://kafka.apache.org/36/documentation/#monitoring -Capacity/Saturation,kafka.consumer_group.lag_sum,true,In Kafka consumer_group telemetry this gauge reports the current observed level for kafka.consumer_group.lag_sum.,{count},gauge,kafka.consumer_group.lag_sum,query-value|timeseries|top-list,max,none,context.topic:12|context.scope:10,context.env|context.team,context.topic|context.scope,Metric pipeline is disabled for this component or no relevant activity occurred in the window.,Use bounded top-N and avoid grouping by pod plus client-id together.,tsuga=kafka.consumer_group.lag_sum,internal-tsuga-metric-scan-2026-02-27|https://kafka.apache.org/36/documentation/#monitoring -Availability/Health,kafka.consumer_group.members,true,Reports current consumer group membership cardinality and helps detect churn and assignment instability.,Unknown,counter,kafka.consumer_group.members,query-value|timeseries|top-list,max,none,context.scope:10|context.team:10,context.env|context.team,context.scope|context.unit,Metric pipeline is disabled for this component or no relevant activity occurred in the window.,Use bounded top-N and avoid grouping by pod plus client-id together.,tsuga=kafka.consumer_group.members,internal-tsuga-metric-scan-2026-02-27|https://kafka.apache.org/36/documentation/#monitoring -Throughput/Usage,kafka.consumer.records_consumed_total,true,In Kafka consumer telemetry this counter reports per-interval event volume for kafka.consumer.records_consumed_total.,{count},counter,kafka.consumer.records_consumed_total,query-value|timeseries|top-list,sum,per-second,context.client-id:12|context.service.instance.id:12,context.env|context.team,context.service.name|context.k8s.cluster.name|context.service.instance.id|context.client-id,Metric pipeline is disabled for this component or no relevant activity occurred in the window.,Use bounded top-N and avoid grouping by pod plus client-id together.,tsuga=kafka.consumer.records_consumed_total,internal-tsuga-metric-scan-2026-02-27|https://kafka.apache.org/36/documentation/#monitoring -Capacity/Saturation,kafka.consumer.records_lag_max,true,Tracks the highest consumer lag observed and indicates worst-case processing delay risk.,{count},gauge,kafka.consumer.records_lag_max,query-value|timeseries|top-list,max,none,context.client-id:12|context.service.instance.id:12,context.env|context.team,context.service.name|context.k8s.cluster.name|context.service.instance.id|context.client-id,Metric pipeline is disabled for this component or no relevant activity occurred in the window.,Use bounded top-N and avoid grouping by pod plus client-id together.,tsuga=kafka.consumer.records_lag_max,internal-tsuga-metric-scan-2026-02-27|https://kafka.apache.org/36/documentation/#monitoring -Throughput/Usage,kafka.consumer.bytes_consumed_total,true,In Kafka consumer telemetry this counter reports per-interval event volume for kafka.consumer.bytes_consumed_total.,By,counter,kafka.consumer.bytes_consumed_total,query-value|timeseries|top-list,sum,per-second,context.client-id:12|context.service.instance.id:12,context.env|context.team,context.service.name|context.k8s.cluster.name|context.service.instance.id|context.client-id,Metric pipeline is disabled for this component or no relevant activity occurred in the window.,Use bounded top-N and avoid grouping by pod plus client-id together.,tsuga=kafka.consumer.bytes_consumed_total,internal-tsuga-metric-scan-2026-02-27|https://kafka.apache.org/36/documentation/#monitoring -Performance/Latency,kafka.consumer.fetch_latency_max,true,In Kafka consumer telemetry this gauge reports the current observed level for kafka.consumer.fetch_latency_max.,ms,gauge,kafka.consumer.fetch_latency_max,query-value|timeseries|top-list,max,none,context.client-id:12|context.service.instance.id:12,context.env|context.team,context.service.name|context.k8s.cluster.name|context.service.instance.id|context.client-id,Metric pipeline is disabled for this component or no relevant activity occurred in the window.,Use bounded top-N and avoid grouping by pod plus client-id together.,tsuga=kafka.consumer.fetch_latency_max,internal-tsuga-metric-scan-2026-02-27|https://kafka.apache.org/36/documentation/#monitoring -Performance/Latency,kafka.consumer.fetch_throttle_time_max,true,In Kafka consumer telemetry this gauge reports the current observed level for kafka.consumer.fetch_throttle_time_max.,Unknown,gauge,kafka.consumer.fetch_throttle_time_max,query-value|timeseries|top-list,max,none,context.client-id:12|context.service.instance.id:12,context.env|context.team,context.service.name|context.k8s.cluster.name|context.service.instance.id|context.client-id,Metric pipeline is disabled for this component or no relevant activity occurred in the window.,Use bounded top-N and avoid grouping by pod plus client-id together.,tsuga=kafka.consumer.fetch_throttle_time_max,internal-tsuga-metric-scan-2026-02-27|https://kafka.apache.org/36/documentation/#monitoring -Errors/Failures,kafka.consumer.failed_rebalance_total,true,In Kafka consumer telemetry this counter reports per-interval event volume for kafka.consumer.failed_rebalance_total.,Unknown,counter,kafka.consumer.failed_rebalance_total,query-value|timeseries|top-list,sum,per-second,context.client-id:12|context.service.instance.id:12,context.env|context.team,context.service.name|context.k8s.cluster.name|context.service.instance.id|context.client-id,Metric pipeline is disabled for this component or no relevant activity occurred in the window.,Use bounded top-N and avoid grouping by pod plus client-id together.,tsuga=kafka.consumer.failed_rebalance_total,internal-tsuga-metric-scan-2026-02-27|https://kafka.apache.org/36/documentation/#monitoring -Throughput/Usage,kafka.consumer.rebalance_total,true,In Kafka consumer telemetry this counter reports per-interval event volume for kafka.consumer.rebalance_total.,Unknown,counter,kafka.consumer.rebalance_total,query-value|timeseries|top-list,sum,per-second,context.client-id:12|context.service.instance.id:12,context.env|context.team,context.service.name|context.k8s.cluster.name|context.service.instance.id|context.client-id,Metric pipeline is disabled for this component or no relevant activity occurred in the window.,Use bounded top-N and avoid grouping by pod plus client-id together.,tsuga=kafka.consumer.rebalance_total,internal-tsuga-metric-scan-2026-02-27|https://kafka.apache.org/36/documentation/#monitoring -Errors/Failures,kafka.consumer.failed_authentication_total,true,In Kafka consumer telemetry this counter reports per-interval event volume for kafka.consumer.failed_authentication_total.,Unknown,counter,kafka.consumer.failed_authentication_total,query-value|timeseries|top-list,sum,per-second,context.client-id:12|context.service.instance.id:12,context.env|context.team,context.service.name|context.k8s.cluster.name|context.service.instance.id|context.client-id,Metric pipeline is disabled for this component or no relevant activity occurred in the window.,Use bounded top-N and avoid grouping by pod plus client-id together.,tsuga=kafka.consumer.failed_authentication_total,internal-tsuga-metric-scan-2026-02-27|https://kafka.apache.org/36/documentation/#monitoring -Throughput/Usage,kafka.consumer.successful_authentication_total,true,In Kafka consumer telemetry this counter reports per-interval event volume for kafka.consumer.successful_authentication_total.,Unknown,counter,kafka.consumer.successful_authentication_total,query-value|timeseries|top-list,sum,per-second,context.client-id:12|context.service.instance.id:12,context.env|context.team,context.service.name|context.k8s.cluster.name|context.service.instance.id|context.client-id,Metric pipeline is disabled for this component or no relevant activity occurred in the window.,Use bounded top-N and avoid grouping by pod plus client-id together.,tsuga=kafka.consumer.successful_authentication_total,internal-tsuga-metric-scan-2026-02-27|https://kafka.apache.org/36/documentation/#monitoring -Performance/Latency,kafka.consumer.commit_latency_max,true,In Kafka consumer telemetry this gauge reports the current observed level for kafka.consumer.commit_latency_max.,ms,gauge,kafka.consumer.commit_latency_max,query-value|timeseries|top-list,max,none,context.client-id:12|context.service.instance.id:12,context.env|context.team,context.service.name|context.k8s.cluster.name|context.service.instance.id|context.client-id,Metric pipeline is disabled for this component or no relevant activity occurred in the window.,Use bounded top-N and avoid grouping by pod plus client-id together.,tsuga=kafka.consumer.commit_latency_max,internal-tsuga-metric-scan-2026-02-27|https://kafka.apache.org/36/documentation/#monitoring -Throughput/Usage,kafka.consumer.commit_total,true,In Kafka consumer telemetry this counter reports per-interval event volume for kafka.consumer.commit_total.,Unknown,counter,kafka.consumer.commit_total,query-value|timeseries|top-list,sum,per-second,context.client-id:12|context.service.instance.id:12,context.env|context.team,context.service.name|context.k8s.cluster.name|context.service.instance.id|context.client-id,Metric pipeline is disabled for this component or no relevant activity occurred in the window.,Use bounded top-N and avoid grouping by pod plus client-id together.,tsuga=kafka.consumer.commit_total,internal-tsuga-metric-scan-2026-02-27|https://kafka.apache.org/36/documentation/#monitoring -Availability/Health,kafka.consumer.connection_count,true,In Kafka consumer telemetry this gauge reports the current observed level for kafka.consumer.connection_count.,{count},gauge,kafka.consumer.connection_count,query-value|timeseries|top-list,max,none,context.client-id:12|context.service.instance.id:12,context.env|context.team,context.service.name|context.k8s.cluster.name|context.service.instance.id|context.client-id,Metric pipeline is disabled for this component or no relevant activity occurred in the window.,Use bounded top-N and avoid grouping by pod plus client-id together.,tsuga=kafka.consumer.connection_count,internal-tsuga-metric-scan-2026-02-27|https://kafka.apache.org/36/documentation/#monitoring -Throughput/Usage,kafka.consumer.request_total,true,In Kafka consumer telemetry this counter reports per-interval event volume for kafka.consumer.request_total.,Unknown,counter,kafka.consumer.request_total,query-value|timeseries|top-list,sum,per-second,context.client-id:12|context.service.instance.id:12,context.env|context.team,context.service.name|context.k8s.cluster.name|context.service.instance.id|context.client-id,Metric pipeline is disabled for this component or no relevant activity occurred in the window.,Use bounded top-N and avoid grouping by pod plus client-id together.,tsuga=kafka.consumer.request_total,internal-tsuga-metric-scan-2026-02-27|https://kafka.apache.org/36/documentation/#monitoring -Performance/Latency,kafka.consumer.request_latency_max,true,In Kafka consumer telemetry this gauge reports the current observed level for kafka.consumer.request_latency_max.,ms,gauge,kafka.consumer.request_latency_max,query-value|timeseries|top-list,max,none,context.client-id:12|context.service.instance.id:12,context.env|context.team,context.service.name|context.k8s.cluster.name|context.service.instance.id|context.client-id,Metric pipeline is disabled for this component or no relevant activity occurred in the window.,Use bounded top-N and avoid grouping by pod plus client-id together.,tsuga=kafka.consumer.request_latency_max,internal-tsuga-metric-scan-2026-02-27|https://kafka.apache.org/36/documentation/#monitoring -Throughput/Usage,kafka.consumer.network_io_total,true,In Kafka consumer telemetry this counter reports per-interval event volume for kafka.consumer.network_io_total.,By,counter,kafka.consumer.network_io_total,query-value|timeseries|top-list,sum,per-second,context.client-id:12|context.service.instance.id:12,context.env|context.team,context.service.name|context.k8s.cluster.name|context.service.instance.id|context.client-id,Metric pipeline is disabled for this component or no relevant activity occurred in the window.,Use bounded top-N and avoid grouping by pod plus client-id together.,tsuga=kafka.consumer.network_io_total,internal-tsuga-metric-scan-2026-02-27|https://kafka.apache.org/36/documentation/#monitoring -Throughput/Usage,kafka_workload_messages_produced_total,true,In Kafka workload telemetry this cumulative counter grows with event activity for kafka_workload_messages_produced_total.,{count},counter,kafka_workload_messages_produced_total,query-value|timeseries|top-list,sum,rate,context.topic:12|context.service.name:10,context.env|context.team,context.service.name|context.k8s.cluster.name|context.service.instance.id|context.topic,Metric pipeline is disabled for this component or no relevant activity occurred in the window.,Use bounded top-N and avoid grouping by pod plus client-id together.,tsuga=kafka_workload_messages_produced_total,internal-tsuga-metric-scan-2026-02-27|https://kafka.apache.org/36/documentation/#monitoring -Throughput/Usage,kafka_workload_messages_consumed_total,true,In Kafka workload telemetry this cumulative counter grows with event activity for kafka_workload_messages_consumed_total.,{count},counter,kafka_workload_messages_consumed_total,query-value|timeseries|top-list,sum,rate,context.topic:12|context.service.name:10,context.env|context.team,context.service.name|context.k8s.cluster.name|context.service.instance.id|context.topic,Metric pipeline is disabled for this component or no relevant activity occurred in the window.,Use bounded top-N and avoid grouping by pod plus client-id together.,tsuga=kafka_workload_messages_consumed_total,internal-tsuga-metric-scan-2026-02-27|https://kafka.apache.org/36/documentation/#monitoring -Performance/Latency,kafka_workload_operation_latency_ms,true,Captures synthetic Kafka operation latency distribution for end-to-end workload response validation.,ms,histogram,kafka_workload_operation_latency_ms,timeseries|query-value,percentile(p95),none,context.topic:12|context.service.name:10,context.env|context.team,context.service.name|context.k8s.cluster.name|context.service.instance.id|context.topic,Metric pipeline is disabled for this component or no relevant activity occurred in the window.,Use bounded top-N and avoid grouping by pod plus client-id together.,tsuga=kafka_workload_operation_latency_ms,internal-tsuga-metric-scan-2026-02-27|https://kafka.apache.org/36/documentation/#monitoring -Cost,NOT_APPLICABLE,false,Kafka metrics in this integration do not provide direct billing data and cost should be inferred from throughput storage and replication pressure.,Unknown,Unknown,NOT_APPLICABLE,query-value,Unknown,none,context.env:5|context.team:10,context.env|context.team,context.service.name,Not applicable for direct cost accounting.,Safe,n/a,https://kafka.apache.org/documentation/#design diff --git a/plugins/tsuga/skills/knowledge-technology/references/kafka/queries.md b/plugins/tsuga/skills/knowledge-technology/references/kafka/queries.md deleted file mode 100644 index 1925b98..0000000 --- a/plugins/tsuga/skills/knowledge-technology/references/kafka/queries.md +++ /dev/null @@ -1,54 +0,0 @@ -# Kafka - -Distributed log. Producers append to partitions, consumers read by offset. Healthy: consumer lag bounded, consume ≈ produce rate, low rebalance churn, zero under-replicated / offline partitions. - -## Incident shapes - -- **Consumer lag growth** — consume rate drops, produce flat → check per-partition lag -- **Rebalance churn** — frequent rebalances halve throughput → check rebalance_rate -- **Broker / partition regression** — under-replicated or offline partitions → check partition health -- **Poison message / stuck partition** — one partition stuck at one offset → drill per-partition lag - -## Key metrics - -| Metric | Unit | Signal | -|---|---|---| -| `kafka.consumer.lag` | records | Per-partition delay. Primary user-visible signal | -| `kafka.consumer.records_consumed_rate` | rec/s | Falling while produce flat = consumer issue | -| `kafka.producer.record_send_rate` | rec/s | Produce baseline | -| `kafka.partition.under_replicated` | count | Any nonzero = replication risk | -| `kafka.partition.offline` | count | Any nonzero = availability incident | -| `kafka.controller.active_count` | count | Should be 1; churn = instability | -| `kafka.consumer.rebalance_rate` | events/s | Frequent rebalances = coordination problem | -| `kafka.consumer.commit_latency` | ms | Commit spikes = replay/duplicate risk | -| `kafka.consumer.fetch.latency` | ms | Poll latency | -| `kafka.request.failure_rate` | fail/s | Broker request failures | -| `kafka.request.throttle_rate` | 1/s | Sustained throttle = contention or quota | -| `kafka.broker.disk_usage` | bytes | Disk fill → writes throttle | -| `kafka.log.flush_time` | ms | Flush spikes precede request-latency climbs | - -## Derived signals - -- First-derivative of `consumer.lag` — lag trajectory. Positive sustained = backlog growing. -- `records_consumed_rate / record_send_rate` — consume-vs-produce ratio. Healthy ≈ 1.0. -- `partition.under_replicated + partition.offline` — partition risk. Any nonzero = active incident. -- Rebalances-per-hour / group-members — churn ratio. > 1 = pathological rebalancing. - -## Log patterns - -- `Shrinking ISR for partition` — replica falling behind -- `Partition [X-N] is under min ISR` — replication risk -- `Controller moved to another broker` — controller churn -- `Group [X] is rebalancing` / `Preparing to rebalance` — rebalance start -- `Marking the coordinator dead` — coordinator connection failure -- `Offset commit failed` — commit-latency / leadership change -- `Attempt to join group failed due to fatal error` — auth / ACL -- `Failed to send record` with back-offs — producer can't reach leader - -## Gotchas - -- Aggregate consumer lag hides stuck partitions. Drill per-partition before concluding "consumer is keeping up." -- Lag in records ≠ lag in time. 10k records at 1 msg/s is much worse than 100k at 100k msg/s. Compute time-to-catch-up. -- A queue "clearing" can be retention dropping records (data loss), not consumers catching up. -- `under_replicated = 0` at investigation time doesn't mean the incident didn't involve replication — check the timeseries. -- Producer retries mask broker degradation: steady produce rate + rising broker failures = silent degradation. diff --git a/plugins/tsuga/skills/knowledge-technology/references/kubernetes/README.md b/plugins/tsuga/skills/knowledge-technology/references/kubernetes/README.md deleted file mode 100644 index eb162fb..0000000 --- a/plugins/tsuga/skills/knowledge-technology/references/kubernetes/README.md +++ /dev/null @@ -1,379 +0,0 @@ -# Kubernetes Integration Context Bundle - -## Metadata - -**Technology:** Kubernetes -**Deployment:** self-hosted -**Environment:** prod -**Persona:** SRE Dev and ops -**Telemetry preference:** mixed (OTel — kubeletstats + k8scluster receivers) -**Integration scope:** core service only -**Primary use-case:** reliability and performance - ---- - -## How to use this bundle - -- `01_kubernetes_metrics.csv` — Source of truth for all metrics: names, units, temporality, safe aggregations, group-bys, and Tsuga field mappings. -- `02_kubernetes_dashboard_plan.yaml` — Dashboard plan: sections, widgets, derived signals, explanation notes, triage chains, and playbooks. -- `03_kubernetes_state.yaml` — Machine-readable state: stage status, unknowns, assumptions, and log intel for Stage 4. -- `04_kubernetes_memory.md` — Human-readable Stage 1 summary: key assumptions, tradeoffs, and what Stage 2 must verify first. -- Stage 2 will create `05_kubernetes_metric_catalog.csv` as the discovered metric catalog (confirmed names + attribute keys + curated descriptions). -- Stage 4 should read `00` "Log intelligence (Stage 4 handoff)" and `03.log_intel` first before designing log routes. - ---- - -## What it is and what "good" looks like - -### Confirmed by sources - -Kubernetes is an open-source container orchestration platform that automates deployment, scaling, and lifecycle management of containerized workloads. Metrics are emitted by two OpenTelemetry receivers: `kubeletstats` (per-node/pod/container resource usage from the kubelet API) and `k8scluster` (cluster-state counts from the Kubernetes API: replicas, phases, conditions). [OTel kubeletstats receiver docs](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/kubeletstatsreceiver), [OTel k8scluster receiver docs](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/k8sclusterreceiver) - -**"Good" looks like:** -- All pods in Running phase; zero in Pending for >5 minutes; Failed count at baseline (near zero) -- Container restarts at a steady low baseline (<3/day for healthy services) -- All nodes in Ready condition; no NotReady nodes -- CPU limit utilization < 70%; memory limit utilization < 80% (headroom for spikes) -- Deployment desired == available for all production workloads -- DaemonSet coverage 100% (every eligible node is scheduled) -- Node filesystem < 70% used (eviction threshold is typically 85%) - -**Paging intent:** Page on pod crash loops (restart rate sudden spike), node NotReady (condition_ready = 0), sustained Pending pods (scheduler can't place work), or deployment replicas unavailable. - -**Top 3 incident shapes:** -1. **OOMKill cascade** → start with "CPU & Memory" section: memory limit utilization > 100% → container restarts → pod phase Failed -2. **Node disk pressure eviction** → start with "Storage" section: filesystem utilization > 85% → pods evicted → Running Pod Ratio drops -3. **Deployment rollout stuck** → start with "Workloads" section: Unavailable Replicas > 0 for >5min → Pod Phase by Deployment shows pending/failed pods - -### Best-practice inference - -- Control plane metrics (kube-apiserver, etcd, scheduler) are not covered in this integration — they require Prometheus scraping of component endpoints and are out of scope for the OTel-based integration. [Inference — standard OTel scope boundary] -- StatefulSet state metrics (desired/current/ready/updated pods) are not currently flowing into Tsuga (k8scluster receiver not deployed or misconfigured). StatefulSet workloads can still be observed via pod-level metrics (pod phase, container restarts, resource usage). [Inference from analysis — receiver config gap] - ---- - -## Key concepts - -### Glossary - -| Term | Definition | Operational meaning | Dashboard section | -|------|-----------|---------------------|-------------------| -| Pod | Smallest schedulable unit; 1+ containers sharing network/storage | A "broken" pod shows as phase=Failed or restart loop | cluster-health | -| Container | Runtime unit inside a pod; has its own resource limits | CPU/memory limits trigger throttling or OOMKill | cpu-memory | -| Node | Worker machine in the cluster (VM or bare metal) | NotReady node = workloads can't run there | nodes | -| Namespace | Logical grouping of resources; multi-tenancy boundary | High restarts in one namespace = team-level issue | cluster-health | -| Deployment | Declares desired pod count + rollout strategy | Unavailable replicas = partial degradation | workloads | -| ReplicaSet | Controller maintaining exact pod count for a Deployment | desired - available = unhealthy pod gap | workloads | -| DaemonSet | Ensures one pod per (matching) node | Coverage < 100% = some nodes unprotected | workloads | -| StatefulSet | Deployment for stateful apps (ordered, stable identity) | State metrics not flowing — observe via pod metrics | workloads | -| Job | Run-to-completion task; creates pods that terminate when done | Failed pods = job didn't complete | jobs | -| CronJob | Scheduled Jobs | active_jobs count growing = stuck previous runs | jobs | -| kubelet | Node agent that manages pod lifecycle on each node | Source of all kubeletstats metrics | nodes | -| k8scluster receiver | OTel receiver polling the Kubernetes API server | Source of replica counts, pod phase, conditions | workloads | -| kubeletstats receiver | OTel receiver polling the kubelet `/stats/summary` API | Source of CPU/memory/network/filesystem metrics | nodes, cpu-memory | -| Nanocores | CPU unit: 1 core = 1,000,000,000 nanocores | Raw unit of k8s.*.cpu.usage — large numbers | cpu-memory | -| Resource Limit | Hard cap on CPU/memory a container can use | Exceeding memory limit → OOMKill; CPU limit → throttle | cpu-memory | -| Resource Request | Soft reservation used by scheduler for placement | Requests >> limits = node overcommit risk | cpu-memory | -| Node Condition | Status flags: Ready, MemoryPressure, DiskPressure, PIDPressure | Ready=false = node unavailable | nodes | -| Eviction | Kubelet forcibly terminates pods under resource pressure | Triggered at filesystem > 85% or memory pressure | storage | -| Taint/Toleration | Node taints repel pods unless they have matching toleration | Misscheduled daemonset nodes = taint mismatch | workloads | -| CrashLoopBackOff | Pod restarts repeatedly with exponential backoff | Application crashes; check pod logs | cluster-health | -| OOMKill | Out-of-memory: container exceeds memory limit | Memory limit utilization > 100% transiently | cpu-memory | -| Phase | Pod lifecycle state: Pending/Running/Succeeded/Failed/Unknown | Only Running is healthy for long-running services | cluster-health | -| Allocatable | Node resources available for pod scheduling (capacity minus OS/kubelet overhead) | Used to compute actual utilization headroom | nodes | - -### Concept Map - -``` -Cluster -> contains -> Node (why: nodes are the execution substrate for all workloads) -Node -> runs -> kubelet (why: kubelet manages pod lifecycle and exposes kubeletstats metrics) -Node -> has -> Condition (Ready/MemoryPressure/DiskPressure) (why: conditions gate pod scheduling) -Deployment -> manages -> ReplicaSet (why: rolling updates create new RS, phase out old RS) -ReplicaSet -> controls -> Pod (why: RS ensures desired pod count is maintained) -DaemonSet -> creates -> Pod per node (why: node-level agents like log collectors run via DaemonSet) -Pod -> runs on -> Node (why: scheduler places pods on nodes with sufficient allocatable resources) -Pod -> contains -> Container (why: containers share network namespace within a pod) -Container -> has -> ResourceRequest + ResourceLimit (why: requests drive scheduling; limits enforce caps) -Container -> consumes -> CPU/memory from Node allocatable pool (why: total container requests must fit within node allocatable) -CronJob -> spawns -> Job (why: CronJob is the scheduler; Job is the execution unit) -Job -> creates -> Pod (why: job work runs inside pods that terminate on completion) -container.cpu.usage (nanocores) -> feeds -> CPU Limit Utilization % derived signal (why: utilization = usage/limit) -k8s.node.memory.usage -> relative to -> k8s.node.allocatable_memory (why: utilization = usage/allocatable, not capacity) -k8s.pod.phase -> aggregated by -> k8s.namespace.name (why: phase breakdown per namespace shows team-level health) -k8s.container.restarts (cumulative) -> rising rate -> CrashLoopBackOff symptom (why: restarts are the OTel proxy for crash loops) -k8s.deployment.desired - k8s.deployment.available -> Unavailable Replicas signal (why: gap = degraded replicas) -k8s.daemonset.desired_scheduled_nodes - k8s.daemonset.current_scheduled_nodes -> scheduling gap (why: gap = nodes without the daemonset pod) -k8s.node.network.io (cumulative) -> rate() -> bandwidth in/out (why: cumulative bytes need rate for per-second throughput) -k8s.node.filesystem.usage / k8s.node.filesystem.capacity -> Filesystem Utilization % (why: raw bytes meaningless without context of total) -Node DiskPressure condition -> triggers -> pod eviction (why: kubelet evicts pods when disk is full) -Node MemoryPressure condition -> triggers -> pod eviction (why: kubelet evicts pods when memory is exhausted) -``` - -### Entities and dimensions - -| Dimension | Why useful | Cardinality risk | Safe top-N | -|-----------|-----------|-----------------|-----------| -| `context.k8s.cluster.name` | Multi-cluster comparison | Low (typically 1-20 clusters) | 10 | -| `context.k8s.namespace.name` | Team/application boundary | Medium (10-100 namespaces) | 20 | -| `context.k8s.node.name` | Node-level triage | Medium (10-500 nodes) | 20 | -| `context.k8s.pod.name` | Per-pod investigation | HIGH — do NOT use as primary group-by in overview; use in deep dive top-lists only | 20 | -| `context.k8s.container.name` | Container-level breakdown | Medium (bounded by workload count) | 20 | -| `context.k8s.deployment.name` | Deployment health | Low-medium | 20 | -| `context.k8s.daemonset.name` | DaemonSet status | Low | 20 | -| `context.k8s.replicaset.name` | ReplicaSet health | Medium (old RS kept during rollouts) | 20 | -| `context.k8s.job.name` | Job tracking | Low-medium | 20 | -| `context.k8s.cronjob.name` | CronJob tracking | Low | 20 | -| `context.direction` | Network I/O direction (receive/transmit) | Fixed (2 values) — safe for group-by | 2 | -| `context.state` | CPU state (idle/user/system/iowait/steal...) / Memory state | Fixed (5-8 values) — safe for group-by | 8 | -| `context.k8s.pod.status.phase` | Pod phase breakdown | Fixed (5 values: Running/Pending/Failed/Succeeded/Unknown) | 6 | -| `context.env` | Environment separation | Fixed (2-5 values) | 5 | -| `context.team` | Team ownership | Low (10-50 teams) | 20 | - -**Do NOT group-by:** `context.k8s.pod.name` in timeseries for large clusters (causes 100s of series). Use top-list widgets for pod-level breakdowns instead. - -### Tsuga field mapping - -**Confirmed by sources:** `context.env` and `context.team` are confirmed in `.env`. - -**Best-practice inference (OTel → context.* naming convention, must be verified in Stage 2):** - -| Vendor/exporter dimension | Recommended context.* key | Must-exist vs optional | -|--------------------------|--------------------------|----------------------| -| `k8s.cluster.name` | `context.k8s.cluster.name` | must-exist (all k8s metrics) | -| `k8s.namespace.name` | `context.k8s.namespace.name` | must-exist (pod/workload metrics) | -| `k8s.node.name` | `context.k8s.node.name` | must-exist (node + kubeletstats metrics) | -| `k8s.pod.name` | `context.k8s.pod.name` | must-exist (pod-level metrics) | -| `k8s.container.name` | `context.k8s.container.name` | must-exist (container-level metrics) | -| `k8s.deployment.name` | `context.k8s.deployment.name` | optional (workload metrics) | -| `k8s.daemonset.name` | `context.k8s.daemonset.name` | optional (daemonset metrics) | -| `k8s.replicaset.name` | `context.k8s.replicaset.name` | optional (replicaset metrics) | -| `k8s.statefulset.name` | `context.k8s.statefulset.name` | optional (not flowing yet) | -| `k8s.job.name` | `context.k8s.job.name` | optional (job metrics) | -| `k8s.cronjob.name` | `context.k8s.cronjob.name` | optional (cronjob metrics) | -| `direction` (network I/O) | `context.direction` | must-exist (network I/O metrics) | -| `state` (CPU/memory) | `context.state` | must-exist (system.cpu.utilization, system.memory.*) | -| `k8s.pod.status.phase` | `context.k8s.pod.status.phase` | must-exist (k8s.pod.phase metric) | -| `env` | `context.env` | must-exist (from .env) | -| `team` | `context.team` | must-exist (from .env) | - ---- - -## Golden signals - -### Traffic (throughput) - -**What it means for Kubernetes:** Network I/O (bytes/s in and out) at node and pod level. Application-level request throughput is not in scope here — it requires application-specific instrumentation. - -**Degradation causes:** unusual pod-to-pod chatting, log floods, large data transfers, misconfigured sidecar proxies. - -**Best telemetry:** `k8s.node.network.io`, `k8s.pod.network.io` (cumulative sums → use `rate()`) - -**Section questions:** Is node/pod network I/O within normal ranges? Are any pods dominating network usage? - -### Errors - -**What it means for Kubernetes:** Container restarts (proxy for crash loops), pod phase failures, network errors, and failed job pods. - -**Degradation causes:** application bugs (OOMKill, unhandled exceptions), image pull errors, misconfigured resource limits, broken network policies, disk full. - -**What people page on:** sustained container restart rate spike (CrashLoopBackOff), pods stuck in Failed phase, Unavailable Replicas > 0. - -**Best telemetry:** `k8s.container.restarts` (cumulative → `rate()` or `increase()`), `k8s.pod.phase` (filter phase:failed), `k8s.job.failed_pods`, `k8s.node.network.errors` - -**Section questions:** Are container restart rates spiking? Are pods in Failed/Pending phase? Are jobs completing successfully? - -### Latency - -**What it means for Kubernetes:** At the platform level, "latency" manifests as pod scheduling delay (time pods spend in Pending), not request latency (which is application-level). CPU throttling from limit hits indirectly increases response times. - -**Best telemetry:** Pod phase timeseries (Pending count over time); CPU limit utilization (> 90% = throttling). - -**Section questions:** How long are pods staying in Pending? Are containers hitting CPU limits? - -**Confirmed by sources:** CPU throttling from cgroup limit enforcement is a well-documented Kubernetes behavior. [Kubernetes resource limits docs](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/) - -**Best-practice inference:** Scheduling latency (time to transition from Pending to Running) is not directly exposed as an OTel metric. Pod phase timeseries is the best available proxy. - -### Saturation - -**What it means for Kubernetes:** Node-level resource saturation (CPU, memory, disk) determines whether new pods can be scheduled and whether existing pods will be evicted. Container-level saturation (usage vs limit) determines throttling and OOMKill risk. - -**Degradation causes:** memory overcommit (requests << limits), log volume filling node disks, noisy neighbor pods. - -**What people page on:** Node MemoryPressure or DiskPressure condition (precursor to eviction), memory limit utilization > 90% across many pods. - -**Best telemetry:** `k8s.node.memory.usage / k8s.node.allocatable_memory`, `k8s.node.filesystem.usage / k8s.node.filesystem.capacity`, `container.memory.usage / k8s.container.memory_limit`, `k8s.node.condition_ready` - -**Section questions:** Are node memory/filesystem utilizations approaching eviction thresholds? Are containers near their limits? - ---- - -## Telemetry sources - -| Source type | How collected | What it provides | Pros | Cons | Common "no data" meaning | -|------------|--------------|-----------------|------|------|--------------------------| -| OTel `kubeletstats` receiver | Polls kubelet `/stats/summary` API on each node | CPU/memory/network/filesystem usage at node, pod, container granularity | Rich resource usage; includes nanocores, bytes, network I/O | No cluster-state counts (replicas, phases); CPU metrics in nanocores (hard to read) | Receiver not deployed or wrong node RBAC | -| OTel `k8scluster` receiver | Polls Kubernetes API server | Pod phase, container readiness, replica counts, conditions, resource requests/limits | Cluster-state truth; captures Pending/Failed pods | Doesn't include resource usage (only config values) | Receiver not deployed or API server RBAC missing | -| `system.*` metrics | Host-level OTel `hostmetrics` receiver (or kubeletstats node-level) | Node CPU utilization by state, memory by state | Familiar % format; useful for node CPU utilization | May be emitted per-pod/per-container in Kubernetes context — groupBy carefully | Hostmetrics receiver not deployed | - -**Optional features that change metric availability:** -- `k8s.pod.memory.node.utilization` (kubeletstats): requires `k8s_api_config` + explicit opt-in in receiver config -- `k8s.node.allocatable_cpu` (k8scluster): requires `allocatable_types_to_report: [cpu]` in receiver config -- StatefulSet metrics: require k8scluster receiver to be deployed (currently not flowing) - -**Confirmed by sources:** OTel kubeletstats and k8scluster receiver documentation. [kubeletstats](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/kubeletstatsreceiver) [k8scluster](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/k8sclusterreceiver) - -**Best-practice inference:** The `hostmetrics` receiver may run alongside the Kubernetes receivers for node-level `system.*` metrics. The exact deployment topology varies by organization. - ---- - -## Log intelligence (Stage 4 handoff) - -### Confirmed by sources - -**Log sources matrix:** - -| Source | Access path | Typical format | Structured vs unstructured | Evidence | -|--------|------------|----------------|---------------------------|---------| -| Container stdout/stderr | Kubelet log driver → fluentd/fluent-bit → Tsuga | Kubernetes log line with metadata prefix OR raw app log | Unstructured (depends on app) | Kubernetes logging docs | -| Kubernetes audit log | API server → configurable audit sink | JSON structured | Structured JSON | Kubernetes audit log docs | -| Kubelet system log | Node journald or /var/log/kubelet.log | Go log format (key=value style) | Semi-structured | Kubernetes node logs docs | -| kube-system pods | Container stdout (scheduler, controller-manager) | Go log format or structured JSON | Mixed | Standard k8s component logs | - -**Known log formats:** - -Format 1 — Kubernetes container log (containerd/CRI format): -``` -2024-01-15T10:30:00.123456789Z stdout F {"level":"info","msg":"request processed","method":"GET","path":"/health","status":200,"duration_ms":5} -``` -- Fields: `timestamp` (RFC3339 nanoseconds), `stream` (stdout/stderr), `flags` (F=full line, P=partial), `message` (rest of line) -- Timestamp: RFC3339 with nanoseconds -- JSON application logs are common but NOT guaranteed — depends on the application - -Format 2 — Kubernetes container log (Docker legacy format): -``` -{"log":"INFO: Starting up server on port 8080\n","stream":"stdout","time":"2024-01-15T10:30:00.123456789Z"} -``` -- JSON wrapper with `log`, `stream`, `time` fields - -Format 3 — Go structured log (kube-system components): -``` -I0115 10:30:00.123456 1 server.go:123] "Starting controller" name="deployments" -``` -- Format: `