Added 1.0 observability topics - #533
kristin-kronstain-brown wants to merge 4 commits into
Conversation
Signed-off-by: Kristin Brown <kristin.brown@solo.io>
Docs preview
Both are uploaded Worker versions and serve no production traffic. |
| weight: 20 | ||
| author: kagent.dev | ||
| --- | ||
|
|
There was a problem hiding this comment.
Random thought, and i dont think there is a precedent for this in sibling oss projects - but i wonder whether we could provide a packaged script for quick install at the top of this page, in case anyone doesnt want to go through each individual step on the whole page? just as an option. or maybe that is what the lightweight version is meant to be? food for thought
| | Signal | Where it goes | Details | | ||
| | ------ | ------------- | ------- | | ||
| | Actor logs | The stdout of the Worker pod that runs the Actor | The Actor's own output, and records for each suspend and resume. See [Read an Actor's logs](#read-an-actors-logs). | | ||
| | Component logs | Over OTLP, when an OTLP endpoint is set | Logs from the Agent Substrate components, including an access log entry for every request that the router receives. | |
There was a problem hiding this comment.
including an access log entry
interesting, do we have a way to configure access logs then??
| | `ate.template.atespace` | Where the template lives, which matches `ate.atespace` for a kagent agent. | | ||
| | `ate.actor.container.name` | The container that wrote the line, such as `kagent`. Records for a suspend or resume omit this label, because the Actor writes them rather than a container. | | ||
|
|
||
| Agent Substrate keeps the labels that identify one Actor off its metrics, so that the number of time series does not grow with the number of conversations. Metrics carry the template and WorkerPool labels instead. To look into one agent, use its logs. |
There was a problem hiding this comment.
Agent Substrate keeps the labels that identify one Actor off its metrics
not totally understanding this wording
|
|
||
| Agent Substrate keeps the labels that identify one Actor off its metrics, so that the number of time series does not grow with the number of conversations. Metrics carry the template and WorkerPool labels instead. To look into one agent, use its logs. | ||
|
|
||
| ## Read an Actor's logs |
There was a problem hiding this comment.
I got a little lost with how much is happening in just 2 commands. I tried my hand at expanding it a bit for clarity. WDYT?
Read an Actor's logs.md (sorry was having trouble w formatting here in the comment, had to put it in a file)
| ## Learn more | ||
|
|
||
| The Agent Substrate project documents its telemetry in full, including the complete metric registry, in [Actor observability](https://github.com/agent-substrate/substrate/blob/main/docs/observability.md). That page describes the newest Agent Substrate, which can be ahead of version {{< reuse "kagent-docs/versions/agent-substrate.md" >}} that this guide uses. Records that it describes might not exist in your version, such as actor state change events and per-Actor usage samples. | ||
|
|
||
| ## Next steps | ||
|
|
||
| {{< cards >}} | ||
| {{< card link=`{{< link path="observability/tracing" >}}` title="Tracing" subtitle="Read the spans of an agent request, and the separate Agent Substrate traces." >}} | ||
| {{< card link=`{{< link path="substrate-runtime/suspend-and-resume" >}}` title="Suspend and resume" subtitle="Understand what happens to an Actor between turns." >}} | ||
| {{< /cards >}} |
There was a problem hiding this comment.
I know we usually use "next steps" so far in this repo, but I would be down with just having a ## Learn more or ## More information etc for pages like this that are largely conceptual and it's more like, 'check out some other resources' (as you have here) rather than 'these are actions to take next'.
so maybe like:
| ## Learn more | |
| The Agent Substrate project documents its telemetry in full, including the complete metric registry, in [Actor observability](https://github.com/agent-substrate/substrate/blob/main/docs/observability.md). That page describes the newest Agent Substrate, which can be ahead of version {{< reuse "kagent-docs/versions/agent-substrate.md" >}} that this guide uses. Records that it describes might not exist in your version, such as actor state change events and per-Actor usage samples. | |
| ## Next steps | |
| {{< cards >}} | |
| {{< card link=`{{< link path="observability/tracing" >}}` title="Tracing" subtitle="Read the spans of an agent request, and the separate Agent Substrate traces." >}} | |
| {{< card link=`{{< link path="substrate-runtime/suspend-and-resume" >}}` title="Suspend and resume" subtitle="Understand what happens to an Actor between turns." >}} | |
| {{< /cards >}} | |
| ## Learn more | |
| The Agent Substrate project documents its telemetry in full, including the complete metric registry, in [Actor observability](https://github.com/agent-substrate/substrate/blob/main/docs/observability.md). That page describes the newest Agent Substrate, which can be ahead of version {{< reuse "kagent-docs/versions/agent-substrate.md" >}} that this guide uses. Records that it describes might not exist in your version, such as actor state change events and per-Actor usage samples. | |
| You can also check out the following resources in this documentation set. | |
| {{< cards >}} | |
| {{< card link=`{{< link path="observability/tracing" >}}` title="Tracing" subtitle="Read the spans of an agent request, and the separate Agent Substrate traces." >}} | |
| {{< card link=`{{< link path="substrate-runtime/suspend-and-resume" >}}` title="Suspend and resume" subtitle="Understand what happens to an Actor between turns." >}} | |
| {{< /cards >}} |
Doesnt have to be that exactly, but just a thought; lmk
Rachael-Graham
left a comment
Rachael-Graham
left a comment
There was a problem hiding this comment.
Reviewed against the 1.0.0-alpha2 chart, and tested by running the OTel stack guide end to end on a kind cluster plus a clean alpha2 install. Findings inline, most important first: the Loki step cannot complete as written, and both stack guides still promise the audit logging that the audit prompts page was pulled for.
The controller metrics half works exactly as documented, the Agent Substrate metric names all check out against a live scrape, and the revision-wait loops are a genuinely good addition.
| auth_enabled: false | ||
| singleBinary: | ||
| replicas: 1 | ||
| minio: |
There was a problem hiding this comment.
Blocking. This step cannot complete. The Loki 6.24.0 subchart pins quay.io/minio/minio:RELEASE.2024-04-18T19-09-19Z, which now returns 401 to anonymous pulls, so the install fails with failed post-install: timed out waiting for the condition. Reproduced on a clean kind cluster.
Either bump the Loki chart to a version whose MinIO image is pullable, or drop MinIO: minio.enabled: false with loki.storage.type: filesystem and object_store: filesystem. If you take the filesystem route, singleBinary.persistence.enabled must be true, or Loki crashloops on mkdir /var/loki: read-only file system.
| {{% tab name="Logs" %}} | ||
| Select the **Loki** data source, and run the following query to show the audit events of your agent. | ||
| ```text | ||
| {service_name="my-first-agent-my-first-harness"} |
There was a problem hiding this comment.
Blocking. This query does not return audit events, for two reasons that are not the same kind of thing.
Content never reaches log records by design: docs/architecture/telemetry.md in kagent-dev/kagent puts content on "invoke_agent and inference spans, under the capture opt-in" (line 30) and gives OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT exactly two values, SPAN_ONLY or NO_CONTENT (line 59). There is no EVENT_ONLY, so the ADK can never put content in a log record.
Separately, and this one is a real bug: Providers.ForceFlush flushes the tracer and meter providers but never the logger provider, so a suspended Actor loses its log events.
The same promise is still made at lines 55, 368 and 413, and at lightweight-otel-stack.md line 19. Line 368's "by default" implies a setting turns content on, and none does; line 467 presents the missing flush as ordinary batching.
Prompt auditing does work, through span capture. Please point readers there instead, as tracing.md line 85 already does. More on what this means for the page itself in my comment on audit-prompts.txt.
| --namespace kagent \ | ||
| --reuse-values \ | ||
| --values - <<EOF | ||
| otel: |
There was a problem hiding this comment.
kagent#2909 is merged to main and renames every key in this block: otel.tracing to otel.traces, otel.logging to otel.logs, otel.captureSensitiveContent to otel.capture.messageContent, and the per-signal exporter.otlp blocks collapse into one top-level otel.exporter.otlp. The old keys are gone with no fallback.
This is correct for the pinned 1.0.0-alpha2 today, but Helm accepts unknown values silently: verified with helm template against main, this block renders OTEL_TRACES_EXPORTER: "none" and no endpoint, while helm upgrade still exits 0. A silent no-op is hard for a reader to diagnose.
Same exposure in lightweight-otel-stack.md, tracing.md, and agent-harness.md. Worth a tracking issue even if you write against alpha2 for now.
| kubectl rollout status deployment/kagent-controller -n kagent --timeout=300s | ||
| ``` | ||
|
|
||
| 4. Wait for kagent to recompile the pair. The controller rebuilds each pair after the controller restarts, and an AgentInstance that you create before the rebuild finishes starts from the previous revision, without the new settings. The following command prints `Recompiled` when the new revision is ready. |
There was a problem hiding this comment.
"The controller rebuilds each pair after the controller restarts" did not hold in testing: the revision was unchanged across two controller restarts with new telemetry settings, and only an AgentTemplate spec change triggered a recompile. The loop then always falls through to the "settings were already in place" message, which sends the reader the wrong way.
Observed against main rather than alpha2, so worth re-checking before rewording. Same step in lightweight-otel-stack.md line 235.
|
|
||
| 1. Create a new AgentInstance. An AgentInstance keeps the runtime configuration that it was created with, so only a new AgentInstance exports traces and audit events. | ||
| ```bash | ||
| kagent create agent-instance --harness my-first-harness --agent-template my-first-agent |
There was a problem hiding this comment.
This command needs the 1.0 CLI. The 0.x CLI has no agent-instance subcommand, so a reader with an older build on their PATH gets a bare unknown command and nothing pointing at the cause. Worth naming the required CLI version in Before you begin.
Unrelated, for whenever installation.md is next touched: its example kubectl get pods -n ate-system output still lists a dns- pod, which Agent Substrate 0.2.0-beta5 no longer ships.
|
|
||
| - **The controller** reports as `kagent-controller` in the `kagent` service namespace. Its spans also carry the pod, node, and namespace that the controller runs on. | ||
| - **Each agent runtime** reports as its own service, named for the {{< gloss "AgentTemplate" >}}AgentTemplate{{< /gloss >}} and {{< gloss "Harness" >}}Harness{{< /gloss >}} pair it was compiled from, with hyphens replaced by underscores. The `my-first-agent` template on the `my-first-harness` Harness reports as `my_first_agent_my_first_harness`. | ||
| - **The Agent Substrate router** reports as `agentgateway`, the proxy that the router runs. |
There was a problem hiding this comment.
The router pod runs two containers and reports two OTel services: atenet-router (the router's own spans, ServiceName in the substrate source) and agentgateway (the proxy sidecar). Both appear in Tempo; this bullet names only the second and attributes it to the router.
Suggest naming both and saying which is which. That also reconciles this page with otel-stack.md line 459 and lightweight-otel-stack.md line 296, which both tell readers to select atenet-router.
| | `atelet_snapshot_size_bytes` | `atelet` | Histogram | Uncompressed size of each snapshot. | | ||
| | `ate_imagecache_requests_total` | `atelet` | Counter | Lookups in the node-local image cache, by the `ate_imagecache_outcome` label. A cache miss pays for pulling and unpacking the image, so a low hit ratio predicts slow resumes. | | ||
| | `ate_actor_crashes_total` | `ateapi` | Counter | Actors that moved to the terminal crashed state, by failure reason. The metric appears only after the first crash. | | ||
| | `rpc_server_call_duration_seconds`, `rpc_client_call_duration_seconds` | `ateapi`, `atelet` | Histogram | Latency, rate, and errors of the gRPC calls between Agent Substrate components. | |
There was a problem hiding this comment.
Four metrics the cluster actually reports are missing from this table: ate_actor_stats_cpu_time_seconds, ate_actor_stats_memory_usage_bytes, ate_actor_stats_memory_working_set_bytes, and ate_actor_stats_sampled_actors. Confirmed live against Agent Substrate 0.2.0-beta5.
Separately, atenet_router_route_duration is in the release's own expected-metric list but did not appear in my scrape, so confirm it is emitted before adding it.
|
|
||
| Every {{< reuse "kagent-docs/snippets/name-product.md" >}} agent runs as an {{< gloss "Actor" >}}Actor{{< /gloss >}} on Agent Substrate. An Actor suspends after each turn, and can resume on a different {{< gloss "Worker" >}}Worker{{< /gloss >}} pod the next time, so telemetry that is keyed to a pod loses track of an agent between turns. To keep one agent's telemetry together, Agent Substrate labels its telemetry with the identity of the Actor rather than the pod that ran it. | ||
|
|
||
| ## What Agent Substrate reports |
There was a problem hiding this comment.
This page has no Before you begin, but its steps need kubectl-ate, jq, and the my-first-agent / my-first-harness pair. The installation guide already covers installing kubectl-ate, so a short prerequisites section pointing there would be enough.
| | `ate.template.atespace` | Where the template lives, which matches `ate.atespace` for a kagent agent. | | ||
| | `ate.actor.container.name` | The container that wrote the line, such as `kagent`. Records for a suspend or resume omit this label, because the Actor writes them rather than a container. | | ||
|
|
||
| Agent Substrate keeps the labels that identify one Actor off its metrics, so that the number of time series does not grow with the number of conversations. Metrics carry the template and WorkerPool labels instead. To look into one agent, use its logs. |
There was a problem hiding this comment.
This is factually correct. I confirmed live that ate_actor_stats_memory_usage_bytes carries ate_template_name and ate_workerpool_name and no Actor identity. It just reads awkwardly.
Suggest something closer to: "Agent Substrate metrics carry template and WorkerPool labels rather than Actor identity, so the number of time series does not grow with the number of conversations."
| --- | ||
| title: Agent Substrate telemetry | ||
| description: Follow an agent's Actor across suspend and resume by using the Actor identity that Agent Substrate adds to its logs, traces, and metrics. | ||
| weight: 35 |
There was a problem hiding this comment.
weight 35 places this page between Tracing (30) and Metrics (40), but it links forward to Metrics for the Agent Substrate metric reference. Consider ordering it after Metrics.
Rachael-Graham
left a comment
Rachael-Graham
left a comment
There was a problem hiding this comment.
Follow-up on my comment on otel-stack.md line 464, and a correction.
| title: Audit prompts | ||
| description: Export the prompts and replies that your agents exchange with a model as OpenTelemetry log events, then query them in a logging backend. | ||
| weight: 30 | ||
| weight: 50 |
There was a problem hiding this comment.
The log-event route this page documents is excluded by design, not awaiting a fix. docs/architecture/telemetry.md in kagent-dev/kagent puts content on "invoke_agent and inference spans, under the capture opt-in" (line 30) and gives OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT exactly two values, SPAN_ONLY or NO_CONTENT (line 59). There is no EVENT_ONLY in the contract, and no open issue or PR proposes adding one. (The unflushed logger provider is a separate, real bug — see my comment on otel-stack.md line 464.)
So this file can be deleted outright rather than renamed to .txt: there is nothing to restore later, and git history keeps it if anyone needs it. No redirect is needed — the page carries a noindex tag, and the only links to it are the two cards in this PR.
Rather than leaving those two cards commented out, please repoint them at the new prompt-auditing section in tracing.md once it exists — examples/agentgateway.md line 389 and observability/launch-ui.md line 190. A reader following either one then lands on content that answers the question.
Rachael-Graham
left a comment
Rachael-Graham
left a comment
There was a problem hiding this comment.
Follow-up on the prompt-auditing replacement.
|
|
||
| > [!WARNING] | ||
| > Spans for a model call carry the full serialized request and response as the `gcp.vertex.agent.llm_request` and `gcp.vertex.agent.llm_response` attributes. Prompts and replies therefore reach your tracing backend. Payloads larger than 32 KiB are truncated to a prefix. To keep this content out of traces, set `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT` to `false` in the Harness `spec.env`. Note that the same variable has the opposite default for audit logging, where content is withheld until you set it to `true`. For more information, see [Audit prompts]({{< link path="observability/audit-prompts" >}}). The `codex` and `claude` runtimes do not read this variable. On those two runtimes, the `otel.captureSensitiveContent` Helm setting controls the same content, as described in the agent harness [telemetry content settings]({{< link path="agents/agent-harness#telemetry-content-settings" >}}). | ||
| > When the `otel.captureSensitiveContent` Helm setting is `true`, prompts and replies reach your tracing backend. The spans for a model call then carry the full serialized request and response as the `gcp.vertex.agent.llm_request` and `gcp.vertex.agent.llm_response` attributes, truncated to a prefix when a payload is larger than 32 KiB. The setting defaults to `false`, which leaves both attributes as `{}`. For how the setting applies to each runtime, see the agent harness [telemetry content settings]({{< link path="agents/agent-harness#telemetry-content-settings" >}}). |
There was a problem hiding this comment.
If a section on prompt auditing lands here to replace the removed page, spans are only part of the story. docs/architecture/telemetry.md notes that suppressing span attributes "does not establish privacy for native runtime events or log bodies, which have their own settings" — on the claude runtime, otel.capture.rawApiBodies logs complete provider request and response bodies, which is a stronger audit record than spans give you. agent-harness.md already documents it.
So the accurate framing is: span capture on every configured runtime, plus raw API body logging on claude. Worth not writing "spans only".
Also, for what it is worth: I searched kagent-dev/kagent and found no open issue or PR proposing event-based content capture, so the span-only contract looks settled rather than interim.
2 participants
No description provided.