From 0e017798ad990d0c3a7dd1f156edb58f61c1cd42 Mon Sep 17 00:00:00 2001 From: Kristin Brown Date: Thu, 24 Sep 2026 10:46:59 -0400 Subject: [PATCH 1/5] Added 1.0 observability topics Signed-off-by: Kristin Brown --- .../versions/kube-prometheus-stack.md | 1 + .../assets/kagent-docs/versions/prometheus.md | 1 + .../kagent/1.x/agents/agent-harness.md | 2 +- .../kagent/1.x/examples/agentgateway.md | 2 +- .../kagent/1.x/observability/_index.md | 2 +- .../{audit-prompts.md => audit-prompts.txt} | 8 +- .../kagent/1.x/observability/launch-ui.md | 2 +- .../observability/lightweight-otel-stack.md | 343 ++++++++++++ .../kagent/1.x/observability/metrics.md | 111 ++++ .../kagent/1.x/observability/otel-stack.md | 518 ++++++++++++++++++ .../kagent/1.x/observability/tracing.md | 169 +++--- .../operations/operational-considerations.md | 12 +- 12 files changed, 1090 insertions(+), 81 deletions(-) create mode 100644 docs-site/assets/kagent-docs/versions/kube-prometheus-stack.md create mode 100644 docs-site/assets/kagent-docs/versions/prometheus.md rename docs-site/content/kagent/1.x/observability/{audit-prompts.md => audit-prompts.txt} (98%) create mode 100644 docs-site/content/kagent/1.x/observability/lightweight-otel-stack.md create mode 100644 docs-site/content/kagent/1.x/observability/metrics.md create mode 100644 docs-site/content/kagent/1.x/observability/otel-stack.md diff --git a/docs-site/assets/kagent-docs/versions/kube-prometheus-stack.md b/docs-site/assets/kagent-docs/versions/kube-prometheus-stack.md new file mode 100644 index 00000000..c090543e --- /dev/null +++ b/docs-site/assets/kagent-docs/versions/kube-prometheus-stack.md @@ -0,0 +1 @@ +75.6.1 \ No newline at end of file diff --git a/docs-site/assets/kagent-docs/versions/prometheus.md b/docs-site/assets/kagent-docs/versions/prometheus.md new file mode 100644 index 00000000..096534ce --- /dev/null +++ b/docs-site/assets/kagent-docs/versions/prometheus.md @@ -0,0 +1 @@ +29.33.0 \ No newline at end of file diff --git a/docs-site/content/kagent/1.x/agents/agent-harness.md b/docs-site/content/kagent/1.x/agents/agent-harness.md index 7b628a92..0b685bf8 100644 --- a/docs-site/content/kagent/1.x/agents/agent-harness.md +++ b/docs-site/content/kagent/1.x/agents/agent-harness.md @@ -146,7 +146,7 @@ otel: | `otel.captureSensitiveContent` | Prompts, tool details, and assistant replies in the runtime's telemetry. On the `claude` runtime, tool results require tracing, and assistant replies require audit logging. | `codex`, `claude` | | `otel.logging.captureRawApiBodies` | The complete provider API request and response bodies. This setting returns more than `otel.captureSensitiveContent` does, and it takes effect only when `otel.logging.enabled` is `true`. | `claude` | -The `kagent` runtime honors neither setting. To include message content for an agent on that runtime, set `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT` in the Harness `spec.env` field. That variable defaults differently for each signal, so check the [audit prompt onfiguration]({{< link path="observability/audit-prompts#configuration" >}}) before you set it. +The `kagent` runtime honors neither setting. To include message content for an agent on that runtime, set `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT` in the Harness `spec.env` field. The controller sends the `byo` runtime no telemetry configuration, so neither setting reaches it. A `byo` image that implements OpenTelemetry itself reads whatever the Harness `spec.env` field holds. For more information, see [Tracing]({{< link path="observability/tracing#about-trace-coverage" >}}). diff --git a/docs-site/content/kagent/1.x/examples/agentgateway.md b/docs-site/content/kagent/1.x/examples/agentgateway.md index 30d2f0d1..f128ddf2 100644 --- a/docs-site/content/kagent/1.x/examples/agentgateway.md +++ b/docs-site/content/kagent/1.x/examples/agentgateway.md @@ -386,6 +386,6 @@ An `AgentgatewayPolicy` holds the same `promptGuard` configuration, and one poli {{< cards >}} {{< card link=`{{< link path="setup/model-providers/byo-agentgateway" >}}` title="agentgateway model routing" subtitle="Review the Gateway, AgentgatewayModel, and ModelConfig that carry an agent's model traffic." >}} - {{< card link=`{{< link path="observability/audit-prompts" >}}` title="Audit prompts" subtitle="Export the prompts and replies that your agents exchange with a model, and query them in a logging backend." >}} + {{< card link="https://agentgateway.dev/docs/kubernetes/latest/documentation/llm/guardrails/regex/#block-requests-with-pii" title="agentgateway regex guardrails" subtitle="Read the full set of regex guards, masking rules, and built-in patterns in the agentgateway documentation." >}} {{< /cards >}} diff --git a/docs-site/content/kagent/1.x/observability/_index.md b/docs-site/content/kagent/1.x/observability/_index.md index 9e7489d6..bacd4f29 100644 --- a/docs-site/content/kagent/1.x/observability/_index.md +++ b/docs-site/content/kagent/1.x/observability/_index.md @@ -1,6 +1,6 @@ --- title: Observability -description: Open the kagent UI, trace agent requests across the controller and the Actors that run your agents, and audit the prompts your agents send. +description: Open the kagent UI, set up an OpenTelemetry stack, and review the traces, metrics, and audit events that kagent and Agent Substrate report. weight: 70 author: kagent.dev --- \ No newline at end of file diff --git a/docs-site/content/kagent/1.x/observability/audit-prompts.md b/docs-site/content/kagent/1.x/observability/audit-prompts.txt similarity index 98% rename from docs-site/content/kagent/1.x/observability/audit-prompts.md rename to docs-site/content/kagent/1.x/observability/audit-prompts.txt index 08ad96c2..a8dd75a6 100644 --- a/docs-site/content/kagent/1.x/observability/audit-prompts.md +++ b/docs-site/content/kagent/1.x/observability/audit-prompts.txt @@ -1,7 +1,7 @@ --- title: Audit prompts description: Export the prompts and replies that your agents exchange with a model as OpenTelemetry log events, then query them in a logging backend. -weight: 30 +weight: 50 author: kagent.dev --- @@ -304,10 +304,10 @@ Turning on auditing takes two changes. The chart setting installs the log export kubectl port-forward -n telemetry svc/loki 3100:3100 ``` -4. Query the events for the agent's service. The runtime builds the service name from the AgentTemplate name and the Harness name, and replaces each hyphen with an underscore. For example, `my-first-agent` on `my-first-harness` reports as `my_first_agent_my_first_harness`. +4. Query the events for the agent's service. The runtime builds the service name from the AgentTemplate name and the Harness name. For example, `my-first-agent` on `my-first-harness` reports as `my-first-agent-my-first-harness`. ```bash curl -s -G 'http://localhost:3100/loki/api/v1/query_range' \ - --data-urlencode 'query={service_name="my_first_agent_my_first_harness"}' \ + --data-urlencode 'query={service_name="my-first-agent-my-first-harness"}' \ --data-urlencode "start=$(( $(date +%s) - 3600 ))000000000" \ --data-urlencode "end=$(date +%s)000000000" | jq ``` @@ -321,7 +321,7 @@ Turning on auditing takes two changes. The chart setting installs the log export "result": [ { "stream": { - "service_name": "my_first_agent_my_first_harness", + "service_name": "my-first-agent-my-first-harness", "service_namespace": "kagent", "scope_name": "gcp.vertex.agent", "trace_id": "3d34d2f1b74f30a5cce0d5ed8571e928", diff --git a/docs-site/content/kagent/1.x/observability/launch-ui.md b/docs-site/content/kagent/1.x/observability/launch-ui.md index 6c71faf2..16a0f4b8 100644 --- a/docs-site/content/kagent/1.x/observability/launch-ui.md +++ b/docs-site/content/kagent/1.x/observability/launch-ui.md @@ -187,5 +187,5 @@ The `parentRefs` field is required when `enabled` is `true`, and it must referen {{< cards >}} {{< card link=`{{< link path="get-started/your-first-agent" >}}` title="Create your first agent" subtitle="Pair an AgentTemplate with a Harness, then hold a conversation with the result." >}} {{< card link=`{{< link path="observability/tracing" >}}` title="Tracing" subtitle="Follow one agent request from the controller through to the Actor that ran it." >}} - {{< card link=`{{< link path="observability/audit-prompts" >}}` title="Audit prompts" subtitle="Export every prompt and reply as a log event for security and compliance review." >}} + {{< /cards >}} diff --git a/docs-site/content/kagent/1.x/observability/lightweight-otel-stack.md b/docs-site/content/kagent/1.x/observability/lightweight-otel-stack.md new file mode 100644 index 00000000..b456dd2f --- /dev/null +++ b/docs-site/content/kagent/1.x/observability/lightweight-otel-stack.md @@ -0,0 +1,343 @@ +--- +title: Lightweight OTel stack +description: Install an OpenTelemetry collector with Jaeger and Prometheus, then send traces and metrics from kagent and Agent Substrate to it. +weight: 25 +author: kagent.dev +--- + +Deploy a small observability stack based on OpenTelemetry (OTel) that collects traces and metrics from both kagent and Agent Substrate. The stack includes the following components: + +- **Traces**: Distributed tracing with [Jaeger](https://github.com/jaegertracing/jaeger). +- **Metrics**: Time-series metrics collection with [Prometheus](https://github.com/prometheus/prometheus). +- **Collection**: Telemetry collection and routing with the [OpenTelemetry Collector](https://github.com/open-telemetry/opentelemetry-collector). +- **Logs**: No logging backend. The collector prints the logs that it receives to its own output. + +This stack follows the setup that Agent Substrate uses for its own local testing, and it needs less memory than the [OTel stack]({{< link path="observability/otel-stack" >}}) with Grafana. + +## About the stack + +The stack has no logging backend. The collector prints the logs that it receives to its own output, which is enough to confirm that logs arrive. To store and query audit events, use the [OTel stack]({{< link path="observability/otel-stack" >}}), which adds Loki. +

+ +```mermaid +flowchart LR + subgraph sources["Your cluster"] + kagent["kagent controller
and agent runtimes"] + substrate["Agent Substrate"] + end + collector["OTel collector"] + subgraph backends["Backends"] + jaeger["Jaeger"] + prom["Prometheus"] + end + kagent -->|traces, logs| collector + substrate -->|traces, logs, metrics| collector + kagent -->|metrics scrape| prom + collector --> jaeger + collector -->|metrics scrape| prom + classDef boundary fill:#a78bfa26,stroke:#a78bfa,stroke-width:2px + classDef inner fill:#80808033,stroke:#9ca3af,stroke-width:1px + class sources,backends boundary + class kagent,substrate,collector,jaeger,prom inner +``` + +For what kagent and Agent Substrate each send, see [About the stack]({{< link path="observability/otel-stack#about-the-stack" >}}) in the OTel stack guide. + +## Before you begin + +1. [Install kagent]({{< link path="setup/installation" >}}). +2. [Create your first agent]({{< link path="get-started/your-first-agent" >}}), so that you have a Harness and an AgentTemplate to send requests to. + +## Install Jaeger + +Install [Jaeger](https://www.jaegertracing.io/) in all-in-one mode, which stores traces in memory and needs no other components. + +```bash +helm upgrade --install jaeger jaeger \ + --repo https://jaegertracing.github.io/helm-charts \ + --version {{< reuse "kagent-docs/versions/jaeger.md" >}} \ + --namespace telemetry --create-namespace \ + --values - <}} \ + --namespace telemetry \ + --values - <}} \ + --namespace telemetry \ + --values - <}}/{{< reuse "kagent-docs/snippets/helm-kagent.md" >}} \ + --version {{< reuse "kagent-docs/versions/kagent.md" >}} \ + --namespace kagent \ + --reuse-values \ + --values - <}} \ + --namespace ate-system \ + --reuse-values \ + --set otel.endpoint=http://otel-collector.telemetry.svc.cluster.local:4317 \ + --set otel.traces.samplingRatio=1.0 \ + --wait --timeout 10m +``` + +## Send a request + +1. Create a new AgentInstance. An AgentInstance keeps the runtime configuration that it was created with, so only a new AgentInstance exports traces. + ```bash + kagent create agent-instance --harness my-first-harness --agent-template my-first-agent + ``` + +2. Confirm that the AgentInstance runs the current revision of the pair. The command waits until kagent finishes compiling the pair, then compares that revision with the one that the AgentInstance started from. If the command prints `Outdated`, the AgentInstance was created from an earlier revision, and exports without the new settings. Create another AgentInstance, and run the command again. + ```bash + for i in $(seq 1 60); do + REVISIONS=$(kubectl get agenttemplate my-first-agent -n kagent \ + -o jsonpath='{.status.harnesses[0].desiredRevision} {.status.harnesses[0].latestSuccessfulRevision}') + [ "${REVISIONS% *}" = "${REVISIONS#* }" ] && break + sleep 5 + done + INSTANCE_REVISION=$(kagent get agent-instance -o json \ + | jq -r '[.agentInstances[] | select(.agentTemplate.name == "my-first-agent")] | sort_by(.createdAt) | last | .preparedRevision') + [ "$INSTANCE_REVISION" = "${REVISIONS#* }" ] && echo "Current" || echo "Outdated" + ``` + +3. Send a request to produce telemetry. + ```bash + export INSTANCE_ID=$(kagent get agent-instance -o json \ + | jq -r '[.agentInstances[] | select(.agentTemplate.name == "my-first-agent")] | sort_by(.createdAt) | last | .id') + kagent invoke --agent-instance $INSTANCE_ID --task "What is 2+2?" + ``` + +## Review the telemetry + +1. Review the traces in Jaeger. + 1. Forward the Jaeger query port, and leave the command running. + ```bash + kubectl port-forward -n telemetry svc/jaeger 16686:16686 + ``` + 2. In your browser, open Jaeger at [http://localhost:16686](http://localhost:16686). + 3. From the **Service** list, select `my-first-agent-my-first-harness`, and click **Find Traces**. To see Agent Substrate's own work, select `ateapi`, `atenet-router`, or `atelet` instead. + +2. Review the metrics in Prometheus. + 1. Forward the Prometheus port, and leave the command running. + ```bash + kubectl port-forward -n telemetry svc/prometheus-server 9090:80 + ``` + 2. In your browser, open Prometheus at [http://localhost:9090/targets](http://localhost:9090/targets), and confirm that the `otel-collector` and `kagent-controller` targets are `UP`. + 3. Open the **Query** page, and run a query such as `ate_workerpool_ready_workers`. For more metrics, see [Metrics]({{< link path="observability/metrics" >}}). + +3. Check that the collector receives logs. Each line reports a batch of log records from one source. + ```bash + kubectl logs -n telemetry deploy/otel-collector | grep '"otelcol.signal": "logs"' + ``` + +## Clean up + +1. Turn off telemetry export in kagent and Agent Substrate. + ```bash + helm upgrade kagent \ + {{< reuse "kagent-docs/snippets/helm-path.md" >}}/{{< reuse "kagent-docs/snippets/helm-kagent.md" >}} \ + --version {{< reuse "kagent-docs/versions/kagent.md" >}} \ + --namespace kagent --reuse-values \ + --set otel.tracing.enabled=false \ + --set otel.logging.enabled=false \ + --set controller.metrics.enabled=false + helm upgrade substrate \ + oci://ghcr.io/kagent-dev/substrate/helm/substrate \ + --version {{< reuse "kagent-docs/versions/agent-substrate.md" >}} \ + --namespace ate-system --reuse-values \ + --set otel.endpoint="" \ + --set otel.traces.samplingRatio=0.01 + ``` + +2. Remove the role binding, the collector, the backends, and the `telemetry` namespace. + ```bash + kubectl delete clusterrolebinding kagent-metrics-reader-prometheus + helm uninstall otel-collector prometheus jaeger -n telemetry + kubectl delete namespace telemetry + ``` + +## Next steps + +{{< cards >}} + {{< card link=`{{< link path="observability/tracing" >}}` title="Tracing" subtitle="Read the spans and attributes of an agent request." >}} + {{< card link=`{{< link path="observability/metrics" >}}` title="Metrics" subtitle="Review the metrics that kagent and Agent Substrate report." >}} + {{< card link=`{{< link path="observability/otel-stack" >}}` title="OTel stack" subtitle="Add Loki and Grafana to store logs and view every signal in one place." >}} +{{< /cards >}} diff --git a/docs-site/content/kagent/1.x/observability/metrics.md b/docs-site/content/kagent/1.x/observability/metrics.md new file mode 100644 index 00000000..554325ce --- /dev/null +++ b/docs-site/content/kagent/1.x/observability/metrics.md @@ -0,0 +1,111 @@ +--- +title: Metrics +description: Scrape the kagent controller's Prometheus metrics, export Agent Substrate metrics over OpenTelemetry, and review what each metric measures. +weight: 40 +author: kagent.dev +--- + +{{< reuse "kagent-docs/snippets/name-product.md" >}} and Agent Substrate report metrics in two different ways. The kagent controller serves a Prometheus `/metrics` endpoint that Prometheus scrapes. Agent Substrate pushes its metrics over the OpenTelemetry Protocol (OTLP) to a collector, which serves them for Prometheus to scrape. Both are off by default. The [OTel stack]({{< link path="observability/otel-stack" >}}) and [Lightweight OTel stack]({{< link path="observability/lightweight-otel-stack" >}}) guides turn both on and set up Prometheus to collect them. + +## kagent controller metrics + +### Enable the metrics endpoint + +The controller serves its metrics over HTTPS, and accepts a scrape only from a ServiceAccount that holds the `-metrics-reader` ClusterRole. Enabling the endpoint creates a dedicated metrics Service and that ClusterRole. + +```yaml +controller: + metrics: + enabled: true + bindAddress: ":8443" + secureServing: true +``` + +Change the port through `bindAddress` rather than through `controller.env`, because the chart derives the Service `targetPort` and the pod `containerPort` from `bindAddress` at template time. Overriding `METRICS_BIND_ADDRESS` directly moves the listener and leaves the Service pointing at the old port. + +Grant your Prometheus instance access in one of the following ways. + +- **With the Prometheus Operator**: Set `controller.metrics.serviceMonitor.enabled` to `true`. The chart creates a ServiceMonitor for the metrics Service. Also set `controller.metrics.serviceMonitor.prometheusServiceAccount.name` and `.namespace` to the ServiceAccount that Prometheus runs as, and the chart binds the metrics reader ClusterRole to it. The chart renders the ServiceMonitor only when the cluster serves the `monitoring.coreos.com/v1` API. For an example, see [OTel stack]({{< link path="observability/otel-stack#send-kagent-telemetry-to-the-collector" >}}). +- **Without the Prometheus Operator**: Bind the `-metrics-reader` ClusterRole to the Prometheus ServiceAccount yourself, and add a scrape job for the `-controller-metrics` Service on port `8443`. The job must use HTTPS, send the ServiceAccount token, and skip verification of the controller's self-signed certificate. For an example, see [Lightweight OTel stack]({{< link path="observability/lightweight-otel-stack#install-prometheus" >}}). + +### Metric reference + +Besides the standard Go runtime and process metrics, the controller reports the following metrics. + +| Metric | Type | Measures | +| ------ | ---- | -------- | +| `kagent_grpc_server_requests_total` | Counter | Requests to the controller's gRPC API, by `method`, `code`, and `rpc_type`. The `method` label names the full gRPC method, such as `/kagent.api.v1alpha1.AgentInstanceService/CreateAgentInstance`. | +| `kagent_grpc_server_request_duration_seconds` | Histogram | Latency of requests to the controller's gRPC API, with the same labels. | +| `controller_runtime_reconcile_total`, `controller_runtime_reconcile_errors_total`, `controller_runtime_reconcile_time_seconds` | Counter, counter, histogram | Reconciliations of kagent resources, by controller, and how long each one takes. A rising error count means that the controller cannot bring a resource to its desired state. | +| `workqueue_depth`, `workqueue_queue_duration_seconds`, `workqueue_retries_total` | Gauge, histogram, counter | Work waiting for a reconciler, how long it waits, and how often it is retried. | +| `leader_election_master_status` | Gauge | `1` on the controller replica that holds the leader lease. With several replicas, exactly one reports `1`. | +| `rest_client_requests_total` | Counter | Requests from the controller to the Kubernetes API server, by status code. | + +The Agent Substrate `atecontroller` component reports the same `controller_runtime_*` and `workqueue_*` metrics for its own reconcilers. To keep the two apart in a query, filter by the scrape job, such as `job="kagent-controller-metrics"` in the OTel stack. + +## Agent Substrate metrics + +### Export the metrics + +Agent Substrate exports metrics over OTLP when its Helm release has an OTLP endpoint. The same setting also turns on Agent Substrate traces and logs. + +```bash +helm upgrade substrate \ + oci://ghcr.io/kagent-dev/substrate/helm/substrate \ + --version {{< reuse "kagent-docs/versions/agent-substrate.md" >}} \ + --namespace ate-system \ + --reuse-values \ + --set otel.endpoint=http://otel-collector.telemetry.svc.cluster.local:4317 +``` + +To send metrics somewhere other than traces and logs, set `otel.metrics.endpoint` instead. To stop exporting metrics while keeping the other signals, set `otel.metrics.enabled` to `false`. Agent Substrate exports a batch of metrics every 60 seconds, so a new installation shows its first values about a minute after you set the endpoint. + +### Metric reference + +The collector's Prometheus exporter converts each OpenTelemetry metric name to a Prometheus name. It replaces dots with underscores, and adds a unit suffix, such as `_seconds` or `_bytes`, and a `_total` suffix for counters. The following table lists the Prometheus names. Every metric also carries a `service_name` label for the Agent Substrate component that reported it, when the collector copies resource attributes to labels, as both stack guides configure. + +| Metric | Reported by | Type | Measures | +| ------ | ----------- | ---- | -------- | +| `ate_workerpool_desired_workers` | `atecontroller` | Gauge | Worker pods that a WorkerPool requests, from its `spec.replicas`. | +| `ate_workerpool_ready_workers` | `atecontroller` | Gauge | Worker pods that are ready in a WorkerPool. A value below `ate_workerpool_desired_workers` means that the pool cannot run as many Actors as it should. | +| `ate_workerpool_workers` | `ateapi` | Gauge | Live Workers per pool, split by the `ate_worker_state` label into `idle` and `assigned`. Few idle Workers means that new and resumed Actors are about to wait for capacity. | +| `ate_scheduler_assignment_duration_seconds` | `ateapi` | Histogram | Time to assign an Actor to a Worker, by the `ate_scheduler_outcome` label: `assigned`, `no_free_worker`, or `error`. | +| `ate_scheduler_eligible_workers` | `ateapi` | Histogram | Unassigned Workers that could take an Actor at scheduling time. | +| `ate_actor_lifecycle_operation_duration_seconds` | `ateapi` | Histogram | Time that each Actor operation takes, by the `ate_actor_operation_name` label, such as `create`, `resume`, and `suspend`. A failed operation carries an `error_type` label. | +| `ate_actor_restore_duration_seconds` | `atelet` | Histogram | Time for each phase of restoring an Actor from a snapshot on its Worker's node, by the `ate_snapshot_phase` label, such as `download`, `oci_unpack`, and `ateom_restore`. This is where most of the time to resume an Actor goes. | +| `ate_actor_checkpoint_duration_seconds` | `atelet` | Histogram | Time for each phase of writing a snapshot when an Actor suspends. | +| `atelet_snapshot_size_bytes` | `atelet` | Histogram | Uncompressed size of each snapshot. | +| `ate_imagecache_requests_total` | `atelet` | Counter | Lookups in the node-local image cache, by the `ate_imagecache_outcome` label. A cache miss pays for pulling and unpacking the image, so a low hit ratio predicts slow resumes. | +| `ate_actor_crashes_total` | `ateapi` | Counter | Actors that moved to the terminal crashed state, by failure reason. The metric appears only after the first crash. | +| `rpc_server_call_duration_seconds`, `rpc_client_call_duration_seconds` | `ateapi`, `atelet` | Histogram | Latency, rate, and errors of the gRPC calls between Agent Substrate components. | + +## Example queries + +Run the following queries in Prometheus or Grafana. + +- Ready Workers compared with desired Workers, per WorkerPool. + ```text + ate_workerpool_ready_workers / ate_workerpool_desired_workers + ``` + +- The 95th percentile time to resume an Actor, per phase. + ```text + histogram_quantile(0.95, sum by (le, ate_snapshot_phase) (rate(ate_actor_restore_duration_seconds_bucket[5m]))) + ``` + +- The rate of requests to the kagent gRPC API, by method. + ```text + sum by (method) (rate(kagent_grpc_server_requests_total[5m])) + ``` + +- The rate of failed reconciliations in the kagent controller, by reconciler. The query uses the job name from the OTel stack. In the lightweight stack, the job is `kagent-controller`. + ```text + sum by (controller) (rate(controller_runtime_reconcile_errors_total{job="kagent-controller-metrics"}[5m])) + ``` + +## Next steps + +{{< cards >}} + {{< card link=`{{< link path="observability/otel-stack" >}}` title="OTel stack" subtitle="Collect traces, logs, and metrics in Grafana." >}} + {{< card link=`{{< link path="operations/tune-agent-substrate" >}}` title="Tune Agent Substrate" subtitle="Size WorkerPools and adjust the runtime settings that these metrics measure." >}} +{{< /cards >}} diff --git a/docs-site/content/kagent/1.x/observability/otel-stack.md b/docs-site/content/kagent/1.x/observability/otel-stack.md new file mode 100644 index 00000000..ecc9d8cd --- /dev/null +++ b/docs-site/content/kagent/1.x/observability/otel-stack.md @@ -0,0 +1,518 @@ +--- +title: OTel stack +description: Install an OpenTelemetry collector with Grafana Tempo, Loki, Prometheus, and Grafana, then send traces, logs, and metrics from kagent and Agent Substrate to it. +weight: 20 +author: kagent.dev +--- + +Deploy an open source observability stack based on OpenTelemetry (OTel) that collects telemetry from both kagent and Agent Substrate. The stack includes the following components: + +- **Logs**: Log collection and storage with Grafana [Loki](https://github.com/grafana/loki). +- **Traces**: Distributed tracing with Grafana [Tempo](https://github.com/grafana/tempo). +- **Metrics**: Time-series metrics collection with [Prometheus](https://github.com/prometheus/prometheus). +- **Collection**: Telemetry collection and routing with the [OpenTelemetry Collector](https://github.com/open-telemetry/opentelemetry-collector). +- **Visualization**: Dashboards and queries across all three signals with [Grafana](https://github.com/grafana/grafana). + +For a smaller stack with Jaeger and Prometheus only, see [Lightweight OTel stack]({{< link path="observability/lightweight-otel-stack" >}}). + +## About the stack + +{{< reuse "kagent-docs/snippets/name-product.md" >}} and Agent Substrate each export their own telemetry over the OpenTelemetry Protocol (OTLP), and neither exports anything by default. The collector gives both a single address to send to, and holds the rules for where each signal goes next. The kagent controller also serves Prometheus metrics, which Prometheus scrapes directly. +

+ +```mermaid +flowchart LR + subgraph sources["Your cluster"] + kagent["kagent controller
and agent runtimes"] + substrate["Agent Substrate"] + end + collector["OTel collector"] + subgraph backends["Backends"] + tempo["Tempo"] + loki["Loki"] + prom["Prometheus"] + end + grafana["Grafana"] + kagent -->|traces, logs| collector + substrate -->|traces, logs, metrics| collector + kagent -->|metrics scrape| prom + collector --> tempo + collector --> loki + collector -->|metrics scrape| prom + tempo --> grafana + loki --> grafana + prom --> grafana + classDef boundary fill:#a78bfa26,stroke:#a78bfa,stroke-width:2px + classDef inner fill:#80808033,stroke:#9ca3af,stroke-width:1px + class sources,backends boundary + class kagent,substrate,collector,tempo,loki,prom,grafana inner +``` + +The following table lists what each source sends. For the meaning of each signal, see [Tracing]({{< link path="observability/tracing" >}}) and [Metrics]({{< link path="observability/metrics" >}}). + +| Source | Traces | Logs | Metrics | +| ------ | ------ | ---- | ------- | +| kagent controller and agent runtimes | One trace per agent request, across the controller, the Agent Substrate proxy, and the agent runtime. | Audit events for the prompts and replies that agents exchange with a model. | Reconciliation, work queue, and gRPC API metrics, scraped from the controller's `/metrics` endpoint. | +| Agent Substrate | Separate traces for its own work, such as routing a request to a Worker and resuming an Actor. | Actor lifecycle events and the router access log. | Actor lifecycle, scheduling, snapshot, image cache, and WorkerPool capacity metrics. | + +Agent Substrate traces do not join the trace of the agent request that caused them. To see why a request was slow to start, look up the Agent Substrate trace from the same time window. + +## Before you begin + +1. [Install kagent]({{< link path="setup/installation" >}}). +2. [Create your first agent]({{< link path="get-started/your-first-agent" >}}), so that you have a Harness and an AgentTemplate to send requests to. +3. Make sure that your cluster has about 1.5 GB of memory available in addition to kagent. On a kind cluster, the memory limit is the memory that you give Docker. + +## Install Tempo and Loki + +Install the backends that store traces and logs. Both run as a single replica without persistent storage, which suits evaluation. For production, follow the Grafana guidance for [Tempo](https://grafana.com/docs/tempo/latest/setup/helm-chart/) and [Loki](https://grafana.com/docs/loki/latest/setup/install/helm/). + +1. Install Tempo, with an OTLP receiver for traces. + ```bash + helm upgrade --install tempo tempo \ + --repo https://grafana.github.io/helm-charts \ + --version {{< reuse "kagent-docs/versions/tempo.md" >}} \ + --namespace telemetry --create-namespace \ + --values - <}} \ + --namespace telemetry \ + --values - <}} \ + --namespace telemetry \ + --values - <}} + + | Setting | Description | + | ------- | ----------- | + | `serviceMonitorSelectorNilUsesHelmValues` | Set to `false` so that Prometheus scrapes every ServiceMonitor in the cluster, including the ones that the kagent chart and this guide create. By default, Prometheus scrapes only the ServiceMonitors that carry the chart's release label. | + | `additionalDataSources` | Adds Tempo and Loki to Grafana. Tempo serves its query API on port `3100` in this chart version. The chart adds Prometheus as a data source on its own. | + +2. Verify that Prometheus and Grafana are running. + ```bash + kubectl get pods -n telemetry -l 'app.kubernetes.io/name in (prometheus,grafana)' + ``` + Example output: + ```console + NAME READY STATUS RESTARTS AGE + kube-prometheus-stack-grafana-7599d6f796-lb4l5 3/3 Running 0 2m + prometheus-kube-prometheus-stack-prometheus-0 2/2 Running 0 2m + ``` + +## Install the OTel collector + +Install a collector that receives OTLP on ports `4317` for gRPC and `4318` for HTTP, and forwards each signal to its backend. The collector sends traces to Tempo and logs to Loki, and serves metrics on port `8889` for Prometheus to scrape. The configuration uses the `contrib` collector image, because the Prometheus exporter is not part of the core image. + +1. Install the collector. + ```bash + helm upgrade --install otel-collector opentelemetry-collector \ + --repo https://open-telemetry.github.io/opentelemetry-helm-charts \ + --version {{< reuse "kagent-docs/versions/otel-collector.md" >}} \ + --namespace telemetry \ + --values - <}}/{{< reuse "kagent-docs/snippets/helm-kagent.md" >}} \ + --version {{< reuse "kagent-docs/versions/kagent.md" >}} \ + --namespace kagent \ + --reuse-values \ + --values - <}} + + | Setting | Description | + | ------- | ----------- | + | `otel.tracing` | Exports traces from the controller and from the agent runtimes that it starts. For each field, see [Tracing]({{< link path="observability/tracing#enable-tracing" >}}). | + | `otel.logging` | Exports audit events from the agent runtimes. By default, kagent withholds message content, so each event body reads ``. | + | `controller.metrics.enabled` | Serves the controller's Prometheus metrics on port `8443` over HTTPS. | + | `controller.metrics.serviceMonitor` | Creates a ServiceMonitor for the metrics endpoint. The `prometheusServiceAccount` setting binds the metrics reader role to the Prometheus ServiceAccount, which authorizes the scrape. | + +3. Wait for the controller to roll out. + ```bash + kubectl rollout status deployment/kagent-controller -n kagent --timeout=300s + ``` + +4. Wait for kagent to recompile the pair. The controller rebuilds each pair after it restarts, and an AgentInstance that you create before the rebuild finishes starts from the previous revision, without the new settings. The following command prints `Recompiled` when the new revision is ready. + ```bash + for i in $(seq 1 60); do + [ "$(kubectl get agenttemplate my-first-agent -n kagent \ + -o jsonpath='{.status.harnesses[0].latestSuccessfulRevision}')" != "$OLD_REVISION" ] \ + && echo "Recompiled" && break + sleep 5 + done + ``` + If the command finishes without printing `Recompiled`, the upgrade did not change the pair, for example because the settings were already in place. + +## Send Agent Substrate telemetry to the collector + +Point Agent Substrate at the collector. A single `otel.endpoint` setting turns on traces, logs, and metrics for every Agent Substrate component. + +1. Upgrade the Agent Substrate Helm release. + ```bash + helm upgrade substrate \ + oci://ghcr.io/kagent-dev/substrate/helm/substrate \ + --version {{< reuse "kagent-docs/versions/agent-substrate.md" >}} \ + --namespace ate-system \ + --reuse-values \ + --set otel.endpoint=http://otel-collector.telemetry.svc.cluster.local:4317 \ + --set otel.traces.samplingRatio=1.0 \ + --wait --timeout 10m + ``` + + Agent Substrate keeps 1% of its traces by default, so a few test requests rarely produce one. A `samplingRatio` of `1.0` keeps every trace so that you can see results right away. Lower it again for production, because the router then records a trace for every request that it forwards. + +2. Verify that the Agent Substrate pods are running. + ```bash + kubectl get pods -n ate-system + ``` + +## Send a request + +1. Create a new AgentInstance. An AgentInstance keeps the runtime configuration that it was created with, so only a new AgentInstance exports traces and audit events. + ```bash + kagent create agent-instance --harness my-first-harness --agent-template my-first-agent + ``` + +2. Confirm that the AgentInstance runs the current revision of the pair. The command waits until kagent finishes compiling the pair, then compares that revision with the one that the AgentInstance started from. If the command prints `Outdated`, the AgentInstance was created from an earlier revision, and exports without the new settings. Create another AgentInstance, and run the command again. + ```bash + for i in $(seq 1 60); do + REVISIONS=$(kubectl get agenttemplate my-first-agent -n kagent \ + -o jsonpath='{.status.harnesses[0].desiredRevision} {.status.harnesses[0].latestSuccessfulRevision}') + [ "${REVISIONS% *}" = "${REVISIONS#* }" ] && break + sleep 5 + done + INSTANCE_REVISION=$(kagent get agent-instance -o json \ + | jq -r '[.agentInstances[] | select(.agentTemplate.name == "my-first-agent")] | sort_by(.createdAt) | last | .preparedRevision') + [ "$INSTANCE_REVISION" = "${REVISIONS#* }" ] && echo "Current" || echo "Outdated" + ``` + +3. Send a few requests to produce telemetry. + ```bash + export INSTANCE_ID=$(kagent get agent-instance -o json \ + | jq -r '[.agentInstances[] | select(.agentTemplate.name == "my-first-agent")] | sort_by(.createdAt) | last | .id') + kagent invoke --agent-instance $INSTANCE_ID --task "What is 2+2?" + kagent invoke --agent-instance $INSTANCE_ID --task "What did I just ask you?" + ``` + +## Explore the telemetry in Grafana + +1. Get the Grafana password for the `admin` user. + ```bash + kubectl get secret -n telemetry kube-prometheus-stack-grafana \ + -o jsonpath='{.data.admin-password}' | base64 --decode; echo + ``` + +2. Forward the Grafana port, and leave the command running. + ```bash + kubectl port-forward -n telemetry svc/kube-prometheus-stack-grafana 3000:80 + ``` + +3. In your browser, open Grafana at [http://localhost:3000](http://localhost:3000), and log in as `admin` with the password from the first step. + +4. Open **Explore**, and review each signal. + {{< tabs >}} + {{% tab name="Traces" %}} + Select the **Tempo** data source, then select the **Search** query type. From the **Service Name** list, select `my-first-agent-my-first-harness`, the service that the AgentTemplate and Harness pair reports as, and run the query. Open a trace to see the controller, proxy, and agent runtime spans of one request. + + To see Agent Substrate's own work, select `ateapi`, `atenet-router`, or `atelet` from the **Service Name** list instead. + {{% /tab %}} + {{% tab name="Logs" %}} + Select the **Loki** data source, and run the following query to show the audit events of your agent. + ```text + {service_name="my-first-agent-my-first-harness"} + ``` + + The runtime exports audit events in batches, and Agent Substrate suspends the Actor as soon as a response completes. The events of the most recent request can therefore appear only after you send the next one. + + To see Actor lifecycle events from Agent Substrate, query `{service_name="ateapi"}` instead. + {{% /tab %}} + {{% tab name="Metrics" %}} + Select the **Prometheus** data source, and run a query. For example, the following query returns the number of ready Workers in each WorkerPool. + ```text + ate_workerpool_ready_workers + ``` + + The following query returns the rate of gRPC requests that the kagent controller serves, by method. + ```text + sum by (method) (rate(kagent_grpc_server_requests_total[5m])) + ``` + + For more metrics, see [Metrics]({{< link path="observability/metrics" >}}). + {{% /tab %}} + {{< /tabs >}} + +## Clean up + +1. Turn off telemetry export in kagent and Agent Substrate. + ```bash + helm upgrade kagent \ + {{< reuse "kagent-docs/snippets/helm-path.md" >}}/{{< reuse "kagent-docs/snippets/helm-kagent.md" >}} \ + --version {{< reuse "kagent-docs/versions/kagent.md" >}} \ + --namespace kagent --reuse-values \ + --set otel.tracing.enabled=false \ + --set otel.logging.enabled=false \ + --set controller.metrics.enabled=false \ + --set controller.metrics.serviceMonitor.enabled=false + helm upgrade substrate \ + oci://ghcr.io/kagent-dev/substrate/helm/substrate \ + --version {{< reuse "kagent-docs/versions/agent-substrate.md" >}} \ + --namespace ate-system --reuse-values \ + --set otel.endpoint="" \ + --set otel.traces.samplingRatio=0.01 + ``` + +2. Remove the collector, the backends, and the `telemetry` namespace. + ```bash + kubectl delete servicemonitor otel-collector -n telemetry + helm uninstall otel-collector kube-prometheus-stack loki tempo -n telemetry + kubectl delete namespace telemetry + ``` + +## Next steps + +{{< cards >}} + {{< card link=`{{< link path="observability/tracing" >}}` title="Tracing" subtitle="Read the spans and attributes of an agent request." >}} + {{< card link=`{{< link path="observability/metrics" >}}` title="Metrics" subtitle="Review the metrics that kagent and Agent Substrate report." >}} +{{< /cards >}} diff --git a/docs-site/content/kagent/1.x/observability/tracing.md b/docs-site/content/kagent/1.x/observability/tracing.md index 3571ebb9..93791e67 100644 --- a/docs-site/content/kagent/1.x/observability/tracing.md +++ b/docs-site/content/kagent/1.x/observability/tracing.md @@ -1,15 +1,15 @@ --- title: Tracing description: Enable OpenTelemetry tracing for kagent, then read a trace that runs from the controller through to the Actor that executed your agent. -weight: 20 +weight: 30 author: kagent.dev --- -A trace records one agent request as a tree of timed spans, so you can see where a slow or failed request spent its time and which model and tool calls it made along the way. In {{< reuse "kagent-docs/snippets/name-product.md" >}} 1.0 a single request crosses two processes, the controller and the {{< gloss "Actor" >}}Actor{{< /gloss >}} that runs the agent, and a trace ties both halves together. +A trace records one agent request as a tree of timed spans, so you can see where a slow or failed request spent its time and which model and tool calls it made along the way. In {{< reuse "kagent-docs/snippets/name-product.md" >}} 1.0 a single request crosses the controller, the Agent Substrate router, and the {{< gloss "Actor" >}}Actor{{< /gloss >}} that runs the agent, and one trace ties all three together. ## About trace coverage -Tracing spans two processes, and the link between them is a W3C Trace Context header that the controller passes to the Actor. The following diagram traces one request through both. +A trace follows a W3C Trace Context header that the controller passes along with each request. The following diagram traces one request from the caller to the agent runtime.

```mermaid @@ -19,6 +19,9 @@ flowchart LR grpc["gRPC API"] gateway["A2A gateway"] end + subgraph substrateproc["Agent Substrate"] + router["Router"] + end subgraph actorproc["Actor"] runtime["Agent runtime"] end @@ -26,14 +29,15 @@ flowchart LR %% mermaid assigns a node to the subgraph that first references it. caller --> grpc grpc --> gateway - gateway -->|traceparent| runtime + gateway -->|traceparent| router + router --> runtime classDef boundary fill:#a78bfa26,stroke:#a78bfa,stroke-width:2px classDef inner fill:#80808033,stroke:#9ca3af,stroke-width:1px - class controllerproc,actorproc boundary - class grpc,gateway,runtime inner + class controllerproc,substrateproc,actorproc boundary + class grpc,gateway,router,runtime inner ``` -A caller reaches the gRPC API on the kagent controller, which starts the trace. The controller hands the request to its A2A gateway, which opens an {{< gloss "A2A" >}}A2A{{< /gloss >}} (Agent-to-Agent) connection to the AgentInstance's Actor and injects a `traceparent` header into that call. The agent runtime inside the Actor reads the header and continues the same trace, so the model and tool spans it produces hang off the controller's spans rather than starting a trace of their own. +A caller reaches the gRPC API on the kagent controller, which starts the trace. The controller hands the request to its A2A gateway, which opens an {{< gloss "A2A" >}}A2A{{< /gloss >}} (Agent-to-Agent) connection to the AgentInstance's Actor and injects a `traceparent` header into that call. The Agent Substrate router forwards the call to the Worker that runs the Actor, and adds its own spans to the trace. The agent runtime inside the Actor reads the header and continues the same trace, so the model and tool spans it produces hang off the controller's spans rather than starting a trace of their own. > [!IMPORTANT] > The controller passes its tracing configuration to the `kagent`, `codex`, and `claude` runtimes. Each of the three exports on its own instrumentation, so the span names in this page describe the `kagent` runtime and do not carry over to the other two. An agent on the `byo` runtime receives no tracing configuration, and its half of the trace is missing. For the available runtimes, see [Choose a runtime]({{< link path="agents/agent-harness#choose-a-runtime" >}}). @@ -41,10 +45,11 @@ A caller reaches the gRPC API on the kagent controller, which starts the trace. > [!NOTE] > A `byo` image that implements OTel itself reads the exporter variables from the Harness `spec.env`, which the controller leaves alone for this runtime. Its spans still do not reach a collector inside the cluster, because kagent adds the collector to an Actor's egress allowlist only for the runtimes it configures, and no field adds a host to that list by hand. For more information, see [Networking and egress control]({{< link path="substrate-runtime/networking-and-egress#policy-generation" >}}). -Both processes report themselves as separate OpenTelemetry (OTel) services. A tracing backend uses these service names to group the spans. +Each hop reports itself as a separate OpenTelemetry (OTel) service. A tracing backend uses these service names to group the spans. - **The controller** reports as `kagent-controller` in the `kagent` service namespace. Its spans also carry the pod, node, and namespace that the controller runs on. -- **Each agent runtime** reports as its own service, named for the {{< gloss "AgentTemplate" >}}AgentTemplate{{< /gloss >}} and {{< gloss "Harness" >}}Harness{{< /gloss >}} pair it was compiled from, with hyphens replaced by underscores. The `my-first-agent` template on the `my-first-harness` Harness reports as `my_first_agent_my_first_harness`. +- **The Agent Substrate router** reports as `agentgateway`, the proxy that the router runs. +- **Each agent runtime** reports as its own service, named for the {{< gloss "AgentTemplate" >}}AgentTemplate{{< /gloss >}} and {{< gloss "Harness" >}}Harness{{< /gloss >}} pair it was compiled from. The `my-first-agent` template on the `my-first-harness` Harness reports as `my-first-agent-my-first-harness`. > [!NOTE] > A service per template and Harness pair is a change from kagent 0.x, where every agent reported under one `kagent` service. A backend that you filter by service now shows one entry for each pair, and adding an agent adds a service. @@ -56,8 +61,9 @@ The `kagent` runtime creates the same spans for every agent, and most span names | Span | When it is created | | ---- | ------------------ | | `POST /lf.a2a.v1.A2AService/SendMessage` | Once per request, as the root of the runtime's half of the trace. The runtime creates it when it accepts the A2A call from the controller. | +| `a2a.request` | Once per request. Records the A2A method and the final state of the task in the `a2a.method` and `a2a.task.state` attributes. | | `invocation` | Once per request, as the parent of the agent's own work. | -| `invoke_agent ` | Once per request, named for the AgentTemplate and Harness pair that serves it. | +| `invoke_agent ` | Once per request, named for the AgentTemplate and Harness pair that serves it. Unlike the service name, the span name replaces hyphens with underscores, such as `invoke_agent my_first_agent_my_first_harness`. | | `generate_content ` | Once per model call, named for the model that was called. | | `execute_tool ` | Once per tool call, named for the tool that was called. | | `execute_tool (merged)` | Once per model turn that calls more than one tool, as the parent of that turn's `execute_tool` spans. A turn that calls a single tool creates no merged span. | @@ -76,63 +82,43 @@ A trace tells you which request you are looking at through attributes on its spa The runtime also adds each scalar value in the A2A message's metadata as an `a2a.message.metadata.` attribute, so a client can tag a request and search for it later. Unlike the four correlation attributes, these tags stay on the `invocation` span alone, so a search on one returns that span instead of the whole subtree. > [!WARNING] -> Spans for a model call carry the full serialized request and response as the `gcp.vertex.agent.llm_request` and `gcp.vertex.agent.llm_response` attributes. Prompts and replies therefore reach your tracing backend. Payloads larger than 32 KiB are truncated to a prefix. To keep this content out of traces, set `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT` to `false` in the Harness `spec.env`. Note that the same variable has the opposite default for audit logging, where content is withheld until you set it to `true`. For more information, see [Audit prompts]({{< link path="observability/audit-prompts" >}}). The `codex` and `claude` runtimes do not read this variable. On those two runtimes, the `otel.captureSensitiveContent` Helm setting controls the same content, as described in the agent harness [telemetry content settings]({{< link path="agents/agent-harness#telemetry-content-settings" >}}). +> Spans for a model call carry the full serialized request and response as the `gcp.vertex.agent.llm_request` and `gcp.vertex.agent.llm_response` attributes. Prompts and replies therefore reach your tracing backend. Payloads larger than 32 KiB are truncated to a prefix. To keep this content out of traces, set `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT` to `false` in the Harness `spec.env`. The `codex` and `claude` runtimes do not read this variable. On those two runtimes, the `otel.captureSensitiveContent` Helm setting controls the same content, as described in the agent harness [telemetry content settings]({{< link path="agents/agent-harness#telemetry-content-settings" >}}). ## Before you begin 1. [Install kagent]({{< link path="setup/installation" >}}). 2. [Create your first agent]({{< link path="get-started/your-first-agent" >}}), so that you have an {{< gloss "AgentInstance" >}}AgentInstance{{< /gloss >}} to send a request to. +3. Set up a tracing backend. The [OTel stack]({{< link path="observability/otel-stack" >}}) sends traces to Tempo, and the [Lightweight OTel stack]({{< link path="observability/lightweight-otel-stack" >}}) sends traces to Jaeger. Both guides turn on tracing for you, so you can skip to [Review a trace](#review-a-trace). -## Install Jaeger - -Install a backend that accepts OpenTelemetry Protocol (OTLP) traces. The following steps install [Jaeger](https://www.jaegertracing.io/) in all-in-one mode, which stores traces in memory and needs no other components. +## Enable tracing -1. Create a `jaeger.yaml` configuration file. - ```yaml - cat << 'EOF' > jaeger.yaml - provisionDataStore: - cassandra: false - allInOne: - enabled: true - storage: - type: memory - agent: - enabled: false - collector: - enabled: false - query: - enabled: false - EOF - ``` +Tracing is off by default. Turning it on is a Helm change, because the controller reads its tracing configuration from the environment and passes it to the agent runtimes it starts. The following steps send traces to the collector that both stack guides install. To send traces to another OTLP backend, change the endpoint. -2. Install Jaeger. +1. Save the current revision of your Harness and AgentTemplate pair. A later step uses it to tell when kagent recompiles the pair with the new settings. The command first waits for any recompile that is still in progress, such as one from an earlier Helm upgrade, so that it saves a finished revision. ```bash - helm repo add jaegertracing https://jaegertracing.github.io/helm-charts - helm repo update - helm upgrade --install jaeger jaegertracing/jaeger \ - --namespace jaeger --create-namespace \ - --history-max 3 \ - --values jaeger.yaml \ - --version {{< reuse "kagent-docs/versions/jaeger.md" >}} + for i in $(seq 1 60); do + REVISIONS=$(kubectl get agenttemplate my-first-agent -n kagent \ + -o jsonpath='{.status.harnesses[0].desiredRevision} {.status.harnesses[0].latestSuccessfulRevision}') + [ "${REVISIONS% *}" = "${REVISIONS#* }" ] && break + sleep 5 + done + export OLD_REVISION=${REVISIONS#* } + echo "Current revision: $OLD_REVISION" ``` -## Enable tracing - -Tracing is off by default. Turning it on is a Helm change, because the controller reads its tracing configuration from the environment and passes it to the agent runtimes it starts. - -1. Get your current Helm values for kagent. +2. Get your current Helm values for kagent. ```shell helm get values kagent -n kagent -o yaml > values.yaml ``` -2. Add the tracing settings to the values file, pointing the exporter at Jaeger. +3. Add the tracing settings to the values file. ```yaml otel: tracing: enabled: true exporter: otlp: - endpoint: http://jaeger.jaeger.svc.cluster.local:4317 + endpoint: http://otel-collector.telemetry.svc.cluster.local:4317 protocol: grpc timeout: 15000 insecure: true @@ -148,7 +134,7 @@ Tracing is off by default. Turning it on is a Helm change, because the controlle | `exporter.otlp.timeout` | The export timeout in milliseconds. Defaults to `15000`. | | `exporter.otlp.insecure` | Whether to skip Transport Layer Security (TLS) for the exporter connection. Defaults to `true`. | -3. Upgrade the kagent Helm release. +4. Upgrade the kagent Helm release. ```bash helm upgrade kagent \ {{< reuse "kagent-docs/snippets/helm-path.md" >}}/{{< reuse "kagent-docs/snippets/helm-kagent.md" >}} \ @@ -157,11 +143,35 @@ Tracing is off by default. Turning it on is a Helm change, because the controlle --values values.yaml ``` -4. Create a new AgentInstance, so that its Actor starts from a runtime that has the tracing configuration. +5. Wait for kagent to recompile the pair. The controller rebuilds each pair after it restarts, and an AgentInstance that you create before the rebuild finishes starts from the previous revision, without the new settings. The following command prints `Recompiled` when the new revision is ready. + ```bash + for i in $(seq 1 60); do + [ "$(kubectl get agenttemplate my-first-agent -n kagent \ + -o jsonpath='{.status.harnesses[0].latestSuccessfulRevision}')" != "$OLD_REVISION" ] \ + && echo "Recompiled" && break + sleep 5 + done + ``` + If the command finishes without printing `Recompiled`, the upgrade did not change the pair, for example because the settings were already in place. + +6. Create a new AgentInstance, so that its Actor starts from a runtime that has the tracing configuration. ```bash kagent create agent-instance --harness my-first-harness --agent-template my-first-agent ``` +7. Confirm that the AgentInstance runs the current revision of the pair. The command waits until kagent finishes compiling the pair, then compares that revision with the one that the AgentInstance started from. If the command prints `Outdated`, the AgentInstance was created from an earlier revision, and exports without the new settings. Create another AgentInstance, and run the command again. + ```bash + for i in $(seq 1 60); do + REVISIONS=$(kubectl get agenttemplate my-first-agent -n kagent \ + -o jsonpath='{.status.harnesses[0].desiredRevision} {.status.harnesses[0].latestSuccessfulRevision}') + [ "${REVISIONS% *}" = "${REVISIONS#* }" ] && break + sleep 5 + done + INSTANCE_REVISION=$(kagent get agent-instance -o json \ + | jq -r '[.agentInstances[] | select(.agentTemplate.name == "my-first-agent")] | sort_by(.createdAt) | last | .preparedRevision') + [ "$INSTANCE_REVISION" = "${REVISIONS#* }" ] && echo "Current" || echo "Outdated" + ``` + ## Review a trace 1. Send a request to the AgentInstance to produce a trace. @@ -171,20 +181,59 @@ Tracing is off by default. Turning it on is a Helm change, because the controlle kagent invoke --agent-instance $INSTANCE_ID --task "What is 2+2?" ``` -2. Forward the Jaeger query port, and leave the command running. - ```bash - kubectl port-forward -n jaeger svc/jaeger 16686:16686 +2. Open the trace in your tracing backend. + {{< tabs >}} + {{% tab name="Grafana with Tempo" %}} + 1. Forward the Grafana port, and leave the command running. + ```bash + kubectl port-forward -n telemetry svc/kube-prometheus-stack-grafana 3000:80 + ``` + 2. In your browser, open Grafana at [http://localhost:3000](http://localhost:3000), and log in. For the password, see [Explore the telemetry in Grafana]({{< link path="observability/otel-stack#explore-the-telemetry-in-grafana" >}}). + 3. Open **Explore**, select the **Tempo** data source, and select the **Search** query type. + 4. From the **Service Name** list, select `my-first-agent-my-first-harness`, and run the query. Selecting `kagent-controller` instead returns the same traces from the controller's side. + 5. Click a trace to open it. + {{% /tab %}} + {{% tab name="Jaeger" %}} + 1. Forward the Jaeger query port, and leave the command running. + ```bash + kubectl port-forward -n telemetry svc/jaeger 16686:16686 + ``` + 2. In your browser, open Jaeger at [http://localhost:16686](http://localhost:16686). + 3. From the **Service** list, select `my-first-agent-my-first-harness`. Selecting `kagent-controller` instead returns the same traces from the controller's side. + 4. Leave **Operation** on `all`, or select `invocation` to start from the agent's own work rather than from the A2A call that carries it, and click **Find Traces**. + 5. Click a trace to open it. + {{% /tab %}} + {{< /tabs >}} + +3. Review the span tree. The trace starts with the controller's spans, continues through the Agent Substrate router, and ends with the agent runtime's spans. The following example shows the spans of one request, with the service that reported each span. + ```console + lf.a2a.v1.A2AService/SendMessage kagent-controller + lf.a2a.v1.A2AService/SendMessage kagent-controller + POST /* agentgateway + POST agentgateway + POST /lf.a2a.v1.A2AService/SendMessage my-first-agent-my-first-harness + a2a.request my-first-agent-my-first-harness + invocation my-first-agent-my-first-harness + invoke_agent my_first_agent_my_first_harness my-first-agent-my-first-harness + generate_content gpt-4.1-mini my-first-agent-my-first-harness + HTTP POST my-first-agent-my-first-harness ``` -3. In your browser, open the Jaeger user interface at [http://localhost:16686](http://localhost:16686). +4. To narrow a search to one conversation, search by a correlation attribute, such as `gen_ai.conversation.id=`. -4. From the **Service** dropdown, select `my_first_agent_my_first_harness`, the service that the AgentTemplate and Harness pair reports as. Selecting `kagent-controller` instead returns the same traces from the controller's side. +## Agent Substrate traces -5. Leave **Operation** on `all`, or select `invocation` to start from the agent's own work rather than from the A2A call that carries it, and click **Find Traces**. +Agent Substrate records traces for its own work, such as scheduling an Actor onto a Worker and restoring it from a snapshot. These traces are separate from the agent request trace. They do not share its trace ID, so a request trace does not show how long the Actor took to resume. To investigate a slow start, look up the Agent Substrate traces from the same time window. -6. Click a trace to open it. The span tree shows the controller's gRPC and gateway spans, followed by the runtime's `POST /lf.a2a.v1.A2AService/SendMessage` span, and finally the `invocation`, `invoke_agent`, `generate_content`, and `execute_tool` spans. +| Service | Reports | +| ------- | ------- | +| `atenet-router` | Requests that the router receives, and its calls to `ateapi` to find or resume the Actor for each request. | +| `ateapi` | Actor lifecycle operations, such as create, resume, and suspend, and the scheduling of Actors onto Workers. | +| `atelet` | Work on a Worker's node, such as restoring an Actor from a snapshot. | +| `ateom-gvisor` | Work inside the sandbox that runs the Actor. | +| `atecontroller` | Reconciliation of Agent Substrate resources, such as WorkerPools. | -7. To narrow a search to one conversation, put a correlation attribute in the **Tags** field, such as `gen_ai.conversation.id=`. +Agent Substrate exports traces only when its Helm release sets `otel.endpoint`, and it keeps 1% of its traces by default. To keep more, raise `otel.traces.samplingRatio`, as the stack guides do. For the steps, see [Send Agent Substrate telemetry to the collector]({{< link path="observability/otel-stack#send-agent-substrate-telemetry-to-the-collector" >}}). ## Traces from a suspended Actor @@ -207,15 +256,11 @@ This behavior allows a kagent trace to arrive promptly rather than on the export 2. Create a new AgentInstance to pick up the change, because an existing Actor keeps the configuration it started with. -3. Remove Jaeger. - ```bash - helm uninstall jaeger -n jaeger - kubectl delete namespace jaeger - ``` +3. To remove the tracing backend, follow the cleanup steps in the [OTel stack]({{< link path="observability/otel-stack#clean-up" >}}) or [Lightweight OTel stack]({{< link path="observability/lightweight-otel-stack#clean-up" >}}) guide. ## Next steps {{< cards >}} - {{< card link=`{{< link path="observability/audit-prompts" >}}` title="Audit prompts" subtitle="Export every prompt and reply as a log event for security and compliance review." >}} + {{< card link=`{{< link path="observability/metrics" >}}` title="Metrics" subtitle="Review the metrics that kagent and Agent Substrate report." >}} {{< card link=`{{< link path="substrate-runtime/suspend-and-resume" >}}` title="Suspend and resume" subtitle="Learn what happens to an Actor between the turns of a conversation." >}} {{< /cards >}} diff --git a/docs-site/content/kagent/1.x/operations/operational-considerations.md b/docs-site/content/kagent/1.x/operations/operational-considerations.md index 4d6feada..b4688899 100644 --- a/docs-site/content/kagent/1.x/operations/operational-considerations.md +++ b/docs-site/content/kagent/1.x/operations/operational-considerations.md @@ -131,17 +131,7 @@ The controller rewrites internally built Kubernetes URLs to the proxy and sets t ## Scrape controller metrics -The controller serves a Prometheus-style `/metrics` endpoint, turned off by default. Enabling it provisions a dedicated metrics Service and the ClusterRoles that an authenticated scrape needs. - -```yaml -controller: - metrics: - enabled: true - bindAddress: ":8443" - secureServing: true -``` - -Bind the `-metrics-reader` ClusterRole to your Prometheus ServiceAccount to grant scrape access. Change the port through `bindAddress` rather than through `controller.env`, because the chart derives the Service `targetPort` and the pod `containerPort` from `bindAddress` at template time. Overriding `METRICS_BIND_ADDRESS` directly moves the listener and leaves the Service pointing at the old port. +The controller serves a Prometheus `/metrics` endpoint, turned off by default. To enable it, grant Prometheus access, and review the metrics that it reports, see [Metrics]({{< link path="observability/metrics" >}}). ## Isolate what an agent can do From 89128c00a2d1e407611f276056cfbef27e469835 Mon Sep 17 00:00:00 2001 From: Kristin Brown Date: Thu, 24 Sep 2026 10:53:24 -0400 Subject: [PATCH 2/5] Updates Signed-off-by: Kristin Brown --- docs-site/content/kagent/1.x/agents/agent-harness.md | 4 ++-- docs-site/content/kagent/1.x/observability/tracing.md | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/docs-site/content/kagent/1.x/agents/agent-harness.md b/docs-site/content/kagent/1.x/agents/agent-harness.md index 0b685bf8..2f517fc7 100644 --- a/docs-site/content/kagent/1.x/agents/agent-harness.md +++ b/docs-site/content/kagent/1.x/agents/agent-harness.md @@ -143,10 +143,10 @@ otel: | Setting | What it includes | Applies to | | ------- | ---------------- | ---------- | -| `otel.captureSensitiveContent` | Prompts, tool details, and assistant replies in the runtime's telemetry. On the `claude` runtime, tool results require tracing, and assistant replies require audit logging. | `codex`, `claude` | +| `otel.captureSensitiveContent` | Prompts, tool details, and assistant replies in the runtime's telemetry. On the `kagent` runtime, the content appears in the spans for each model call. On the `claude` runtime, tool results require tracing, and assistant replies require audit logging. | `kagent`, `codex`, `claude` | | `otel.logging.captureRawApiBodies` | The complete provider API request and response bodies. This setting returns more than `otel.captureSensitiveContent` does, and it takes effect only when `otel.logging.enabled` is `true`. | `claude` | -The `kagent` runtime honors neither setting. To include message content for an agent on that runtime, set `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT` in the Harness `spec.env` field. +The controller sets `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT` in every compiled runtime from `otel.captureSensitiveContent`, so setting that variable in the Harness `spec.env` field has no effect. The controller sends the `byo` runtime no telemetry configuration, so neither setting reaches it. A `byo` image that implements OpenTelemetry itself reads whatever the Harness `spec.env` field holds. For more information, see [Tracing]({{< link path="observability/tracing#about-trace-coverage" >}}). diff --git a/docs-site/content/kagent/1.x/observability/tracing.md b/docs-site/content/kagent/1.x/observability/tracing.md index 93791e67..101ccc04 100644 --- a/docs-site/content/kagent/1.x/observability/tracing.md +++ b/docs-site/content/kagent/1.x/observability/tracing.md @@ -82,7 +82,7 @@ A trace tells you which request you are looking at through attributes on its spa The runtime also adds each scalar value in the A2A message's metadata as an `a2a.message.metadata.` attribute, so a client can tag a request and search for it later. Unlike the four correlation attributes, these tags stay on the `invocation` span alone, so a search on one returns that span instead of the whole subtree. > [!WARNING] -> Spans for a model call carry the full serialized request and response as the `gcp.vertex.agent.llm_request` and `gcp.vertex.agent.llm_response` attributes. Prompts and replies therefore reach your tracing backend. Payloads larger than 32 KiB are truncated to a prefix. To keep this content out of traces, set `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT` to `false` in the Harness `spec.env`. The `codex` and `claude` runtimes do not read this variable. On those two runtimes, the `otel.captureSensitiveContent` Helm setting controls the same content, as described in the agent harness [telemetry content settings]({{< link path="agents/agent-harness#telemetry-content-settings" >}}). +> When the `otel.captureSensitiveContent` Helm setting is `true`, prompts and replies reach your tracing backend. The spans for a model call then carry the full serialized request and response as the `gcp.vertex.agent.llm_request` and `gcp.vertex.agent.llm_response` attributes, truncated to a prefix when a payload is larger than 32 KiB. The setting defaults to `false`, which leaves both attributes as `{}`. For how the setting applies to each runtime, see the agent harness [telemetry content settings]({{< link path="agents/agent-harness#telemetry-content-settings" >}}). ## Before you begin From c6ce7e3883f467e1035810897e51d45f9183ec67 Mon Sep 17 00:00:00 2001 From: Kristin Brown Date: Thu, 24 Sep 2026 12:12:41 -0400 Subject: [PATCH 3/5] Added substrate topic Signed-off-by: Kristin Brown --- .../1.x/observability/substrate-telemetry.md | 118 ++++++++++++++++++ .../kagent/1.x/observability/tracing.md | 2 +- 2 files changed, 119 insertions(+), 1 deletion(-) create mode 100644 docs-site/content/kagent/1.x/observability/substrate-telemetry.md diff --git a/docs-site/content/kagent/1.x/observability/substrate-telemetry.md b/docs-site/content/kagent/1.x/observability/substrate-telemetry.md new file mode 100644 index 00000000..1c7d2f35 --- /dev/null +++ b/docs-site/content/kagent/1.x/observability/substrate-telemetry.md @@ -0,0 +1,118 @@ +--- +title: Agent Substrate telemetry +description: Follow an agent's Actor across suspend and resume by using the Actor identity that Agent Substrate adds to its logs, traces, and metrics. +weight: 35 +author: kagent.dev +--- + +Every {{< reuse "kagent-docs/snippets/name-product.md" >}} agent runs as an {{< gloss "Actor" >}}Actor{{< /gloss >}} on Agent Substrate. An Actor suspends after each turn, and can resume on a different {{< gloss "Worker" >}}Worker{{< /gloss >}} pod the next time, so telemetry that is keyed to a pod loses track of an agent between turns. To keep one agent's telemetry together, Agent Substrate labels its telemetry with the identity of the Actor rather than the pod that ran it. + +## What Agent Substrate reports + +Agent Substrate reports telemetry from its own components, separately from the telemetry that kagent reports for each agent request. + +| Signal | Where it goes | Details | +| ------ | ------------- | ------- | +| Actor logs | The stdout of the Worker pod that runs the Actor | The Actor's own output, and records for each suspend and resume. See [Read an Actor's logs](#read-an-actors-logs). | +| Component logs | Over OTLP, when an OTLP endpoint is set | Logs from the Agent Substrate components, including an access log entry for every request that the router receives. | +| Traces | Over OTLP, when an OTLP endpoint is set | Scheduling, resume, and suspend work. These traces are separate from the agent request trace. See [Agent Substrate traces]({{< link path="observability/tracing#agent-substrate-traces" >}}). | +| Metrics | Over OTLP, when an OTLP endpoint is set | Worker capacity, scheduling, and snapshot timing. See [Agent Substrate metrics]({{< link path="observability/metrics#agent-substrate-metrics" >}}). | + +The Agent Substrate Helm chart sets one OTLP endpoint for traces, metrics, and logs through `otel.endpoint`, and samples traces at a ratio of `0.01` by default through `otel.traces.samplingRatio`. The [OTel stack]({{< link path="observability/otel-stack#send-agent-substrate-telemetry-to-the-collector" >}}) and [Lightweight OTel stack]({{< link path="observability/lightweight-otel-stack" >}}) guides set both. + +## Actor identity + +Agent Substrate adds the following labels to Actor logs and to its component logs about an Actor. Each label maps to a kagent object, so you can find an agent's telemetry from the AgentInstance or the AgentTemplate that you already know. + +| Label | Value for a kagent agent | +| ----- | ------------------------ | +| `ate.actor.name` | `ai-`, one Actor for each AgentInstance. | +| `ate.actor.uid` | A unique ID for the lifetime of the Actor. A deleted and recreated Actor gets a new one. | +| `ate.atespace` | The {{< gloss "Atespace" >}}atespace{{< /gloss >}} of the Actor, which is the namespace of the AgentTemplate and Harness, such as `kagent`. The Actors that build a pair's first snapshot run in `ate-golden` instead. | +| `ate.template.name` | The AgentTemplate name, the Harness name, and the short form of the {{< gloss "Revision" >}}revision{{< /gloss >}}, joined by hyphens, such as `my-first-agent-my-first-harness-a995d20d30a7`. All the AgentInstances of one pair at one revision share this value. | +| `ate.template.atespace` | Where the template lives, which matches `ate.atespace` for a kagent agent. | +| `ate.actor.container.name` | The container that wrote the line, such as `kagent`. Records for a suspend or resume omit this label, because the Actor writes them rather than a container. | + +Agent Substrate keeps the labels that identify one Actor off its metrics, so that the number of time series does not grow with the number of conversations. Metrics carry the template and WorkerPool labels instead. To look into one agent, use its logs. + +## Read an Actor's logs + +1. Find the Actor for an AgentInstance. The Actor name is `ai-` followed by the AgentInstance ID. + ```bash + export INSTANCE_ID=$(kagent get agent-instance -o json \ + | jq -r '[.agentInstances[] | select(.agentTemplate.name == "my-first-agent")] | sort_by(.createdAt) | last | .id') + kubectl ate get actors --atespace kagent + ``` + + Example output: + ```console + ATESPACE NAME TEMPLATE STATE WORKER POD WORKER IP VERSION AGE + kagent ai-01a0d409-d249-728e-865c-ee58b55dff04 kagent/my-first-agent-my-first-harness-a995d20d30a7 ACTOR_STATE_SUSPENDED 5 32m + ``` + +2. Read the logs from the Workers of the WorkerPool that the Harness runs on. A Worker writes the output of every Actor that it runs, so filter the output by the Actor name. The filter returns the Actor's lines from every current Worker in the pool, including the lines from before the Actor last suspended. Worker output mixes JSON records with plain text lines, so the filter skips any line that is not JSON. + ```bash + export WORKER_POOL=$(kubectl get harness my-first-harness -n kagent \ + -o jsonpath='{.spec.substrate.workerPoolRef.name}') + kubectl logs -n kagent -l ate.dev/worker-pool=$WORKER_POOL --tail=-1 \ + | jq -cR "fromjson? | select(.labels[\"ate.actor.name\"] == \"ai-${INSTANCE_ID}\")" + ``` + + Example output: + ```console + {"labels":{"ate.actor.name":"ai-01a0d409-d249-728e-865c-ee58b55dff04","ate.actor.uid":"c91579a8-675c-474c-9437-4482ac628774","ate.atespace":"kagent","ate.template.atespace":"kagent","ate.template.name":"my-first-agent-my-first-harness-a995d20d30a7"},"message":"Actor restoring","span_id":"d81bb304af04c945","time":"2026-09-24T16:02:52.008842377Z","trace_flags":"00","trace_id":"3b8987f98057610937b38639136d32b5"} + {"labels":{"ate.actor.name":"ai-01a0d409-d249-728e-865c-ee58b55dff04","ate.actor.uid":"c91579a8-675c-474c-9437-4482ac628774","ate.atespace":"kagent","ate.template.atespace":"kagent","ate.template.name":"my-first-agent-my-first-harness-a995d20d30a7"},"message":"Actor checkpointed","span_id":"4914db77fcc2f59b","time":"2026-09-24T15:30:09.671009219Z","trace_flags":"01","trace_id":"c4971ca2e614428819443f64fe70f81c"} + ``` + + `kubectl logs` reads only the Worker pods that exist now. When a Worker pod is replaced, the output that it held is gone. To keep an Actor's history across Worker replacements, collect the stdout of the Worker pods with a log agent that stores structured JSON, and query it by the same labels. Neither OTel stack guide collects pod stdout. + +The `kubectl ate logs actors` command streams the output of a single Actor, but only while the Actor is running on a Worker. A kagent agent runs only during a turn, and the command returns an error the rest of the time. + +```console +Error: actor kagent/ai-01a0d409-d249-728e-865c-ee58b55dff04 is not currently running on any worker pod +``` + +## Suspend and resume records + +The records that Agent Substrate writes for an Actor mark each step of a suspend and resume cycle. Each record carries the Actor labels and a `message` field, and no container label. To list only these records for an Actor, filter out the lines that have a container label. + +```bash +kubectl logs -n kagent -l ate.dev/worker-pool=$WORKER_POOL --tail=-1 \ + | jq -cR "fromjson? | select(.labels[\"ate.actor.name\"] == \"ai-${INSTANCE_ID}\" and .labels[\"ate.actor.container.name\"] == null) | {time, message}" +``` + +Example output: +```console +{"time":"2026-09-24T15:30:08.965994177Z","message":"Actor restoring"} +{"time":"2026-09-24T15:30:09.04801126Z","message":"Actor restored"} +{"time":"2026-09-24T15:30:09.582808844Z","message":"Actor checkpointing"} +{"time":"2026-09-24T15:30:09.671009219Z","message":"Actor checkpointed"} +{"time":"2026-09-24T16:02:52.008842377Z","message":"Actor restoring"} +{"time":"2026-09-24T16:02:52.093494544Z","message":"Actor restored"} +{"time":"2026-09-24T16:02:52.481758336Z","message":"Actor checkpointing"} +{"time":"2026-09-24T16:02:52.588873294Z","message":"Actor checkpointed"} +``` + +| `message` | Written when | +| --------- | ------------ | +| `Actor restoring` | A Worker begins to restore the Actor from a snapshot. An AgentInstance's Actor restores even for its first turn, from the snapshot that kagent built for the pair, so this is the first record for every AgentInstance. | +| `Actor restored` | The restore finishes and the Actor can serve the request. | +| `Actor checkpointing` | The Actor begins to suspend, and Agent Substrate begins to write its snapshot. | +| `Actor checkpointed` | The snapshot is written. The Actor is suspended, and the Worker is free for another Actor. | + +Each turn produces one cycle of the four records. The time between `Actor restoring` and `Actor restored` is how long the agent took to wake up for the turn. + +The Actor that builds a pair's first snapshot, in the `ate-golden` atespace, writes `Actor starting` and `Actor started` instead of the restore records, because it has no snapshot to restore from. + +Each record also carries `trace_id`, `span_id`, and `trace_flags`, which join it to the Agent Substrate trace for the same operation. A `trace_flags` value of `01` means that the trace was sampled and is in your tracing backend. A value of `00` means that the trace was not sampled, which is the case for most requests at the default ratio of `0.01`. + +## Learn more + +The Agent Substrate project documents its telemetry in full, including the complete metric registry, in [Actor observability](https://github.com/agent-substrate/substrate/blob/main/docs/observability.md). That page describes the newest Agent Substrate, which can be ahead of version {{< reuse "kagent-docs/versions/agent-substrate.md" >}} that this guide uses. Records that it describes might not exist in your version, such as actor state change events and per-Actor usage samples. + +## Next steps + +{{< cards >}} + {{< card link=`{{< link path="observability/tracing" >}}` title="Tracing" subtitle="Read the spans of an agent request, and the separate Agent Substrate traces." >}} + {{< card link=`{{< link path="substrate-runtime/suspend-and-resume" >}}` title="Suspend and resume" subtitle="Understand what happens to an Actor between turns." >}} +{{< /cards >}} diff --git a/docs-site/content/kagent/1.x/observability/tracing.md b/docs-site/content/kagent/1.x/observability/tracing.md index 101ccc04..c4943e51 100644 --- a/docs-site/content/kagent/1.x/observability/tracing.md +++ b/docs-site/content/kagent/1.x/observability/tracing.md @@ -223,7 +223,7 @@ Tracing is off by default. Turning it on is a Helm change, because the controlle ## Agent Substrate traces -Agent Substrate records traces for its own work, such as scheduling an Actor onto a Worker and restoring it from a snapshot. These traces are separate from the agent request trace. They do not share its trace ID, so a request trace does not show how long the Actor took to resume. To investigate a slow start, look up the Agent Substrate traces from the same time window. +Agent Substrate records traces for its own work, such as scheduling an Actor onto a Worker and restoring it from a snapshot. These traces are separate from the agent request trace. They do not share its trace ID, so a request trace does not show how long the Actor took to resume. To investigate a slow start, look up the Agent Substrate traces from the same time window, or read the Actor's [suspend and resume records]({{< link path="observability/substrate-telemetry#suspend-and-resume-records" >}}), which carry the trace ID of each operation. | Service | Reports | | ------- | ------- | From b56771db8d2f22bc57ca590ed3abbe18b0ed0210 Mon Sep 17 00:00:00 2001 From: Kristin Brown Date: Thu, 24 Sep 2026 12:51:05 -0400 Subject: [PATCH 4/5] Clarity Signed-off-by: Kristin Brown --- docs-site/content/kagent/1.x/agents/agent-harness.md | 2 +- .../kagent/1.x/observability/lightweight-otel-stack.md | 2 +- docs-site/content/kagent/1.x/observability/metrics.md | 2 +- docs-site/content/kagent/1.x/observability/otel-stack.md | 2 +- .../kagent/1.x/observability/substrate-telemetry.md | 4 ++-- docs-site/content/kagent/1.x/observability/tracing.md | 8 ++++---- .../kagent/1.x/operations/operational-considerations.md | 2 +- 7 files changed, 11 insertions(+), 11 deletions(-) diff --git a/docs-site/content/kagent/1.x/agents/agent-harness.md b/docs-site/content/kagent/1.x/agents/agent-harness.md index 2f517fc7..bc4f7f2f 100644 --- a/docs-site/content/kagent/1.x/agents/agent-harness.md +++ b/docs-site/content/kagent/1.x/agents/agent-harness.md @@ -116,7 +116,7 @@ Some supported combinations still carry restrictions. | `claude` with `AnthropicVertexAI` | Accepts only `projectID` and `location`. The Secret must hold a `service_account` key whose `project_id` matches and whose `token_uri` is `https://oauth2.googleapis.com`. | > [!IMPORTANT] -> Neither `codex` nor `claude` accepts a ModelConfig that sets `defaultHeaders`, `tls`, or `apiKeyPassthrough`. Separately, the `kagent` and `byo` runtimes cannot use a ModelConfig whose credential is a file rather than a string. This restriction rules out both Vertex AI providers there. For more information about that limitation, see [About model providers]({{< link path="setup/model-providers/about-model-providers" >}}). +> Neither `codex` nor `claude` accepts a ModelConfig that sets `defaultHeaders`, `tls`, or `apiKeyPassthrough`. Separately, the `kagent` and `byo` runtimes cannot use a ModelConfig whose credential is a file rather than a string. The file credential restriction rules out both Vertex AI providers on the `kagent` and `byo` runtimes. For more information about that limitation, see [About model providers]({{< link path="setup/model-providers/about-model-providers" >}}). ## Tool and skill support diff --git a/docs-site/content/kagent/1.x/observability/lightweight-otel-stack.md b/docs-site/content/kagent/1.x/observability/lightweight-otel-stack.md index b456dd2f..0398d33b 100644 --- a/docs-site/content/kagent/1.x/observability/lightweight-otel-stack.md +++ b/docs-site/content/kagent/1.x/observability/lightweight-otel-stack.md @@ -232,7 +232,7 @@ EOF kubectl rollout status deployment/kagent-controller -n kagent --timeout=300s ``` -5. Wait for kagent to recompile the pair. The controller rebuilds each pair after it restarts, and an AgentInstance that you create before the rebuild finishes starts from the previous revision, without the new settings. The following command prints `Recompiled` when the new revision is ready. +5. Wait for kagent to recompile the pair. The controller rebuilds each pair after the controller restarts, and an AgentInstance that you create before the rebuild finishes starts from the previous revision, without the new settings. The following command prints `Recompiled` when the new revision is ready. ```bash for i in $(seq 1 60); do [ "$(kubectl get agenttemplate my-first-agent -n kagent \ diff --git a/docs-site/content/kagent/1.x/observability/metrics.md b/docs-site/content/kagent/1.x/observability/metrics.md index 554325ce..d5d96b01 100644 --- a/docs-site/content/kagent/1.x/observability/metrics.md +++ b/docs-site/content/kagent/1.x/observability/metrics.md @@ -62,7 +62,7 @@ To send metrics somewhere other than traces and logs, set `otel.metrics.endpoint ### Metric reference -The collector's Prometheus exporter converts each OpenTelemetry metric name to a Prometheus name. It replaces dots with underscores, and adds a unit suffix, such as `_seconds` or `_bytes`, and a `_total` suffix for counters. The following table lists the Prometheus names. Every metric also carries a `service_name` label for the Agent Substrate component that reported it, when the collector copies resource attributes to labels, as both stack guides configure. +The collector's Prometheus exporter converts each OpenTelemetry metric name to a Prometheus name. The exporter replaces dots with underscores, and adds a unit suffix, such as `_seconds` or `_bytes`, and a `_total` suffix for counters. The following table lists the Prometheus names. Every metric also carries a `service_name` label for the Agent Substrate component that reported it, when the collector copies resource attributes to labels, as both stack guides configure. | Metric | Reported by | Type | Measures | | ------ | ----------- | ---- | -------- | diff --git a/docs-site/content/kagent/1.x/observability/otel-stack.md b/docs-site/content/kagent/1.x/observability/otel-stack.md index ecc9d8cd..8e19a4c6 100644 --- a/docs-site/content/kagent/1.x/observability/otel-stack.md +++ b/docs-site/content/kagent/1.x/observability/otel-stack.md @@ -374,7 +374,7 @@ Turn on the kagent trace and log exporters, and point both at the collector. Als kubectl rollout status deployment/kagent-controller -n kagent --timeout=300s ``` -4. Wait for kagent to recompile the pair. The controller rebuilds each pair after it restarts, and an AgentInstance that you create before the rebuild finishes starts from the previous revision, without the new settings. The following command prints `Recompiled` when the new revision is ready. +4. Wait for kagent to recompile the pair. The controller rebuilds each pair after the controller restarts, and an AgentInstance that you create before the rebuild finishes starts from the previous revision, without the new settings. The following command prints `Recompiled` when the new revision is ready. ```bash for i in $(seq 1 60); do [ "$(kubectl get agenttemplate my-first-agent -n kagent \ diff --git a/docs-site/content/kagent/1.x/observability/substrate-telemetry.md b/docs-site/content/kagent/1.x/observability/substrate-telemetry.md index 1c7d2f35..de3a0caa 100644 --- a/docs-site/content/kagent/1.x/observability/substrate-telemetry.md +++ b/docs-site/content/kagent/1.x/observability/substrate-telemetry.md @@ -95,7 +95,7 @@ Example output: | `message` | Written when | | --------- | ------------ | -| `Actor restoring` | A Worker begins to restore the Actor from a snapshot. An AgentInstance's Actor restores even for its first turn, from the snapshot that kagent built for the pair, so this is the first record for every AgentInstance. | +| `Actor restoring` | A Worker begins to restore the Actor from a snapshot. An AgentInstance's Actor restores even for its first turn, from the snapshot that kagent built for the pair, so `Actor restoring` is the first record for every AgentInstance. | | `Actor restored` | The restore finishes and the Actor can serve the request. | | `Actor checkpointing` | The Actor begins to suspend, and Agent Substrate begins to write its snapshot. | | `Actor checkpointed` | The snapshot is written. The Actor is suspended, and the Worker is free for another Actor. | @@ -104,7 +104,7 @@ Each turn produces one cycle of the four records. The time between `Actor restor The Actor that builds a pair's first snapshot, in the `ate-golden` atespace, writes `Actor starting` and `Actor started` instead of the restore records, because it has no snapshot to restore from. -Each record also carries `trace_id`, `span_id`, and `trace_flags`, which join it to the Agent Substrate trace for the same operation. A `trace_flags` value of `01` means that the trace was sampled and is in your tracing backend. A value of `00` means that the trace was not sampled, which is the case for most requests at the default ratio of `0.01`. +Each record also carries `trace_id`, `span_id`, and `trace_flags`, which join the record to the Agent Substrate trace for the same operation. A `trace_flags` value of `01` means that the trace was sampled and is in your tracing backend. A value of `00` means that the trace was not sampled, which is the case for most requests at the default ratio of `0.01`. ## Learn more diff --git a/docs-site/content/kagent/1.x/observability/tracing.md b/docs-site/content/kagent/1.x/observability/tracing.md index c4943e51..22c2dce8 100644 --- a/docs-site/content/kagent/1.x/observability/tracing.md +++ b/docs-site/content/kagent/1.x/observability/tracing.md @@ -92,7 +92,7 @@ The runtime also adds each scalar value in the A2A message's metadata as an `a2a ## Enable tracing -Tracing is off by default. Turning it on is a Helm change, because the controller reads its tracing configuration from the environment and passes it to the agent runtimes it starts. The following steps send traces to the collector that both stack guides install. To send traces to another OTLP backend, change the endpoint. +Tracing is off by default. Turning it on is a Helm change, because the controller reads its tracing configuration from the environment and passes that configuration to the agent runtimes that the controller starts. The following steps send traces to the collector that both stack guides install. To send traces to another OTLP backend, change the endpoint. 1. Save the current revision of your Harness and AgentTemplate pair. A later step uses it to tell when kagent recompiles the pair with the new settings. The command first waits for any recompile that is still in progress, such as one from an earlier Helm upgrade, so that it saves a finished revision. ```bash @@ -143,7 +143,7 @@ Tracing is off by default. Turning it on is a Helm change, because the controlle --values values.yaml ``` -5. Wait for kagent to recompile the pair. The controller rebuilds each pair after it restarts, and an AgentInstance that you create before the rebuild finishes starts from the previous revision, without the new settings. The following command prints `Recompiled` when the new revision is ready. +5. Wait for kagent to recompile the pair. The controller rebuilds each pair after the controller restarts, and an AgentInstance that you create before the rebuild finishes starts from the previous revision, without the new settings. The following command prints `Recompiled` when the new revision is ready. ```bash for i in $(seq 1 60); do [ "$(kubectl get agenttemplate my-first-agent -n kagent \ @@ -223,7 +223,7 @@ Tracing is off by default. Turning it on is a Helm change, because the controlle ## Agent Substrate traces -Agent Substrate records traces for its own work, such as scheduling an Actor onto a Worker and restoring it from a snapshot. These traces are separate from the agent request trace. They do not share its trace ID, so a request trace does not show how long the Actor took to resume. To investigate a slow start, look up the Agent Substrate traces from the same time window, or read the Actor's [suspend and resume records]({{< link path="observability/substrate-telemetry#suspend-and-resume-records" >}}), which carry the trace ID of each operation. +Agent Substrate records traces for its own work, such as scheduling an Actor onto a Worker and restoring it from a snapshot. These traces are separate from the agent request trace. Agent Substrate traces do not share the request trace's ID, so a request trace does not show how long the Actor took to resume. To investigate a slow start, look up the Agent Substrate traces from the same time window, or read the Actor's [suspend and resume records]({{< link path="observability/substrate-telemetry#suspend-and-resume-records" >}}), which carry the trace ID of each operation. | Service | Reports | | ------- | ------- | @@ -241,7 +241,7 @@ Agent Substrate {{< gloss "Checkpoint" >}}checkpoints{{< /gloss >}} an Actor as To avoid losing them, the controller sets `KAGENT_PRE_RESPONSE_TRACE_FLUSH` to `true` on the `kagent` and `codex` runtimes, and the runtime flushes its span buffer before each response completes. The flush waits up to three seconds, which you can change with `KAGENT_TRACE_FLUSH_TIMEOUT_MS` in the Harness `spec.env`. The `claude` runtime gets no such flush, so its spans arrive on its exporter's own schedule and a conversation's last turn can lose them. -This behavior allows a kagent trace to arrive promptly rather than on the exporter's own schedule. To understand what suspension does to an Actor, see [Suspend and resume]({{< link path="substrate-runtime/suspend-and-resume" >}}). +The flush lets a kagent trace arrive promptly rather than on the exporter's own schedule. To understand what suspension does to an Actor, see [Suspend and resume]({{< link path="substrate-runtime/suspend-and-resume" >}}). ## Turn tracing off diff --git a/docs-site/content/kagent/1.x/operations/operational-considerations.md b/docs-site/content/kagent/1.x/operations/operational-considerations.md index b4688899..45b9dc49 100644 --- a/docs-site/content/kagent/1.x/operations/operational-considerations.md +++ b/docs-site/content/kagent/1.x/operations/operational-considerations.md @@ -131,7 +131,7 @@ The controller rewrites internally built Kubernetes URLs to the proxy and sets t ## Scrape controller metrics -The controller serves a Prometheus `/metrics` endpoint, turned off by default. To enable it, grant Prometheus access, and review the metrics that it reports, see [Metrics]({{< link path="observability/metrics" >}}). +The controller serves a Prometheus `/metrics` endpoint, turned off by default. To enable the endpoint, grant Prometheus access to it, and review the metrics that the controller reports, see [Metrics]({{< link path="observability/metrics" >}}). ## Isolate what an agent can do From b4cb7b51bd07cd1961751cb1a24b47c026b846f0 Mon Sep 17 00:00:00 2001 From: Kristin Brown Date: Fri, 25 Sep 2026 10:12:07 -0400 Subject: [PATCH 5/5] Address review comments on 1.0 observability topics - OTel stack: run Loki on filesystem storage instead of MinIO, whose pinned image no longer pulls anonymously - Replace the Loki audit query with Agent Substrate Actor state changes; kagent puts prompts and replies on spans, not log records - Rewrite Audit prompts around span capture, per runtime, at the same path - Name both router services, add atecontroller and ateom-gvisor - Add the four ate_actor_stats metrics; correct otel.metrics.enabled - Agent Substrate telemetry: add Before you begin, expand the Actor log steps, explain the otel.logs settings, move after Metrics - Require the 1.0 CLI and jq in Before you begin; add framing sentences Co-Authored-By: Claude Opus 5.5 (1M context) Signed-off-by: Kristin Brown --- .../kagent/1.x/agents/agent-harness.md | 4 +- .../kagent/1.x/examples/agentgateway.md | 2 +- .../kagent/1.x/observability/_index.md | 2 +- .../kagent/1.x/observability/audit-prompts.md | 190 ++++++++ .../1.x/observability/audit-prompts.txt | 404 ------------------ .../kagent/1.x/observability/launch-ui.md | 2 +- .../observability/lightweight-otel-stack.md | 17 +- .../kagent/1.x/observability/metrics.md | 6 +- .../kagent/1.x/observability/otel-stack.md | 51 ++- .../1.x/observability/substrate-telemetry.md | 90 +++- .../kagent/1.x/observability/tracing.md | 12 +- 11 files changed, 330 insertions(+), 450 deletions(-) create mode 100644 docs-site/content/kagent/1.x/observability/audit-prompts.md delete mode 100644 docs-site/content/kagent/1.x/observability/audit-prompts.txt diff --git a/docs-site/content/kagent/1.x/agents/agent-harness.md b/docs-site/content/kagent/1.x/agents/agent-harness.md index bc4f7f2f..5816272b 100644 --- a/docs-site/content/kagent/1.x/agents/agent-harness.md +++ b/docs-site/content/kagent/1.x/agents/agent-harness.md @@ -132,7 +132,7 @@ The `kagent` and `byo` runtimes take the full set. For more information about wh ## Telemetry content settings -Tracing and audit logging both carry the prompts and replies that an agent exchanges with a model. Two settings in the kagent Helm chart decide whether that content leaves the runtime, and each one reaches a different set of runtimes. Both default to `false`, and both take effect only where tracing or audit logging is already enabled. +Tracing carries the prompts and replies that an agent exchanges with a model, and so does log export on the `claude` runtime. Two settings in the kagent Helm chart decide whether that content leaves the runtime, and each one reaches a different set of runtimes. Both default to `false`, and both take effect only where tracing or log export is already enabled. ```yaml otel: @@ -143,7 +143,7 @@ otel: | Setting | What it includes | Applies to | | ------- | ---------------- | ---------- | -| `otel.captureSensitiveContent` | Prompts, tool details, and assistant replies in the runtime's telemetry. On the `kagent` runtime, the content appears in the spans for each model call. On the `claude` runtime, tool results require tracing, and assistant replies require audit logging. | `kagent`, `codex`, `claude` | +| `otel.captureSensitiveContent` | Prompts, tool details, and assistant replies in the runtime's telemetry. On the `kagent` runtime, the content appears in the spans for each model call. On the `claude` runtime, tool results require tracing, and assistant replies require log export through `otel.logging`. | `kagent`, `codex`, `claude` | | `otel.logging.captureRawApiBodies` | The complete provider API request and response bodies. This setting returns more than `otel.captureSensitiveContent` does, and it takes effect only when `otel.logging.enabled` is `true`. | `claude` | The controller sets `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT` in every compiled runtime from `otel.captureSensitiveContent`, so setting that variable in the Harness `spec.env` field has no effect. diff --git a/docs-site/content/kagent/1.x/examples/agentgateway.md b/docs-site/content/kagent/1.x/examples/agentgateway.md index f128ddf2..4855df8e 100644 --- a/docs-site/content/kagent/1.x/examples/agentgateway.md +++ b/docs-site/content/kagent/1.x/examples/agentgateway.md @@ -386,6 +386,6 @@ An `AgentgatewayPolicy` holds the same `promptGuard` configuration, and one poli {{< cards >}} {{< card link=`{{< link path="setup/model-providers/byo-agentgateway" >}}` title="agentgateway model routing" subtitle="Review the Gateway, AgentgatewayModel, and ModelConfig that carry an agent's model traffic." >}} - + {{< card link=`{{< link path="observability/audit-prompts" >}}` title="Audit prompts" subtitle="Review where each runtime records the prompts and replies that your agents exchange with a model, and how to capture them for an audit." >}} {{< card link="https://agentgateway.dev/docs/kubernetes/latest/documentation/llm/guardrails/regex/#block-requests-with-pii" title="agentgateway regex guardrails" subtitle="Read the full set of regex guards, masking rules, and built-in patterns in the agentgateway documentation." >}} {{< /cards >}} diff --git a/docs-site/content/kagent/1.x/observability/_index.md b/docs-site/content/kagent/1.x/observability/_index.md index bacd4f29..3b61a2d8 100644 --- a/docs-site/content/kagent/1.x/observability/_index.md +++ b/docs-site/content/kagent/1.x/observability/_index.md @@ -1,6 +1,6 @@ --- title: Observability -description: Open the kagent UI, set up an OpenTelemetry stack, and review the traces, metrics, and audit events that kagent and Agent Substrate report. +description: Open the kagent UI, set up an OpenTelemetry stack, and review the traces, metrics, and logs that kagent and Agent Substrate report. weight: 70 author: kagent.dev --- \ No newline at end of file diff --git a/docs-site/content/kagent/1.x/observability/audit-prompts.md b/docs-site/content/kagent/1.x/observability/audit-prompts.md new file mode 100644 index 00000000..82d97935 --- /dev/null +++ b/docs-site/content/kagent/1.x/observability/audit-prompts.md @@ -0,0 +1,190 @@ +--- +title: Audit prompts +description: Capture the prompts and replies that your agents exchange with a model in your traces, then find them in your tracing backend for security and compliance review. +weight: 50 +author: kagent.dev +--- + +Audit the prompts (inputs) and replies (outputs) that pass between your agents and their models. Security and compliance teams use these records to review how people use your {{< reuse "kagent-docs/snippets/name-product.md" >}} environment. For example, you can confirm that no request sends personally identifiable information (PII) to a model. You can also reconstruct the instructions that an agent received in an earlier conversation. + +## About prompt auditing + +{{< reuse "kagent-docs/snippets/name-product.md" >}} records prompts and replies in traces. When you turn on content capture, the spans for each model call carry the request that the agent sent and the reply that it received. You then find the records in your tracing backend, by agent, by conversation, or by the text that they contain. + +The `kagent` runtime does not write prompts or replies to log records, so a logging backend on its own cannot hold a prompt audit trail. Capture is off by default, because the content can include sensitive user and model data. + +### What each runtime records + +Each runtime records the content on its own instrumentation, so where the content lands differs by runtime. For the available runtimes, see [Choose a runtime]({{< link path="agents/agent-harness#choose-a-runtime" >}}). + +| Runtime | Where the content goes | Settings | +| ------- | ---------------------- | -------- | +| `kagent` | The `generate_content` span of each model call, in two attributes. See [What a record holds](#what-a-record-holds). | `otel.captureSensitiveContent` | +| `codex` | The runtime's own spans. | `otel.captureSensitiveContent` | +| `claude` | Prompts and tool details on spans, and assistant replies in the runtime's own log records. With `otel.logging.captureRawApiBodies`, the log records also carry the complete provider request and response bodies, which is a fuller record than the spans give you. | `otel.captureSensitiveContent`, `otel.logging.captureRawApiBodies` | +| `byo` | Nowhere. The controller sends this runtime no telemetry configuration. | None | + +For each setting, see the agent harness [telemetry content settings]({{< link path="agents/agent-harness#telemetry-content-settings" >}}). + +### What a record holds + +On the `kagent` runtime, each `generate_content` span carries the following two attributes, as JSON. + +| Attribute | What it holds | +| --------- | ------------- | +| `gcp.vertex.agent.llm_request` | The whole request that the runtime sent to the model. | +| `gcp.vertex.agent.llm_response` | The model's reply. On a turn that calls a tool, the reply holds the tool call and its arguments instead of text. | + +The request holds more than the prompts that your team wrote. + +- **The system instruction**, which holds the `systemPrompt` field of your AgentTemplate followed by instructions that the runtime appends. +- **The message history**, including the person's messages, the agent's earlier turns, and tool results. +- **The tools** that the agent offered the model, with their definitions. + +Each model call carries the full history again, so a long conversation repeats its earlier messages in every span. Account for that volume when you set a retention period. + +A payload larger than 32 KiB is cut to a prefix. The attribute then holds a JSON object with `truncated` set to `true`, the `original_size` of the payload in bytes, and the first 32 KiB in `payload_prefix`. Your tracing backend might also limit the size of an attribute, so check its limits before you rely on it for long conversations. + +### Delivery + +Agent Substrate suspends an Actor as soon as a response completes. The controller therefore sets the `kagent` and `codex` runtimes to flush their spans before each response completes, so that the records of a turn reach your backend before the Actor suspends. The `claude` runtime gets no such flush, so the records of a conversation's last turn can be lost. For more information, see [Traces from a suspended Actor]({{< link path="observability/tracing#traces-from-a-suspended-actor" >}}). + +> [!IMPORTANT] +> Traces are a best-effort record. An exporter drops spans without an error when the collector is unreachable or its queue is full, and nothing in the trace shows that a record is missing. Treat captured spans as a review aid, not as a complete or tamper-proof compliance log. + +## Before you begin + +1. [Install kagent]({{< link path="setup/installation" >}}). +2. [Create your first agent]({{< link path="get-started/your-first-agent" >}}), so that you have a Harness and an {{< gloss "AgentTemplate" >}}AgentTemplate{{< /gloss >}} to send requests to. That guide also installs the kagent CLI. The steps on this page need the {{< reuse "kagent-docs/versions/kagent.md" >}} CLI, because earlier CLI versions have no `agent-instance` commands and fail with `unknown command`. To check your version, run `kagent version`. +3. Install [`jq`](https://jqlang.org/download/), to read the AgentInstance ID and revision out of the CLI's JSON output. +4. Set up a tracing backend, and turn on tracing. The [OTel stack]({{< link path="observability/otel-stack" >}}) sends traces to Tempo, and the [Lightweight OTel stack]({{< link path="observability/lightweight-otel-stack" >}}) sends traces to Jaeger. Both guides turn on tracing for you. + +## Turn on content capture + +Turn on content capture in the kagent Helm release, then create an AgentInstance that picks up the new setting. + +1. Save the current revision of your Harness and AgentTemplate pair. A later step uses it to tell when kagent rebuilds the pair with the new setting. The command first waits for any rebuild that is still in progress, such as one from an earlier Helm upgrade, so that it saves a finished revision. + ```bash + for i in $(seq 1 60); do + REVISIONS=$(kubectl get agenttemplate my-first-agent -n kagent \ + -o jsonpath='{.status.harnesses[0].desiredRevision} {.status.harnesses[0].latestSuccessfulRevision}') + [ "${REVISIONS% *}" = "${REVISIONS#* }" ] && break + sleep 5 + done + export OLD_REVISION=${REVISIONS#* } + echo "Current revision: $OLD_REVISION" + ``` + +2. Upgrade the kagent Helm release. The `--reuse-values` flag keeps the tracing settings that you already set. + ```bash + helm upgrade kagent \ + {{< reuse "kagent-docs/snippets/helm-path.md" >}}/{{< reuse "kagent-docs/snippets/helm-kagent.md" >}} \ + --version {{< reuse "kagent-docs/versions/kagent.md" >}} \ + --namespace kagent \ + --reuse-values \ + --set otel.captureSensitiveContent=true + ``` + + For an agent on the `claude` runtime, also set `otel.logging.enabled` to `true`, and send the logs to a backend that stores them, such as Loki in the [OTel stack]({{< link path="observability/otel-stack" >}}). Without log export, the replies of a `claude` agent are not recorded. + +3. Wait for the controller to roll out. + ```bash + kubectl rollout status deployment/kagent-controller -n kagent --timeout=300s + ``` + +4. Wait for kagent to rebuild the pair. The controller rebuilds each pair after the controller restarts, and an AgentInstance that you create before the rebuild finishes starts from the previous revision, without the new setting. The following command prints `Recompiled` when the new revision is ready. + ```bash + for i in $(seq 1 60); do + [ "$(kubectl get agenttemplate my-first-agent -n kagent \ + -o jsonpath='{.status.harnesses[0].latestSuccessfulRevision}')" != "$OLD_REVISION" ] \ + && echo "Recompiled" && break + sleep 5 + done + ``` + If the command finishes without printing `Recompiled`, the upgrade did not change the settings that kagent compiles into the pair. Either the setting was already in place, or the chart did not recognize the key. Helm accepts a key that a chart does not define without an error, so check that you upgraded to version {{< reuse "kagent-docs/versions/kagent.md" >}} of the chart, which uses the key on this page. + +5. Create a new AgentInstance. An AgentInstance keeps the runtime configuration that it was created with, so only a new AgentInstance captures content. + ```bash + kagent create agent-instance --harness my-first-harness --agent-template my-first-agent + ``` + +6. Confirm that the AgentInstance runs the current revision of the pair. If the command prints `Outdated`, the AgentInstance was created from an earlier revision, and does not capture content. Create another AgentInstance, and run the command again. + ```bash + for i in $(seq 1 60); do + REVISIONS=$(kubectl get agenttemplate my-first-agent -n kagent \ + -o jsonpath='{.status.harnesses[0].desiredRevision} {.status.harnesses[0].latestSuccessfulRevision}') + [ "${REVISIONS% *}" = "${REVISIONS#* }" ] && break + sleep 5 + done + INSTANCE_REVISION=$(kagent get agent-instance -o json \ + | jq -r '[.agentInstances[] | select(.agentTemplate.name == "my-first-agent")] | sort_by(.createdAt) | last | .preparedRevision') + [ "$INSTANCE_REVISION" = "${REVISIONS#* }" ] && echo "Current" || echo "Outdated" + ``` + +## Verify the setup + +Send a request that contains a distinctive phrase, then find that phrase in the captured request. + +1. Send a request to the new AgentInstance. + ```bash + export INSTANCE_ID=$(kagent get agent-instance -o json \ + | jq -r '[.agentInstances[] | select(.agentTemplate.name == "my-first-agent")] | sort_by(.createdAt) | last | .id') + kagent invoke --agent-instance $INSTANCE_ID --task "Audit check: what is 2+2?" + ``` + +2. Find the captured request in your tracing backend. + {{< tabs >}} + {{% tab name="Grafana with Tempo" %}} + 1. Forward the Grafana port, and leave the command running. + ```bash + kubectl port-forward -n telemetry svc/kube-prometheus-stack-grafana 3000:80 + ``` + 2. In your browser, open Grafana at [http://localhost:3000](http://localhost:3000), and log in. For the password, see [Explore the telemetry in Grafana]({{< link path="observability/otel-stack#explore-the-telemetry-in-grafana" >}}). + 3. Open **Explore**, select the **Tempo** data source, and select the **TraceQL** query type. + 4. Run the following query, which returns the model calls whose request contains the phrase. + ```text + { span.gcp.vertex.agent.llm_request =~ ".*Audit check.*" } + ``` + 5. Open a trace, and select its `generate_content` span. The **Span Attributes** section shows the two attributes that [What a record holds](#what-a-record-holds) describes. + {{% /tab %}} + {{% tab name="Jaeger" %}} + 1. Forward the Jaeger query port, and leave the command running. + ```bash + kubectl port-forward -n telemetry svc/jaeger 16686:16686 + ``` + 2. In your browser, open Jaeger at [http://localhost:16686](http://localhost:16686). + 3. From the **Service** list, select `my-first-agent-my-first-harness`. From the **Operation** list, select the `generate_content` operation for your model, such as `generate_content gpt-4.1-mini`, and click **Find Traces**. + 4. Open the most recent trace, and expand the `generate_content` span. The **Tags** section shows the two attributes that [What a record holds](#what-a-record-holds) describes. + {{% /tab %}} + {{< /tabs >}} + + If both attributes read `{}`, the AgentInstance started without content capture. Check that the previous section printed `Current`, and create a new AgentInstance if it did not. + +3. To collect every model call of one conversation, search by its conversation ID. Every span of the conversation carries the ID in the `gen_ai.conversation.id` attribute. For the other attributes that you can search by, see [Correlation attributes]({{< link path="observability/tracing#correlation-attributes" >}}). + +> [!CAUTION] +> Anyone who can read your tracing or logging backend can now read the prompts and replies of every agent. Restrict access to the backend, and set a retention period that meets your compliance requirements. + +## Turn off content capture + +Turn off content capture, then create a new AgentInstance so that the change takes effect. + +1. Turn off content capture in the kagent Helm release. Tracing stays on. + ```bash + helm upgrade kagent \ + {{< reuse "kagent-docs/snippets/helm-path.md" >}}/{{< reuse "kagent-docs/snippets/helm-kagent.md" >}} \ + --version {{< reuse "kagent-docs/versions/kagent.md" >}} \ + --namespace kagent --reuse-values \ + --set otel.captureSensitiveContent=false + ``` + +2. Create a new AgentInstance, because an existing Actor keeps the configuration that it started with. The spans of an AgentInstance that still captures content keep carrying it until you delete the AgentInstance. + +3. Delete the captured content from your backend when your retention policy requires it. Turning off capture does not remove the spans that your backend already stores. + +## Next steps + +{{< cards >}} + {{< card link=`{{< link path="observability/tracing" >}}` title="Tracing" subtitle="Read the spans and attributes of an agent request." >}} + {{< card link=`{{< link path="agents/agent-harness#telemetry-content-settings" >}}` title="Telemetry content settings" subtitle="Review how each content setting applies to each runtime." >}} +{{< /cards >}} diff --git a/docs-site/content/kagent/1.x/observability/audit-prompts.txt b/docs-site/content/kagent/1.x/observability/audit-prompts.txt deleted file mode 100644 index a8dd75a6..00000000 --- a/docs-site/content/kagent/1.x/observability/audit-prompts.txt +++ /dev/null @@ -1,404 +0,0 @@ ---- -title: Audit prompts -description: Export the prompts and replies that your agents exchange with a model as OpenTelemetry log events, then query them in a logging backend. -weight: 50 -author: kagent.dev ---- - -Audit every prompt (input) and reply (output) that passes between your agents and their models. Security and compliance teams use these records to review how people use your {{< reuse "kagent-docs/snippets/name-product.md" >}} environment. For example, you can confirm that no request sends personally identifiable information (PII) to a model. You can also reconstruct the instructions that an agent received in an earlier conversation. - -## About prompt auditing - -The agent runtime emits each message as an OpenTelemetry (OTel) log event. You export these events over the OpenTelemetry Protocol (OTLP) to a logging backend or to a security information and event management (SIEM) system. - -### Trace correlation - -The runtime emits each event from inside the model call. Each event records the trace ID and the span ID of the request that produced it. Those IDs let you match an audit record to the trace of the same request. The runtime populates both IDs whether or not you enable tracing, but only an enabled tracing pipeline exports the matching trace. With tracing disabled, a lookup of the trace ID in your tracing backend returns nothing. For more information, see [Tracing]({{< link path="observability/tracing" >}}). - -### Events - -The runtime emits three event names for each model call. The system prompt and the model's reply each produce one event. The message history produces one event for every entry that it holds. - -| Event name | What it holds | -| ---------- | ------------- | -| `gen_ai.system.message` | The system prompt for the request, as one concatenated string. | -| `gen_ai.user.message` | One entry from the request's message history. The entry holds a person's message, an earlier agent turn, or a tool result. | -| `gen_ai.choice` | The model's reply, with the reply content and a `finish_reason`. On a turn that calls a tool, the reply content holds the tool call and its arguments instead of text. | - -An audit returns more than the prompts that your team wrote. A `gen_ai.system.message` body holds the `systemPrompt` field of your AgentTemplate followed by instructions that the runtime appends, which name the agent and repeat its description. Tool traffic is included as well, because a tool call reaches the log with its arguments, and the tool's output returns as a `gen_ai.user.message` that holds the tool response. - -> [!NOTE] -> The runtime labels every history entry as `gen_ai.user.message`, including the agent's own earlier turns and tool results. The `content.role` field in the event body names the speaker. To select only the messages that a person sent, filter on `content.role` instead of on the event name. Each turn also re-emits the full history, so a long conversation produces repeated events. Account for that volume when you set a retention period. - -### Configuration - -Audit output comes from two places. The kagent Helm chart decides whether the runtime exports events and where it sends them. The {{< gloss "Harness" >}}Harness{{< /gloss >}} decides whether those events carry message content. - -| Setting | Where you set it | What it does | -| ------- | ---------------- | ------------ | -| `otel.logging.enabled` | kagent Helm chart | Installs the log exporter in the agent runtime. The default value is `false`, and the runtime then emits no audit events, regardless of the other settings. | -| `otel.logging.exporter.otlp.endpoint` | kagent Helm chart | The address that the runtime exports events to. Set it to the address of your collector. | -| `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT` | `Harness.spec.env` | Includes message content in the events. The default value is `false`, and the runtime then replaces each message body with ``. The event metadata and the trace IDs remain. Those fields still record which agent handled a request, and when. | - -> [!IMPORTANT] -> The chart's `otel.captureSensitiveContent` setting does not reach the `kagent` runtime. It applies to the Claude and Codex runtimes only. To include message content in an audit of a `kagent` agent, set `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT` on the Harness, as shown in the following steps. - -> [!NOTE] -> The controller compiles the chart's logging settings into every runtime revision, and its values override a Harness `spec.env` entry for the same variable. The variables it owns are `OTEL_LOGGING_ENABLED`, `OTEL_EXPORTER_OTLP_LOGS_ENDPOINT`, and `OTEL_EXPORTER_OTLP_LOGS_PROTOCOL`, together with their tracing equivalents and the endpoint and protocol variables that cover both signals. If `otel.logging.enabled` is `false`, the controller compiles no logging variable, and a `spec.env` entry takes effect as written. - -### Runtime support - -Only the `kagent` runtime emits these events. The runtime emits them from the model call itself, not from a provider-specific instrumentation library. Auditing therefore covers every model provider that the `kagent` runtime supports. For the available runtimes, see [Choose a runtime]({{< link path="agents/agent-harness#choose-a-runtime" >}}). - -## Before you begin - -1. [Install kagent]({{< link path="setup/installation" >}}). -2. [Create your first agent]({{< link path="get-started/your-first-agent" >}}), so that you have a Harness and an {{< gloss "AgentTemplate" >}}AgentTemplate{{< /gloss >}} to configure. - -## Install a collector and a logging backend - -Set up the path that audit events take from the agent runtime to a logging backend. The runtime exports to an OpenTelemetry collector, and the collector forwards the events to the backend. These steps install Grafana Loki as that backend, because Loki supports the queries that this guide runs later. Datadog, Splunk, and other OTLP-compatible systems work in the same way. - -Export to a collector rather than directly to the backend. The collector holds the rules for which content and metadata leave your cluster. Audit events carry prompt text, so those rules matter more than they do for other telemetry. The collector also lets you change the rules without creating a new AgentInstance. - -1. Add the OpenTelemetry Helm repository. - ```bash - helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts - helm repo update - ``` - -2. Install Loki in single-binary mode. The values file disables the two Loki memcached caches, because the chart requests roughly 10 GB of memory for them by default and a single-node cluster cannot schedule that request. - ```yaml - helm upgrade --install loki loki \ - --repo https://grafana.github.io/helm-charts \ - --version {{< reuse "kagent-docs/versions/loki.md" >}} \ - --namespace telemetry \ - --create-namespace \ - --values - < otel-collector-audit.yaml <}} \ - --values otel-collector-audit.yaml - ``` - -6. Verify that the collector is running. - ```bash - kubectl get pods -n telemetry -l app.kubernetes.io/name=opentelemetry-collector - ``` - Example output: - ```console - NAME READY STATUS RESTARTS AGE - opentelemetry-collector-audit-xxxxxxxxx-xxxxx 1/1 Running 0 30s - ``` - -## Turn on audit logging - -Turning on auditing takes two changes. The chart setting installs the log exporter in every agent runtime that the controller starts, and the Harness setting decides whether the exported events carry message content. A Harness applies to every AgentTemplate that it admits, so auditing covers an entire Harness rather than a single agent. - -1. Upgrade kagent to export audit events to the collector. The controller compiles these settings into every runtime revision that it builds from now on. - ```bash - helm upgrade kagent \ - {{< reuse "kagent-docs/snippets/helm-path.md" >}}/{{< reuse "kagent-docs/snippets/helm-kagent.md" >}} \ - --version {{< reuse "kagent-docs/versions/kagent.md" >}} \ - --namespace kagent --reuse-values \ - --set otel.logging.enabled=true \ - --set otel.logging.exporter.otlp.endpoint=http://opentelemetry-collector-audit.telemetry.svc.cluster.local:4317 - ``` - - To export over HTTP instead of gRPC, add `--set otel.logging.exporter.otlp.protocol=http/protobuf` and use port `4318`. - -2. Add the message content variable to the Harness. Keep the rest of its configuration unchanged. - ```yaml - kubectl apply -f - <}} - env: - - name: OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT - value: "true" - substrate: - workerPoolRef: - name: kagent-default - snapshotPolicy: - location: gs:///kagent/ - allowedAgentTemplates: - selector: - matchLabels: - kagent.dev/harness: my-first-harness - EOF - ``` - - The `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT` variable includes message content in the events. If you omit it, each event body reads ``, and the runtime exports only the metadata and the trace IDs. Those fields still record which agent handled a request, and when. Set the export destination through the chart rather than here, because the controller's compiled values override a `spec.env` entry for a variable that it owns. For every other field that a Harness takes, see [Agent harness]({{< link path="agents/agent-harness" >}}). - -3. Confirm that kagent compiled a new {{< gloss "Revision" >}}revision{{< /gloss >}} for the edited Harness. The Harness is current when `latestSuccessfulRevision` matches `desiredRevision`. - ```bash - kubectl get agenttemplate my-first-agent -n kagent \ - -o jsonpath='{range .status.harnesses[*]}{.harness}{"\t"}{.desiredRevision}{"\t"}{.latestSuccessfulRevision}{"\n"}{end}' - ``` - - Example output: - ```console - my-first-harness 4b8e1d3f5a7c9e2b0d4f6a8c1e3b5d7f9a2c4e6b8d0f2a4c6e8b0d2f4a6c8e0b 4b8e1d3f5a7c9e2b0d4f6a8c1e3b5d7f9a2c4e6b8d0f2a4c6e8b0d2f4a6c8e0b - ``` - -4. Create a new AgentInstance. An {{< gloss "AgentInstance" >}}AgentInstance{{< /gloss >}} pins the revision that it was created from, so an existing instance continues to run without auditing. - ```bash - kagent create agent-instance --harness my-first-harness --agent-template my-first-agent - ``` - -## Verify the setup - -1. Send a request to the new AgentInstance to produce audit events. - ```bash - export INSTANCE_ID=$(kagent get agent-instance -o json \ - | jq -r '[.agentInstances[] | select(.agentTemplate.name == "my-first-agent")] | sort_by(.createdAt) | last | .id') - kagent invoke --agent-instance $INSTANCE_ID --task "What is 2+2?" - ``` - -2. Check that the collector received the events. The collector logs its own metrics to the same stream, so filter the output for the audit records. - ```bash - kubectl -n telemetry logs -l app.kubernetes.io/name=opentelemetry-collector --tail=200 \ - | grep -B 5 -A 4 "EventName: gen_ai" - ``` - Example output: - ```console - LogRecord #1 - ObservedTimestamp: 2026-09-03 19:26:18.48324493 +0000 UTC - Timestamp: 1970-01-01 00:00:00 +0000 UTC - SeverityText: - SeverityNumber: Unspecified(0) - EventName: gen_ai.user.message - Body: Map({"content":{"parts":[{"text":"What is 2+2?"}],"role":"user"}}) - Trace ID: 3d34d2f1b74f30a5cce0d5ed8571e928 - Span ID: 12671255711f5511 - Flags: 1 - ``` - - The runtime leaves the `Timestamp` field unset, so every record reports `1970-01-01 00:00:00`. Read `ObservedTimestamp` instead, which records when the collector received the event. - - > [!NOTE] - > The runtime buffers audit events and exports them in batches, and Agent Substrate suspends an Actor as soon as its response completes. A short conversation can therefore finish before the runtime exports its events, and this command then returns nothing. Send another request to the AgentInstance and check again. - -3. Forward the Loki query port. Leave the command running. - ```bash - kubectl port-forward -n telemetry svc/loki 3100:3100 - ``` - -4. Query the events for the agent's service. The runtime builds the service name from the AgentTemplate name and the Harness name. For example, `my-first-agent` on `my-first-harness` reports as `my-first-agent-my-first-harness`. - ```bash - curl -s -G 'http://localhost:3100/loki/api/v1/query_range' \ - --data-urlencode 'query={service_name="my-first-agent-my-first-harness"}' \ - --data-urlencode "start=$(( $(date +%s) - 3600 ))000000000" \ - --data-urlencode "end=$(date +%s)000000000" | jq - ``` - - Each entry holds the message content in the log line. The `stream` object holds the agent identity in the `service_name` and `service_namespace` labels, and holds the trace IDs as structured metadata. Loki does not record the event name, so the response carries no `event_name` field, and every event from one request shares a single stream. Example output: - ```json - { - "status": "success", - "data": { - "resultType": "streams", - "result": [ - { - "stream": { - "service_name": "my-first-agent-my-first-harness", - "service_namespace": "kagent", - "scope_name": "gcp.vertex.agent", - "trace_id": "3d34d2f1b74f30a5cce0d5ed8571e928", - "span_id": "12671255711f5511", - "flags": "1" - }, - "values": [ - [ - "1788463578483244930", - "{\"content\":{\"parts\":[{\"text\":\"What is 2+2?\"}],\"role\":\"user\"}}" - ], - [ - "1788463578483063303", - "{\"content\":\"You are a concise, helpful assistant. ...\"}" - ] - ] - } - ] - } - } - ``` - -## Refine audit queries - -An audit usually needs a narrower set of events than the full message history of one agent. Loki does not index the event name, so each of the following examples selects an event type by a field in the event body instead. The examples use the Loki query language. Adapt each example to the query language of your own backend. - -- Return only the model's replies. Only a `gen_ai.choice` event carries a `finish_reason` field, so that field selects the replies. - ```bash - curl -s -G 'http://localhost:3100/loki/api/v1/query_range' \ - --data-urlencode 'query={service_namespace="kagent"} | json reason="finish_reason" | reason != ""' \ - --data-urlencode "start=$(( $(date +%s) - 3600 ))000000000" \ - --data-urlencode "end=$(date +%s)000000000" | jq - ``` - -- Return only the messages that a person sent, and exclude the agent's replayed history. The filter reads `content.role` from the event body, because only a person's message sets that field to `user`. - ```bash - curl -s -G 'http://localhost:3100/loki/api/v1/query_range' \ - --data-urlencode 'query={service_namespace="kagent"} | json role="content.role" | role="user"' \ - --data-urlencode "start=$(( $(date +%s) - 3600 ))000000000" \ - --data-urlencode "end=$(date +%s)000000000" | jq - ``` - -- Return every message from every agent that contains a given string. For example, this query checks whether a request sent a credential to a model. - ```bash - curl -s -G 'http://localhost:3100/loki/api/v1/query_range' \ - --data-urlencode 'query={service_namespace="kagent"} |= "password"' \ - --data-urlencode "start=$(( $(date +%s) - 3600 ))000000000" \ - --data-urlencode "end=$(date +%s)000000000" | jq - ``` - -To follow a request from its audit records into its trace, take the `trace_id` from any entry and look it up in your tracing backend. The lookup returns a trace only when [tracing]({{< link path="observability/tracing" >}}) is also enabled. With tracing disabled, the record still carries a trace ID, but no pipeline exported the trace that the ID names. - -## Turn off audit logging - -1. Turn the log exporter off again. - ```bash - helm upgrade kagent \ - {{< reuse "kagent-docs/snippets/helm-path.md" >}}/{{< reuse "kagent-docs/snippets/helm-kagent.md" >}} \ - --version {{< reuse "kagent-docs/versions/kagent.md" >}} \ - --namespace kagent --reuse-values \ - --set otel.logging.enabled=false - ``` - -2. Remove the `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT` variable from the `spec.env` field of the Harness. - -3. Create a new AgentInstance, so that its Actor starts without auditing. - -4. Remove the collector and the logging backend. - ```bash - helm uninstall opentelemetry-collector-audit -n telemetry - helm uninstall loki -n telemetry - kubectl delete namespace telemetry - ``` - -## Next steps - -{{< cards >}} - {{< card link=`{{< link path="observability/tracing" >}}` title="Tracing" subtitle="Follow one request from the controller through to the Actor that ran your agent." >}} - {{< card link=`{{< link path="agents/agent-harness" >}}` title="Agent harness" subtitle="Review every field that a Harness takes, including the environment that its runtime receives." >}} -{{< /cards >}} diff --git a/docs-site/content/kagent/1.x/observability/launch-ui.md b/docs-site/content/kagent/1.x/observability/launch-ui.md index 16a0f4b8..827bee09 100644 --- a/docs-site/content/kagent/1.x/observability/launch-ui.md +++ b/docs-site/content/kagent/1.x/observability/launch-ui.md @@ -187,5 +187,5 @@ The `parentRefs` field is required when `enabled` is `true`, and it must referen {{< cards >}} {{< card link=`{{< link path="get-started/your-first-agent" >}}` title="Create your first agent" subtitle="Pair an AgentTemplate with a Harness, then hold a conversation with the result." >}} {{< card link=`{{< link path="observability/tracing" >}}` title="Tracing" subtitle="Follow one agent request from the controller through to the Actor that ran it." >}} - + {{< card link=`{{< link path="observability/audit-prompts" >}}` title="Audit prompts" subtitle="Capture every prompt and reply in your traces for security and compliance review." >}} {{< /cards >}} diff --git a/docs-site/content/kagent/1.x/observability/lightweight-otel-stack.md b/docs-site/content/kagent/1.x/observability/lightweight-otel-stack.md index 0398d33b..80cf7669 100644 --- a/docs-site/content/kagent/1.x/observability/lightweight-otel-stack.md +++ b/docs-site/content/kagent/1.x/observability/lightweight-otel-stack.md @@ -16,7 +16,7 @@ This stack follows the setup that Agent Substrate uses for its own local testing ## About the stack -The stack has no logging backend. The collector prints the logs that it receives to its own output, which is enough to confirm that logs arrive. To store and query audit events, use the [OTel stack]({{< link path="observability/otel-stack" >}}), which adds Loki. +The stack has no logging backend. The collector prints the logs that it receives to its own output, which is enough to confirm that logs arrive. To store and query logs, such as the Actor state changes that Agent Substrate records, use the [OTel stack]({{< link path="observability/otel-stack" >}}), which adds Loki.

```mermaid @@ -46,7 +46,8 @@ For what kagent and Agent Substrate each send, see [About the stack]({{< link pa ## Before you begin 1. [Install kagent]({{< link path="setup/installation" >}}). -2. [Create your first agent]({{< link path="get-started/your-first-agent" >}}), so that you have a Harness and an AgentTemplate to send requests to. +2. [Create your first agent]({{< link path="get-started/your-first-agent" >}}), so that you have a Harness and an AgentTemplate to send requests to. That guide also installs the kagent CLI. The steps on this page need the {{< reuse "kagent-docs/versions/kagent.md" >}} CLI, because earlier CLI versions have no `agent-instance` commands and fail with `unknown command`. To check your version, run `kagent version`. +3. Install [`jq`](https://jqlang.org/download/), to read the AgentInstance ID and revision out of the CLI's JSON output. ## Install Jaeger @@ -114,6 +115,8 @@ Install the [Prometheus](https://github.com/prometheus-community/helm-charts/tre The kagent controller serves its metrics over HTTPS with a self-signed certificate, and it accepts a scrape only from a ServiceAccount that holds its metrics reader role. The `kagent-controller` job therefore skips certificate verification and sends the Prometheus ServiceAccount token. + The `kagent-controller` target reads `DOWN` until you turn on the controller metrics in [Send kagent telemetry to the collector](#send-kagent-telemetry-to-the-collector), because the controller metrics Service does not exist before then. + 2. Verify that Prometheus is running. ```bash kubectl get pods -n telemetry -l app.kubernetes.io/name=prometheus @@ -179,6 +182,8 @@ EOF ## Send kagent telemetry to the collector +Turn on the kagent trace and log exporters, and point both at the collector. Also turn on the controller's metrics endpoint, and allow Prometheus to scrape it. + 1. Save the current revision of your Harness and AgentTemplate pair. A later step uses it to tell when kagent recompiles the pair with the new settings. The command first waits for any recompile that is still in progress, such as one from an earlier Helm upgrade, so that it saves a finished revision. ```bash for i in $(seq 1 60); do @@ -241,7 +246,7 @@ EOF sleep 5 done ``` - If the command finishes without printing `Recompiled`, the upgrade did not change the pair, for example because the settings were already in place. + If the command finishes without printing `Recompiled`, the upgrade did not change the settings that kagent compiles into the pair. Either the settings were already in place, or the chart did not recognize the `otel` keys. Helm accepts a key that a chart does not define without an error, so check that you upgraded to version {{< reuse "kagent-docs/versions/kagent.md" >}} of the chart, which uses the keys on this page. ## Send Agent Substrate telemetry to the collector @@ -260,6 +265,8 @@ helm upgrade substrate \ ## Send a request +Create an AgentInstance that picks up the new telemetry settings, and send it a request to produce traces and metrics. + 1. Create a new AgentInstance. An AgentInstance keeps the runtime configuration that it was created with, so only a new AgentInstance exports traces. ```bash kagent create agent-instance --harness my-first-harness --agent-template my-first-agent @@ -287,13 +294,15 @@ helm upgrade substrate \ ## Review the telemetry +Check each signal in turn: traces in Jaeger, metrics in Prometheus, and logs in the collector's own output. + 1. Review the traces in Jaeger. 1. Forward the Jaeger query port, and leave the command running. ```bash kubectl port-forward -n telemetry svc/jaeger 16686:16686 ``` 2. In your browser, open Jaeger at [http://localhost:16686](http://localhost:16686). - 3. From the **Service** list, select `my-first-agent-my-first-harness`, and click **Find Traces**. To see Agent Substrate's own work, select `ateapi`, `atenet-router`, or `atelet` instead. + 3. From the **Service** list, select `my-first-agent-my-first-harness`, and click **Find Traces**. To see Agent Substrate's own work, select one of its services instead: `ateapi`, `atenet-router`, `atelet`, `atecontroller`, or `ateom-gvisor`. For what each service reports, see [Agent Substrate traces]({{< link path="observability/tracing#agent-substrate-traces" >}}). 2. Review the metrics in Prometheus. 1. Forward the Prometheus port, and leave the command running. diff --git a/docs-site/content/kagent/1.x/observability/metrics.md b/docs-site/content/kagent/1.x/observability/metrics.md index d5d96b01..99a24b93 100644 --- a/docs-site/content/kagent/1.x/observability/metrics.md +++ b/docs-site/content/kagent/1.x/observability/metrics.md @@ -58,7 +58,7 @@ helm upgrade substrate \ --set otel.endpoint=http://otel-collector.telemetry.svc.cluster.local:4317 ``` -To send metrics somewhere other than traces and logs, set `otel.metrics.endpoint` instead. To stop exporting metrics while keeping the other signals, set `otel.metrics.enabled` to `false`. Agent Substrate exports a batch of metrics every 60 seconds, so a new installation shows its first values about a minute after you set the endpoint. +To send metrics somewhere other than traces and logs, set `otel.metrics.endpoint` instead. To stop exporting metrics while keeping the other signals, leave `otel.endpoint` empty, and set `otel.traces.endpoint` and `otel.logs.endpoint` instead. Setting `otel.metrics.enabled` to `false` is not enough in version {{< reuse "kagent-docs/versions/agent-substrate.md" >}}, because only the router's agentgateway reads it, and the other Agent Substrate components keep exporting. Agent Substrate exports a batch of metrics every 60 seconds, so a new installation shows its first values about a minute after you set the endpoint. ### Metric reference @@ -76,6 +76,10 @@ The collector's Prometheus exporter converts each OpenTelemetry metric name to a | `ate_actor_checkpoint_duration_seconds` | `atelet` | Histogram | Time for each phase of writing a snapshot when an Actor suspends. | | `atelet_snapshot_size_bytes` | `atelet` | Histogram | Uncompressed size of each snapshot. | | `ate_imagecache_requests_total` | `atelet` | Counter | Lookups in the node-local image cache, by the `ate_imagecache_outcome` label. A cache miss pays for pulling and unpacking the image, so a low hit ratio predicts slow resumes. | +| `ate_actor_stats_cpu_time_seconds_total` | `atelet` | Counter | CPU time of the Actors that run on a node, by template. Use `rate()` to find the templates that use a node's CPU. | +| `ate_actor_stats_memory_usage_bytes` | `atelet` | Gauge | Memory of the Actors that run on a node, by template, including page cache that the node can reclaim. | +| `ate_actor_stats_memory_working_set_bytes` | `atelet` | Gauge | Working set memory of the Actors that run on a node, by template, without reclaimable page cache. Compare this value with a memory limit. | +| `ate_actor_stats_sampled_actors` | `atelet` | Gauge | Actors on a node that have a current resource measurement. Divide the other `ate_actor_stats_*` metrics by this value to get a per-Actor average for a template. | | `ate_actor_crashes_total` | `ateapi` | Counter | Actors that moved to the terminal crashed state, by failure reason. The metric appears only after the first crash. | | `rpc_server_call_duration_seconds`, `rpc_client_call_duration_seconds` | `ateapi`, `atelet` | Histogram | Latency, rate, and errors of the gRPC calls between Agent Substrate components. | diff --git a/docs-site/content/kagent/1.x/observability/otel-stack.md b/docs-site/content/kagent/1.x/observability/otel-stack.md index 8e19a4c6..87da2d32 100644 --- a/docs-site/content/kagent/1.x/observability/otel-stack.md +++ b/docs-site/content/kagent/1.x/observability/otel-stack.md @@ -52,20 +52,21 @@ The following table lists what each source sends. For the meaning of each signal | Source | Traces | Logs | Metrics | | ------ | ------ | ---- | ------- | -| kagent controller and agent runtimes | One trace per agent request, across the controller, the Agent Substrate proxy, and the agent runtime. | Audit events for the prompts and replies that agents exchange with a model. | Reconciliation, work queue, and gRPC API metrics, scraped from the controller's `/metrics` endpoint. | -| Agent Substrate | Separate traces for its own work, such as routing a request to a Worker and resuming an Actor. | Actor lifecycle events and the router access log. | Actor lifecycle, scheduling, snapshot, image cache, and WorkerPool capacity metrics. | +| kagent controller and agent runtimes | One trace per agent request, across the controller, the Agent Substrate proxy, and the agent runtime. | Log records from the agent runtimes. Prompts and replies are not among them on the `kagent` runtime, which puts them on spans instead. For how to capture them, see [Audit prompts]({{< link path="observability/audit-prompts" >}}). | Reconciliation, work queue, and gRPC API metrics, scraped from the controller's `/metrics` endpoint. | +| Agent Substrate | Separate traces for its own work, such as routing a request to a Worker and resuming an Actor. | A record of every Actor state change, and the router access log. | Actor lifecycle, scheduling, snapshot, image cache, and WorkerPool capacity metrics. | Agent Substrate traces do not join the trace of the agent request that caused them. To see why a request was slow to start, look up the Agent Substrate trace from the same time window. ## Before you begin 1. [Install kagent]({{< link path="setup/installation" >}}). -2. [Create your first agent]({{< link path="get-started/your-first-agent" >}}), so that you have a Harness and an AgentTemplate to send requests to. -3. Make sure that your cluster has about 1.5 GB of memory available in addition to kagent. On a kind cluster, the memory limit is the memory that you give Docker. +2. [Create your first agent]({{< link path="get-started/your-first-agent" >}}), so that you have a Harness and an AgentTemplate to send requests to. That guide also installs the kagent CLI. The steps on this page need the {{< reuse "kagent-docs/versions/kagent.md" >}} CLI, because earlier CLI versions have no `agent-instance` commands and fail with `unknown command`. To check your version, run `kagent version`. +3. Install [`jq`](https://jqlang.org/download/), to read the AgentInstance ID and revision out of the CLI's JSON output. +4. Make sure that your cluster has about 1.5 GB of memory available in addition to kagent. On a kind cluster, the memory limit is the memory that you give Docker. ## Install Tempo and Loki -Install the backends that store traces and logs. Both run as a single replica without persistent storage, which suits evaluation. For production, follow the Grafana guidance for [Tempo](https://grafana.com/docs/tempo/latest/setup/helm-chart/) and [Loki](https://grafana.com/docs/loki/latest/setup/install/helm/). +Install the backends that store traces and logs. Both run as a single replica, which suits evaluation. Tempo keeps its traces in memory, and Loki keeps its logs on a small persistent volume, so your cluster needs a default StorageClass, which a kind cluster has. For production, follow the Grafana guidance for [Tempo](https://grafana.com/docs/tempo/latest/setup/helm-chart/) and [Loki](https://grafana.com/docs/loki/latest/setup/install/helm/). 1. Install Tempo, with an OTLP receiver for traces. ```bash @@ -87,7 +88,7 @@ Install the backends that store traces and logs. Both run as a single replica wi EOF ``` -2. Install Loki in single-binary mode. The values file disables the two Loki memcached caches, because the chart requests roughly 10 GB of memory for them by default and a single-node cluster cannot schedule that request. +2. Install Loki in single-binary mode, with its logs on the local filesystem. The values file disables the two Loki memcached caches, because the chart requests roughly 10 GB of memory for them by default and a single-node cluster cannot schedule that request. ```bash helm upgrade --install loki loki \ --repo https://grafana.github.io/helm-charts \ @@ -101,16 +102,21 @@ Install the backends that store traces and logs. Both run as a single replica wi configs: - from: 2024-04-01 store: tsdb - object_store: s3 + object_store: filesystem schema: v13 index: prefix: loki_index_ period: 24h auth_enabled: false + storage: + type: filesystem singleBinary: replicas: 1 + persistence: + enabled: true + size: 2Gi minio: - enabled: true + enabled: false gateway: enabled: false test: @@ -165,10 +171,9 @@ Install the backends that store traces and logs. Both run as a single replica wi ``` Example output: ```console - NAME READY STATUS RESTARTS AGE - loki-0 2/2 Running 0 90s - loki-minio-0 1/1 Running 0 90s - tempo-0 1/1 Running 0 2m + NAME READY STATUS RESTARTS AGE + loki-0 2/2 Running 0 90s + tempo-0 1/1 Running 0 2m ``` ## Install Prometheus and Grafana @@ -365,7 +370,7 @@ Turn on the kagent trace and log exporters, and point both at the collector. Als | Setting | Description | | ------- | ----------- | | `otel.tracing` | Exports traces from the controller and from the agent runtimes that it starts. For each field, see [Tracing]({{< link path="observability/tracing#enable-tracing" >}}). | - | `otel.logging` | Exports audit events from the agent runtimes. By default, kagent withholds message content, so each event body reads ``. | + | `otel.logging` | Exports log records from the agent runtimes. The `kagent` runtime never puts prompts or replies in a log record. For where that content goes, see [Audit prompts]({{< link path="observability/audit-prompts" >}}). | | `controller.metrics.enabled` | Serves the controller's Prometheus metrics on port `8443` over HTTPS. | | `controller.metrics.serviceMonitor` | Creates a ServiceMonitor for the metrics endpoint. The `prometheusServiceAccount` setting binds the metrics reader role to the Prometheus ServiceAccount, which authorizes the scrape. | @@ -383,7 +388,7 @@ Turn on the kagent trace and log exporters, and point both at the collector. Als sleep 5 done ``` - If the command finishes without printing `Recompiled`, the upgrade did not change the pair, for example because the settings were already in place. + If the command finishes without printing `Recompiled`, the upgrade did not change the settings that kagent compiles into the pair. Either the settings were already in place, or the chart did not recognize the `otel` keys. Helm accepts a key that a chart does not define without an error, so check that you upgraded to version {{< reuse "kagent-docs/versions/kagent.md" >}} of the chart, which uses the keys on this page. ## Send Agent Substrate telemetry to the collector @@ -410,7 +415,9 @@ Point Agent Substrate at the collector. A single `otel.endpoint` setting turns o ## Send a request -1. Create a new AgentInstance. An AgentInstance keeps the runtime configuration that it was created with, so only a new AgentInstance exports traces and audit events. +Create an AgentInstance that picks up the new telemetry settings, and send it a few requests to produce traces, logs, and metrics. + +1. Create a new AgentInstance. An AgentInstance keeps the runtime configuration that it was created with, so only a new AgentInstance exports telemetry. ```bash kagent create agent-instance --harness my-first-harness --agent-template my-first-agent ``` @@ -438,6 +445,8 @@ Point Agent Substrate at the collector. A single `otel.endpoint` setting turns o ## Explore the telemetry in Grafana +Log in to Grafana, and query each backend from the **Explore** view. + 1. Get the Grafana password for the `admin` user. ```bash kubectl get secret -n telemetry kube-prometheus-stack-grafana \ @@ -456,17 +465,19 @@ Point Agent Substrate at the collector. A single `otel.endpoint` setting turns o {{% tab name="Traces" %}} Select the **Tempo** data source, then select the **Search** query type. From the **Service Name** list, select `my-first-agent-my-first-harness`, the service that the AgentTemplate and Harness pair reports as, and run the query. Open a trace to see the controller, proxy, and agent runtime spans of one request. - To see Agent Substrate's own work, select `ateapi`, `atenet-router`, or `atelet` from the **Service Name** list instead. + To see Agent Substrate's own work, select one of its services from the **Service Name** list instead: `ateapi`, `atenet-router`, `atelet`, `atecontroller`, or `ateom-gvisor`. For what each service reports, see [Agent Substrate traces]({{< link path="observability/tracing#agent-substrate-traces" >}}). {{% /tab %}} {{% tab name="Logs" %}} - Select the **Loki** data source, and run the following query to show the audit events of your agent. + Select the **Loki** data source, and run the following query to show the state changes of your agent's Actor. Agent Substrate names the Actor `ai-` followed by the AgentInstance ID, so replace `` with the value of `$INSTANCE_ID` from the previous section. ```text - {service_name="my-first-agent-my-first-harness"} + {service_name="ateapi"} | ate_actor_name="ai-" ``` - The runtime exports audit events in batches, and Agent Substrate suspends the Actor as soon as a response completes. The events of the most recent request can therefore appear only after you send the next one. + Each record carries the new state of the Actor, such as `resuming`, `running`, or `suspended`. For the meaning of each record, see [Actor state changes]({{< link path="observability/substrate-telemetry#actor-state-changes" >}}). + + To see the router access log, with an entry for every request that the router forwards, query `{service_name="agentgateway"}` instead. - To see Actor lifecycle events from Agent Substrate, query `{service_name="ateapi"}` instead. + Loki does not hold the prompts and replies of the `kagent` runtime, because that runtime puts them on spans rather than in log records. To audit them, see [Audit prompts]({{< link path="observability/audit-prompts" >}}). {{% /tab %}} {{% tab name="Metrics" %}} Select the **Prometheus** data source, and run a query. For example, the following query returns the number of ready Workers in each WorkerPool. diff --git a/docs-site/content/kagent/1.x/observability/substrate-telemetry.md b/docs-site/content/kagent/1.x/observability/substrate-telemetry.md index de3a0caa..6fcc2d49 100644 --- a/docs-site/content/kagent/1.x/observability/substrate-telemetry.md +++ b/docs-site/content/kagent/1.x/observability/substrate-telemetry.md @@ -1,7 +1,7 @@ --- title: Agent Substrate telemetry description: Follow an agent's Actor across suspend and resume by using the Actor identity that Agent Substrate adds to its logs, traces, and metrics. -weight: 35 +weight: 45 author: kagent.dev --- @@ -14,12 +14,14 @@ Agent Substrate reports telemetry from its own components, separately from the t | Signal | Where it goes | Details | | ------ | ------------- | ------- | | Actor logs | The stdout of the Worker pod that runs the Actor | The Actor's own output, and records for each suspend and resume. See [Read an Actor's logs](#read-an-actors-logs). | -| Component logs | Over OTLP, when an OTLP endpoint is set | Logs from the Agent Substrate components, including an access log entry for every request that the router receives. | +| Component logs | The stdout of each component, and over OTLP when an OTLP endpoint is set | Logs from the Agent Substrate components, including a record of every Actor state change and an access log entry for every request that the router receives. See [Actor state changes](#actor-state-changes). | | Traces | Over OTLP, when an OTLP endpoint is set | Scheduling, resume, and suspend work. These traces are separate from the agent request trace. See [Agent Substrate traces]({{< link path="observability/tracing#agent-substrate-traces" >}}). | | Metrics | Over OTLP, when an OTLP endpoint is set | Worker capacity, scheduling, and snapshot timing. See [Agent Substrate metrics]({{< link path="observability/metrics#agent-substrate-metrics" >}}). | The Agent Substrate Helm chart sets one OTLP endpoint for traces, metrics, and logs through `otel.endpoint`, and samples traces at a ratio of `0.01` by default through `otel.traces.samplingRatio`. The [OTel stack]({{< link path="observability/otel-stack#send-agent-substrate-telemetry-to-the-collector" >}}) and [Lightweight OTel stack]({{< link path="observability/lightweight-otel-stack" >}}) guides set both. +The router access log and the Actor state change records are the two sources of Agent Substrate logs over OTLP, and the `otel.logs` settings control both together. To send them to a different backend than traces and metrics, set `otel.logs.endpoint`. To stop exporting them, set `otel.logs.enabled` to `false`. The chart has no setting for the access log alone. + ## Actor identity Agent Substrate adds the following labels to Actor logs and to its component logs about an Actor. Each label maps to a kagent object, so you can find an agent's telemetry from the AgentInstance or the AgentTemplate that you already know. @@ -33,15 +35,30 @@ Agent Substrate adds the following labels to Actor logs and to its component log | `ate.template.atespace` | Where the template lives, which matches `ate.atespace` for a kagent agent. | | `ate.actor.container.name` | The container that wrote the line, such as `kagent`. Records for a suspend or resume omit this label, because the Actor writes them rather than a container. | -Agent Substrate keeps the labels that identify one Actor off its metrics, so that the number of time series does not grow with the number of conversations. Metrics carry the template and WorkerPool labels instead. To look into one agent, use its logs. +Agent Substrate metrics carry template and WorkerPool labels rather than the Actor identity, so that the number of time series does not grow with the number of conversations. To look into one agent, use its logs. + +## Before you begin + +1. [Install kagent]({{< link path="setup/installation" >}}), including the `kubectl-ate` plugin that the installation guide describes. +2. [Create your first agent]({{< link path="get-started/your-first-agent" >}}), and send it at least one message, so that the `my-first-agent` AgentTemplate and the `my-first-harness` Harness have an Actor with logs to read. That guide also installs the kagent CLI. +3. Install [`jq`](https://jqlang.org/download/), to read the AgentInstance ID and filter the JSON log records. ## Read an Actor's logs -1. Find the Actor for an AgentInstance. The Actor name is `ai-` followed by the AgentInstance ID. +An Actor writes to the stdout of whichever Worker pod runs it, and a suspended Actor is assigned to no Worker at all. Find the Actor by its AgentInstance ID, then read the Worker pods of its WorkerPool and filter by the Actor's name. + +1. Save the ID of the AgentInstance that you want to follow. ```bash export INSTANCE_ID=$(kagent get agent-instance -o json \ | jq -r '[.agentInstances[] | select(.agentTemplate.name == "my-first-agent")] | sort_by(.createdAt) | last | .id') - kubectl ate get actors --atespace kagent + echo $INSTANCE_ID + ``` + + A value of `null` means that no AgentInstance matched. The CLI lists only the AgentInstances that you created, so confirm that you are using the identity that created the agent. + +2. Confirm that the Actor exists, and check its state. The Actor name is `ai-` followed by the AgentInstance ID. + ```bash + kubectl ate get actors ai-$INSTANCE_ID --atespace kagent ``` Example output: @@ -50,12 +67,25 @@ Agent Substrate keeps the labels that identify one Actor off its metrics, so tha kagent ai-01a0d409-d249-728e-865c-ee58b55dff04 kagent/my-first-agent-my-first-harness-a995d20d30a7 ACTOR_STATE_SUSPENDED 5 32m ``` -2. Read the logs from the Workers of the WorkerPool that the Harness runs on. A Worker writes the output of every Actor that it runs, so filter the output by the Actor name. The filter returns the Actor's lines from every current Worker in the pool, including the lines from before the Actor last suspended. Worker output mixes JSON records with plain text lines, so the filter skips any line that is not JSON. + `ACTOR_STATE_SUSPENDED` with a `WORKER POD` of `` is the resting state between turns. Because a suspended Actor names no Worker, the remaining steps read every Worker in the pool rather than one pod. + +3. Find the WorkerPool that the Harness runs on. ```bash export WORKER_POOL=$(kubectl get harness my-first-harness -n kagent \ -o jsonpath='{.spec.substrate.workerPoolRef.name}') + echo $WORKER_POOL + ``` + + Example output: + ```console + kagent-default + ``` + +4. Read the Actor's lines from the Workers in the pool. A Worker writes the output of every Actor that it runs, so the filter selects a single Actor by name. Worker output mixes JSON records with plain text, so the filter also drops any line that is not a JSON object. + ```bash kubectl logs -n kagent -l ate.dev/worker-pool=$WORKER_POOL --tail=-1 \ - | jq -cR "fromjson? | select(.labels[\"ate.actor.name\"] == \"ai-${INSTANCE_ID}\")" + | jq -cR --arg actor "ai-$INSTANCE_ID" \ + 'fromjson? | objects | select(.labels["ate.actor.name"] == $actor)' ``` Example output: @@ -64,9 +94,17 @@ Agent Substrate keeps the labels that identify one Actor off its metrics, so tha {"labels":{"ate.actor.name":"ai-01a0d409-d249-728e-865c-ee58b55dff04","ate.actor.uid":"c91579a8-675c-474c-9437-4482ac628774","ate.atespace":"kagent","ate.template.atespace":"kagent","ate.template.name":"my-first-agent-my-first-harness-a995d20d30a7"},"message":"Actor checkpointed","span_id":"4914db77fcc2f59b","time":"2026-09-24T15:30:09.671009219Z","trace_flags":"01","trace_id":"c4971ca2e614428819443f64fe70f81c"} ``` - `kubectl logs` reads only the Worker pods that exist now. When a Worker pod is replaced, the output that it held is gone. To keep an Actor's history across Worker replacements, collect the stdout of the Worker pods with a log agent that stores structured JSON, and query it by the same labels. Neither OTel stack guide collects pod stdout. + The filter returns the Actor's lines from every Worker that currently runs in the pool, including the lines from before the Actor last suspended. -The `kubectl ate logs actors` command streams the output of a single Actor, but only while the Actor is running on a Worker. A kagent agent runs only during a turn, and the command returns an error the rest of the time. +`kubectl logs` reads only the Worker pods that exist now. When a Worker pod is replaced, the output that it held is gone. To keep an Actor's history across Worker replacements, collect the stdout of the Worker pods with a log agent that stores structured JSON, and query it by the same labels. Neither the OTel stack guide nor the Lightweight OTel stack guide collects pod stdout, because both configure the Collector with the OTLP receiver only. + +Agent Substrate can also stream a single Actor's output directly, without the pool-wide filter. + +```bash +kubectl ate logs actors ai-$INSTANCE_ID --atespace kagent +``` + +The command reads the Worker that the Actor runs on, so it works only while the Actor is running. A kagent agent runs only during a turn, and the command returns an error the rest of the time. ```console Error: actor kagent/ai-01a0d409-d249-728e-865c-ee58b55dff04 is not currently running on any worker pod @@ -78,7 +116,8 @@ The records that Agent Substrate writes for an Actor mark each step of a suspend ```bash kubectl logs -n kagent -l ate.dev/worker-pool=$WORKER_POOL --tail=-1 \ - | jq -cR "fromjson? | select(.labels[\"ate.actor.name\"] == \"ai-${INSTANCE_ID}\" and .labels[\"ate.actor.container.name\"] == null) | {time, message}" + | jq -cR --arg actor "ai-$INSTANCE_ID" \ + 'fromjson? | objects | select(.labels["ate.actor.name"] == $actor and .labels["ate.actor.container.name"] == null) | {time, message}' ``` Example output: @@ -106,11 +145,38 @@ The Actor that builds a pair's first snapshot, in the `ate-golden` atespace, wri Each record also carries `trace_id`, `span_id`, and `trace_flags`, which join the record to the Agent Substrate trace for the same operation. A `trace_flags` value of `01` means that the trace was sampled and is in your tracing backend. A value of `00` means that the trace was not sampled, which is the case for most requests at the default ratio of `0.01`. +## Actor state changes + +The Agent Substrate API server, `ateapi`, writes an `Actor state changed` record each time an Actor moves to a new state. The record carries the Actor identity labels, the operation that caused the change in `ate.actor.operation.name`, and the new state in `ate.actor.state`. The last record for an Actor tells you its state, and when the Actor entered it. + +| `ate.actor.state` | Meaning | +| ----------------- | ------- | +| `resuming` | A turn woke the Actor, and a Worker is restoring it. | +| `running` | The Actor is on a Worker and can serve the turn. | +| `suspending` | The turn ended, and the Actor is writing its snapshot. | +| `suspended` | The snapshot is written, and the Actor holds no Worker. | +| `deleting`, `deleted` | The AgentInstance was deleted. `deleted` is the last record that an Actor gets. | + +`ateapi` runs more than one replica, and each replica writes only the changes that it handles. Read the logs of every replica. +```bash +for pod in $(kubectl get pods -n ate-system -o name | grep ate-api-server); do + kubectl logs -n ate-system "$pod" +done | grep '"msg":"Actor state changed"' | grep "ai-${INSTANCE_ID}" +``` + +Example output: +```console +{"time":"2026-09-24T17:00:43.007925845Z","level":"INFO","msg":"Actor state changed","ate.atespace":"kagent","ate.actor.name":"ai-01a0d45b-c729-7d67-85c6-daa053202bff","ate.actor.uid":"831c62c5-b5ac-411f-b15e-7addfef97d73","ate.template.atespace":"kagent","ate.template.name":"my-first-agent-my-first-harness-699a5ed2f709","ate.actor.operation.name":"resume","ate.actor.state":"resuming","trace_id":"402701838ffd50932dc0aec8e43a1604","span_id":"1f8f032ff37422c7","trace_flags":"00"} +{"time":"2026-09-24T17:00:43.194765137Z","level":"INFO","msg":"Actor state changed","ate.atespace":"kagent","ate.actor.name":"ai-01a0d45b-c729-7d67-85c6-daa053202bff","ate.actor.uid":"831c62c5-b5ac-411f-b15e-7addfef97d73","ate.template.atespace":"kagent","ate.template.name":"my-first-agent-my-first-harness-699a5ed2f709","ate.actor.operation.name":"resume","ate.actor.state":"running","trace_id":"402701838ffd50932dc0aec8e43a1604","span_id":"f92aecd47d89ad68","trace_flags":"00"} +``` + +When the Agent Substrate release sets an OTLP endpoint, `ateapi` also exports each record as an OTLP log with the same attributes, so a logging backend receives the state changes of every Actor without reading pod output. Do not sample this stream. A dropped record leaves the last known state wrong, with nothing to show that a record is missing. + ## Learn more -The Agent Substrate project documents its telemetry in full, including the complete metric registry, in [Actor observability](https://github.com/agent-substrate/substrate/blob/main/docs/observability.md). That page describes the newest Agent Substrate, which can be ahead of version {{< reuse "kagent-docs/versions/agent-substrate.md" >}} that this guide uses. Records that it describes might not exist in your version, such as actor state change events and per-Actor usage samples. +The Agent Substrate project documents its telemetry in full, including the complete metric registry, in [Actor observability](https://github.com/agent-substrate/substrate/blob/main/docs/observability.md). That page describes the newest Agent Substrate, which can be ahead of version {{< reuse "kagent-docs/versions/agent-substrate.md" >}} that this guide uses. Records that it describes might not exist in your version, such as per-Actor usage samples and the restore timing breakdown. -## Next steps +You can also check out the following resources in this documentation set. {{< cards >}} {{< card link=`{{< link path="observability/tracing" >}}` title="Tracing" subtitle="Read the spans of an agent request, and the separate Agent Substrate traces." >}} diff --git a/docs-site/content/kagent/1.x/observability/tracing.md b/docs-site/content/kagent/1.x/observability/tracing.md index 22c2dce8..8945927b 100644 --- a/docs-site/content/kagent/1.x/observability/tracing.md +++ b/docs-site/content/kagent/1.x/observability/tracing.md @@ -48,7 +48,7 @@ A caller reaches the gRPC API on the kagent controller, which starts the trace. Each hop reports itself as a separate OpenTelemetry (OTel) service. A tracing backend uses these service names to group the spans. - **The controller** reports as `kagent-controller` in the `kagent` service namespace. Its spans also carry the pod, node, and namespace that the controller runs on. -- **The Agent Substrate router** reports as `agentgateway`, the proxy that the router runs. +- **The Agent Substrate router** reports as two services, because its pod runs two containers. The router's own spans, such as its lookup of the Actor for a request, report as `atenet-router`. The spans of the agentgateway proxy that forwards the request to the Worker report as `agentgateway`. Only the `agentgateway` spans join the agent request trace. The `atenet-router` spans form separate [Agent Substrate traces](#agent-substrate-traces). - **Each agent runtime** reports as its own service, named for the {{< gloss "AgentTemplate" >}}AgentTemplate{{< /gloss >}} and {{< gloss "Harness" >}}Harness{{< /gloss >}} pair it was compiled from. The `my-first-agent` template on the `my-first-harness` Harness reports as `my-first-agent-my-first-harness`. > [!NOTE] @@ -82,12 +82,12 @@ A trace tells you which request you are looking at through attributes on its spa The runtime also adds each scalar value in the A2A message's metadata as an `a2a.message.metadata.` attribute, so a client can tag a request and search for it later. Unlike the four correlation attributes, these tags stay on the `invocation` span alone, so a search on one returns that span instead of the whole subtree. > [!WARNING] -> When the `otel.captureSensitiveContent` Helm setting is `true`, prompts and replies reach your tracing backend. The spans for a model call then carry the full serialized request and response as the `gcp.vertex.agent.llm_request` and `gcp.vertex.agent.llm_response` attributes, truncated to a prefix when a payload is larger than 32 KiB. The setting defaults to `false`, which leaves both attributes as `{}`. For how the setting applies to each runtime, see the agent harness [telemetry content settings]({{< link path="agents/agent-harness#telemetry-content-settings" >}}). +> When the `otel.captureSensitiveContent` Helm setting is `true`, prompts and replies reach your tracing backend. The spans for a model call then carry the full serialized request and response as the `gcp.vertex.agent.llm_request` and `gcp.vertex.agent.llm_response` attributes, truncated to a prefix when a payload is larger than 32 KiB. The setting defaults to `false`, which leaves both attributes as `{}`. For how to use this content as an audit record, see [Audit prompts]({{< link path="observability/audit-prompts" >}}). ## Before you begin 1. [Install kagent]({{< link path="setup/installation" >}}). -2. [Create your first agent]({{< link path="get-started/your-first-agent" >}}), so that you have an {{< gloss "AgentInstance" >}}AgentInstance{{< /gloss >}} to send a request to. +2. [Create your first agent]({{< link path="get-started/your-first-agent" >}}), so that you have an {{< gloss "AgentInstance" >}}AgentInstance{{< /gloss >}} to send a request to. That guide also installs the kagent CLI. The steps on this page need the {{< reuse "kagent-docs/versions/kagent.md" >}} CLI, because earlier CLI versions have no `agent-instance` commands and fail with `unknown command`. To check your version, run `kagent version`. 3. Set up a tracing backend. The [OTel stack]({{< link path="observability/otel-stack" >}}) sends traces to Tempo, and the [Lightweight OTel stack]({{< link path="observability/lightweight-otel-stack" >}}) sends traces to Jaeger. Both guides turn on tracing for you, so you can skip to [Review a trace](#review-a-trace). ## Enable tracing @@ -152,7 +152,7 @@ Tracing is off by default. Turning it on is a Helm change, because the controlle sleep 5 done ``` - If the command finishes without printing `Recompiled`, the upgrade did not change the pair, for example because the settings were already in place. + If the command finishes without printing `Recompiled`, the upgrade did not change the settings that kagent compiles into the pair. Either the settings were already in place, or the chart did not recognize the `otel` keys. Helm accepts a key that a chart does not define without an error, so check that you upgraded to version {{< reuse "kagent-docs/versions/kagent.md" >}} of the chart, which uses the keys on this page. 6. Create a new AgentInstance, so that its Actor starts from a runtime that has the tracing configuration. ```bash @@ -174,6 +174,8 @@ Tracing is off by default. Turning it on is a Helm change, because the controlle ## Review a trace +Send a request to the new AgentInstance, then find its trace in the backend that you set up. + 1. Send a request to the AgentInstance to produce a trace. ```bash export INSTANCE_ID=$(kagent get agent-instance -o json \ @@ -245,6 +247,8 @@ The flush lets a kagent trace arrive promptly rather than on the exporter's own ## Turn tracing off +Turn off the trace exporter, then create a new AgentInstance so that the change takes effect. + 1. Disable tracing in the kagent Helm release. ```bash helm upgrade kagent \