feat(traces): Add error sampler, rescue errored p0 trace chunks - #169
lucaspimentel wants to merge 9 commits into
Conversation
|
@DataDog review |
![datadog-datadog-prod-us1-2[bot]](https://avatars.githubusercontent.com/u/365230?s=60&v=4){kind=small}
There was a problem hiding this comment.
Concurrent requests can send times to the shared sampler in reverse order at a bucket limit. This can clear valid counts and permit too many rescued traces.
🤖 Datadog Autotest · Commit f316a2a · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest
|
@DataDog review |
There was a problem hiding this comment.
A backward system-clock change can move the sampler to an old bucket. The sampler then uses stale counts and applies the TPS limit incorrectly.
🤖 Datadog Autotest · Commit e5fcd5e · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest
Wire the shared datadog-agent-trace-sampler into the SCL trace processor so error chunks with automatic-drop priority (0) get a second look before being forwarded. On a keep, the sampler's rate is stamped as a positive `_dd.errors_sr` metric on the chunk's root span, which the APM backend treats as an `error` ingestion reason and retains without any priority promotion. Chunks are never dropped locally: unrescued chunks are forwarded unchanged for the backend to discard. - Default to rate-limited rescue at 10 error TPS per process, with a shared budget across requests and processor clones - Configure via DD_APM_ERROR_SAMPLER_MODE (rate_limited | always_keep) and DD_APM_ERROR_TPS (<= 0 disables rescue in either mode); invalid values warn and fall back to defaults - Gate rescue on agent stats computation and submit all chunks to the stats concentrator before any rescue stamping
e5fcd5e to
bfbe164
Compare
lucaspimentel
changed the title
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: bfbe16481b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Moderate issues remain in sampler test equivalence, mutex contention, and integration-test error propagation.
Get a fresh assessment by requesting another Copilot review.
Review effort: Lite
Findings: 2
Open (3)
What changed in this PR
Adds shared error sampling to rescue eligible errored P0 trace chunks while preserving priorities and stats behavior.
Changes:
- Integrates configurable, shared error sampling.
- Stamps
_dd.errors_sron rescued root spans. - Adds configuration, dependency wiring, and unit/integration coverage.
| File | Review summary |
|---|---|
crates/datadog-trace-agent/tests/integration_test.rs |
Moderate findings (1 vote each): propagate mini-agent startup and shutdown/task errors. |
crates/datadog-trace-agent/src/trace_processor.rs |
Moderate findings: align test environments in sampler oracles (2 votes each); avoid holding a blocking mutex during the full payload walk (1 vote). Nit: document root-resolution fallback behavior (2 votes). |
crates/datadog-trace-agent/src/stats_processor.rs |
Test configuration fixtures updated; no findings. |
crates/datadog-trace-agent/src/config.rs |
Sampler configuration parsing updated; no findings. |
crates/datadog-trace-agent/Cargo.toml |
Adds the sampler dependency; no findings. |
crates/datadog-serverless-compat/src/main.rs |
Initializes the shared sampler; no findings. |
Cargo.lock |
Records dependency updates; no findings. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Extracts the "prefer tracer payload env, fall back to agent config env" rule into one function reused by stats flushing and error-rescue sampling, and removes duplicated sampler lock/disabled-check and SpanView-construction code in the error sampler and its tests.
The clock read and timestamp clamp were happening before the sampler lock was acquired, letting concurrent requests deliver timestamps out of lock-acquisition order and undercount TPS in the rolling window. 🤖
What does this PR do?
Wires the shared error sampler (
datadog-agent-trace-sampler) into the SCL trace processor. Chunks with automatic-drop priority (0) containing a span with a non-zero error flag get a second look; on a keep, the sampler's rate is stamped as a positive_dd.errors_sron the root span, which the backend treats as anerroringestion reason and retains without priority promotion. Chunks are not dropped locally (yet): unrescued P0 chunks are forwarded unchanged for the backend to discard. This will be done in a separate PR.RateLimitedat 10 error TPS per process, one budget shared across requests and processor clones;AlwaysKeepavailableDD_APM_ERROR_SAMPLER_MODEandDD_APM_ERROR_TPS(<= 0disables rescue); invalid values warn and fall back to defaultsagent_stats_computation_enabled; only the root's_dd.errors_sris added, priorities and decision makers are untouchedMotivation
APMSVLS-472. Under tracer sampling, errored P0 chunks are discarded by the backend's ordinary P0 drop; a positive
_dd.errors_sris sufficient for the backend to bypass it.Describe how to test/QA your changes
New config tests (env parsing, defaults, fallbacks), deterministic processor tests covering the full rescue matrix (compared against a standalone shared sampler), and an integration test that decodes the outbound protobuf payload and agent-computed stats.
cargo check/clippy/fmt,cargo nextest run --workspace --no-fail-fast(429/430; the one failure is a pre-existing local port-8126 collision onmain), and doc tests all pass.