Repository navigation
Ingestion reliability: SDK backoff and spool, idempotent batch, silence alert - #59
Merged
Merged
Conversation
…d a silence alert SDK OTLPExporter: exponential backoff with jitter and Retry-After, retries only retryable failures, optional persistent spool (0600, bounded, drained oldest first), stats(). /v1/traces/batch no longer fails when a retried batch re-sends stored span ids and its limit is RATE_LIMIT_BATCH (120/minute, it was 10 while the SDK flushes every 5 s). GET /v1/ingest/health, alert metric ingest_silence_minutes, an intake card on Overview. Quota and ingest alerts no longer depend on min_samples. Roadmap step 8 done.
josedaniel-dev
force-pushed
the
feat/ingest-reliability
branch
from
October 16, 2026 07:58
e7f0ae1 to
3adbe4a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Step 8 of docs/roadmap.md.
Findings fixed: the SDK exporter retried instantly with no backoff, retried permanent 4xx, dropped silently and kept no counters;
/v1/traces/batchwas capped at 10/minute while the SDK flushes every 5 s, and answered 500 when a retried batch re-sent span ids already stored; nothing said a project had stopped sending.OTLPExporter: retries only network errors and 408/425/429/5xx with exponential backoff, jitter andRetry-After; permanent 4xx are counted asrejected; optional persistent spool (spool_dir; files 0600, unencrypted, bounded byspool_max_bytes, oldest evicted, drained oldest first, survives restarts);stats()= exported, retried, rejected, dropped, spooled, recovered, spool_evicted, pending. Existing constructor arguments and behaviour are unchanged by default (no spool)./v1/traces/batchskips span ids already stored (a lost-answer retry no longer fails); its limit is nowRATE_LIMIT_BATCH, default120/minute(was a fixed 10/minute);GET /v1/ingest/health(last trace + 5 m/1 h/24 h counts per project); alert metricingest_silence_minutes; quota and ingest alerts no longer depend onmin_samples(a quota rule created in the UI with the default 10 could never fire).Tests: SDK 9 new (backoff, retry classification, spool order/restart/bound/permanent reject, network errors), backend 5 new (idempotent batch, health, silence alert fire/resolve, never-sent project); backend 215 passed, SDK 46, dashboard 40, ruff/tsc/build clean. Checked in a browser on a synthetic instance: the intake card showed a 1-minute, a 2-hour and a 2-day silent project; a silence rule created with min_samples=10 opened its alert; no overflow at 390 px.
Not done: encrypting the SDK spool; sender-reported drop counters inside Observer (the SDK exposes
stats()locally).Merge note: this branch predates #58, so
docs/roadmap.mdanddashboard/src/i18n-en.tswill conflict with it; I will rebase whichever merges second.