Repository navigation
observability: built-in OpenTelemetry Collector (Consul SD → OTLP) - #24
Merged
Merged
Conversation
…OTLP)
A managed OTel Collector that discovers services via Consul and ships their
Prometheus metrics to a configurable OTLP backend (SigNoz/Grafana/etc.) — the
general app-metrics layer on top of the LB/Nomad metrics.
- setup/otel-collector.ts: on-disk config ({enabled, otlpEndpoint, otlpHeaders,
scrapeIntervalSeconds}); generates the collector's prometheus receiver
(consul_sd_configs against the hub Consul 10.10.0.1:8500 — keeps the Traefik
instances on :8082 and any service tagged nomploy.metrics.port=<port>) → OTLP
exporter; deploy/stop as a count=1 Nomad service job that reschedules forever
(survives a node death, restarts elsewhere). Mirrors the HA-Traefik subsystem.
- settings router: getObservability / saveObservability (adminProcedure; save
deploys when enabled, purges when disabled; audited).
- Settings → Web Server: new Observability card (enable, OTLP endpoint, headers,
scrape interval).
Caveat: cross-node scraping works for host-networked services (node IPs);
bridge/CNI alloc IPs aren't cross-node routable here, so those must be
co-located until the mesh/CNI routing fix lands.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A built-in OpenTelemetry Collector subsystem: a managed Nomad job that discovers services via Consul service discovery, scrapes their Prometheus metrics, and ships them to a configurable OTLP backend (SigNoz, Grafana/Tempo/Mimir, Honeycomb, …). This is the general app-metrics layer on top of the existing LB metrics + Nomad CPU/mem telemetry.
How
setup/otel-collector.ts: on-disk config ({enabled, otlpEndpoint, otlpHeaders, scrapeIntervalSeconds}); generates the collector config (prometheus receiver withconsul_sd_configsagainst the hub Consul10.10.0.1:8500→ OTLP exporter) and acount=1Nomad service job that reschedules forever (survives a node death, restarts elsewhere).deployOtelCollector()/stopOtelCollector()mirror the HA-Traefik subsystem.:8082) + any service taggednomploy.metrics.port=<port>(per-service opt-in via a Consul tag).getObservability/saveObservability(adminProcedure; enable → deploy, disable → purge; audited).Scope / caveats (v1)
otlp); point it at SigNoz:443/:4317. (otlphttp/4318 is a later toggle.)Validation
scripts/check.shgreen (server prod build, 529 tests, app-tsc unchanged / 8 pre-existing). Not yet live-tested — recommend merging, rolling, then enabling it in Settings with an OTLP endpoint and confirming the collector job runs + scrapes Traefik.