Repository navigation
Head-to-head races, logs, traces and eBPF request metrics - #1
Merged
Merged
Conversation
…rics - Add bench/k6/src/match.ts and runner /run/compare for 2-3 server races, plus a Head to Head dashboard - Add Loki + Alloy (container logs), Tempo 3 (traces) and Beyla (eBPF RED metrics and sampled traces for all 13 servers, observe-only) - Give each server its own internal port from servers.json so Beyla's generated config instruments only them - Instrument the runner with OpenTelemetry spans and slog JSON logs carrying trace IDs; link Loki, Tempo and Prometheus datasources - Add Inside the Servers dashboard and eBPF, logs and traces rows on server, Live, Head to Head and Stack dashboards - Fix runner links in Grafana (target=_self / new tab) that Grafana's router turned into a 404 - Add optional webhook alert notifications via deploy.sh, alert lists on dashboards, and a server 5xx alert - Move k6 rate/match tags onto scenarios, since CLI --tag replaces options.tags - Disable uvicorn access logs in FastAPI to match Flask and Django
…hem on deploy - Validate Loki (-verify-config), Alloy (validate) and Tempo (-config.verify) configs with their own images in the checks job - Fail the checks job when infra/beyla/beyla.yaml drifts from generate.py or the runner is not gofmt-clean - Reload Alloy on SIGHUP alongside Prometheus and blackbox in infra/deploy.sh - Restart Loki, Tempo or Beyla only when their config file changed since the container started - Fail the deploy unless Grafana reaches the Loki and Tempo datasources
…ces views - Give every timeseries that reads Beyla metrics a 15s minimum interval, since the 1s datasource interval made $__rate_interval shorter than Beyla's 5s scrape - Set Beyla duration buckets from 0.1 ms upward so sub-millisecond server latency gets real p50/p99 instead of the 5 ms default bucket - Limit Slowest recent requests to the 13 servers and a 5 ms threshold, so the runner's 30s load-test traces no longer fill it - Make All on Inside the Servers mean the servers and the runner, keeping the stack's own logs opt-in - Sum Prometheus's samples-appended counter, which Prometheus 3 splits by sample type - Document Beyla's kernel and privilege needs, memory caps, retention, the new CI checks and deploy reloads
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this adds
/run/compare): 2-3 servers get the same rate at the same moment; new Head to Head dashboard with winners, scoreboard and overlaid graphs.trace_id(link to the trace), spans link to their service's logs and request metrics.ALERT_WEBHOOK_URLininfra/.env(load-test alerts stay dashboard-only), alert lists on dashboards, and a server-side 5xx alert./run/<server>links into its 404 page; k6rate/matchtags were dropped because CLI--tagreplacesoptions.tags; FastAPI no longer writes an access log per request (Flask and Django already did not).Behaviour changes to know before merging
servers.jsonport inside its container (Fiber on 3106 everywhere), so Beyla can match servers by port.Testing
deploy.sh's config-change restart verified against running containers; every CI check passes locally (generator, gofmt/vet/build, typecheck, compose, promtool, Loki, Alloy, Tempo, nginx, shellcheck).After merging (on the VPS)
sudo ufw allow from 172.30.0.0/24 to 172.30.0.1 port 9100 proto tcp comment 'prometheus -> node-exporter'so the Host dashboard gets data.ALERT_WEBHOOK_TYPE/ALERT_WEBHOOK_URLininfra/.envfor chat notifications.