LLM Observatory

Observer is a local-first, self-hosted observability and evaluation platform for LLM and agentic systems. It stores traces in SQLite or PostgreSQL that you control, works with local or hosted models, and requires no Observer cloud account.

The dashboard above was rendered from synthetic telemetry. The clean-runner quickstart smoke test still captures its own screenshot as a CI artifact.

What's new: a redesigned dashboard

The dashboard was rebuilt as a dense, dark console: sidebar navigation, a global time range and environment filter, live updates, and charts that use the whole screen. The interface is available in English and Spanish: pick the language with the ES/EN switch in the header (the choice is remembered; the default follows the browser language).

• Overview — turns, error rate, p50/p95 latency, time to first token, fallback rate and cost (always shown, even at $0), an activity/latency chart, latency histogram, per-stage time and model share. The series and latency KPIs come from the metrics query, so 7 d and 30 d ranges use every turn.
• Providers — your LLM fallback chain in the order it is tried, with per-provider success rate, 429/timeout/error counts, p50/p95 latency, how often each one answered, fallback depth, common routes and the average turn score. It flags providers that are configured but never reached.

• Alerts — threshold rules over a sliding window (error rate, p95 latency, fallback rate, provider failures, providers out of quota, ...), a global banner and optional desktop notifications.

• Evaluations — every turn gets an operational-health score without a model (reliability, completeness, degradation, fallback, latency, time to first token); answer correctness is shown as unknown until measured. Health by model and a per-criterion breakdown of each turn.

• Explore — a query builder over turns, provider attempts and evaluations (measure, aggregation, group by, granularity). The query lives in the URL so it can be shared, and views can be saved.

• Trace detail — a hierarchical waterfall, a per-stage time bar and, for llm.chain spans, the provider-by-provider attempts of that turn.

The Providers page reads llm.chain spans and an optional metadata-only provider snapshot (POST /v1/providers/snapshot, never keys); see docs/manitos-integration.md. These screenshots were rendered from synthetic telemetry.

Features

• Multi-framework tracing — Auto-instrument OpenAI, Anthropic, LangChain
• Agent runtime integration — Versioned adapters for metadata-only telemetry
• Cost tracking — Per-model pricing for 20+ models (GPT-4o, Claude 4, Gemini 2.0, etc.)
• Evaluation engine — LLM-as-Judge with YAML rubric, rule-based metrics, human feedback
• Real-time dashboard — Live trace streaming via WebSocket, time-range filtering
• CLI tools — Query, export, import, and evaluate traces from the terminal
• SQLite & PostgreSQL — Local dev with SQLite, production with PostgreSQL

Privacy and data collection

Observer has no built-in product analytics or managed telemetry service. Your Observer data plane runs on infrastructure you control. What enters that data plane depends on how you instrument the application:

• Generic Python and TypeScript auto-instrumentation captures model inputs and outputs, token usage, timing, cost, status, model/provider identifiers, and user-supplied span attributes by default. Inputs and outputs can contain raw prompts or model responses.
• Direct HTTP instrumentation can be metadata-only: omit input and output and send only the operational fields your retention policy permits.
• The reference runtime integration is documented for metadata-only telemetry. Its compatibility schema still accepts generic input and output fields, so exporters must omit content and deployments should verify stored data.
• Observer does not require actor identity, provider credentials, or secrets in trace payloads. Do not place API keys, personal data, or secret-bearing URLs in spans or attributes.
• Rule-based evaluation stays local. Enabling LLM-as-Judge with OPENAI_API_KEY sends the selected evaluation context to the configured model provider under that provider's data policy.

Treat the database as sensitive whenever content capture is enabled. Configure authentication, TLS, access control, backups, retention, and deletion before a shared deployment. See First-time setup.

How Observer compares

Observer deliberately favors a small, inspectable self-hosted deployment over the broadest integration catalog. This table is a starting point, not a claim that one tool fits every team.

Project	Strongest fit	Where it is stronger than Observer	Where Observer is simpler
Observer	A compact local data plane with traces, evaluations, cost/latency analytics, CLI, and dashboard	—	SQLite for a single-node start; API, dashboard, and database are the complete core stack
Langfuse	Mature end-to-end LLM engineering workflows	Prompt management, datasets, experiments, custom dashboards, and production-scale ingestion	Observer avoids the ClickHouse, Redis/Valkey, and object-storage services used by production Langfuse self-hosting
Arize Phoenix	OpenTelemetry/OpenInference tracing and systematic experimentation	Broader instrumentation, datasets, experiments, prompt playground, and a larger ecosystem	Observer offers a narrower API and SQLite path when those workflows are unnecessary
Helicone	Gateway-first routing and observability	Provider gateway, routing, and proxy-based onboarding	Observer is not in the model request path and can ingest traces directly
OpenLLMetry	OpenTelemetry instrumentation for an existing observability stack	Much broader provider, framework, vector database, and OTLP backend coverage	Observer includes its own storage, analytics API, evaluation engine, CLI, and dashboard

Comparison checked against the linked project documentation on 2026-08-07.

Quick Start

For a guided installation, first trace, application integration, privacy choices, and production checklist, see First-time setup.

Copy .env.example when you need to customize the backend. The checked-in values are development examples, not production credentials.

Docker Compose (recommended)

git clone https://github.com/Magic-Lab-Studio/observer.git
cd observer
docker compose up -d

Service	URL
Backend	http://localhost:8000
Dashboard	http://localhost:5173
PostgreSQL	localhost:5432

Podman Compose

podman-compose up -d

Manual Setup

1. Start PostgreSQL:

podman run -d --name postgres -p 5432:5432 \
  -e POSTGRES_PASSWORD=postgres -e POSTGRES_DB=observatory \
  postgres:16-alpine

2. Backend:

cd backend
pip install -e ".[dev,sqlite]"
alembic upgrade head
uvicorn app.main:app --reload

3. Dashboard:

cd dashboard
npm install
npm run dev

Project Structure

observer/
├── backend/                  # FastAPI backend + evaluation engine
│   ├── app/
│   │   ├── api/              # Route handlers (traces, evaluations, analytics)
│   │   ├── evaluators/       # LLM-as-Judge, rubric engine (YAML/JSON)
│   │   ├── models/           # SQLAlchemy models
│   │   └── main.py           # App entrypoint, health check
│   └── alembic/              # Database migrations
├── dashboard/                # React + Vite + Tailwind
│   └── src/
│       ├── pages/            # Overview, Traces, TraceDetail, Evaluations
│       ├── components/       # TraceWaterfall, ErrorBoundary, shared
│       ├── api.ts            # Centralized API client
│       └── types.ts          # Shared TypeScript interfaces
├── sdk/
│   ├── python/               # magic-lab-observer PyPI package
│   └── typescript/           # @magic-lab-studio/observer npm package
├── cli/                      # llm-observatory CLI
├── docker-compose.yml
├── podman-compose.yml
└── .github/workflows/ci.yml

Backend API

Base URL: http://localhost:8000

Method	Endpoint	Description
GET	/health	Health check (db status, version)
POST	/v1/traces/batch	Ingest traces
POST	/v1/ingest/manitos/traces	Versioned integration ingestion
GET	/v1/traces	List traces (paginated)
GET	/v1/traces/{id}	Get trace detail
DELETE	/v1/traces/{id}	Delete trace
POST	/v1/traces/batch-delete	Bulk delete traces
GET	/v1/traces/{id}/evaluations	Evaluations for a trace
GET	/v1/traces/export	Export traces as JSON
POST	/v1/evaluations	Create evaluation
POST	/v1/evaluations/run	Run evaluator on a trace
GET	/v1/evaluations/summary	Aggregated eval stats
GET	/v1/analytics/summary	Trace analytics
GET	/v1/analytics/timeline	Trace counts over time
GET	/v1/analytics/cost-by-model	Cost breakdown by model
GET	/v1/analytics/sessions	Unique sessions with stats
GET	/v1/analytics/manitos-quality	Integration quality and latency
GET	/v1/providers/	LLM provider chain statistics
POST	/v1/providers/snapshot	Provider configuration snapshot
GET	/v1/alerts/metrics	Metrics an alert rule can watch
GET	/v1/alerts/rules	List alert rules
POST	/v1/alerts/rules	Create an alert rule
PATCH	/v1/alerts/rules/{id}	Edit or enable/disable a rule
DELETE	/v1/alerts/rules/{id}	Delete a rule and its events
GET	/v1/alerts/events	Alert events (?state=open)
POST	/v1/alerts/events/{id}/ack	Mark an alert as seen
POST	/v1/alerts/evaluate	Evaluate all rules now
POST	/v1/otlp/v1/traces	Standard OTLP/HTTP trace receiver
GET	/v1/privacy/projects	Capture mode per project + what is stored
PUT	/v1/privacy/projects/{id}	Set a project's capture mode
DELETE	/v1/privacy/projects/{id}	Back to the path defaults
POST	/v1/privacy/projects/{id}/enforce	Rewrite stored data to the mode (confirm)
GET	/v1/datasets	List test-case datasets
POST	/v1/datasets	Create a dataset
GET	/v1/datasets/{id}	A dataset with its cases
GET	/v1/datasets/{id}/export	Portable observer.dataset.v1
POST	/v1/datasets/{id}/items	Save a case (optionally from a trace)
PATCH	/v1/datasets/items/{id}	Edit a case
DELETE	/v1/datasets/items/{id}	Delete a case
DELETE	/v1/datasets/{id}	Delete a dataset and its cases
POST	/v1/datasets/{id}/runs	Record a run: version + a trace per case
GET	/v1/datasets/{id}/runs	Runs of a dataset with summaries
GET	/v1/datasets/{id}/compare	Case-by-case diff (?base=&candidate=)
GET	/v1/datasets/runs/{id}	One run with every case result
DELETE	/v1/datasets/runs/{id}	Delete a run
GET	/v1/ingest/health	Last trace and counts per project
GET	/v1/evaluations/by-model	Turn health by model and provider
POST	/v1/analytics/query	Flexible metrics query
GET	/v1/analytics/query/schema	Views, measures and dimensions

The integration endpoint uses the versioned manitos.telemetry.v1 envelope, accepts opaque session identifiers, and safely deduplicates exporter retries. See docs/manitos-integration.md for the published contract.

Metrics explorer

POST /v1/analytics/query aggregates stored telemetry by dimension and time, in the spirit of Langfuse's metrics API. A request names a view (turns, provider_attempts or evaluations), metrics (a measure plus count, sum, avg, min, max, p50, p95 or p99; avg of a 0/1 field is a rate), up to two dimensions to group by (environment, model, provider, language, intent, status, ...), filters (=, !=, in), a range (hours or from_time/to_time), an optional granularity (minute, hour, day or auto) and a row_limit (at most 1000). Every field is checked against a fixed list per view (GET /v1/analytics/query/schema), so nothing in the request becomes a column or an expression; session and turn ids can filter but never group.

{"view": "turns", "hours": 168, "granularity": "day",
 "metrics": [{"measure": "duration_ms", "aggregation": "p95"}],
 "dimensions": ["provider"],
 "filters": [{"field": "environment", "op": "=", "value": "prod"}]}

The dashboard's Explore page builds these queries from selectors, keeps the query in the URL so it can be shared, and saves named views in the browser. The Overview reads its series and latency KPIs from the same endpoint, so 7 d and 30 d ranges use every turn.

Automatic turn evaluations

Every Manitos turn is scored without a model by a deterministic rubric (manitos_turn@2) of operational health: reliability (error / tool error), completeness (truncated), degradation, fallback, latency and time to first token, with fixed weights; a turn passes at 0.6 unless it failed outright. These signals say how the turn ran, not whether the answer was right, so correctness is reported as unknown and the overall score is capped at 0.89 (grade B) until a real signal exists. Scores of the previous rubric are replaced when the turn is re-scored. A background sweep (OBSERVATORY_AUTO_EVAL_INTERVAL_SEC, default 60; OBSERVATORY_AUTO_EVAL=0 turns it off; OBSERVATORY_AUTO_EVAL_DAYS, default 7) scores recent turns that have no evaluation yet and also backfills history. Evaluations are idempotent (the id derives from the trace and the rubric). The Evaluations page shows health by model, both axes, the grade and the per-criterion breakdown.

Capture privacy per project

Each project has a capture mode, set on the Privacy page or with PUT /v1/privacy/projects/{id}: metadata (no inputs/outputs; content-like attributes dropped), redacted (everything kept, but emails, phone numbers, card and ID numbers, tokens and keys are masked) or full. It applies on every ingestion path (SDK, Manitos, OTLP). Without a project mode each path keeps its default (SDK and Manitos store what they send, OTLP stores metadata only); OBSERVATORY_CAPTURE_DEFAULT overrides those defaults. The page also reports, from the latest spans, how many hold an input, an output or sensitive patterns, so a mode can be checked instead of trusted, and Apply to stored data rewrites what is already stored (irreversible, asks for confirmation). Redaction is pattern-based: it does not detect names or addresses. Details: docs/privacy.md.

Standard OpenTelemetry (OTLP) ingestion

Point any stock OpenTelemetry exporter at OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:8100/v1/otlp (protobuf or JSON, optionally gzipped) and its traces appear without the Observer SDK. GenAI conventions map to Observer span types, tokens and tools. Content-like attributes are dropped by default (OBSERVATORY_OTLP_CAPTURE=metadata). Mapping, privacy and limits: docs/otlp.md.

Security

Compose binds to loopback, does not publish PostgreSQL and requires POSTGRES_PASSWORD; the API is open unless OBSERVATORY_API_KEY is set. Untrusted input has size limits and the rate limiter ignores unvalidated tokens. Read docs/security.md before exposing Observer on a network.

Read-only MCP server

observer-mcp lets Claude Code, Codex or any MCP client investigate Observer without screenshots: failed traces with reasons, one trace's spans, tools and provider chain, alerts, datasets, version comparisons and metric queries. Read-only; trace input/output and any attribute outside an allow-list are never returned unless OBSERVER_MCP_CONTENT=1. Setup and tools: docs/mcp.md.

Ingestion reliability

The Python SDK exporter retries only what can succeed later (network errors, 408/425/429/5xx) with exponential backoff and jitter, honours Retry-After, and counts permanent 4xx answers as rejected instead of retrying them. Pass spool_dir to keep batches that still fail on disk (mode 0600, unencrypted, bounded by spool_max_bytes) and resend them oldest first after the backend returns or the process restarts; exporter.stats() reports exported, retried, rejected, dropped, spooled, recovered, spool_evicted, pending. /v1/traces/batch is idempotent (a retried batch does not fail on spans already stored) and rate-limited by RATE_LIMIT_BATCH (default 120/minute). GET /v1/ingest/health and the Overview card show when each project last sent a trace; the alert metric Minutes without traces (ingest_silence_minutes) notifies when one goes quiet.

Test cases (datasets)

On a trace page, Save as case stores a failure as a reusable test case in a dataset. A case is metadata only: nothing from the trace's input or output is copied. It records what was observed (intent, tool names, model, health signals) and what a person expects of the next run: tools that must or must not be used, health conditions (no error, no truncation, no degradation, no fallback, maximum duration), phrases the reply must or must not contain (typed by the person), and an optional human verdict (correct / incorrect). Cases keep their source trace id but outlive it when retention purges the trace. GET /v1/datasets/{id}/export returns the observer.dataset.v1 JSON that later steps (run comparison, a CI regression command) read. Note: Manitos exports one metadata-only tool span per executed call (name, outcome, duration; never arguments or results) since its PR #36, so observed.tools is filled for turns recorded after that version; earlier traces have none.

Runs and version comparison

Observer does not execute the agent. Whoever ran the cases (a developer, CI) reports a run: POST /v1/datasets/{id}/runs with the version (commit, model, a small config map; the rubric version is added by Observer) and the trace each case produced. Observer checks every case against its trace: health conditions are verified; tool expectations are verified only when the trace has tool spans, and text phrases only when reply text is available. Anything it cannot decide is unknown, never a silent pass, and a run's pass rate counts decided cases only. The Cases page lists the runs of a dataset and compares two of them case by case (regressed, improved, same, no data, missing), with the difference in health score, duration and configuration.

Regression gate for CI

observer-regression decides from stored runs whether a version may ship. After the pipeline has run the dataset's cases, point it at the traces they produced:

observer-regression --dataset regresiones \
  --results results.json --name "$GITHUB_SHA" --commit "$GITHUB_SHA" --model gemma3-4b \
  --baseline previous --max-regressions 0 --report regression.json

results.json is a list of {"case_trace": "<trace the case was saved from>", "trace_id": "<trace this run produced>"} (or item_id instead of case_trace). Without --results it gates an already recorded run (--candidate latest by default). Exit codes: 0 passes, 1 regression or threshold failure (reasons on stderr, JSON on stdout, optional observer.regression.v1 report), 2 it could not decide (Observer unreachable, unknown dataset or run, bad arguments).

Rules: with a baseline, more than --max-regressions cases going pass → fail blocks (default 0); without one, any failed case blocks (--max-failures changes it). --min-pass-rate and --max-health-drop add thresholds. Cases it cannot verify are unknown and reported apart; --fail-on-unknown blocks on them, and a run where nothing could be verified fails unless --allow-unverified is given, so the gate never goes green on no evidence. OBSERVER_URL and OBSERVATORY_API_KEY supply the address and key.

Alerts

Rules watch a metric over a sliding window (error rate, p50/p95 latency, share of turns on the fallback path, degraded/truncated replies, provider failures, providers out of quota) and fire when it crosses a threshold with enough samples. A rule resolves once the value is back inside the threshold by a 10 % margin, then waits for its cooldown before it can fire again. Open alerts show as a banner in the dashboard and, optionally, as browser desktop notifications. Four default rules are created on first start (OBSERVATORY_ALERTS_DEFAULTS=0 skips them); OBSERVATORY_ALERTS=0 turns the evaluator off and OBSERVATORY_ALERT_INTERVAL_SEC (default 60) sets its period.

Example: Ingest a trace

curl -X POST http://localhost:8000/v1/traces/batch \
  -H "Content-Type: application/json" \
  -d '{
    "spans": [{
      "trace_id": "demo-001",
      "name": "gpt-4o-call",
      "span_type": "llm",
      "start_time": "2025-01-01T00:00:00Z",
      "end_time": "2025-01-01T00:00:01Z",
      "status": "ok",
      "tokens_input": 150,
      "tokens_output": 50,
      "cost_usd": 0.000875
    }]
  }'

Python SDK

pip install magic-lab-observer==0.1.1

For editable SDK development from a checkout:

python -m pip install "./sdk/python[openai,anthropic,langchain]"

See First-time setup for package identities and integration examples.

from llm_observatory import instrument, trace
from llm_observatory.tracer import Tracer

# Auto-instrument LLM libraries
instrument(openai=True, anthropic=True, langchain=True)

# Or trace functions manually
@trace(name="summarize")
def summarize(text: str) -> str:
    return openai.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": text}]
    )

# Custom tracer with OTLP exporter
tracer = Tracer(service_name="my-app")
from llm_observatory.exporters.otlp import OTLPExporter
tracer.add_exporter(OTLPExporter(endpoint="http://localhost:8000"))

Supported providers

Provider	Auto-instrument	Manual trace
OpenAI	Yes	Yes
Anthropic	Yes	Yes
LangChain	Yes	Yes
Gemini	Manual only	Yes

TypeScript SDK

npm install @magic-lab-studio/observer

import { instrument, trace, asyncTrace, Tracer } from "@magic-lab-studio/observer";

// Auto-instrument
instrument({ openai: true, anthropic: true });

// Trace async functions
const result = await asyncTrace("summarize", async (span) => {
  const response = await openai.chat.completions.create({
    model: "gpt-4o",
    messages: [{ role: "user", content: text }],
  });
  span.attributes["model"] = "gpt-4o";
  return response;
});

// Custom tracer with OTLP exporter
const tracer = new Tracer({ serviceName: "my-app" });
import { OTLPExporter } from "@magic-lab-studio/observer";
tracer.addExporter(new OTLPExporter({ endpoint: "http://localhost:8000" }));

CLI

pip install magic-lab-observer-cli==0.1.1

For editable CLI development from a checkout:

python -m pip install ./cli

# List traces from local SQLite
llm-observatory traces --db observatory.db

# Inspect a specific trace and its spans
llm-observatory inspect <trace-id>

# Show summary statistics
llm-observatory stats

# Export traces to JSON
llm-observatory export -o traces.json

# Import traces from JSON
llm-observatory import traces.json

# Dry-run import (preview only)
llm-observatory import traces.json --dry-run

# Run an evaluation
llm-observatory evaluate --trace-id abc123 --evaluator llm_judge

# Check server status
llm-observatory status

Evaluation System

The LLM-as-Judge evaluator uses a YAML rubric with 7 criteria:

Criterion	Weight	Scale
Relevance	1.5	0-5
Accuracy	1.5	0-5
Completeness	1.0	0-5
Coherence	1.0	0-5
Conciseness	0.8	0-5
Safety	1.2	0-5
Hallucination	1.5	0-5 (inverted)

Configure the judge model:

export OPENAI_API_KEY=sk-...
export OBSERVATORY_JUDGE_MODEL=gpt-4o  # default

Development

Run tests

# Backend (170 tests)
cd backend && pytest

# Python SDK (37 tests)
cd sdk/python && pytest

# TypeScript SDK (30 tests)
cd sdk/typescript && npx vitest run

# CLI (63 tests)
cd cli && pytest

Lint & typecheck

ruff check backend/ sdk/python/ cli/
cd sdk/typescript && npx tsc --noEmit
cd dashboard && npx tsc --noEmit

Architecture

┌─────────────────────────────────────────────────────────────────────┐
│                          LLM Observatory                            │
├──────────────┬──────────────┬───────────────┬───────────────────────┤
│  Python SDK  │ TypeScript   │    Backend    │     Dashboard         │
│  (pip)       │ SDK (npm)    │   (FastAPI)   │   (React + Vite)      │
│              │              │               │                       │
│  OpenAI      │  OpenAI      │  /v1/traces   │   Overview            │
│  Anthropic   │  Anthropic   │  /v1/evals    │   Traces              │
│  LangChain   │              │  /v1/analytics│   Trace Detail        │
│  OTLP export │  OTLP export │  WebSocket    │   Evaluations         │
└──────┬───────┴──────┬───────┴───────┬───────┴───────────┬───────────┘
       │              │               │                   │
       └──────────────┴───────────────┴───────────────────┘
                                     │
                          ┌──────────┴──────────┐
                          │     PostgreSQL      │
                          │   (or SQLite)       │
                          └─────────────────────┘

License

Apache License 2.0 — see LICENSE for details.

Contributions are welcome. See CONTRIBUTING.md for setup, verification, compatibility, and privacy requirements.