A compact, deterministic, vendor-neutral fingerprint format for bounded-size summaries of log streams.
Status: Draft — the 0.x line. SPEC.md declares the current version on its
first line and is normative; CHANGELOG.md is the dated release history.
Editor: the InSight project (reference implementation: insight-metalog)
License: Spec text — CC-BY-4.0. Reference schemas — MIT.
A MetaLog is a bounded-size statistical and structural fingerprint of a window of log behaviour. It answers a single question:
What was this log stream doing in the last N minutes — in a document bounded by the caps it declares, not by how many lines the stream carried?
A MetaLog is not a log, not a metric, and not an alert. It is a new primitive that sits between raw logs (high entropy, low signal, expensive to store) and pre-defined metrics (low entropy, lossy, defined ahead of time):
┌──────────────────┐ ┌──────────────────┐ ┌──────────────────┐
│ Raw logs │ │ MetaLog │ │ Metrics │
│ (GB / hour) │ ─▶ │ (KB / window) │ ─▶ │ (B / sample) │
│ high entropy │ │ defined by │ │ defined ahead │
│ defined nowhere │ │ the data itself │ │ of time │
└──────────────────┘ └──────────────────┘ └──────────────────┘
observable compressible queryable
A MetaLog is derivable from raw logs (one-way), composable across sources and windows, diff-able between consecutive windows, and small enough to feed to humans, dashboards, alerting rules, and LLMs without further reduction.
The observability industry stores logs at $0.50–$3 per GB ingested and queries them with full-text search. This works at small scale and collapses at large scale: companies routinely drop logs after 7 days because retention is unaffordable, then have no incident forensics.
The industry response has been more storage, more indexes, more
dashboards. The MetaLog response is compress the meaning, then store
that. A MetaLog is bounded by the structure a window contains —
its top templates, its salient tail, its n-grams, its branching, its
cube cells — each capped by a size the producer declares in the
document itself (§11.1) — and not by how many lines the window
carried. §11 makes that bound computable rather than quoted: §11.3's
formula prices a document from the caps it declares, and a document
that also carries behavior or cube pays for those blocks by the same
arithmetic. No size is published per line count, because a document's
size is set by its caps and not by its line count (§11.5).
That is the value proposition: the artifact stops growing where the
stream does not.
That is a bound, not a measured ratio, and no ratio is quoted anywhere in this spec. A compression ratio is a measurement, and it means nothing without the population it was measured on. This spec has none to cite — which is exactly why §11.5 asks producers to report their actual
envelope_bytes, the only exact figure. An implementation that publishes a ratio should publish the corpus, the configuration and the run with it. This spec learned that at its own expense: until 0.10.0 it published a size target — ≤ 4 KB per MetaLog covering ≥ 1 M log lines — that no implementation met, and it withdrew it (adr/0006).
For this primitive to be useful across vendors — for an SRE to be able to switch their log analyzer without re-training their dashboards, alerts, and LLM prompts — the format itself must be open, versioned, and vendor-neutral.
That is what this spec is.
| File | Purpose |
|---|---|
SPEC.md |
The normative specification. |
RATIONALE.md |
Design decisions and the alternatives that were rejected. |
schema/metalog.v0.schema.json |
JSON Schema for the v0 MetaLog envelope. |
schema/metalog.v0.example.json |
Worked example MetaLog. |
conformance/ |
§8 clause 1, made runnable — validate your documents against the shipped schemas. |
CHANGELOG.md |
Versioned history of the spec. |
GOVERNANCE.md |
How the spec evolves; who can propose changes. |
adr/0001-v1-freeze-policy.md |
Conditions for freezing the first stable MetaLog version. |
This is a draft — v0.x — published to reserve the term, the schema, and the design intent before incumbents publish a proprietary equivalent. The format will change in incompatible ways before v1.
The first stable version (v1.0) will land alongside the InSight
reference implementation when the insight-metalog producer and
downstream consumers have enough compatibility evidence
to freeze the schema. After v1.0, breaking changes follow semver and
GOVERNANCE.md.
§8 says "there is no central conformance authority — the schema is the test".
conformance/ is that test, shipped with the standard so
you can run it yourself:
pip install -r conformance/requirements.txt
python3 conformance/metalog_validate.py --selftest # prove the tool has teeth
python3 conformance/metalog_validate.py my_metalogs.jsonl # judge your documents
It reports schema violations separately from members that an open container permits but no schema describes — the second is legal, and conflating the two misprices both. It covers §8 clause 1 only; clauses 2–4 are not tested, and the tool says so on every run.
| Implementation | Language | License | Status |
|---|---|---|---|
| insight-metalog (repo) | C++23 | CodeRoast-owned package | Declares metalog_version 0.10.0 — producer, compose, and diff. It declares diff_version 0.6.0 on the diffs it publishes, and that is non-conformant as of 0.10.0 (adr/0004, §13.1.1): measured 2026-09-02 over the determinism evidence linked below, 9 of 9 published MetaLogDiff documents declare a version four MINOR releases older than the comparison_outcome member they carry. GOVERNANCE §3 resolves it in the spec's favour; the producer repoints. Its published documents meet §8 clause 1 — measured 2026-08-22 with conformance/ over both published surfaces: the determinism evidence (17 documents) and the two published diff documents — zero schema violations, zero undescribed members. The member that failed here before (stats.top_k[].ordinal_histograms, bare in a closed object) now rides in stats.top_k[].extensions under a reverse-DNS key, per §7: its bins ride an unfrozen ladder, so it stays vendor data until an RFC freezes the ladder. The two members the producer emitted that no spec text described — run_outcome (MetaLog root) and reservoir_delta (MetaLogDiff root) — are described as of 0.9.0 (§2.5 and §13.7), the adoption arm of adr/0003 (Accepted, 2026-08-22); neither had reached a published surface. One divergence remains and the spec wins it (GOVERNANCE.md §3): §2.5's enum is lower-case, and this producer currently spells run_outcome in upper case, so it is non-conformant on that member until it normalises — a producer-side fix affecting no published byte, since the member has never been published. The two vendor-shaped members that ADR relocates — ruleset and service_edge_delta — now ride in extensions under reverse-DNS keys. The diff-root container that carries the second of them is granted in §7's placement table as of 0.9.0, drafted on main and unreleased at the time of writing (GOVERNANCE.md §7): a consumer pinned to the 0.8.0 Release still reads that container as legal-but-undescribed. |
| Your implementation here | — | — | PRs welcome |
The spec is deliberately implementation-agnostic. Any language that can serialise the JSON envelope and compute the required statistics can produce conformant MetaLogs.
To avoid scope creep and to keep the spec implementable, the following are explicitly out of scope for MetaLog:
- Log storage or retention. MetaLog says nothing about where raw logs live. It is a derived artifact.
- Log routing or transport. Use Vector, Fluent Bit, OTel Collector, etc. MetaLog is what you produce from a log stream, not how you move logs around.
- Alerting policy. A MetaLog enables alerting; it does not prescribe alerts. Wire your own rules on top of MetaLog fields.
- Querying raw logs. A MetaLog is lossy by design. If you need the original log lines, keep them; MetaLog is not a replacement index.
- Anomaly detection algorithms. A MetaLog is the input to a detector. The detector itself is not part of this spec.
- A wire protocol. MetaLogs are JSON documents. Move them with HTTP, Kafka, files, object storage, or any transport that fits your system.
This spec is in the reserve-the-term phase. The most useful contributions right now are:
- Implement the v0 envelope in your favourite language and open a PR adding it to the reference implementations table.
- Open issues with concrete log streams where the v0 schema does not capture something you'd want a MetaLog to express.
- Argue with
RATIONALE.md. Every rejected alternative is documented; if you think one was wrong, say so on the issue tracker with a specific scenario.
Process and review rules live in GOVERNANCE.md.
"MetaLog" is used here as a generic technical term for the artifact described by this spec. Any vendor may produce or consume spec-conformant MetaLogs and describe their tool as "MetaLog producer" or "MetaLog consumer". The spec is permissively licensed precisely so this term can become standard usage.