Documentation extraction for Elixir, without a renderer attached.
Installation | Quick start | Configuration | Try it interactively | Artifacts | Serving
The Livebook notebooks are executable tutorials, not extra API reference pages. Start with the build-pipeline notebook in a browser:
- The Build Pipeline - Follow one build from source files to generated artifacts.
- Artifact Contract - Inspect the public JSON shapes, envelopes, manifests, and presentation indexes.
- OpenAPI Adapters - Work through the default document, shipped adapters, custom adapters, and validation errors.
- Serving Artifacts - Walk through static serving, runtime caching, reloads, gates, and controller usage.
DocShell reads module documentation, Markdown guides, Livebook notebooks, release notes, and OpenAPI documents into versioned JSON. Hosts can reuse those artifacts in a documentation site, in-product help, search index, or knowledge graph. Rendering, routing, and authorization belong to the host.
def deps do
[
{:doc_shell, "~> 0.3.0"}
]
endTwo integrations are optional and only needed if you use them:
def deps do
[
{:doc_shell, "~> 0.3.0"},
# Derive the OpenAPI document from Ash domains
{:ash_oaskit, "~> 0.4"},
# Serve artifacts over HTTP
{:plug, "~> 1.16"}
]
endRequires Elixir 1.17 or later.
Build the artifacts:
mix doc_shell.buildThat documents every module in the current application, picks up Markdown under
guides/ and notebooks under notebooks/, and writes JSON to
priv/doc_shell/. It works with no configuration at all — a project that has
set nothing still gets a complete, well-formed artifact tree.
If the host application starts dev servers, sockets, or other side effects that are not needed for static docs generation, compile and load the app spec without starting the supervision tree:
mix doc_shell.build --no-startThe same pipeline is available from code, which is what you want from a release task or when feeding a database rather than a directory:
{:ok, result} =
DocShell.Build.run(
modules: [MyApp.Accounts, MyApp.Billing],
guide_bases: ["guides", "handbook"]
)result holds :modules, :guides, :livebooks, :changelog, :openapi, and
:presentation. Source entries retain their ASTs in memory; the per-source
files omit those bodies, which are stored in content.json for entries included
in the presentation. Optional projector backlinks remain in memory only.
Extraction stops at the first error and names the module or file at fault. A guide with broken frontmatter returns an error naming its source file.
Everything lives under :doc_shell. Per-call options to DocShell.Build.run/1
win over host config, which wins over the package defaults.
config :doc_shell,
# What to document
modules: [MyApp.Accounts, MyApp.Billing],
guide_bases: ["guides"],
changelog_source: DocShell.Generate.Changelog.Sources.MarkdownFile,
changelog_options: [],
changelog_path: "CHANGELOG.md",
livebook_base: "notebooks",
collection: %{
id: "my_app",
title: "My App",
version: "1.4.0",
revision: String.duplicate("a", 40),
tree_digest: "sha256:" <> String.duplicate("b", 64),
artifact_dir: "priv/doc_shell/public",
source_url: "https://example.invalid/my_app",
edit_base_url: "https://example.invalid/my_app/edit/revision"
},
# Where it goes
public_dir: "priv/doc_shell/public",
private_dir: "priv/doc_shell/private",
# Where the API description comes from
open_api_adapter: DocShell.Generate.OpenApi.Adapters.AshOaskit,
open_api_options: [],
domains: [MyApp.Blog],
title: "My API",
api_version: "1.0.0",
security_schemes: %{},
# How it is presented
presentation_source: DocShell.Presentation.StaticGenerator,
path_builder: &MyApp.Docs.path/1,
skip_empty: true,
search_tokens: falseSet search_members: true to include module member names/arities, signatures and
parsed documentation in the containing page's search text. It defaults to false,
does not add member routes or change content ASTs, and does not override
skip_empty. Malformed member Markdown returns a module-tagged error.
Every key is optional, including the OpenAPI adapter — without one the build
emits a valid empty OpenAPI 3.1 document, so openapi.json is always there and
always parseable. changelog_source defaults to Markdown-file extraction, but a
host can replace it with a graph, database, CMS, or service adapter and pass
source-specific changelog_options. changelog_path is kept as the default
Markdown-file shortcut. DocShell.Config documents each key and its default.
DocShell reads :doc_shell and nothing else. It will not look under your
application's key, infer settings from Mix.Project, or reach into another
library's environment.
Set :collection when another documentation site will import this build. The
descriptor binds the generated corpus to a stable lowercase id, source
revision, caller-verified tree digest, and inert source/edit URLs. DocShell
does not invoke Git, fetch either URL, or infer the revision. Optional metadata
includes package, license, locale, audience, source root, and status.
With a collection descriptor, the public manifest includes an enveloped
collection.json. Its doc-shell-collection/v1 payload contains the portable
descriptor, source provenance records, per-artifact SHA-256 digests, and a
canonical aggregate content_digest. The local artifact_dir is omitted from
the portable descriptor. DocShell.Generate.Collection.load/1 validates the
manifest, files, generation IDs, paths, and digests without starting an
application or making a network request, then qualifies document IDs as
collection_id:document_id.
Collection.load/2 and load_many/2 accept finite import budgets:
max_file_bytes (32 MiB), max_total_bytes (2 GiB including the manifest),
max_artifacts (256), max_sources (100,000), max_json_depth (64), and
max_collections (256 per call). Other budgets apply per collection. Limits
are positive integers and may be explicitly raised or lowered. Byte and nesting
checks precede decoding; duplicate JSON keys are rejected. Import requires a
stable caller-owned directory, not a hostile-writer sandbox. Root symlinks,
including / and /. suffix variants, are rejected. Existing ancestors of the
working and system temporary directories are trusted; other symlink components
and parent (..) traversal are rejected.
Collection admission follows the integrity requirements in the
site specification; acceptance status
is tracked in the implementation plan.
The contract distinguishes source files from document records (several releases
can share one changelog), includes the synthetic openapi identity in uniqueness
checks, and requires a collection build to round-trip through its loader.
Custom collection projectors must preserve extracted bodies in content.json
with exact JSON types (including integer versus float);
filtered nonempty bodies must cause a build error before publication rather than
being silently restored to public output. Checksums establish internal consistency;
the caller still verifies source authenticity and owns the import directory.
Loading validates complete provenance, unique identities, source-index shapes
and recursive content. Unknown manifested source indexes referenced by provenance
use the same generic entry contract; other extension artifacts remain opaque.
Embedded index ASTs must agree with content; OpenAPI has no page AST.
Several releases may share CHANGELOG.md, while independent guides or notebooks
cannot claim the same source path. Dynamic sources should keep opaque locators
in metadata such as source_ref; source_path in a collection is a relative
filesystem path.
For a guided walkthrough, start with the build pipeline notebook. The artifact contract notebook documents every file DocShell writes, and the OpenAPI adapters notebook covers adapter selection and implementation. Use the serving artifacts notebook when wiring the runtime cache, Plug, or a host controller.
The default OpenAPI 3.1 document includes info.title and the configured
api_version as info.version (default "0.1.0").
Build.run/1 rejects malformed options and unknown per-call keys before
extraction. Unknown application environment keys remain ignored. Invalid guide
identities, titles, audience, and locale return errors naming the field and file.
Guide IDs and titles accept nonempty strings or numeric/boolean scalars.
Public and private output directories must be disjoint. The optional raw OpenAPI destination must lie outside both. Conflicting paths fail before extraction or writes; use dedicated directories without symlink aliases.
Builds stage all JSON and back up existing files before publishing. Stale
optional artifact deletion runs
under the same locks and rollback as writes. Artifact.Transaction.write/2
accepts delete: [path]; duplicate targets, directories and symlinks are rejected.
Returned publication failures restore earlier files, including deleted artifacts;
rollback failures report retained
backup paths. Cooperating builds use .doc-shell-build.lock directories. After
a process or machine crash, recover retained backups and remove stale locks
before rebuilding. Files still publish individually, so cache reloads validate
generation IDs and keep the last complete snapshot. Use dedicated output
directories without external writers or symlink aliases.
priv/doc_shell/
├── public/
│ ├── manifest.json
│ ├── navigation.json
│ ├── search-index.json
│ ├── content.json
│ ├── openapi.json
│ ├── collection.json # present when :collection is configured
│ ├── modules.json
│ ├── guides.json
│ ├── livebooks.json
│ └── changelog.json
└── private/
└── manifest.json
Every file is wrapped in a versioned envelope:
{
"schema_version": "doc-shell/v1",
"generated_at": "2026-08-05T09:12:44.000000Z",
"generation_id": "Lve95gjOVATpfV8EL5X4nx",
"data": {}
}Every file from one build carries the same opaque generation_id. Runtime
caches use it with each directory's manifest to reject a partially observed
build instead of combining artifacts from different generations.
Markdown from every source becomes the same recursive node shape, so a renderer writes one walker rather than one per source:
{"tag": "p", "attrs": {}, "content": ["text"], "meta": {}}Text nodes are bare strings; everything else is a map with all four keys always present. The AST preserves raw HTML tags, attributes, and link destinations. Renderers must allow-list tags and attributes, validate URL schemes, and escape text for their output context before rendering untrusted documents.
navigation.json, search-index.json, and content.json are the three files a
renderer actually reads. modules.json, guides.json, livebooks.json, and
changelog.json are per-source indexes for hosts that ingest documentation
rather than display it — including the entries the presentation filtered out, so
they double as a coverage report. A document's parsed body is stored once, in
content.json, keyed by the same id.
These shapes are public API. They are consumed by renderers in other repositories on their own release cadence, so a schema-version bump is a coordinated change across all of them, not a local refactor. The notebook documents every file and field.
Within doc-shell/v1, the source catalogue is additive: a producer may add a
new per-source index file to the manifest, and kind is an open string carried
through navigation and search. Generic consumers must ignore unrecognised
manifest entries and kinds; selective consumers may continue reading only the
artifacts they support. Removing an existing artifact, or removing, renaming,
or retyping an existing field, still requires a schema-version change.
Document IDs must be nonempty and unique across all sources, including entries filtered from presentation. Duplicate IDs return an error naming both sources. Overlapping guide directories extract each normalized path once.
Host projectors and changelog sources must provide complete recursive AST nodes
and JSON metadata with string keys. Invalid nested content fails validation
before output is written. DocShell.Ast.valid?/1 checks node lists.
Artifact.read/1 and read_envelope/1 accept legacy v1 envelopes without
generation_id. A present ID must be a nonempty string. Runtime caches require
an ID on every artifact and manifest to verify that they form one generation.
Individual artifact writes use exclusively created random temporary files in the destination directory, so independent BEAM instances cannot share a temporary file. A rename publishes each complete file.
For checked serialization and hashing, use DocShell.Json.Canonical.encode/1
and digest/1. They reject duplicate encoded keys and return tagged errors.
Collection.digest/1 remains a string-returning convenience for encodable input;
invalid input raises ArgumentError. Portable fixtures ship in
priv/contracts/canonical-json-v1.json. Invalid UTF-8 source files return
:invalid_utf8 errors tagged with the file by the extractors/build.
Artifact writes also contain host encoder exceptions and reject malformed or
duplicate-key JSON fragments before replacing existing files. Source extension
fields must remain native JSON even when no collection is requested.
ExDoc member metadata uses checked normalization too: colliding converted keys
return a member-tagged error instead of silently dropping a value.
Metadata preserves JSON scalars and uses UTF-8 string keys. Unsupported terms
become inspected text; improper list tails become a final array value.
DocShell.Json.normalize/1 rejects converted-key collisions. The legacy
stringify/1 keeps string keys when a collision occurs. Guides use normalize/1.
Search content preserves adjacent inline text, including words split by formatting. Block elements and line breaks add separators; image alt text is searchable. Token generation uses this same text.
Each document path is calculated once and reused by navigation and search.
Default paths percent-encode kind and ID as individual URL segments. Use a
custom path_builder when IDs intentionally represent a path hierarchy.
Five extractors, each usable on its own:
DocShell.Generate.ExDocreads compiled modules through the BEAM docs chunk used by IEx and ExDoc. It extracts English documentation and keeps metadata indicating hidden or absent module documentation.DocShell.Generate.Guidesreads Markdown, with optional YAML frontmatter for titles, audience, locale, and anything else you want carried through to the renderer.DocShell.Generate.Livebooksindexes.livemdnotebooks, so runnable onboarding docs stop being invisible to your documentation site.DocShell.Generate.Changelogloads release notes through a source adapter. The built-in Markdown source readsCHANGELOG.md; hosts can supply graph-, database-, CMS-, or service-backed adapters without changing the artifact contract. Its parser accepts git_ops parenthesised dates and Keep a Changelog hyphenated dates, with inline, reference-style, or absent version links and SemVer prereleases.DocShell.Generate.OpenApiloads an API description through a pluggable adapter.
YAML frontmatter accepts LF, CRLF, and CR line endings, including a closing
--- delimiter at end of file.
Guide and notebook titles come from the first top-level parsed H1, including Setext headings. Inline formatting is flattened, and headings inside code examples are ignored. Explicit guide frontmatter titles still take precedence. Guides with explicit titles do no title-only parse. Otherwise guide/notebook titles reuse the body AST when there are no fences; fenced input retains the defensive title parse for permissive Markdown closing-fence behavior.
Every changelog source entry must be a valid entry map; nil and other invalid
entries return {:error, {:invalid_changelog_entry, entry}}.
Changelogs are parsed as complete Markdown documents before splitting on top-level release headings. Code examples remain within their release, and reference links resolve across the whole document. Parse errors in any part of the source return a source-tagged error.
Where the OpenAPI document comes from varies too much to hard-code, so the build talks to an adapter and never to a spec library. Three ship with the package:
| Adapter | For |
|---|---|
Adapters.AshOaskit |
Ash domains, via AshOaskit |
Adapters.OpenApiSpex |
An existing OpenApiSpex module |
Adapters.RawJson |
A map, or a JSON file on disk |
Both library-backed adapters resolve their dependency at runtime, so neither library is a dependency of DocShell. Writing your own means implementing one callback — see the OpenAPI adapters notebook.
Raw and custom adapters accept OpenAPI 3.0, 3.1, and 3.2 documents without rewriting their fields. Validation checks the version and unambiguous JSON encoding; source libraries own schema validation. The default document remains OpenAPI 3.1.
Plug is an optional package dependency because web modules compile against it. AshOaskit is a development/test fixture; hosts using its runtime adapter install AshOaskit themselves. Core consumers do not resolve its dependency tree.
The artifacts are plain JSON files; serving them statically is a perfectly good answer, and none of the following is required.
When documentation is not uniformly public, or should update without a redeploy, add the cache to your supervision tree:
children = [
{DocShell.Web.Cache, dir: "priv/doc_shell/public"}
]and mount the plug:
forward "/docs/api",
to: DocShell.Web.Plug,
init_opts: [gate: &MyApp.Auth.allow_docs?/1]The :gate is where you decide who may read what — a unary function or an MFA
tuple, returning :ok or true to allow the request. DocShell has no view on
sessions, roles, or tenancy, and that callback is the whole extension point.
Hosts that would rather keep their own pipeline can call
DocShell.Web.Controller.show/2 from an ordinary controller action instead.
The serving artifacts notebook walks through static serving, runtime caching, reloads, gates, and controller usage.
A cache child specification uses its registered name as its child ID. Multiple named caches can be listed directly in one supervision tree.
Cache ETS tables permit direct concurrent reads, but only the cache process
may write them. Individual fetches are consistent; separate fetches may cross a
reload. Use Cache.snapshot/2 for a caller-owned copy of all envelopes from one
generation, including the manifest. It returns {:ok, %{generation_id: id, artifacts: envelopes}} and remains valid after reloads, at the cost of retaining
that copy in caller memory. Snapshot calls queue behind reloads.
Cache.reload(server, timeout) and snapshot(server, timeout) default to 5,000
milliseconds. They follow GenServer.call/3: timeouts/unavailable processes exit
the caller, and timing out does not cancel a queued or running reload.
Reusing the active generation ID with changed envelopes returns
{:error, {:generation_id_reused, id}} and preserves the previous snapshot.
Identical envelopes are a safe no-op; a new build needs a new generation ID.
HTTP serving caches encoded JSON and an ETag per generation. GET and HEAD
share headers; matching If-None-Match requests return 304 after authorization.
Other methods return 405 with Allow: GET, HEAD. The host retains control of
Cache-Control and Vary. Encoding happens during cache publication, not requests.
Hosts that ingest documentation into a database or knowledge graph do not need the files at all:
{:ok, result} = DocShell.Build.run(write: false, modules: [MyApp.Accounts])To serve documentation back out of that store, implement
DocShell.Presentation.GraphProjector and point the build at it:
config :doc_shell, presentation_source: MyApp.Docs.GraphProjectorPresentation IDs are nonempty and unique within navigation (including children) and search. Shared IDs must have identical paths. Local leaves/search results need content; navigation groups with children and absolute HTTP(S) links may omit it. Content can be hidden from both indexes. Backlink targets need content; origins may live in a larger host graph but cannot contradict a known path. The default generator sorts by kind, title, then ID, including tied titles.
The pipeline validates the projector's field types, recursive AST content, and metadata before writing files. Hosts use the same presentation contract for extracted files and graph-backed content.
The foundation hardening plan separates implemented behavior from outstanding acceptance work. It covers malformed-input errors, canonical digest fixtures, bounded collection loading, transactional deletion, deterministic presentation, cache snapshots and dependency qualification. The future site renderer and static exporter remain separate planned features.
mix setup # fetch and compile
mix test # run the suite
mix test.cover # run the suite with coverage
mix lint # format check, credo, dialyzer
mix check # the full quality gate
mix ci # setup + lint + coverage in one pass
mix docs # build the documentation
mix bench # run the benchmarks
mix bench.collection # measure collection preparation and complete import
scripts/notebook_smoke.py # execute tutorial cells against this checkout
scripts/consumer_smoke.sh # verify the built package in fresh consumersmix check runs formatting, --warnings-as-errors compilation, strict Credo,
documentation and typespec coverage, tests with coverage, dependency
advisories, Dialyzer, and a compile with the optional dependencies removed. CI also executes the notebooks and packaged consumer checks. CI
runs tests and coverage across Elixir 1.17 through 1.20, with the remaining
quality checks on Elixir 1.18.
The local gate disables automatic retry-only mode: every mix check runs all
configured checks unless you explicitly request a narrower run. Packaged consumer
tests unpack a Hex archive into fresh projects and perform canonical-fixture,
multi-release build/import and cache-snapshot checks for core, Plug and Ash hosts.
They never change this repository's dependency lock or release refs.
scripts/consumer_smoke.sh # locked dependency set (the CI default)
DOC_SHELL_DEPENDENCIES=unlocked scripts/consumer_smoke.sh
DOC_SHELL_DEPENDENCIES=minimum scripts/consumer_smoke.shThe selected minimum set uses EarmarkParser 1.4.8 (its earliest published 1.4 release), YamlElixir 2.11.0 and Jason 1.4.0. The Ash host uses Jason 1.4.5 because its Decimal 3 dependency is incompatible with earlier Jason versions. Optional host/transitive dependencies are solver-selected, not forcibly overridden; this is not an exhaustive combination of every historical dependency version. The old EarmarkParser emits upstream deprecation warnings on modern Elixir; no warnings are suppressed and the library itself still compiles with warnings as errors.
mix docs emits HTML, Markdown, and EPUB. The Markdown formatter is what
produces doc/llms.txt and a .md file per module for machine readers.
mix bench writes Markdown reports to bench/output/, which are published as
the Performance section of the documentation.
The collection suite measures 1,000–16,000 documents with preparation outside
timed work and reports elapsed time and BEAM allocation (not peak RSS). Its CI
smoke mode is console-only. Measurements are diagnostics, not timing assertions.
Contributions are welcome — see CONTRIBUTING.md.
AI coding tools working in this repository should read AGENTS.md. Package
consumers can ingest usage-rules.md through the Hex
usage_rules ecosystem.
MIT. See LICENSE.md.