Skip to content

feat(tools): bound list_directory output; root call is a folders-only overview - #513

Merged
Leo310 merged 4 commits into
mainfrom
feat/list-directory-bounded
Sep 19, 2026
Merged

Leo310 merged 4 commits into
mainfrom
feat/list-directory-bounded

Conversation

@Leo310

@Leo310 Leo310 commented Sep 19, 2026

Copy link
Copy Markdown
Member

What

list_directory had no output cap: it walked every vault file at depth 3 with files included and returned a JSON tree with name, extension and size per entry, so one orientation call in a large vault could fill tens of thousands of tokens of context — and the explore-vault skill told the model to make that call first.

  • Root call is an overview. With no path it returns folders with recursive file counts, 2 levels deep, no file names — a few hundred tokens in any vault. With a path it lists that folder's files (1 level deep by default). includeFiles / maxDepth override both defaults.
  • Bounded, structurally. A context-derived budget (5 % of the window, capped at 40k chars); when exceeded the listing shrinks via tighter per-folder caps, then one level shallower — always a well-formed tree, never cut mid-JSON. moreFiles / moreFolders mark what was omitted, note tells the model how to see it (subfolder path, or search_notes / grep_notes).
  • Compact shape. Files are bare names, sizes dropped, every folder carries fileCount. recursive and includeFolders params removed (maxDepth alone expresses the former).
  • Description pinned to the shipped default (as read_content / search_notes already do) so existing agents aren't stuck describing the old shape.
  • explore-vault 1.1 → 1.2: tags and properties first; list the root only when the layout matters; never walk the tree to find a note. 1.1 retained in src/skills/history/ per the shipped-skill protocol.
  • Chat tool card updated for the new payload (counts, "+N more" rows, no size column).
  • Dataview, when available, takes the structural questions. dataview 1.1 → 1.2: its description (the text in the always-visible skills block) now names the trigger questions — a category of notes, a filter by tag/property/date, counts and groupings — and a new "When to reach for it" section draws the line (structure yes, prose no; verify names first; combine with read_content). explore-vault step 1 loads it first when listed, step 6 prefers a GROUP BY query over gather-then-execute_javascript. Both mentions are conditional, so a vault without Dataview reads one inert line. 1.1 body retained in history.

How I tested it

bun run check, format, lint, test (1863 passing, incl. new budget-collapse tests and the shipped-skill history guard); madge reports no cycles. Built once in slot wt3; not yet exercised in a live vault — that is Leo's call.

AI assistance: Claude Code implemented it from Leo's brief ("the agent lists the vault so often that large vaults fill the context — how to avoid it"); Leo reviewed the approach and will live-test in his vault.

Checklist

  • bun run check, bun run format, bun run lint, and bun run test pass locally
  • I tried the change in a real Obsidian vault (or explained above why that isn't applicable)
  • I read CONTRIBUTING.md, including the section on AI assistance
  • If this adds a provider, a bundled skill, a built-in tool, or changes manifest.json: I noted that the docs site needs updating (see "Documentation" in CONTRIBUTING.md)

… overview

`list_directory` was the one read tool with no output cap: it walked every vault file, defaulted
to depth 3 with files included, and returned a JSON tree carrying name, extension and size per
file. In a vault with thousands of notes a single orientation call could cost tens of thousands
of tokens — and the explore-vault skill told the model to make that call first.

Now:
- Root call (no path) is an overview: folders with recursive file counts, 2 levels deep, no file
  names. It stays a few hundred tokens in any vault. Inside a folder, files are listed (1 level
  deep by default). `includeFiles` / `maxDepth` override either.
- Output is bounded by a context-derived budget (5% of the window, capped at 40k chars) and
  shrinks structurally, never mid-JSON: tighter per-folder caps first, then one level shallower.
  `moreFiles` / `moreFolders` say what was left out, `note` says how to get it.
- Compact shape: files are bare names, sizes dropped, every folder carries `fileCount`.
- `recursive` / `includeFolders` parameters removed (`maxDepth` alone expresses the former).
- Tool description always uses the shipped default rather than a persisted copy, matching
  read_content / search_notes — descriptions aren't user-editable.
- explore-vault 1.1 → 1.2: tags and properties first; list the root only when layout matters;
  never walk the tree to find a note. 1.1 body retained in history per the shipped-skill protocol.
- Chat tool card renders the new shape (counts, "+N more" rows, no size column).

Co-Authored-By: Claude <noreply@anthropic.com>
@greptile-apps

greptile-apps Bot commented Sep 19, 2026

Copy link
Copy Markdown
Contributor

RetriggerConfidence Score: 5/5

The PR appears safe to merge; the previously reported budget, collapse-signaling, and aggregation issues are resolved.

Summary

The PR redesigns list_directory to provide compact root overviews and structurally bounded folder listings, updates the chat renderer for the new payload, and revises the shipped exploration and Dataview guidance.

  • Root listings now show folders and recursive file counts without file names by default.
  • Oversized results collapse through entry caps and reduced depth while preserving valid JSON and omission metadata.
  • Directory cards display counts and omitted-entry indicators.
  • Shipped skills steer structural queries toward Dataview and retain prior skill versions in history.
  • The changes since the previous review exempt aggregate Dataview queries from the display limit, resolving the remaining incomplete-aggregation concern.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[list_directory call] --> B{Path supplied?}
    B -- No --> C[Root overview: folders and counts]
    B -- Yes --> D[Folder listing: files and subfolders]
    C --> E[Render requested depth]
    D --> E
    E --> F{Serialized result within budget?}
    F -- Yes --> G[Return complete JSON tree]
    F -- No --> H[Tighten per-folder caps]
    H --> I{Within budget?}
    I -- Yes --> J[Return tree with omission markers and note]
    I -- No --> K[Reduce depth]
    K --> E
Loading

Reviews (4) · Last reviewed commit: "fix(skills): exempt Dataview aggregation..."

Comment thread src/agent/tools/listDirectory.ts
Comment thread src/agent/tools/listDirectory.ts Outdated
Comment thread src/components/chat/toolOutputRenderModel.ts
Leo310 and others added 2 commits September 19, 2026 20:00
…pse in overviews

Review follow-ups on #513:
- The collapse ladder could fall through with an over-budget listing when even twenty
  top-level folder names were too long for the floor budget. Two last-resort rungs, tried
  only at depth 1, now end at "counts only" — a few hundred characters for any vault.
- A folders-only overview that had to drop folders produced `moreFolders` without a `note`,
  because collapse detection was gated on `includeFiles`. Omitted folders now always count as
  a collapse; omitted files count only when file names were requested.

Co-Authored-By: Claude <noreply@anthropic.com>
…when it is available

The explore-vault procedure sent every "which notes…" question through tags, properties, a broad
search and several reads. Where the Dataview integration is enabled, that is one DQL query
returning a compact table — far cheaper in context and more accurate for counts.

- dataview 1.1 → 1.2: the description (what the model sees in the always-visible skills block)
  now names the trigger questions — a category of notes, a filter by tag/property/date, counts
  and groupings — and says to prefer it over search for structural questions. A new "When to
  reach for it" section draws the line: structure yes, prose no; verify names first; combine
  with read_content for the few notes that matter. 1.1 body retained in history.
- explore-vault (still 1.2, unreleased): step 1 loads the dataview skill first when it is
  listed; step 6 prefers a GROUP BY / sum query over gather-then-execute_javascript.

Both mentions are conditional on the skill being listed, so a vault without Dataview reads one
inert line.

Co-Authored-By: Claude <noreply@anthropic.com>
Comment thread src/skills/defaults/explore-vault/SKILL.md Outdated
…fault

A GROUP BY, count or sum is only correct over the whole result set; the display limit applies
to LIST / TABLE output shown to the user. Review follow-up on #513.

Co-Authored-By: Claude <noreply@anthropic.com>
@Leo310
Leo310 merged commit f71db2b into main Sep 19, 2026
3 checks passed
@Leo310
Leo310 deleted the feat/list-directory-bounded branch September 19, 2026 18:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant