Skip to content

Knowledge search omits nested frontmatter records from metadata indexing #60

Description

@moohancode

Knowledge search cannot find identifiers that appear only in nested frontmatter, although the parser and raw retrieval preserve those values.

Version and code path

Observed at 1d6b0e40c78d7fb1ec2dd459301876adcb499846; main still resolves to this commit when checked on 2026-09-16.

In search_knowledge.go, the metadata search field uses frontmatterSearchText(section.Frontmatter). ContextSection also preserves structured FrontmatterData, but that is not passed into this indexing path. The recursive search-text helper therefore does not see nested records.

Reproduction shape (synthetic data only)

Add this metadata to a valid document in a validated bundle, keeping the test tokens out of its filename, title, and body:

people:
  - id: person-1
    name: Example Person
    identities:
      slack:
        team_id: TTEST123
        user_id: UTEST123
      github:
        id: "123456789"
        login: ExampleLogin

Search with okn search <bundle> UTEST123 --matches --no-expand --format json, then with ExampleLogin.

Expected: the containing document is found through its metadata, with normal path/content evidence and retrieval policy.

Actual: identifiers present only in nested records do not retrieve the directory through knowledge search. Reading the original document preserves the data; this is a search-input omission rather than serialization loss.

Requested correction and validation

  • Feed structured frontmatter into knowledge-search metadata indexing, retaining flat metadata behavior where applicable.
  • Exercise arrays, nested maps, flat scalars, and metadata-only matches through the actual SearchKnowledgeWithVersion path.
  • Verify CLI/MCP behavior and preserve access, trust, freshness, evidence, and snapshot boundaries.

A downstream local patch has exercised nested account searches and a regression test named TestKnowledgeSearchIndexesNestedFrontmatter. This report does not claim an upstream fix or release.

Related retrieval improvement

After locating a structured directory, whole-document retrieval returns the full array when the consumer needs one record. A bounded exact-record selection operation within the existing query surface would help: document path + collection pointer + exact predicates, returning matching rows with document digest and row pointer. Multiple matches should remain multiple results; no match should remain empty. Row/byte limits, policy checks, and generation consistency should be preserved. This enhancement can be split from the indexing bug if preferred.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions