Sync Python SDK with Node SDK 2.2.0 #34

Summary

Bring the Python SDK to Node SDK 2.2.0 HTTP/type parity, then port the storage-first knowledge layer.

main already has v2 page-memory (#31) and BYOK llm_config (#33). That is Node 2.0.0 core + Node 2.2.0 BYOK. It does not yet cover Node 0.7.0–0.8.0 HTTP extras that still ship in 2.2.0 (document metadata, planned document_id on create, auth token provider) or Node 2.1.x knowledge/parsed-storage.

Already on main — do not redo

• v2-only jobs / documents / retrieval (Add v2 page-memory schema support #31)
• BYOK llm_config on parse, job create, and retrieval query (feat: support llm_config BYOK on jobs and retrieval #33)
• retrieval.query omits use_agentic when None (server map-nav default). Node 2.2.0’s omit change is knowledge.search, not HTTP retrieval.
• asset_url / include_asset_urls on document chunks
• Release PR for 2.0.0 is open: release: 2.0.0 #32

Contract

Port from Node main at tag v2.2.0. Python stays snake_case on the wire and in public APIs (document_metadata, not documentMetadata).

Node sources of truth:

• Jobs + metadata: src/resources/jobs.ts, src/lib/document-metadata.ts, src/types/params.ts, src/types/job.ts
• Documents + page citation: src/resources/documents.ts, src/types/document.ts, src/types/page-citation-assets.ts
• Auth: src/client.ts, src/lib/http-client.ts, src/types/client.ts
• Knowledge / storage: src/knowledge/knowledge.ts, src/knowledge/types.ts, src/types/storage.ts, src/storage/parsed-document-storage.ts

P0 — HTTP / type parity

Ship as 2.1.0 after publishing 2.0.0 (see rollout). Split into the PRs below so each is reviewable.

PR A — Publish 2.0.0

No code. Merge #32 so PyPI matches main (v2 + BYOK). Do not wait for the rest of P0.

PR B — Document metadata + telemetry + planned document_id

Request

• Add document_metadata: Optional[Dict[str, Any]] on jobs.create (sync + async) and forward it from Knowhere.parse / AsyncKnowhere.parse.
• Always send metadata. Merge official defaults under caller keys:

{"created_by_client": "python-sdk", "client_version": __version__, **(document_metadata or {})}

Caller keys win. Defaults only fill missing keys (Node mergeDocumentMetadataDefaults).

Response

• Add document_id: Optional[str] = None on Job (create response). JobResult already has it.
• Add document_metadata: Optional[Dict[str, Any]] = None on Document.
• Flip tests that currently assert create responses drop document_id (tests/test_jobs.py, tests/test_models.py).

Files

• src/knowhere/types/params.py — DocumentMetadata alias
• src/knowhere/lib/document_metadata.py — defaults + merge helper (mirror Node)
• src/knowhere/types/job.py, src/knowhere/types/document.py
• src/knowhere/resources/jobs.py, src/knowhere/_client.py
• src/knowhere/__init__.py — export helper + type
• tests/test_jobs.py, tests/test_documents.py, tests/test_models.py, tests/test_parse.py
• docs/usage.md, README.md

Acceptance

• Create with no metadata → body includes created_by_client=python-sdk and client_version.
• Create with {created_by_client: "cli"} → that key is "cli"; client_version still defaulted.
• Create response with document_id parses onto Job.document_id.
• documents.get / list parse document_metadata when present.

PR C — Page citation source + typed assets

HTTP

• Documents.get_page_citation_source(document_id) -> DocumentPageCitationSource
• GET /v2/documents/{document_id}/files/page-citation-source
• Sync + async.

Types (match Node; snake_case)

class DocumentPageCitationSource(BaseModel):
    document_id: str
    namespace: Optional[str] = None
    job_id: Optional[str] = None
    job_result_id: Optional[str] = None
    variant: Optional[str] = None
    file_name: str
    content_type: str
    url: str
    expires_at: Optional[datetime] = None

class PageCitationAsset(BaseModel):
    page_num: int
    artifact_ref: str
    asset_url: Optional[str] = None
    content_type: Literal["image/png", "image/jpeg"]
    width: Optional[int] = None
    height: Optional[int] = None
    source: Literal["knowhere-rendered-page-citation-source"]

Chunk metadata stays Dict[str, Any]. Export PageCitationAsset so callers can parse chunk.metadata["pageAssets"] (Node pageCitationAssetsMetadataKey). Do not invent a second generation path; server descriptors only.

Files

• src/knowhere/types/document.py, src/knowhere/types/page_citation.py (or colocated)
• src/knowhere/resources/documents.py
• src/knowhere/__init__.py
• tests/test_documents.py
• docs/usage.md

Acceptance

• Helper hits the path above and casts the signed-URL payload.
• 404 from the API surfaces as the existing NotFoundError.
• Typed asset model round-trips a Node-shaped descriptor.

PR D — auth_token_provider

Mirror Node: exactly one of api_key (arg or KNOWHERE_API_KEY) or auth_token_provider. If api_key is set, ignore the provider.

• Sync Knowhere: auth_token_provider: Optional[Callable[[], str]] = None
• Async AsyncKnowhere: Optional[Callable[[], Union[str, Awaitable[str]]]] = None
• Call the provider per request when building Authorization: Bearer …. Empty/None token → ValidationError.
• Current constructor still requires a static key today (src/knowhere/_base_client.py). Change that guard.

Files

• src/knowhere/_base_client.py, src/knowhere/_client.py if signatures need documenting
• tests/test_client.py (and retry tests if they assume a frozen Authorization header)
• docs/usage.md Authentication section

Acceptance

• Knowhere(auth_token_provider=lambda: "jwt") authenticates; no api_key required.
• Provider is invoked again on a second request (short-lived tokens).
• Empty provider return raises ValidationError.
• api_key="sk_…" plus a provider uses the static key.

P1 — Knowledge / parsed-storage (Node 2.1.1–2.2.0)

New client.knowledge module. Contract is Node 2.1.2+ (result-relative objects, not old paged snapshots). MCP stays Node-only.

Use these Python names (Node in parentheses):

Storage

• Commit expanded result files: manifest.json, chunks.json, optional sidecars, assets, plus a commit marker.
• Adapter is result-relative (read_object / write_object / optional head_object, get_object_url) per Node ParsedDocumentStorage.
• After validating the commit marker, read committed objects concurrently (Node 2.1.3).
• Disk helper for a cache directory, same as Node’s MCP-oriented disk storage, but no MCP package in this repo.

Reads

• Storage-first read_chunks / grep_chunks / get_document_outline with remote fallback.
• grep_chunks copies source-chunk page_numbers onto matches (Node 2.2.0).
• search omits use_agentic when unset so the API map-nav default applies; False forces classic. Do not change HTTP retrieval.query (already correct).

Acceptance

• Local parse → outline / read / grep against committed files without a network chunk list.
• Missing storage object falls back to remote document chunks and returns remote asset_urls.
• Grep match includes page_numbers when the source chunk has them.
• search(query=...) body has no use_agentic; search(..., use_agentic=False) sends false.

Implement against Node src/knowledge/ + src/storage/. Prefer a follow-up design note in the PR if a Python storage adapter must differ (sync vs async).

Non-goals

• Reintroducing /v1 or an API version switcher.
• Matching Node package version numbers 1:1 (Python 2.0.0 / 2.1.0 vs Node 2.2.0 is fine).
• Porting @ontos-ai/knowhere-mcp / packages/mcp into this repo.
• Regenerating page-citation assets in the SDK (Node already stopped doing that).
• Changing retrieval.query use_agentic omit behavior.

Suggested rollout

1. Merge release: 2.0.0 #32 — publish Python 2.0.0 (v2 + BYOK).
2. P0 PRs B → C → D (or B+C together if small) — release 2.1.0.
3. P1 knowledge + parsed storage — release 2.2.0, using Node 2.1.2+ as the storage contract.

Each PR: tests + docs/usage.md / README.md / examples when public API changes. Follow CONTRIBUTING.md (ruff, mypy, pytest).

References

• Node changelog 2.0.0–2.2.0: https://github.com/Ontos-AI/knowhere-node-sdk/blob/main/CHANGELOG.md
• Node 2.2.0: BYOK llmConfig (already in Python feat: support llm_config BYOK on jobs and retrieval #33), grep pageNumbers, omit unset useAgentic in knowledge.search, created_by_client / client_version
• Python main: Add v2 page-memory schema support #31 v2 page-memory, feat: support llm_config BYOK on jobs and retrieval #33 BYOK llm_config, release: 2.0.0 #32 unpublished 2.0.0 release