diff --git a/AUDIT.md b/AUDIT.md new file mode 100644 index 00000000..9ed6a970 --- /dev/null +++ b/AUDIT.md @@ -0,0 +1,226 @@ +# Improvement audit ledger + +Baseline: `5ca412d5d2ba8e0db4d1e14d361f3443b86df90c` on `main`, verified against GitHub on 2026-10-04. Work branch: `codex/audit-safety`. + +The original 36 findings below came from a source audit, not live exploitation. Their IDs are preserved; later confirmed findings are appended with new IDs. An open item remains a hypothesis until independently investigated. This ledger records evidence, dispositions, checks, and the next bounded work so later batches do not repeat or conflict with earlier work. + +## Boundaries and coordination + +- One separate draft PR; no merge, deployment, new service, credential, billing, hosting, or system/network change. +- The existing `/Users/yoginth/rag.computer` checkout is on `yoginth/remove-dead-code` with substantial uncommitted Elixir migration work. It was inspected read-only. All audit changes use a separate clone of the verified baseline; none of that work is copied or modified. +- No `.agents` directory or nested `AGENTS.md` exists in the baseline checkout. Root `AGENTS.md` and `STYLEGUIDE.md` govern this work. Removed test runners and coverage suites stay removed. Behavioral verification uses temporary local smoke scripts with mock transports and fake credentials outside the repository. +- Tenant findings 4–7, related metadata propagation, and replay authorization need a coherent design before schema, reindex, or provisioning work. Tenant provisioning stays unchanged. Replay implementation is held until explicit policy approval is forwarded; tenant quarantine/redesign requires its own coordination. +- Uncommitted batches are backed up outside this checkout before commit. Each useful batch requires relevant checks and independent review. Do not create changes merely to fill an hourly schedule. +- Parent continuation requires approximately one combined commit per hour, anchored to the latest publication. Journal is running in its separate coordinator; this work must not start or modify Journal. The parent handles continuation scheduling. + +## Findings + +Paths below are relative to `api/bigrag/` unless another package is named. + +| ID | Finding and reported source | Disposition / evidence | +|---|---|---| +| 1 | Saved chat credential can reach a caller-selected endpoint; `services/chat/turn/credentials.py`, `services/chat/provider.py` | Fixed in batch 1. Mock reproduction accepted a different endpoint at baseline. Save-time verification now records a normalized destination authenticated with a versioned HMAC using the existing provider key. Chat and suggestions reject mismatches, forged bindings, and legacy unbound keys. Existing keys need to be saved again. No schema or new secret. | +| 2 | Idempotency replay precedes current auth; `middleware/idempotency.py`, `middleware/principal.py` | Independently confirmed; open. Both replay paths bypass dependencies/handler authorization. Cookie/bearer selection also differs from auth. A middleware login check alone misses admin, scope, collection-body, tenant, and ownership guards. Next work requires reusable read-only authorization guards, fresh selected-credential validation, and safe handling of legacy cache entries; denials must never execute mutations again. | +| 3 | Document metadata omitted from chunk metadata; `services/queue_embedding/chunk_metadata.py`, `insert_batches.py` | Open; coordinate with tenant metadata design before changing stored vector metadata. | +| 4 | Tenant API-key field absent from API models/admin key creation despite auth reading it; `models/auth.py`, admin API-key routes | Open; tenant workstream, no provisioning or schema change authorized by this ledger. | +| 5 | Suggestions and analytics omit tenant guards; `services/chat/questions/api.py`, `routers/analytics.py` | Open; tenant workstream. Batch 1 only fixes session concurrency, not isolation. | +| 6 | Vector upsert/delete and collection truncate miss tenant safeguards | Open; tenant workstream, validate ownership and collection boundaries together. | +| 7 | Upload sessions owned only by issuing user | Open; tenant workstream, review key/tenant ownership together. | +| 8 | Docker context excludes SDK required by UI `COPY` | Existing PR [#77](https://github.com/bigint/rag.computer/pull/77), head `c121513f3f8fd2192f28aefb88191ae69e5b632f`. Read-only review: its targeted exclusion retains Python exclusion while making TypeScript available. Do not duplicate or merge; Docker runtime verification not performed here. | +| 9 | Ready document/source cleanup precedes fallible followups and may lead to destructive retry; queue processing/finalize | Independently confirmed; open. Actual orchestration with in-memory dependencies commits ready then pending/failed after transient success-followup errors, deletes vectors/source, and exhausts retry on missing source. Persistent Redis failure can interrupt compensation and retain ready; ordinary asynchronous EventBus errors do not trigger the downgrade. Repair needs a durable success boundary, safe followup reconciliation, and ready-row redelivery guards. | +| 10 | Failed-document dedup prevents retry | Independently confirmed; open. Single/batch/open-session uploads reuse failed hash matches without ingestion; the oldest failed batch match can shadow a newer ready row. Collection/tenant predicates remain effective in the synthetic checks. Define a shared status-aware reuse/retry policy with freshly staged bytes and guarded ownership before changing IDs or compensation. | +| 11 | Lookup-before-insert hash dedup race with nonunique index | Independently confirmed in source and controlled in-memory flows; open. Single, batch and session uploads can both miss the same scoped hash then stage/enqueue separate documents; repeats within one batch remain deduped. The document hash index is nonunique. This is not PostgreSQL concurrency proof. Shared status-aware dedup ownership must preserve tenant boundaries before locks or constraints are chosen; no schema change. | +| 12 | Upload limits use unlocked stale counts | Partially corrected in published batch 6: failed retry bytes were excluded from the session sum and then subtracted again. Only previously counted existing bytes are now subtracted. Independently reproduced over-cap acceptance and preserved exact-cap/new-file behavior. Snapshot admission races for concurrent distinct items remain open: controlled actual-flow interleavings admit two items into one slot or 1,400,000 logical bytes into a 1 MiB cap. Same-client-ID uniqueness prevents duplicate identities but does not reserve capacity. No PostgreSQL concurrency proof, reservation, lock protocol or schema change. | +| 13 | Queue recovery ignores `lrem` result and sets lease unconditionally | Independently confirmed; open. Ten fake-Redis checks reproduce duplicate recovery, stale TTL decisions, admission gaps, stale renewal/cleanup affecting a replacement lease, and repeated admission release. Checking LREM alone is insufficient; atomic token ownership and compare-token renewal/recovery/finalization need a coherent protocol. No live Redis or queue change. | +| 14 | Mixed text/scanned PDFs skip OCR | Independently reviewed; open. A generated digital/raster/blank PDF plus actual installed PDFium extraction and direct conversion shortcut returns only digital text with OCR enabled. Actual queue branch bodies skip OCR on injected nonempty parse results. Native Docling/OCR recognition and element output remain untested; page coverage, blank pages, ordering, cost and cancellation need a reviewed design. | +| 15 | ProcessPool timeout leaves conversion process running | Independently confirmed; open. Actual wrapper with a synthetic local child times out and releases its semaphore slot while the child keeps running; shutdown waits for completion. Process, admission and staged-file ownership must be designed together. No model or service was invoked. | +| 16 | Whole-document embeddings retained by gather | Independently confirmed; open. Actual orchestration retains all synthetic batch vectors before insertion, schedules every batch despite bounded active concurrency, and propagates a permanent error with seven siblings still pending. Production finalizers do not cancel/join them. Evidence counts retained objects/control flow, not RSS; bounded insertion and joined cancellation must preserve ownership/retry semantics. | +| 17 | SDK exception retries ignore safety flag; Python `_core.py`, TypeScript `core.ts` | Fixed in batch 1. Python mock transport reproduced three unsafe attempts at baseline; exception retries now honor the flag in both SDKs. Timeouts fail immediately, following the style guide. Reads and keyed mutations retain connection-error retries. Multipart/large-body replay limitations remain under 18. | +| 18 | Idempotency skips multipart/large bodies and has a 60-second unrenewed lease | Independently confirmed; open, coordinate with 2. Thirty-nine local mocked SDK/middleware checks show keyed multipart/large JSON retries exceed server replay coverage; lease expiry and stale-owner deletion permit repeated execution, large successful responses lose their completion marker, and unknown-length bodies over 8 MiB can lose an unread tail. These are source/mock claims, not live protocol or concurrency proof. SDK timeout/unkeyed safety from 17 remains effective. Server lease/replay changes remain held; no implementation in this batch. | +| 19 | SSE heartbeat `wait_for(__anext__)` cancels slow provider iterator; `services/chat/completion.py` | Fixed in batch 1. Baseline slow-token smoke returned an empty successful assistant. One pending token task survives heartbeat ticks. Cleanup is shielded from ASGI cancellation, and the raw provider response closes even when disconnected immediately after a delta. | +| 20 | Stale chat finally/collection switch clears newer state; `app/src/features/chat/use-chat-streaming.ts` | Fixed in batch 2. Store-owned controller identity guards start, updates, stop, and finish. Same-collection selection preserves the stream; collection changes, clear, unmount, and account reset abort it. Late completion cannot clear a newer response. | +| 21 | EOF without terminal event appears successful; `app/src/lib/chat-stream.ts` | Fixed in batch 2. Truncated mock SSE reproduced success at baseline. Parser requires done/error/[DONE], handles CRLF and terminal tails, cancels the reader, and ignores post-terminal frames. Partial answers remain visible beside an interruption error. | +| 22 | Password change revokes session then UI logout fails | Fixed in batch 2. Independent review confirmed logout authenticates after password revokes every session. Successful password change now clears the cookie and local session directly, with guarded navigation and no extra logout call. Incorrect-current-password 401 preserves the session. | +| 23 | Logout leaves chat/upload Zustand stores | Fixed in batch 2. Confirmed transitions replace/remount the QueryClient provider, abort generation-owned auth/upload requests and chat, reset stores, and seed the new session. Upload persistence is owner-bound; unowned v1 IDs are discarded. Generation guards prevent late callbacks from restoring tracking. Focus/reconnect and active-tab minute refresh detect confirmed expiry/account changes. | +| 24 | Upload UI ignores returned `item.status=failed` | Fixed in batch 3. Baseline mock returned a failed item under HTTP 201 while the UI reported it queued. File responses now record failed/canceled items, final counts drive warnings, and issue panels remain visible. Failed validation rows keep their status during persistence. | +| 25 | Closed incomplete upload sessions remain uploading | Fixed in batch 3. Baseline closed sessions with missing files remained preparing/uploading. Missing files after closure now fail once accepted work is inactive; active work stays ingesting. New file requests reject closed sessions. The UI closes after all attempts, including all failed requests, and recognizes cancellation conflicts. | +| 26 | Allowed file types and empty-file semantics inconsistent | Partially corrected in published batch 6: Studio cannot save an empty draft as a restricted selection; all-selected retains the existing unrestricted [] contract. Labels/docs scope the filter to Studio. Direct API and upload-session extensionless-file validation differences remain open; no server policy or ingest behavior change. | +| 27 | Personal-key-only UI gate blocks instance chat credentials | Corrected in published batch 8. Session-only computed readiness shares saved/instance selection and destination guards; editable preference presence is not authority. Browser send, generation and retry gates use current account-owned readiness, with checking/blocked/unavailable states, key-change refetch and cancellation/late-callback guards. Personal-key controls remain separate and Stop/history remain available. Local configuration readiness does not promise provider, model, retrieval or collection authorization. Independent backend and UI reviews passed; published as `61660732`, with all six exact-head CI jobs passed. | +| 28 | Analytics gather shares AsyncSession; `services/analytics.py` | Fixed in batch 1. Mock session reproduced overlapping execute calls at baseline. Period queries now run sequentially in one session, preserving output and five-minute cache. | +| 29 | NDCG duplicates document hits and can exceed 1 | Fixed in batch 4. Actual baseline helper scored three hits for one relevant document as 2.1309297535714578. Each relevant ID now earns credit only once at its original rank; hit_ids, recall, MRR, and unique rankings stay unchanged. | +| 30 | ASCII/length tokenizer drops Unicode and short terms | Independently confirmed; open. Synthetic tokenizer/provider-payload/scorer checks drop single-character and Chinese/Cyrillic terms; accented words split or truncate and can score zero even with a supplied candidate. Provider analyzer, punctuation/normalization/boundaries, local scoring, request limits, and query-cache version need explicit examples before choosing a fix. Fresh read-only acceptance preparation also confirms the shared tokenizer affects chat auto-mode: a long hyphenated ID with trailing punctuation stays semantic while the bare ID becomes hybrid, so normalization can change ANN/provider work. No index/reindex or live provider change. | +| 31 | Reranking only final top-k and silent degradation | Independently confirmed; open. Actual mocked pipelines pass only final top-k hits to reranking, including after hybrid expansion. Missing configuration/dependency or provider failures return original hits; client acquisition errors propagate. Transient fallback results can be cached without outcome provenance. Candidate budgets, safe status metadata, score thresholds, cache policy and fan-out/cost limits need coordinated API/SDK examples before changing defaults. | +| 32 | Explicit null settings cannot clear values | Partially fixed in batch 4 for nullable collection default_min_score and metadata_schema. API-key expiry was corrected in batch 5: synthetic actual-router reproduction accepted null but retained expiration; the update now distinguishes explicit null from omission. Existing SDKs and nullable ORM/migration already support it. Published batch 9 (`f6a565ef`, all six CI jobs passed) also corrects webhook collection-filter clearing: the nullable request/database/SDK contract accepts null=all, but the update ignored it. A field-presence guard honors explicit null while preserving omitted/list/empty-list behavior. Other nullable update contracts retain their current semantics pending separate validation. | +| 33 | Retrieved context in system prompt lacks untrusted-data boundary | Open and held for explicit prompt compatibility approval. Read-only assessment recommends exactly `[system, user]`, with unchanged context then the verbatim question in user, and a fixed source-data instruction appended to both default/custom system prompts. The concrete choice and limitations are recorded in `docs/audit/chat-context-options.md`; no production prompt change. | +| 34 | Usage estimates characters/4 with incomplete prices/history | Partially corrected in published batch 6: UI/docs identify current stored-document character-based estimates, a query-only window, model-name rate lookup independent of provider, and unpriced zero values. Provider-measured usage, rate coverage and historical accounting remain open; no API values, rate card or billing behavior changes. | +| 35 | Upload polling loads all rows | Independently confirmed; open. Each session snapshot materializes all joined items twice despite returning at most 20; a 10,000-item session transfers 20,000 tuples per foreground two-second poll. Per-file context/response scans also grow cumulatively. A single aggregate/limited-recent candidate preserves synthetic status counts and ordering, including local in-memory SQLite execution, but a current independent in-memory concurrency counterexample shows baseline identity-mapped items stay stale after a post-commit cancellation while fresh aggregate totals change. Even a second-read-only replacement needs a coherent fresh-item/count/recent policy; a full replacement also needs cancellation, closure and updated_at policy. PostgreSQL plans and lifecycle persistence remain unvalidated. Never globally limit rows used by cancellation or budgets. | +| 36 | Docker Python install ignores verified uv.lock/current Torch | Independently confirmed from source; open. Docker omits the lock, installs unpinned CPU Torch, resolves project ranges again, and removes declared dependencies. Freezing the current lock would include Linux CUDA dependencies; a CPU source/lock policy must preserve both published architectures and be reviewed first. CPU wheel availability and image runtime remain untested. PR 77 stays separate. | +| 37 | Multipart body size compared with raw upload limits; `routers/documents/crud.py`, `routers/documents_batch.py` | Corrected in published batch 7. The settings and streaming counters limit file bytes, but both handlers also compared the larger whole multipart Content-Length with those caps. Actual SDK mock encoding and isolated handler/helper bodies confirm valid near/exact-limit files were rejected; the two whole-body comparisons are removed. Existing per-file/aggregate counters, equality, 100-file cap, validation, authorization, audit attribution and cleanup remain. No wire-body ceiling, parser/proxy policy, schema or SDK change. Independent route/docs review found no blocker; commit `4b4a6b19065c4c4813790e43a22f219cb8bb34dd` was published around 14:00 UTC with all six CI jobs passing. | +| 38 | SDK list contracts omit existing cursor continuation | Independently confirmed on published `f6a565ef`; corrected in published batch 10, commit `3c5c02de8709b363708ac38ef3e49d39dae68db4`. Python rejected cursor keywords; TypeScript omitted runtime cursor values and rejected the public option/response fields during compilation, while both SDKs already preserved raw next_cursor JSON. Additive list options/optional nullable response types now match existing collection/document endpoints. TypeScript offset iterators exclude and suppress cursor to retain prior behavior; Python iterators are unchanged. No API/auth/pagination/default/ranking change. | +| 39 | Malformed SDK HTTP error-envelope containers bypass typed errors; `sdks/python/src/bigrag/_core.py`, `sdks/typescript/src/core.ts` | Independently confirmed on published `3c5c02de`; pending batch 11. Python calls `.get` on non-dict JSON roots and truthy non-dict nested errors; TypeScript dereferences a null JSON root. Shape-only guards now ignore non-object outer/nested containers while retaining field selection, validation arrays, nested code values and status classes. No arbitrary field normalization, top-level code support, retry/transport or server change. Special JavaScript message-coercion failures remain a separate deferred finding. | + +## Batch 1 verification + +Changes: saved provider destination binding, slow-stream heartbeat ownership, safe SDK exception retry policy, sequential analytics queries, and matching documentation. + +Temporary reproductions initially failed for saved-key redirection, slow-stream token loss, unsafe Python connection/timeout retries, and concurrent analytics session use. These now pass. + +- `pnpm check`: passed on the final source, covering Biome, API Ruff lint/format, all workspace typechecks, app/SDK/docs builds, and API compile. The docs build required access for its existing Google font download; an earlier sandboxed build was interrupted after it stalled. No font or network configuration changed. +- Python SDK Ruff lint/format, compile, wheel and source distribution build: passed. +- Backend mocked smoke: eight checks passed, covering saved destination rejection, destination/credential matrix, signed preference binding and redaction/clear, delayed tokens across heartbeats, pending-read cancellation, actual OpenAI AsyncStream response closure after a delta, AnyIO scope cancellation with awaited cleanup, and exclusive analytics session use. +- Python SDK mocked connection/timeout retry matrix: four checks passed. TypeScript SDK runtime smoke: eight checks passed, including generated-key reuse and unkeyed mutation behavior. +- Independent reviewer initially found unsigned legacy preference bindings and provider cleanup gaps. All were reproduced, fixed, and re-reviewed. Final review: no blocking findings. Reviewer separately verified encrypted-key bindings, rejected updates, and legacy rejection for chat and suggestions. +- `git diff --check`: passed. No test runner or coverage suite added. The route generator emits an existing circular-module warning during otherwise successful app typecheck/build. + +Local recovery: verification logs and temporary smoke scripts live beside this checkout in the task workspace. A copy of changed files, binary diff, and manifest is saved under the task's `audit-backups/` before the commit. These copies exclude dependencies, runtime data, and secrets. This ledger is the durable source for later batches; continue in the same draft PR. + +Published batch 1: commit `d5f8c60d64826bdf0779678c6cf8fbbb442b9d5d` in draft PR [#90](https://github.com/bigint/rag.computer/pull/90). All six GitHub checks passed: lint, repo-check, SDK typecheck, Python SDK build, website build, and app build. + +Next coordination: replay authorization (2/18) needs guard extraction design; tenant isolation (3–7) needs explicit coordination with the separate migration work before persistence changes. Other bounded findings remain available for independent reproduction. + +Limitations: no live credential transmission, exploit, production database/vector access, Docker service start, provider billing, or deployed runtime verification. Idempotency authorization and tenant isolation remain open and are not claimed fixed. + +## Batch 2 verification + +Changes: UI stream terminal/ownership handling, password cookie cleanup, guarded auth transitions, retired query-client isolation, and owner-bound upload tracking. No schema, tenant provisioning, replay middleware, service, or credential changes. + +- `pnpm check`: passed, covering repository lint, API Ruff lint/format, workspace typechecks, app/SDK/docs builds, and API compile. Existing route-generator circular-module warning remains. +- Nine external UI runtime checks passed: truncated SSE, done/error/[DONE] termination and reader cancellation, CRLF/chunked tails, atomic stream ownership, legacy persistence migration, owner/session comparisons, generation cancellation/cache isolation including a real late mutation, and same-owner refresh/sign-out behavior. +- Password backend smoke passed: successful password change deletes sessions and the cookie; validation failure retains the cookie. No database or provider connection used. +- Independent source review and 16 mocked runtime checks passed. Review found stale caller navigation, login-versus-session-refresh adoption, and overlapping auth-gate issues; all fixed and re-reviewed. Independent coverage includes actual QueryObserver cancellation, stale auth responses, upload cancellation before late callbacks, and denied localStorage. +- Browser smoke with intercepted API responses passed for truncated-answer rendering, stop/immediate resend, password validation/success, login, confirmed account switch with a late mutation, and confirmed expiry. Vite's versioned module URLs initially made the fixture inspect a duplicate module; the fixture was corrected to use the running app's imports. Incorrect-password 401 preserves the account; successful change reaches sign-in with empty chat/upload tracking and zero logout requests. +- `git diff --check`: passed. Temporary scripts, browser artifacts, and logs stay outside the production checkout; no removed runner or suite restored. + +Limitations: confirmed `/me` responses drive expiry/account-switch cleanup; no claim of instantaneous detection across tabs. Canceling browser work cannot undo API work already accepted, and cross-tab cookie races remain outside this client lifecycle change. Explicit sign-out clears tracking IDs; same-account reload/refresh retains them. + +Published batch 2: commit `02c2f032028b920b4338374705f34c2f7f750a21` in draft PR #90. All six GitHub checks passed on this head. + +## Batch 3 verification + +Changes: upload item failure reporting, preservation of validation failures, closed/missing-file terminal states, rejection of new requests after closure, cancellation handling, and retained issue panels with matching documentation. + +- `pnpm check`: passed, covering repository lint, API Ruff lint/format, workspace typechecks, app/SDK/docs builds, and API compile. The final badge-only follow-up also passed app typecheck/build and Biome lint. +- Eleven backend mocked checks passed: open/closed count and active-work combinations, cancellation preservation, full-count partial success, failed-item persistence, and rejection after closure. +- Eleven UI runtime checks passed: failed/canceled HTTP-success items, accepted wording, closing after all failed requests, missing/late-failure warnings, cancellation from responses and file/finalization conflicts, unrelated conflicts, and issue-panel retention. +- Independent review reran both sets and four additional in-memory checks using the real SDK ConflictError: pending attempts stop after cancellation, unrelated conflicts stay errors, session abort blocks closure, and failed cancellation lookup does not report false cancellation. Review approved the final warning badge for completed sessions containing failed/canceled files. +- Intercepted browser smoke passed for a rejected HTTP 201 file with one completion call, a closed session with missing files, and a completed session with a canceled item. Tracking and issue panels remained visible; no browser runtime errors occurred. Screenshots were visually checked. +- `git diff --check`: passed. Temporary scripts, logs, and browser artifacts remain outside the checkout. No test runner or coverage suite restored. + +Limitations: requests admitted before closure can still finish; stronger serialization, document registration, and quota races need a separate coordinated change. The batch does not claim transactional cancellation, automatic ingestion retry, or tenant isolation. + +Hourly continuation: a thread-heartbeat creation was attempted, but the app returned “Automations are only supported for local threads.” No recurring automation was created. This delegated task needs continuation from its local parent task; no standalone cron workaround was installed. + +Published batch 3: commit `eb9486f9eeeab1e2e3f828cf385126b241f79b3b` in draft PR #90 at approximately 10:00 UTC. All six GitHub checks passed. The parent subsequently confirmed it handles scheduling; no automation setup is needed here. The next routine combined publication is held until approximately 11:00 UTC. + +## Batch 4 verification + +Related retrieval correctness changes: unique nDCG relevance credit and explicit clearing of nullable collection retrieval/validation settings, with SDK types and matching docs. No schema or reindex change. Reproductions and smoke scripts live outside the checkout. + +Design proposals are in [authorization-options.md](docs/audit/authorization-options.md). Replay response quarantine versus shared current-authorization guards and tenant collection quarantine versus shared/separate durable layouts require a decision before larger implementation. Proposals include current reachability, compatibility, legacy handling, and schema/reindex boundaries; this batch implements none of those options. + +Verification before the held publication: + +- `pnpm check`: passed on final code; final documentation build also passed after compatibility clarification. +- 25 external runtime checks and 364 ranking combinations passed. Clearing metadata validation still requires tenant metadata where configured. +- Independent review passed 43,688 ranked-list/label-set comparisons against a first-occurrence oracle, including unchanged recall/MRR and 520 unchanged unique rankings. Isolated handler execution preserved hit IDs, top-k, per-case/aggregate output, and one retrieval per case. +- Actual Python SDK MockTransport and compiled TypeScript SDK mocked fetch preserved explicit null, omission, zero, and empty schema; the public TypeScript null contract compiled. An initial external type-check invocation omitted the SDK's Node type settings; it passed after using the existing Node type root. +- Pre-commit hooks, Python SDK Ruff/format, wheel/source-distribution build, and diff checks passed. The package build initially hit sandbox DNS while resolving its existing Hatchling backend; it passed with authorized network access. No build configuration or dependency declaration changed. +- Independent design review corrected credential-cache precedence, fresh role/session requirements, suggestions generation, global aggregate coverage, tenant creation, T0/replay interaction, and T2 session policy. No blocker remains to presenting the proposals; larger implementation requires the requested policy decision. + +Published batch 4: commit `7eef3b38d1db60a261fe3ac3c63ff21c6b446c4a` in draft PR #90 at 10:59 UTC. The eight reviewed files remained unchanged through publication. Fresh full checks, smoke/oracle/SDK checks, installed hooks, and all six CI jobs passed. The checkout was clean after push. The next routine combined publication is held until approximately 12:00 UTC. + +## Batch 5 verification + +Bounded request-semantics correction: explicit null clears API-key expiration, omission preserves it, and a timestamp replaces it. Only the expiration presence guard changes; admin-session authorization, MCP exclusion, active state, scopes/collection/tenant permissions, key identity/secret, principal invalidation, and audit attribution retain their existing behavior. Docs explain that removing expiration from an active expired key restores expiration eligibility without reactivating an inactive key. + +The user authorized this correction with synthetic-only verification. No credential is created, rotated, transmitted, or changed in a deployed account. Actual route/model/auth-guard bodies run with an in-memory session and fake key; credential generation and unrelated/live dependencies are forbidden. The original reproduction and pre-fix null-clearing failure are retained outside the checkout. Both SDKs already accept and serialize null, and ORM/migration fields already allow NULL; no SDK/schema change is needed. + +Independent route/docs review passed 48 synthetic state/payload combinations and 14 denied requests, covering already-null/expired/inactive keys, preservation of identity/hash/permissions/pins, authorization/validation, audit fields, and invalidation order. Root reran that fixture and its own route smoke. Full pnpm check passed, including lint/format, workspace typechecks, app/SDK/docs builds, and API compile. Credential generation and live dependencies were never called. Independent final three-file review also passed. Pre-commit hooks and all six CI jobs passed on the published head. Backup retained under the task workspace audit-backups/batch-5-20261004T115405Z; no small early push. + +Further replay review establishes a feasible schema-free first step for new versioned completions on POST /v1/collections/{collection_name}/query and POST /v1/evaluation, limited to collections unmarked for tenancy both originally and currently. Fresh credential/scope/pin/stable-collection checks must match saved authorization evidence; tenant, legacy, outer MCP, and unsupported completions remain conflicts without eviction or rerun. This refines the earlier broad options and does not implement them. The full compatibility brief and exact 41-pattern recovery inventory were returned to the parent for informed user coordination. Tenant provisioning, schema/layout choices, replay/tenant code, Journal, merge, and deployment remain unchanged. + + +Published batch 5: commit `d0077c8dcb1405fa597daac724d8f6b1cd148ff8` in draft PR [#90](https://github.com/bigint/rag.computer/pull/90), pushed at 11:59 UTC. All six CI jobs passed on that exact head, verified at 12:03 UTC. The parent received the full decision brief; replay and tenant implementation remain held. The next routine combined publication is targeted for approximately 13:00 UTC. + +## Batch 6 verification + +Bounded corrections for findings 12, 26 and 34: prevent saving an empty file-type draft, require actual all-type membership before writing the unrestricted sentinel, and clarify query-window versus current-document token/cost estimates. A failed upload retry also stops subtracting bytes already excluded from its session sum. Documentation describes retry accounting, the browser-only upload filter and current estimate provenance. File-content validation, usage values, rate card, schemas, tenant/replay policy, and deployed credentials stay unchanged. The byte-cap threshold is unchanged; over-cap failed retries now reject correctly. + +The original actual-hook/save reproduction showed an empty draft saving [] with a misleading zero-type restriction toast; the new expectation failed before the source fix. The actual save handler now rejects before transport, and subset/all-type saves preserve unrelated metadata. An equal-count set of unknown extensions no longer silently removes its filter. Seven existing unrestricted representations and other file-helper edge cases retain their behavior. The temporary VM harness initially used an Error from a different realm; injecting the host constructor corrected the harness without a production change, and both logs are retained. + +The initial UI-only full pnpm check passed: repository lint, API Ruff lint/format, workspace typechecks, app/SDK/docs builds, and API compile. Independent UI/source review passed 20 actual-card states, 11 save-body cases and 10 usage renders; synthetic actual usage SQL/body checks confirm current-document aggregation, model-only rates and query-window scope. Built-UI browser verification passed keyboard subset selection/save/reload, empty Save disabled, all-selected sentinel save/reload, preserved custom metadata, query windows 30/1, unchanged stored-token estimates, and mobile viewport containment with zero page errors. Screenshots were visually inspected. An incomplete admin onboarding fixture initially timed out; the corrected synthetic member fixture passed. No deployed account was involved. The final full pnpm check after the backend addition passed, and the subsequent precise open/nonterminal retry wording passed a final docs build. Independent failed-retry review passed eight additional actual-helper cases, rejecting over-cap and cap-plus-one while preserving exact-cap, under-cap, counted replacement and new-file outcomes. Fresh combined source/docs review and isolated UI/usage/staging reruns found no blocker. Complete fresh eight-file snapshot review passed; all reviewed hashes remained unchanged through publication. All temporary fixtures and receipts remain outside the repository; no package test runner or coverage suite is restored. Back up the reviewed batch before the approximately hourly publication. + + +The failed-retry byte-accounting correction was independently validated against the actual staging helper: a 1 MiB cap admitted 1,100,000 counted bytes because 300,000 excluded failed bytes were subtracted again. Root's separate actual-body fixture failed on the captured baseline and passes 11 cases after the guard, covering over/exact caps, excluded failed rows, counted replacements, new files, canceled bytes and empty files. Its initial AST harness lacked synthetic-node locations; the harness was corrected and original/fixed logs retained. Failure persistence and content validation were mocked, so this does not prove transport, database serialization or concurrent capacity enforcement. The broader finding 12 remains open. + + +Further read-only validation independently exercised eight actual-flow checks for findings 11/12, and root reran them with the API import path after an initial import-path error. Single/batch/session lookup barriers permit duplicate same-scope/hash document creation in synthetic transactions; one batch's repeated hashes remain deduped. Initial session locking is released before staging, so distinct-item stale snapshots permit file-slot/byte overshoot. Actual tiny-byte streaming helpers still enforce per-file and per-batch limits with cleanup. These fixtures forbid socket connections and use in-memory locks/storage/transactions; they establish source reachability, not PostgreSQL concurrency, physical storage billing, global tenant quotas or real framework/proxy behavior. Capacity admission and shared dedup/recovery policy need coordinated ownership/transaction design; neither is implemented here. + + +Published batch 6: commit `e15d6cb8f6bd5c66a08c5af9d5db47d1152992ee` in draft PR [#90](https://github.com/bigint/rag.computer/pull/90), pushed around 13:00 UTC. All three installed hooks and all six CI jobs passed on the exact head, verified at 13:03 UTC. The checkout was clean and committed source matched the independent review and recovery backup. Findings 12, 26 and 34 remain partial. The next routine combined publication is targeted for approximately 14:00 UTC. + +## Batch 7 verification + +Bounded correction for new finding 37: single and batch upload handlers stop comparing whole multipart Content-Length with the configured raw-file and aggregate-file limits. The actual streaming counters remain authoritative, reject only above the limit, and clean temporary files on rejection. Request parameters remain for successful upload audit attribution. Pin, collection, embedding, metadata/tenant and content validation, the 100-file batch cap, persistence and queue logic are unchanged. Documents API docs define the byte units and exact-limit behavior. No SDK, parser/proxy, wire-body policy, schema, replay, tenant or Journal change. + +Independent pre-diff review confirmed the raw-byte contract and minimal deletion. Root's external actual Python SDK/HTTPX MockTransport encoding plus extracted complete handler/helper bodies produced 12 expected failures among 36 cases before the fix and passes all 36 afterward. Cases cover below/equal/above caps, short/Unicode/long filenames, ordinary/large metadata, absent Content-Length, denied pins, unsupported/empty/invalid content, malformed metadata, per-file overcap in a larger batch budget, the 101-file rejection and temporary-file cleanup. Content validation and authorization dependencies are mocked and successful raw staging stops before persistence, so these checks do not prove live ingestion, actual route dispatch, HTTP protocol, database, parser or proxy behavior. No socket or credential is used. The independent prior exact-cap capture and after root evidence are preserved outside the checkout. + +Removing the post-parser header shortcut means an oversized declared body can reach the second raw staging copy before its file counters reject it. FastAPI already parses multipart form data before these handlers; the deleted checks were not a before-parser ingress ceiling. A distinct wire-body policy would need separately chosen semantics and enforcement. Full pnpm check passed, covering lint/format, workspace typechecks, app/SDK/docs builds and API compile. Source and docs match the checked snapshot. Independent route/docs review passed 27 handler cases and five raw-helper cases, including aggregate/content rejection after an earlier staged file; root reran the independent fixture. Module AST comparison confirms only the two header nodes per handler changed, with helper/settings/audit source unchanged. Preserve the reviewed hashes through complete four-file review, recovery backup and installed commit hooks before the held hourly publication. Replay and tenant behavior remain held pending the user's decision. + + +Read-only follow-ups from the previous checkpoint are now recorded durably: finding 18's 39 local mocked checks and root reruns remain external evidence for unsupported retries/lease/request-tail issues, not authorization for replay behavior changes. Finding 27's eight gate/credential/preparation/preferences sources remain identical to the prior independently reviewed configuration-readiness proposal and 27 synthetic checks. Finding 27 is the recommended next bounded candidate; it does not require a new service, credential or schema, and actual chat retains all destination/authorization checks. No additional production change is included here. + + +Published batch 7: commit `4b4a6b19065c4c4813790e43a22f219cb8bb34dd` in draft PR [#90](https://github.com/bigint/rag.computer/pull/90), pushed around 14:00 UTC. The exact committed four-file snapshot matched independent review and recovery backup; applicable Ruff hooks passed, with Biome correctly reporting no applicable staged files. All six CI jobs passed on that exact head. The preceding batch 7 verification text preserves its pre-publication checkpoint. Parent authorized the reviewed finding 27 readiness plan as the next bounded candidate, within existing audit authorization, with the next combined publication held until approximately 15:00 UTC. + +## Pending batch 8 + +Corrects finding 27 by exposing local chat credential readiness for the authenticated session account at `GET /v1/auth/preferences/chat-readiness`. The three-field DTO contains only a boolean, nullable saved/instance source and stable nullable reason; the route uses `require_session`, an authenticated-UUID preference lookup and `Cache-Control: private, no-store`. Saved/instance selection is shared with actual chat. Typed internal exceptions preserve existing chat error categories/messages while letting readiness distinguish recognized configuration failures from unexpected operational failures. Saved-key precedence, signed destination restrictions, request-key bypass, missing decryption configuration and configured-invalid-ciphertext fallback remain unchanged. The endpoint normalizes locally and performs no provider, DNS, model, retrieval or collection request. It does not change writable preference data or the browser's explicit OpenAI/no-request-override contract. Both SDKs add matching typed auth methods and public exports with documentation. + +Studio uses session-owned readiness queries and signals, refreshes on entry/focus/reconnect and each foreground minute, and exposes Check again. Loading, blocked and unavailable states gate sending and question generation without disabling key editing, history or Stop. Actual send and generation callbacks check eligibility, and blocked edit/regenerate returns before truncating history. Key changes remain gated through their settled refetch; unrelated preference updates do not invalidate readiness. Personal-key labels/save/clear remain separate from eligible instance credentials. A small owned-request wrapper combines caller/session cancellation, preserves abort reasons, rejects late retired-account results and removes listeners after completion without requiring native AbortSignal.any. Owned preference/question requests and callbacks are canceled or ignored after confirmed session changes; current non-abort stream/question errors refetch readiness and preferences without automatically resending. + +Verification for the candidate: full `pnpm check` passed, covering repository lint/format, workspace typechecks, app/SDK/docs builds and API compile. Python SDK Ruff/format and wheel/source-distribution build passed. Root's real-service and FastAPI/HTTPX in-memory ASGI fixture passed 20 credential cases and 11 supplemental request-bypass, operational-error, auth/account, no-store and OpenAPI checks. Independent backend review passed 56 checks, including 25 exact baseline/current chat-selection comparisons, typed-error propagation and extracted actual route/session-guard behavior. Root's actual Python SDK MockTransport and compiled TypeScript SDK mocked-fetch fixtures passed four checks each. An initial TypeScript harness imported the types/errors-only browser entry instead of the client entry; the corrected client-entry fixture passed and both logs are retained externally. + +Independent UI review passed 69 actual-source synthetic checks and seven real installed TanStack QueryClient/QueryObserver/MutationObserver lifecycle checks, all repeated with native AbortSignal.any disabled, covering readiness/gates, key pending/refetch, unrelated preference writes, focus refresh, late account reads, generation cache/toasts and stream/Stop/history. Built-UI browser smoke passed eight named scenarios: checking/instance gates, instance generation/send, blocked history retention, unavailable/manual retry, pending save/refetch/personal labels, clear-to-instance fallback, Stop, and broad credential-error refresh without resend. Its first run reached save/clear but had an ambiguous Clear selector; the external fixture was corrected and rerun successfully. After finding that native AbortSignal.any would add a browser requirement, root replaced it with the owned-request wrapper and repeated the full pnpm check and all eight browser scenarios with that native API disabled; both passed. Mobile layout stayed within the viewport and no browser runtime errors occurred. Screenshots and all temporary fixtures/logs remain outside the checkout; no removed test runner is restored. A fresh whole-batch review independently reran the final fixtures and added 15 actual-source owned-request checks for pre-abort, in-flight cancellation, late success, ownership-only retirement, errors, concurrent requests, first abort reason and listener cleanup. Root reran those checks. Preserve the complete independently reviewed source/docs/ledger hashes through backup, installed hooks and the held publication. + +Limitations: authentication lookup, runtime settings and database behavior are mocked in local ASGI checks; synthetic source fixtures do not establish PostgreSQL concurrency or deployed protocol behavior. Browser API responses are intercepted and no real key validation/provider generation occurs. Readiness is a local configuration hint; actual chat retains all checks and may reject a stale positive. Confirmed account changes drive client ownership, and existing unconfirmed cross-tab cookie races remain. Cancellation cannot undo accepted server work. Replay and tenant behavior, Journal, schema/reindex, deployment and system/network settings remain held or outside this batch. No new consequential compatibility decision was required for the reviewed readiness contract. Next useful bounded audit: independently inspect retrieved-context message boundaries (finding 33) and prepare a concrete compatibility-preserving candidate before implementation. + +Published batch 8: commit `616607324569b60f6f1cc3d600b8aae2a88854c0` in draft PR [#90](https://github.com/bigint/rag.computer/pull/90), pushed around 15:00 UTC. The exact 24-file committed snapshot matched the final independent review and recovery backup; all three installed hooks and all six CI jobs passed. The preceding batch 8 section preserves its pre-publication checkpoint. Publication receipts and reviewed file hashes are retained in the external task evidence. No merge or deployment occurred. + +## Pending batch 9 + +Corrects another validated finding 32 case: webhook updates distinguish an omitted collection filter from explicit `collections: null`. The existing nullable database field, request/response models, SDK update types and null=all documentation already support clearing the filter. Previously the route accepted null but retained the old filter and excluded collections from its audit fields. The presence-based guard clears it and records the changed field; omission still preserves it and list/empty-list behavior remains unchanged. Existing empty-list normalization to null/all in response and dispatcher is preserved. Session-admin authorization, destination checks, events, active state, commit/refresh, cache invalidation and audit lifecycle stay unchanged. No webhook registration, credential, secret, delivery or live network operation is performed during verification. + +Finding 33 remains a read-only design deliverable in `docs/audit/chat-context-options.md`. Its exact recommended two-message layout and fixed instruction for custom prompts require explicit user approval before implementation. The independent fixture passed 51 benign checks, reproduced by root, with zero socket/DNS attempts; it does not establish live provider compatibility, instruction following or injection prevention. Question generation remains unchanged. Finding 35 remains deferred after a fresh independent actual-helper/mapping counterexample: fresh aggregate counts and stale ORM recent items need an explicit coherent policy. No upload snapshot optimization is included. + +Root baseline ASGI smoke passed 39 checks and reproduced the null defect. Current baseline-plus-candidate smoke passed 82 checks, covering omission/null/list/empty list, mixed active updates, unchanged unrelated nulls, response serialization, audit/cache lifecycle, admin-session guards, malformed/missing IDs and request validation. Both actual SDK clients passed eight mocked transport checks each for exact PUT payload serialization. Independent webhook review passed 16 actual-model/router/serializer/dispatcher checks, reproduced by root, and found no blocker. Existing empty-list normalization, other-process five-second cache and already queued delivery behavior remain unchanged. Authentication lookup, ORM persistence/cache/audit and target validation are mocked; these checks do not establish PostgreSQL/Redis behavior or webhook delivery. External fixtures remain outside the checkout and no removed test runner is restored. Full `pnpm check` passed, covering repository lint/format, workspace typechecks, app/SDK/docs builds and API compile. The first run stopped at documentation typechecking because four generated `.next/types` files had byte-identical ` 2.ts` copies; those ignored copies were preserved outside the checkout and the full rerun passed without source/configuration changes. Fresh complete four-file review found no blocker and reran the 82 webhook cases, 16 actual imported route/model/cache-loader/dispatcher cases, eight checks per SDK, 51 benign context cases and both actual ORM polling cases. It verified the entire webhook module changes only at the collection guard and the supporting auth/SDK/replay/context/polling contracts remain HEAD-identical. Preserve the final reviewed four-file hashes through recovery backup and installed hooks before the combined publication held until approximately 16:00 UTC. Replay, tenant, Journal, schema/reindex, deployment and system/network settings remain held or outside this batch. + +Published batch 9: commit `f6a565ef409e84e2871ef53a30ac8a1b184aa142` in draft PR [#90](https://github.com/bigint/rag.computer/pull/90), pushed after 16:00 UTC. All four committed blobs and per-file diff blocks matched final independent review and recovery backup; applicable Ruff hooks passed and Biome correctly skipped. Full local checks and all six exact-head CI jobs passed. The preceding pending section preserves its pre-publication checkpoint. External publication/handoff receipts hold final status without an extra bookkeeping commit. Finding 33's concrete prompt policy remains unimplemented pending explicit approval; replay/tenant remain held and finding 35 is deferred. + +## Pending batch 10 + +Adds existing cursor pagination to the collection/document SDK list contracts. Both list endpoints already accept cursor and return next_cursor, but Python rejected the keyword and TypeScript ignored untyped values while omitting the option/response fields from public declarations. Each ordinary list method now forwards an optional opaque cursor through its existing GET encoder; response types expose optional nullable next_cursor to retain compatibility with older responses that omit it. Existing filters, default arguments, nullable totals, offset/include_total handling, response decoding, credentials, retries and errors remain unchanged. No server handler, pagination helper, auth/tenant rule, ranking, schema or service changes. + +TypeScript listAll continues its existing offset algorithm: its option type excludes cursor and its list calls suppress any cursor after the options spread. Exclusion alone would not prevent cursor-bearing variables, JavaScript or casts from repeating a cursor page while offset increments are ignored. This suppression preserves baseline ignored-unknown-property behavior without mutating caller options. Python list_all signatures/bodies remain AST-identical. SDK/API docs show explicit next-page requests with the same collection/filters/order/page size, opaque/missing/null tokens, nullable totals with opt-in counts and document sort=created_at restriction. Existing iterators are documented as offset-based; no cursor iterator or fallback is added. + +Root baseline checks passed eight actual Python SDK cases and six actual compiled TypeScript cases, reproducing the gap. Public type probing produced exactly two missing cursor-option and two missing next_cursor-property diagnostics; the corrected external compiler invocation uses the installed compiler's ignoreConfig flag, with the initial harness diagnostic retained. Independent pre-edit review passed ten mocked SDK cases, compiled the baseline SDK and verified the structural-typing seam. Current root SDK smoke passed 59 Python checks and 81 compiled TypeScript checks, including exact encoding of supplied/empty/omitted cursors, filters, zero/false values, encoded collection paths, nullable/missing tokens, unchanged typed errors and numeric/null total iterator termination. Actual compiled baseline/current TypeScript iterator requests and outputs match for cursor-bearing inputs; the Python iterator AST is unchanged. The public type probe accepts new fields/older response literals while rejecting cursor object literals for offset iterators. + +Full `pnpm check` passed, covering lint/format, workspace typechecks, app/SDK/docs builds and API compile. Independent review caught two unintended cursor/offset descriptions in unrelated chunks/elements tables; root restored their existing 0+ rows, then repeated the full check successfully. Python SDK Ruff/format and wheel/source-distribution build passed. The initial online build hit sandbox DNS restrictions resolving the existing build backend; offline cached build passed with no dependency/source/config change. The wheel contains all four changed Python files byte-for-byte and its actual imports passed the same 59 mocked SDK checks. No removed test runner is restored; all fixtures/build artifacts and failed/corrected harness logs remain outside the checkout. Transport checks use synthetic responses/errors and establish no deployed cursor progression, PostgreSQL concurrency or snapshot semantics. Independent after-review found no blocker in the corrected twelve-file SDK/docs candidate and passed 28 Python plus 30 TypeScript checks, SDK compile and public-type probes, with zero network/DNS attempts. It verified exact cursor forwarding, optional response fields, typed errors, caller input immutability and baseline/current offset iterator behavior. Fresh complete thirteen-file review found no blocker and independently repeated 59 Python source checks, 59 actual-wheel checks, 81 checks against a fresh external TypeScript compilation and the public type probe. It reproduced the read-only 54 nullability checks, 32 tokenizer/boundary/chat cases and nine query transport requests per SDK. Fifteen static scope/artifact checks confirm only the approved cursor additions in complete SDK modules, 399 supporting API/auth/pagination/core/held/schema/dependency files remain HEAD-identical, all 44 tracked Python SDK package files match both wheel and source distribution, and all twelve code/docs hashes match the corrected independent after-review. The isolated type probe first required explicit installed Node type roots; failed/corrected harness logs are retained without product changes. Preserve the final reviewed thirteen-file hashes through recovery backup and installed hooks before the combined publication held until approximately 17:00 UTC. + +Read-only remaining-work preparation selected no additional production patch. The nullable-contract inventory reviewed 38 fields across five models with 54 mocked checks: collection key/min-score/schema clearing and S3 endpoint/token presence already work; null pin/scope aliases broaden authorization and preset URL clearing affects credential destinations/linked fallbacks. Those require explicit contracts rather than a generic null guard. Fresh retrieval acceptance preparation passed 32 tokenizer/boundary/chat cases and nine mocked transport requests per SDK, with all 128 assessed sources unchanged during that read-only review interval. The later approved SDK formatter change is covered by final source/docs receipts; the historical manifests do not claim all 128 still match the final candidate. Tokenization is shared with chat's auto-mode detection, so punctuation/Unicode changes can alter provider/ANN work; rerank budgets, thresholds, failure/cache provenance and multi-query fan-out still need deliberate choices. Exact acceptance matrices/source receipts remain in external audit evidence. Finding 33, replay/tenant, Journal, schema/reindex, deployment and system/network changes remain held or outside this batch; finding 35 stays deferred. + +Published batch 10: commit `3c5c02de8709b363708ac38ef3e49d39dae68db4` in draft PR [#90](https://github.com/bigint/rag.computer/pull/90). All six exact-head CI jobs passed in [run 37220632337](https://github.com/bigint/rag.computer/actions/runs/37220632337). The preceding pending section preserves the pre-publication checkpoint. Finding 33's prompt policy remains unimplemented pending explicit approval; replay/tenant remain held and finding 35 stays deferred. + +## Pending batch 11 + +Prevents malformed JSON error-envelope containers from replacing status-specific SDK exceptions with envelope-access exceptions. The existing JSON parse/catch boundaries remain intact. Python retains only dict outer and nested error containers; TypeScript retains only non-null, non-array object containers. Both read the nested error once, use an empty container for unsupported shapes, and retain their existing error factories and field-selection order. Python keeps truthy fallthrough and its `"Unknown error"` fallback; TypeScript keeps nullish fallthrough and empty status text. Message and nested code values are not normalized or coerced by the new checks, including legitimate FastAPI validation detail arrays. Top-level code is still ignored. No API, authentication, tenant, replay, schema, dependency, retry, body-read or transport change. + +The completed independent read-only audit ran 186 scenarios per SDK against published `3c5c02de` and scratch-only shape-guard prototypes. It reproduced 57 Python and 15 TypeScript envelope crashes, preserved all 129 Python and 165 TypeScript existing APIError results, and retained six separate JavaScript message-coercion failures. Its original fixtures, prototypes, source receipts and results remain intact outside the checkout. Production guards were then implemented freshly from the approved requirements after a new scratch harness failed against the actual unmodified Python source and freshly compiled TypeScript SDK. + +The new regression harness uses 91 explicit expected fixtures through collection GET, multipart upload and initial non-2xx chat-stream entry points: 273 scenarios per SDK, including all eleven named HTTP status classes and generic APIError at 418. Baseline assertion failures were 90 Python envelope-access exceptions and 42 TypeScript null-envelope exceptions. After the repair, all 273 assertions pass against Python source, all 273 pass against the built Python wheel, and all 273 pass against freshly compiled TypeScript. The TypeScript assertions explicitly retain six expected TypeError outcomes for the deferred message-coercion cases; they do not claim those values now produce APIError. All 183 Python and 225 TypeScript successful baseline APIError outcomes remain exactly equal, including message values, code values, status and request/body observations. The six deferred TypeScript outcomes are also unchanged. + +Focused Python SDK Ruff/format and compile checks, TypeScript Biome/typecheck/build, and Python wheel build passed. The wheel was built with the existing cached Hatchling backend and dependencies without installation. Each mocked scenario made one request with retries disabled, emitted no Authorization header, preserved upload-only idempotency headers for these entry points, and closed Python responses or consumed TypeScript bodies. Network guards recorded zero external attempts. Three actual FastAPI validation-response fixtures from the read-only audit preserve Python's original list argument and JavaScript's existing array-to-message coercion. No live API, provider, proxy, database, retry timing or deployed transport behavior was tested, and no removed test runner was restored. Full repository checks and fresh independent review remain required before publication; the next combined batch is held until no earlier than 18:28:15 UTC. + +Final cloud verification passed the complete `pnpm check`, covering repository lint/format, workspace typechecks, app/SDK/docs builds and API compilation. Only the check process's CPU affinity was limited to two CPUs; source and build configuration were unchanged. Fresh independent review found no blocker and reran all 273 scenarios against Python source, the actual wheel and an isolated fresh TypeScript compilation. Reconstructed baseline imports reproduced the exact 90 Python and 42 TypeScript failures. All successful baseline outcomes and the six deferred TypeScript outcomes remained unchanged. Independent receipts verify the five-file scope, all 776 other tracked files, all eleven original audit-evidence files, all 44 wheel package files, and all 86 TypeScript compiler outputs; only compiled `core.js` differs from baseline. One external baseline compilation initially lacked the correct ESM module context; correcting the scratch context produced the expected baseline failures without a product change. Logs and both failed/corrected evidence remain outside the checkout. Preserve the final reviewed hashes through backup and configured hooks before publication. + +Deferred separately: top-level server-code parity needs a nested-versus-top-level precedence contract; non-string message/code normalization needs an accepted-value policy that preserves structured validation output; special JavaScript message objects can still fail primitive coercion during Error construction. Findings 2–7, 33 and 35, tenant/replay, Journal, schema/reindex, merge, deployment and system/network changes remain held or outside this batch. diff --git a/api/bigrag/models/chat_readiness.py b/api/bigrag/models/chat_readiness.py new file mode 100644 index 00000000..1fdd7273 --- /dev/null +++ b/api/bigrag/models/chat_readiness.py @@ -0,0 +1,13 @@ +from __future__ import annotations + +from typing import Literal + +from pydantic import BaseModel + + +class ChatReadinessResponse(BaseModel): + credential_ready: bool + credential_source: Literal["saved", "instance"] | None = None + reason: ( + Literal["missing_credentials", "credential_blocked", "decryption_unavailable"] | None + ) = None diff --git a/api/bigrag/routers/admin_api_keys.py b/api/bigrag/routers/admin_api_keys.py index 7f6be8dd..3aabd6ea 100644 --- a/api/bigrag/routers/admin_api_keys.py +++ b/api/bigrag/routers/admin_api_keys.py @@ -166,7 +166,7 @@ async def update_api_key( if body.active is not None: key.active = body.active fields.append("active") - if body.expires_at is not None: + if "expires_at" in body.model_fields_set: key.expires_at = body.expires_at fields.append("expires_at") existing = dict(key.permissions or {}) diff --git a/api/bigrag/routers/auth.py b/api/bigrag/routers/auth.py index c1cd5566..5d74be77 100644 --- a/api/bigrag/routers/auth.py +++ b/api/bigrag/routers/auth.py @@ -276,6 +276,7 @@ async def whoami(user: dict = Depends(get_current_user)) -> WhoamiResponse: async def change_password( body: ChangePasswordRequest, request: Request, + response: Response, user: dict = Depends(require_session), session: AsyncSession = Depends(get_session), ) -> StatusResponse: @@ -289,6 +290,7 @@ async def change_password( await session.execute(sa.delete(UserSession).where(UserSession.user_id == target.id)) await session.commit() await invalidate_auth_principals() + await clear_session_cookie(response) audit.record( request, user=user, diff --git a/api/bigrag/routers/documents/crud.py b/api/bigrag/routers/documents/crud.py index 6c9beccc..f495d076 100644 --- a/api/bigrag/routers/documents/crud.py +++ b/api/bigrag/routers/documents/crud.py @@ -59,12 +59,6 @@ async def upload_document( upload_limits = await get_values(["max_upload_size_mb"]) max_upload_size_mb = upload_limits["max_upload_size_mb"] max_size = max_upload_size_mb * 1024 * 1024 - content_length = request.headers.get("content-length") - if content_length and int(content_length) > max_size: - raise HTTPException( - status_code=413, - detail=f"File too large. Max size: {max_upload_size_mb}MB", - ) tmp_path, content_hash, file_size = await validated_upload_to_temp( file, file_ext, max_size=max_size diff --git a/api/bigrag/routers/documents_batch.py b/api/bigrag/routers/documents_batch.py index ea8303ce..96cb0356 100644 --- a/api/bigrag/routers/documents_batch.py +++ b/api/bigrag/routers/documents_batch.py @@ -121,12 +121,6 @@ async def batch_upload_documents( max_batch_upload_size_mb = upload_limits["max_batch_upload_size_mb"] max_size = max_upload_size_mb * 1024 * 1024 batch_max_size = max_batch_upload_size_mb * 1024 * 1024 - content_length = request.headers.get("content-length") - if content_length and int(content_length) > batch_max_size: - raise HTTPException( - status_code=413, - detail=f"Batch upload too large. Max size: {max_batch_upload_size_mb}MB", - ) budget = UploadBudget(batch_max_size) shared_meta = metadata_or_400( collection, diff --git a/api/bigrag/routers/evaluation.py b/api/bigrag/routers/evaluation.py index 89f49191..ae5abb92 100644 --- a/api/bigrag/routers/evaluation.py +++ b/api/bigrag/routers/evaluation.py @@ -68,10 +68,11 @@ def _ndcg_at_k(hit_ids: list[str], expected: set[str]) -> float: if not expected: return 0.0 dcg = 0.0 + credited: set[str] = set() for idx, h in enumerate(hit_ids, start=1): - rel = 1 if h in expected else 0 - if rel: + if h in expected and h not in credited: dcg += 1.0 / math.log2(idx + 1) + credited.add(h) k = len(hit_ids) ideal = sum(1.0 / math.log2(i + 1) for i in range(1, min(len(expected), k) + 1)) return dcg / ideal if ideal else 0.0 diff --git a/api/bigrag/routers/preferences.py b/api/bigrag/routers/preferences.py index fb12142e..bcc2d2be 100644 --- a/api/bigrag/routers/preferences.py +++ b/api/bigrag/routers/preferences.py @@ -3,7 +3,7 @@ import uuid import sqlalchemy as sa -from fastapi import APIRouter, Depends +from fastapi import APIRouter, Depends, Response from sqlalchemy.dialects.postgresql import insert as pg_insert from sqlalchemy.ext.asyncio import AsyncSession @@ -11,6 +11,8 @@ from bigrag.db.session import get_session from bigrag.logging import get_logger from bigrag.middleware.auth import require_session +from bigrag.models.chat_readiness import ChatReadinessResponse +from bigrag.services.chat.readiness import get_chat_readiness from bigrag.services.preferences import ( _deep_merge, _encrypt_sensitive, @@ -25,6 +27,16 @@ router = APIRouter(prefix="/v1/auth/preferences", tags=["auth"]) +@router.get("/chat-readiness", response_model=ChatReadinessResponse) +async def chat_readiness( + response: Response, + user: dict = Depends(require_session), + session: AsyncSession = Depends(get_session), +) -> ChatReadinessResponse: + response.headers["Cache-Control"] = "private, no-store" + return await get_chat_readiness(session, user) + + @router.get("", response_model=dict[str, dict]) async def get_preferences( user: dict = Depends(require_session), @@ -53,7 +65,7 @@ async def update_preferences( ) existing = dict(existing_row.data) if existing_row else {} incoming = _normalize_sensitive(incoming) - await _validate_sensitive(incoming) + incoming = await _validate_sensitive(incoming) incoming = _encrypt_sensitive(incoming) merged = _remove_cleared_sensitive(_deep_merge(existing, incoming), incoming) diff --git a/api/bigrag/routers/webhooks/crud.py b/api/bigrag/routers/webhooks/crud.py index 84bd586c..0649ea0f 100644 --- a/api/bigrag/routers/webhooks/crud.py +++ b/api/bigrag/routers/webhooks/crud.py @@ -142,7 +142,7 @@ async def update_webhook( if body.events is not None: wh.events = body.events fields.append("events") - if body.collections is not None: + if "collections" in body.model_fields_set: wh.collections = body.collections fields.append("collections") if body.active is not None: diff --git a/api/bigrag/services/analytics.py b/api/bigrag/services/analytics.py index 2eaee42d..7e112413 100644 --- a/api/bigrag/services/analytics.py +++ b/api/bigrag/services/analytics.py @@ -1,7 +1,5 @@ from __future__ import annotations -import asyncio - import sqlalchemy as sa from bigrag.db.engine import session_factory @@ -49,11 +47,9 @@ async def collection_analytics(collection_name: str) -> dict: ) ).all() - stats_24h, stats_7d, stats_30d = await asyncio.gather( - _period_stats(session, collection_name, 1), - _period_stats(session, collection_name, 7), - _period_stats(session, collection_name, 30), - ) + stats_24h = await _period_stats(session, collection_name, 1) + stats_7d = await _period_stats(session, collection_name, 7) + stats_30d = await _period_stats(session, collection_name, 30) result = { "collection": collection_name, diff --git a/api/bigrag/services/chat/completion.py b/api/bigrag/services/chat/completion.py index 71370f88..27453ba0 100644 --- a/api/bigrag/services/chat/completion.py +++ b/api/bigrag/services/chat/completion.py @@ -2,8 +2,10 @@ import asyncio from collections.abc import AsyncIterator +from contextlib import suppress from datetime import UTC, datetime +import anyio from fastapi import Request from bigrag.db.engine import session_factory @@ -87,28 +89,38 @@ async def stream_chat_completion( content_parts: list[str] = [] stream = _stream_model(prepared).__aiter__() + pending_delta: asyncio.Task[str] | None = None try: while True: if request is not None and await request.is_disconnected(): return - try: - delta = await asyncio.wait_for( - stream.__anext__(), timeout=_HEARTBEAT_INTERVAL_SECONDS - ) - except TimeoutError: + if pending_delta is None: + pending_delta = asyncio.create_task(stream.__anext__()) + completed, _ = await asyncio.wait( + {pending_delta}, timeout=_HEARTBEAT_INTERVAL_SECONDS + ) + if not completed: yield _HEARTBEAT continue + try: + delta = pending_delta.result() except StopAsyncIteration: break + pending_delta = None content_parts.append(delta) yield _sse("delta", {"delta": delta}) finally: - aclose = getattr(stream, "aclose", None) - if aclose is not None: - try: - await aclose() - except Exception: - pass + with anyio.CancelScope(shield=True): + if pending_delta is not None: + pending_delta.cancel() + with suppress(asyncio.CancelledError, Exception): + await pending_delta + aclose = getattr(stream, "aclose", None) + if aclose is not None: + try: + await aclose() + except Exception: + pass assistant = _chat_message_response( id=str(uuid7()), diff --git a/api/bigrag/services/chat/provider.py b/api/bigrag/services/chat/provider.py index 83cede14..ce8e9e00 100644 --- a/api/bigrag/services/chat/provider.py +++ b/api/bigrag/services/chat/provider.py @@ -99,13 +99,14 @@ async def _stream_model(prepared: PreparedChatTurn) -> AsyncIterator[str]: ), timeout=_MODEL_TIMEOUT_SECONDS, ) - async for chunk in stream: - choices = getattr(chunk, "choices", None) or [] - if not choices: - continue - delta = getattr(choices[0].delta, "content", None) - if delta: - yield delta + async with stream: + async for chunk in stream: + choices = getattr(chunk, "choices", None) or [] + if not choices: + continue + delta = getattr(choices[0].delta, "content", None) + if delta: + yield delta return except Exception as exc: last_error = exc diff --git a/api/bigrag/services/chat/readiness.py b/api/bigrag/services/chat/readiness.py new file mode 100644 index 00000000..2cae58f2 --- /dev/null +++ b/api/bigrag/services/chat/readiness.py @@ -0,0 +1,36 @@ +from __future__ import annotations + +from sqlalchemy.ext.asyncio import AsyncSession + +from bigrag.models.chat_readiness import ChatReadinessResponse +from bigrag.services.chat.turn.credentials import ( + ChatCredentialDecryptionError, + ChatCredentialDestinationError, + assert_credentials_allowed_for_base_url, + resolve_stored_chat_credentials, +) +from bigrag.services.runtime_settings import get_values +from bigrag.services.url_security import UnsafeOutboundUrlError, normalize_url_root + + +async def get_chat_readiness(session: AsyncSession, user: dict) -> ChatReadinessResponse: + runtime = await get_values(["chat_base_url"]) + try: + credentials = await resolve_stored_chat_credentials(session, user) + except ChatCredentialDecryptionError: + return ChatReadinessResponse(credential_ready=False, reason="decryption_unavailable") + if not credentials: + return ChatReadinessResponse(credential_ready=False, reason="missing_credentials") + try: + raw_base_url = runtime["chat_base_url"] + base_url = normalize_url_root(raw_base_url) if raw_base_url else None + except (UnsafeOutboundUrlError, ValueError): + return ChatReadinessResponse(credential_ready=False, reason="credential_blocked") + try: + assert_credentials_allowed_for_base_url(credentials, base_url, request_base_url=None) + except ChatCredentialDestinationError: + return ChatReadinessResponse(credential_ready=False, reason="credential_blocked") + return ChatReadinessResponse( + credential_ready=True, + credential_source="saved" if credentials[0].source == "saved chat key" else "instance", + ) diff --git a/api/bigrag/services/chat/turn/credentials.py b/api/bigrag/services/chat/turn/credentials.py index 46e7b30f..a4262732 100644 --- a/api/bigrag/services/chat/turn/credentials.py +++ b/api/bigrag/services/chat/turn/credentials.py @@ -12,6 +12,7 @@ from bigrag.services import crypto from bigrag.services.chat.types import ProviderCredential from bigrag.services.preferences import decrypt_preferences +from bigrag.services.provider_key_binding import bound_chat_key_destination from bigrag.services.url_security import ( UnsafeOutboundUrlError, normalize_url_root, @@ -22,6 +23,14 @@ _DEFAULT_OPENAI_CHAT_BASE_URL = "https://api.openai.com/v1" +class ChatCredentialDecryptionError(ServerError): + pass + + +class ChatCredentialDestinationError(ValidationError): + pass + + def _resolve_provider(provider: str | None, default_provider: str | None = "openai") -> str: value = (provider or default_provider or "openai").strip().lower() if value not in _PROVIDERS: @@ -37,23 +46,36 @@ async def _resolve_api_credentials( if body.provider_api_key and body.provider_api_key.strip(): return [ProviderCredential(api_key=body.provider_api_key.strip(), source="request")] + credentials = await resolve_stored_chat_credentials(session, user) + if not credentials: + raise ValidationError( + "Save an OpenAI API key in Chat settings, or pass provider_api_key " + "with the chat request." + ) + return credentials + + +async def resolve_stored_chat_credentials( + session: AsyncSession, + user: dict, +) -> list[ProviderCredential]: credentials: list[ProviderCredential] = [] data = await session.scalar( sa.select(UserPreference.data).where(UserPreference.user_id == UUID(user["id"])) ) prefs = decrypt_preferences(dict(data)) if isinstance(data, dict) else {} chat = prefs.get("chat") if isinstance(prefs.get("chat"), dict) else {} - _append_credential(credentials, chat.get("openai_key"), "saved chat key") + _append_credential( + credentials, + chat.get("openai_key"), + "saved chat key", + base_url=bound_chat_key_destination(chat), + ) if not credentials: env_key = os.environ.get("BIGRAG_CHAT_API_KEY") _append_credential(credentials, env_key, "instance chat key") - if not credentials: - raise ValidationError( - "Save an OpenAI API key in Chat settings, or pass provider_api_key " - "with the chat request." - ) return credentials @@ -61,16 +83,18 @@ def _append_credential( credentials: list[ProviderCredential], raw_api_key: object, source: str, + *, + base_url: str | None = None, ) -> None: if not isinstance(raw_api_key, str) or not raw_api_key.strip(): return api_key = raw_api_key.strip() if api_key.startswith(crypto._FERNET_PREFIX): msg = f"{source} cannot be decrypted. Configure BIGRAG_MASTER_KEY." - raise ServerError(msg, public_message=msg) + raise ChatCredentialDecryptionError(msg, public_message=msg) if any(existing.api_key == api_key for existing in credentials): return - credentials.append(ProviderCredential(api_key=api_key, source=source)) + credentials.append(ProviderCredential(api_key=api_key, source=source, base_url=base_url)) async def _resolve_base_url(raw_base_url: str | None, default_base_url: str | None) -> str | None: @@ -89,14 +113,33 @@ def assert_credentials_allowed_for_base_url( *, request_base_url: str | None, ) -> None: + for credential in credentials: + if credential.source != "saved chat key": + continue + if not credential.base_url: + raise ChatCredentialDestinationError( + "Save the chat key again in Chat settings to bind it to the configured provider " + "URL." + ) + try: + matches = normalize_url_root(credential.base_url) == normalize_url_root( + base_url or _DEFAULT_OPENAI_CHAT_BASE_URL + ) + except (UnsafeOutboundUrlError, ValueError): + matches = False + if not matches: + raise ChatCredentialDestinationError( + "The saved chat key belongs to a different provider URL. " + "Save a key for the configured provider, or pass provider_api_key with the request." + ) has_instance_key = _has_instance_chat_key(credentials) if request_base_url is not None and has_instance_key: - raise ValidationError( + raise ChatCredentialDestinationError( "provider_base_url requires provider_api_key or a saved chat key; " "the instance chat key cannot be sent to a custom base URL." ) if base_url is not None and not _is_default_openai_chat_base_url(base_url) and has_instance_key: - raise ValidationError( + raise ChatCredentialDestinationError( "The instance chat key cannot be sent to a non-default chat base URL; " "save a chat key in Chat settings or pass provider_api_key." ) @@ -125,6 +168,8 @@ async def _clear_saved_chat_key(session: AsyncSession, user: dict) -> None: return cleaned_chat = {**chat} cleaned_chat.pop("openai_key", None) + cleaned_chat.pop("openai_key_base_url", None) + cleaned_chat.pop("openai_key_base_url_signature", None) cleaned = {**data, "chat": cleaned_chat} await session.execute( sa.update(UserPreference) diff --git a/api/bigrag/services/chat/types.py b/api/bigrag/services/chat/types.py index 13c3f467..5a000553 100644 --- a/api/bigrag/services/chat/types.py +++ b/api/bigrag/services/chat/types.py @@ -10,6 +10,7 @@ class ProviderCredential: api_key: str source: str + base_url: str | None = None @dataclass diff --git a/api/bigrag/services/collections/apply_update.py b/api/bigrag/services/collections/apply_update.py index 2d15779c..e7c9f6e2 100644 --- a/api/bigrag/services/collections/apply_update.py +++ b/api/bigrag/services/collections/apply_update.py @@ -63,7 +63,7 @@ async def apply_collection_update( if body.default_top_k is not None: collection.default_top_k = body.default_top_k fields.append("default_top_k") - if body.default_min_score is not None: + if "default_min_score" in body.model_fields_set: collection.default_min_score = body.default_min_score fields.append("default_min_score") if body.default_search_mode is not None: @@ -72,7 +72,7 @@ async def apply_collection_update( if body.chunk_strategy is not None: collection.chunk_strategy = body.chunk_strategy fields.append("chunk_strategy") - if body.metadata_schema is not None: + if "metadata_schema" in body.model_fields_set: collection.metadata_schema = body.metadata_schema fields.append("metadata_schema") return fields diff --git a/api/bigrag/services/preferences.py b/api/bigrag/services/preferences.py index ec5fa5b7..fdbc8823 100644 --- a/api/bigrag/services/preferences.py +++ b/api/bigrag/services/preferences.py @@ -5,6 +5,8 @@ from bigrag.logging import get_logger from bigrag.services import crypto from bigrag.services.credential_check import CredentialCheckError, verify_provider_credentials +from bigrag.services.provider_key_binding import chat_key_binding_signature +from bigrag.services.url_security import UnsafeOutboundUrlError, normalize_url_root logger = get_logger("bigrag.services.preferences") @@ -24,6 +26,11 @@ def _deep_merge(left: dict, right: dict) -> dict: def _normalize_sensitive(data: dict) -> dict: out = {**data} + if isinstance(out.get("chat"), dict): + chat = {**out["chat"]} + chat.pop("openai_key_base_url", None) + chat.pop("openai_key_base_url_signature", None) + out["chat"] = chat for parent, key in _SENSITIVE_PATHS: sub = out.get(parent) if not isinstance(sub, dict) or key not in sub: @@ -34,14 +41,14 @@ def _normalize_sensitive(data: dict) -> dict: return out -async def _validate_sensitive(data: dict) -> None: +async def _validate_sensitive(data: dict) -> dict: chat = data.get("chat") if not isinstance(chat, dict) or "openai_key" not in chat: - return + return data value = chat.get("openai_key") if value is None or value == "": - return + return data if not isinstance(value, str): raise HTTPException(status_code=422, detail="OpenAI API key must be a string.") @@ -49,13 +56,14 @@ async def _validate_sensitive(data: dict) -> None: runtime = await get_values(["chat_provider", "chat_base_url"]) try: + base_url = normalize_url_root(runtime["chat_base_url"] or "https://api.openai.com/v1") await verify_provider_credentials( runtime["chat_provider"], value, - runtime["chat_base_url"], + base_url, url_purpose="chat", ) - except CredentialCheckError as exc: + except (CredentialCheckError, UnsafeOutboundUrlError, ValueError) as exc: raise HTTPException( status_code=422, detail=( @@ -63,6 +71,14 @@ async def _validate_sensitive(data: dict) -> None: "provider and base URL, then try again." ), ) from exc + return { + **data, + "chat": { + **chat, + "openai_key_base_url": base_url, + "openai_key_base_url_signature": chat_key_binding_signature(value, base_url), + }, + } def _encrypt_sensitive(data: dict) -> dict: @@ -95,6 +111,9 @@ def _remove_cleared_sensitive(merged: dict, incoming: dict) -> dict: continue cleaned = {**sub_out} cleaned.pop(key, None) + if (parent, key) == ("chat", "openai_key"): + cleaned.pop("openai_key_base_url", None) + cleaned.pop("openai_key_base_url_signature", None) out[parent] = cleaned return out @@ -143,6 +162,8 @@ def _public_preferences(data: dict) -> dict: if not isinstance(sub, dict): continue cleaned = {**sub} + cleaned.pop("openai_key_base_url", None) + cleaned.pop("openai_key_base_url_signature", None) if key in cleaned: value = cleaned.pop(key) cleaned[f"has_{key}"] = bool(value) diff --git a/api/bigrag/services/provider_key_binding.py b/api/bigrag/services/provider_key_binding.py new file mode 100644 index 00000000..08960c49 --- /dev/null +++ b/api/bigrag/services/provider_key_binding.py @@ -0,0 +1,21 @@ +from __future__ import annotations + +import hashlib +import hmac + + +def chat_key_binding_signature(api_key: str, base_url: str) -> str: + payload = f"bigrag:chat-provider-url:v1:{base_url}".encode() + return hmac.new(api_key.encode(), payload, hashlib.sha256).hexdigest() + + +def bound_chat_key_destination(chat: dict) -> str | None: + api_key = chat.get("openai_key") + base_url = chat.get("openai_key_base_url") + signature = chat.get("openai_key_base_url_signature") + if not all(isinstance(value, str) and value for value in (api_key, base_url, signature)): + return None + expected = chat_key_binding_signature(api_key, base_url) + if not hmac.compare_digest(signature.encode(), expected.encode()): + return None + return base_url diff --git a/api/bigrag/services/upload_session_files/context.py b/api/bigrag/services/upload_session_files/context.py index fd72fb85..c42e894f 100644 --- a/api/bigrag/services/upload_session_files/context.py +++ b/api/bigrag/services/upload_session_files/context.py @@ -34,7 +34,7 @@ async def load_context( session_id, user_id=uuid.UUID(user["id"]), ) - if upload_session.status in TERMINAL_SESSION_STATUSES: + if upload_session.closed_at is not None or upload_session.status in TERMINAL_SESSION_STATUSES: raise HTTPException(status_code=409, detail="Upload session is closed") item_key = client_item_id or str(uuid7()) diff --git a/api/bigrag/services/upload_session_files/staging.py b/api/bigrag/services/upload_session_files/staging.py index 3bfa4ed6..43370e32 100644 --- a/api/bigrag/services/upload_session_files/staging.py +++ b/api/bigrag/services/upload_session_files/staging.py @@ -64,7 +64,11 @@ async def staged_rejection( item.file_size for item, _status, _error in context.rows if item.status != "failed" ) max_session_bytes = limits["max_upload_session_size_mb"] * 1024 * 1024 - existing_size = context.existing.file_size if context.existing is not None else 0 + existing_size = ( + context.existing.file_size + if context.existing is not None and context.existing.status != "failed" + else 0 + ) if uploaded_bytes - existing_size + staged.size > max_session_bytes: return await failure_response( db, diff --git a/api/bigrag/services/upload_sessions.py b/api/bigrag/services/upload_sessions.py index 2ed14ced..f1da86c9 100644 --- a/api/bigrag/services/upload_sessions.py +++ b/api/bigrag/services/upload_sessions.py @@ -108,10 +108,13 @@ def session_status_value(upload_session: UploadSession, counts: dict[str, int]) if upload_session.status == "canceled": return "canceled" active = counts["queued_files"] + counts["processing_files"] - if counts["uploaded_files"] < upload_session.total_files: + missing_files = counts["uploaded_files"] < upload_session.total_files + if missing_files and upload_session.closed_at is None: return "uploading" if counts["uploaded_files"] else "preparing" if active: return "ingesting" + if missing_files: + return "failed" if not counts["completed_files"] and (counts["failed_files"] or counts["canceled_files"]): return "failed" return "complete" @@ -129,7 +132,7 @@ async def upload_session_response( active = counts["queued_files"] + counts["processing_files"] if persist_counts: for item, document_status, _error in rows: - if deleted_document_item(item, document_status): + if item.status != "failed" and deleted_document_item(item, document_status): item.status = "canceled" item.error_message = item.error_message or "Document deleted" upload_session.status = status diff --git a/app/src/components/navigation/user-menu.tsx b/app/src/components/navigation/user-menu.tsx index 3feaf2eb..534e7493 100644 --- a/app/src/components/navigation/user-menu.tsx +++ b/app/src/components/navigation/user-menu.tsx @@ -1,5 +1,4 @@ import { Menu } from "@base-ui/react/menu"; -import { useNavigate } from "@tanstack/react-router"; import { LogOut } from "lucide-react"; import { ThemeControl } from "@/features/theme/theme-control"; import { useLogout, useSession } from "@/hooks/use-auth"; @@ -14,7 +13,6 @@ const initials = (name: string, email: string) => { }; export const UserMenu = () => { - const navigate = useNavigate(); const { data } = useSession(); const logout = useLogout(); const user = data?.user; @@ -23,7 +21,6 @@ export const UserMenu = () => { const onSignOut = async () => { await logout.mutateAsync(); - navigate({ to: "/login", replace: true }); }; return ( diff --git a/app/src/features/chat/assistant-message.tsx b/app/src/features/chat/assistant-message.tsx index cd0cfa5e..8955ec7e 100644 --- a/app/src/features/chat/assistant-message.tsx +++ b/app/src/features/chat/assistant-message.tsx @@ -12,7 +12,6 @@ import { type MutableRefObject, memo, useCallback, useEffect, useRef, useState } import { type ChatMessage, formatWholeMs } from "@/features/chat/chat-message-types"; import { MarkdownContent } from "@/features/chat/markdown/cited-markdown"; import { SourcesPanel } from "@/features/chat/sources-panel"; -import { cn } from "@/lib/cn"; const useHighlightTimerCleanup = (highlightTimer: MutableRefObject) => { useEffect( @@ -131,27 +130,24 @@ export const AssistantMessage = memo( -
- {hasError ? ( - - - {message.errorMessage} - - ) : message.content ? ( + {hasError && ( +
+ + {message.errorMessage} +
+ )} +
+ {message.content ? ( - ) : ( + ) : hasError ? null : ( - Retrieving context and drafting answer... + {message.status === "stopped" + ? "Response stopped." + : "Retrieving context and drafting answer..."} )} {isStreaming && ( diff --git a/app/src/features/chat/chat-input.tsx b/app/src/features/chat/chat-input.tsx index aba7783d..7529bc93 100644 --- a/app/src/features/chat/chat-input.tsx +++ b/app/src/features/chat/chat-input.tsx @@ -9,6 +9,7 @@ import { } from "lucide-react"; import { type KeyboardEvent, useCallback, useEffect, useRef, useState } from "react"; import { Button } from "@/components/ui/button"; +import type { ChatReadinessStatus } from "@/hooks/use-chat-readiness"; import { cn } from "@/lib/cn"; import type { Collection } from "@/types/bigrag"; import { @@ -21,6 +22,7 @@ import { SettingsMenu, ToolbarPopover, } from "./chat-input-controls"; +import { ChatReadinessNotice } from "./chat-readiness-notice"; export type { ChatState } from "./chat-input-controls"; @@ -35,6 +37,7 @@ interface Props { onPatch: (patch: ChatPatch) => void; onSend: (text: string) => void; onStop: () => void; + readiness: ChatReadinessStatus; saving: boolean; state: ChatState; } @@ -48,6 +51,7 @@ export const ChatInput = ({ onPatch, onSend, onStop, + readiness, saving, state, }: Props) => { @@ -82,7 +86,7 @@ export const ChatInput = ({ const selectedModelLabel = OPENAI_MODELS.find((m) => m.value === state.model)?.label ?? state.model; const keyIsSet = state.hasOpenAIKey; - const readyLabel = keyIsSet ? "API key saved" : "Add API key"; + const readyLabel = keyIsSet ? "Personal key saved" : "Add personal key"; return (
@@ -156,7 +160,10 @@ export const ChatInput = ({ onOpenChange={(nextOpen) => setOpen(nextOpen ? "key" : null)} trigger={ )}
+
); diff --git a/app/src/features/chat/chat-page.tsx b/app/src/features/chat/chat-page.tsx index 9dac3ebe..6f6ff949 100644 --- a/app/src/features/chat/chat-page.tsx +++ b/app/src/features/chat/chat-page.tsx @@ -21,7 +21,7 @@ export const ChatPage = () => { collection={chat.collection} collectionCount={chat.collections.length} disabled={chat.disabled} - hasOpenAIKey={chat.state.hasOpenAIKey} + readiness={chat.readiness} isGeneratingQuestions={chat.generateQuestionsPending} onGenerateQuestions={chat.handleGenerateQuestions} onSelect={chat.handleSend} @@ -46,6 +46,7 @@ export const ChatPage = () => { onPatch={chat.patchState} onSend={chat.handleSend} onStop={chat.stopStreaming} + readiness={chat.readiness} saving={chat.saving} state={chat.state} /> diff --git a/app/src/features/chat/chat-readiness-notice.tsx b/app/src/features/chat/chat-readiness-notice.tsx new file mode 100644 index 00000000..c55bad91 --- /dev/null +++ b/app/src/features/chat/chat-readiness-notice.tsx @@ -0,0 +1,20 @@ +import { RefreshCcw } from "lucide-react"; +import { Button } from "@/components/ui/button"; +import type { ChatReadinessStatus } from "@/hooks/use-chat-readiness"; + +export const ChatReadinessNotice = ({ readiness }: { readiness: ChatReadinessStatus }) => ( +
+ + {readiness.message} + + +
+); diff --git a/app/src/features/chat/chat-store.ts b/app/src/features/chat/chat-store.ts index abf7be4b..c9cc8b8d 100644 --- a/app/src/features/chat/chat-store.ts +++ b/app/src/features/chat/chat-store.ts @@ -5,11 +5,13 @@ type ChatStoreState = { collection: string; isStreaming: boolean; messages: ChatMessage[]; - appendMessages: (messages: ChatMessage[]) => void; + activeStream: AbortController | null; clearMessages: () => void; + reset: () => void; selectCollection: (collection: string) => void; - selectFirstCollection: (collections: readonly { name: string }[]) => void; - setStreaming: (isStreaming: boolean) => void; + startStream: (collection: string, messages: ChatMessage[]) => AbortController | null; + finishStream: (controller: AbortController) => void; + stopStream: (controller?: AbortController) => void; setMessages: (messages: ChatMessage[]) => void; updateMessage: (id: string, update: (message: ChatMessage) => ChatMessage) => void; }; @@ -18,33 +20,51 @@ const initialState = { collection: "", isStreaming: false, messages: [], -} satisfies Pick; + activeStream: null, +} satisfies Pick; -export const useChatStore = create()((set) => ({ +export const useChatStore = create()((set, get) => ({ ...initialState, - appendMessages: (messages) => - set((state) => ({ - messages: [...state.messages, ...messages], - })), - clearMessages: () => + clearMessages: () => { + get().activeStream?.abort(); + set({ activeStream: null, isStreaming: false, messages: [] }); + }, + reset: () => { + get().activeStream?.abort(); + set(initialState); + }, + selectCollection: (collection) => { + if (get().collection === collection) return; + get().activeStream?.abort(); + set({ collection, activeStream: null, isStreaming: false, messages: [] }); + }, + startStream: (collection, messages) => { + const state = get(); + if (state.activeStream || state.collection !== collection) return null; + const controller = new AbortController(); set({ - isStreaming: false, - messages: [], - }), - selectCollection: (collection) => - set((state) => ({ - collection, - messages: state.collection === collection ? state.messages : [], - isStreaming: false, - })), - selectFirstCollection: (collections) => - set((state) => { - const first = collections[0]; - if (state.collection || !first) return state; - return { collection: first.name }; - }), + activeStream: controller, + isStreaming: true, + messages: [...state.messages, ...messages], + }); + return controller; + }, + finishStream: (controller) => { + if (get().activeStream !== controller) return; + set({ activeStream: null, isStreaming: false }); + }, + stopStream: (controller) => { + const state = get(); + if (!state.activeStream || (controller && state.activeStream !== controller)) return; + state.activeStream.abort(); + const messages = state.messages.map((message, index) => + index === state.messages.length - 1 && message.role === "assistant" && !message.status + ? { ...message, status: "stopped" as const } + : message, + ); + set({ activeStream: null, isStreaming: false, messages }); + }, setMessages: (messages) => set({ messages }), - setStreaming: (isStreaming) => set({ isStreaming }), updateMessage: (id, update) => set((state) => { const index = state.messages.findIndex((message) => message.id === id); diff --git a/app/src/features/chat/empty-prompts.tsx b/app/src/features/chat/empty-prompts.tsx index 8076bb05..e8c852b7 100644 --- a/app/src/features/chat/empty-prompts.tsx +++ b/app/src/features/chat/empty-prompts.tsx @@ -1,11 +1,12 @@ import { ArrowUpRight, RefreshCcw, Shuffle, TriangleAlert } from "lucide-react"; import { Button } from "@/components/ui/button"; +import type { ChatReadinessStatus } from "@/hooks/use-chat-readiness"; interface Props { collection: string; collectionCount: number; disabled?: boolean; - hasOpenAIKey: boolean; + readiness: ChatReadinessStatus; isGeneratingQuestions: boolean; onGenerateQuestions: () => void; onSelect: (text: string) => void; @@ -16,19 +17,20 @@ export const EmptyPrompts = ({ collection, collectionCount, disabled, - hasOpenAIKey, + readiness, isGeneratingQuestions, onGenerateQuestions, onSelect, questions, }: Props) => { const missingCollection = collectionCount === 0 || !collection; - const notice = hasOpenAIKey + const notice = readiness.ready ? missingCollection ? "Choose a collection to start asking questions." : null - : "Add an API key to start asking questions."; - const canGenerateQuestions = !disabled && !missingCollection && !isGeneratingQuestions; + : readiness.message; + const canGenerateQuestions = + readiness.ready && !disabled && !missingCollection && !isGeneratingQuestions; const hasQuestions = questions.length > 0; return ( @@ -45,7 +47,7 @@ export const EmptyPrompts = ({ {notice && (
- {notice} + {notice}
)} @@ -105,7 +107,8 @@ export const EmptyPrompts = ({ {isGeneratingQuestions ? "Generating" : "Generate 5 questions"}

- Uses your saved AI key to create questions from ready documents in this collection. + Uses eligible chat credentials to create questions from ready documents in this + collection.

)} diff --git a/app/src/features/chat/use-chat-page-controller.ts b/app/src/features/chat/use-chat-page-controller.ts index 38892ad0..7fcec961 100644 --- a/app/src/features/chat/use-chat-page-controller.ts +++ b/app/src/features/chat/use-chat-page-controller.ts @@ -1,3 +1,4 @@ +import { useQueryClient } from "@tanstack/react-query"; import { useCallback, useEffect, useMemo } from "react"; import { toast } from "sonner"; import { useShallow } from "zustand/react/shallow"; @@ -6,8 +7,10 @@ import { defaultChatState } from "@/features/chat/chat-page-defaults"; import { useChatStore } from "@/features/chat/chat-store"; import { useChatStreaming } from "@/features/chat/use-chat-streaming"; import { useChatQuestionSuggestions, useGenerateChatQuestions } from "@/hooks/use-chat"; +import { useChatReadiness } from "@/hooks/use-chat-readiness"; import { useCollections } from "@/hooks/use-collections"; import { usePreferences, useUpdatePreferences } from "@/hooks/use-preferences"; +import { isCurrentSession } from "@/lib/session-state"; const useSelectAvailableCollection = ( collections: { name: string }[], @@ -27,34 +30,24 @@ const useSelectAvailableCollection = ( }; export const useChatPageController = () => { + const queryClient = useQueryClient(); + const readiness = useChatReadiness(); const prefsQuery = usePreferences(); const updatePrefs = useUpdatePreferences(); const generateQuestions = useGenerateChatQuestions(); const { data: collectionsData, isPending: collectionsLoading } = useCollections(); const collections = useMemo(() => collectionsData?.collections ?? [], [collectionsData]); - const { - appendMessages, - clearMessages, - collection, - isStreaming, - messages, - selectCollection, - setMessages, - setStreaming, - updateMessage, - } = useChatStore( - useShallow((state) => ({ - appendMessages: state.appendMessages, - clearMessages: state.clearMessages, - collection: state.collection, - isStreaming: state.isStreaming, - messages: state.messages, - selectCollection: state.selectCollection, - setMessages: state.setMessages, - setStreaming: state.setStreaming, - updateMessage: state.updateMessage, - })), - ); + const { clearMessages, collection, isStreaming, messages, selectCollection, setMessages } = + useChatStore( + useShallow((state) => ({ + clearMessages: state.clearMessages, + collection: state.collection, + isStreaming: state.isStreaming, + messages: state.messages, + selectCollection: state.selectCollection, + setMessages: state.setMessages, + })), + ); useSelectAvailableCollection(collections, collection, selectCollection); @@ -76,6 +69,7 @@ export const useChatPageController = () => { const patchState = useCallback( (patch: Partial & { openaiKey?: string }) => { + if (!isCurrentSession(queryClient)) return; const mapped: Record = {}; if (patch.openaiKey !== undefined) mapped.openai_key = patch.openaiKey; if (patch.model !== undefined) mapped.model = patch.model; @@ -89,11 +83,13 @@ export const useChatPageController = () => { { chat: mapped }, { onSuccess: () => { + if (!isCurrentSession(queryClient)) return; if (patch.openaiKey !== undefined) { toast.success(patch.openaiKey ? "OpenAI key saved" : "OpenAI key cleared"); } }, onError: (error) => { + if (!isCurrentSession(queryClient)) return; if (patch.openaiKey !== undefined) { toast.error(error instanceof Error ? error.message : "Could not save OpenAI key"); } @@ -101,17 +97,14 @@ export const useChatPageController = () => { }, ); }, - [updatePrefs], + [queryClient, updatePrefs], ); const questionsQuery = useChatQuestionSuggestions(collection); const { handleSend, stopStreaming } = useChatStreaming({ - appendMessages, + canUseCredentials: readiness.canUseCredentials, collection, - isStreaming, - setStreaming, state, - updateMessage, }); const handleCollectionChange = useCallback( @@ -122,13 +115,13 @@ export const useChatPageController = () => { ); const handleGenerateQuestions = useCallback(() => { - if (!collection) return; + if (!collection || !readiness.canUseCredentials() || generateQuestions.isPending) return; generateQuestions.mutate({ collection: collection, model: state.model, temperature: state.temperature, }); - }, [collection, generateQuestions, state.model, state.temperature]); + }, [collection, generateQuestions, readiness.canUseCredentials, state.model, state.temperature]); const handleClear = useCallback(() => { stopStreaming(); @@ -137,11 +130,16 @@ export const useChatPageController = () => { const resendFrom = useCallback( (messageIndex: number, content: string) => { + if (!readiness.canUseCredentials()) { + toast.error("Check chat credentials before retrying"); + return; + } + stopStreaming(); const currentMessages = useChatStore.getState().messages; setMessages(currentMessages.slice(0, messageIndex)); void handleSend(content); }, - [handleSend, setMessages], + [handleSend, readiness.canUseCredentials, setMessages, stopStreaming], ); const handleEditUserMessage = useCallback( @@ -175,7 +173,7 @@ export const useChatPageController = () => { return { collection, collections, - disabled: !state.hasOpenAIKey || !collection, + disabled: !readiness.status.ready || !collection, generateQuestionsPending: generateQuestions.isPending, handleClear, handleCollectionChange, @@ -188,6 +186,7 @@ export const useChatPageController = () => { messages, patchState, questions: questionsQuery.data?.questions ?? [], + readiness: readiness.status, saving: updatePrefs.isPending, state, stopStreaming, diff --git a/app/src/features/chat/use-chat-streaming.ts b/app/src/features/chat/use-chat-streaming.ts index 3c898e15..4ffbe0f7 100644 --- a/app/src/features/chat/use-chat-streaming.ts +++ b/app/src/features/chat/use-chat-streaming.ts @@ -5,52 +5,49 @@ import type { ChatState } from "@/features/chat/chat-input"; import type { ChatMessage } from "@/features/chat/chat-messages"; import { createChatMessageId } from "@/features/chat/chat-page-defaults"; import { normalizeTimings, timingsFromRetrieval } from "@/features/chat/chat-page-timings"; -import type { useChatStore } from "@/features/chat/chat-store"; +import { useChatStore } from "@/features/chat/chat-store"; +import { refreshChatCredentials } from "@/hooks/use-chat-readiness"; import { streamChat } from "@/lib/chat-stream"; -import { queryKeys } from "@/lib/query-keys"; - -type ChatStoreState = ReturnType; +import { withSessionSignal } from "@/lib/session-request"; +import { isCurrentSession } from "@/lib/session-state"; type ChatStreamingOptions = { - appendMessages: ChatStoreState["appendMessages"]; + canUseCredentials: () => boolean; collection: string; - isStreaming: boolean; - setStreaming: ChatStoreState["setStreaming"]; state: ChatState; - updateMessage: ChatStoreState["updateMessage"]; }; export const useChatStreaming = ({ - appendMessages, + canUseCredentials, collection, - isStreaming, - setStreaming, state, - updateMessage, }: ChatStreamingOptions) => { const queryClient = useQueryClient(); const abortRef = useRef(null); + const flushRef = useRef<(() => void) | null>(null); + const { startStream, finishStream, stopStream, updateMessage } = useChatStore.getState(); useEffect( () => () => { - abortRef.current?.abort(); + flushRef.current?.(); + if (abortRef.current) stopStream(abortRef.current); abortRef.current = null; - setStreaming(false); }, - [setStreaming], + [stopStream], ); const stopStreaming = useCallback(() => { - abortRef.current?.abort(); + flushRef.current?.(); + if (abortRef.current) stopStream(abortRef.current); abortRef.current = null; - setStreaming(false); - }, [setStreaming]); + }, [stopStream]); const handleSend = useCallback( async (text: string) => { - if (isStreaming) return; - if (!state.hasOpenAIKey) { - toast.error("Add your OpenAI API key first"); + if (useChatStore.getState().activeStream) return; + if (!isCurrentSession(queryClient)) return; + if (!canUseCredentials()) { + toast.error("Check chat credentials before sending"); return; } if (!collection) { @@ -63,15 +60,20 @@ export const useChatStreaming = ({ let currentAssistantId = assistantId; const userMsg: ChatMessage = { id: userId, role: "user", content: text }; const assistantMsg: ChatMessage = { id: assistantId, role: "assistant", content: "" }; - appendMessages([userMsg, assistantMsg]); - setStreaming(true); - - const controller = new AbortController(); + const controller = startStream(collection, [userMsg, assistantMsg]); + if (!controller) return; abortRef.current = controller; + const ownsStream = () => + isCurrentSession(queryClient) && useChatStore.getState().activeStream === controller; let deltaBuffer = ""; let deltaFrame: number | null = null; const flushDelta = () => { + if (deltaFrame !== null) window.cancelAnimationFrame(deltaFrame); deltaFrame = null; + if (!ownsStream()) { + deltaBuffer = ""; + return; + } if (!deltaBuffer) return; const delta = deltaBuffer; deltaBuffer = ""; @@ -80,79 +82,88 @@ export const useChatStreaming = ({ content: message.content + delta, })); }; + flushRef.current = flushDelta; const enqueueDelta = (delta: string) => { deltaBuffer += delta; if (deltaFrame === null) { deltaFrame = window.requestAnimationFrame(flushDelta); } }; - const refreshPreferencesIfCredentialError = (message: string) => { - if (message.includes("OpenAI rejected") || message.includes("Save an OpenAI API key")) { - queryClient.invalidateQueries({ queryKey: queryKeys.preferences() }); - } - }; - try { - await streamChat({ - signal: controller.signal, - body: { - message: text, - collection: collection, - model_provider: "openai", - model: state.model, - temperature: state.temperature, - top_k: state.topK, - search_mode: state.searchMode, - rerank: state.rerank, - multimodal: state.multimodal, - system_prompt: state.systemPrompt, - }, - onEvent: (event) => { - if (event.event === "sources") { - updateMessage(currentAssistantId, (message) => ({ - ...message, - meta: { - collection: event.data.collection, - sources: event.data.sources, - timings: normalizeTimings(event.data.timings), - }, - })); - return; - } - if (event.event === "delta") { - enqueueDelta(event.data.delta); - return; - } - if (event.event === "assistant_message") { - flushDelta(); - updateMessage(currentAssistantId, (message) => ({ - ...message, - id: event.data.id, - content: event.data.content, - status: event.data.status, - errorMessage: event.data.error_message, - meta: { - collection: collection, - sources: event.data.sources, - timings: timingsFromRetrieval(event.data), - }, - })); - currentAssistantId = event.data.id; - return; - } - if (event.event === "error") { - flushDelta(); - updateMessage(currentAssistantId, (message) => ({ - ...message, - status: "error", - errorMessage: event.data.error, - })); - refreshPreferencesIfCredentialError(event.data.error); - toast.error(event.data.error); - } - }, - }); + await withSessionSignal( + queryClient, + (signal) => + streamChat({ + signal, + body: { + message: text, + collection: collection, + model_provider: "openai", + model: state.model, + temperature: state.temperature, + top_k: state.topK, + search_mode: state.searchMode, + rerank: state.rerank, + multimodal: state.multimodal, + system_prompt: state.systemPrompt, + }, + onEvent: (event) => { + if (!ownsStream()) return; + if (event.event === "sources") { + updateMessage(currentAssistantId, (message) => ({ + ...message, + meta: { + collection: event.data.collection, + sources: event.data.sources, + timings: normalizeTimings(event.data.timings), + }, + })); + return; + } + if (event.event === "delta") { + enqueueDelta(event.data.delta); + return; + } + if (event.event === "assistant_message") { + flushDelta(); + updateMessage(currentAssistantId, (message) => ({ + ...message, + id: event.data.id, + content: event.data.content, + status: event.data.status, + errorMessage: event.data.error_message, + meta: { + collection: collection, + sources: event.data.sources, + timings: timingsFromRetrieval(event.data), + }, + })); + currentAssistantId = event.data.id; + return; + } + if (event.event === "error") { + flushDelta(); + updateMessage(currentAssistantId, (message) => ({ + ...message, + status: "error", + errorMessage: event.data.error, + })); + void refreshChatCredentials(queryClient); + toast.error(event.data.error); + } + }, + }), + controller.signal, + ); + if (ownsStream()) { + flushDelta(); + updateMessage(currentAssistantId, (message) => ({ + ...message, + status: message.status ?? "complete", + })); + } } catch (err) { + if (!ownsStream()) return; if (err instanceof DOMException && err.name === "AbortError") { flushDelta(); updateMessage(currentAssistantId, (chatMessage) => ({ @@ -167,7 +178,7 @@ export const useChatStreaming = ({ status: "error", errorMessage: message, })); - refreshPreferencesIfCredentialError(message); + void refreshChatCredentials(queryClient); toast.error(message); } } finally { @@ -175,17 +186,19 @@ export const useChatStreaming = ({ window.cancelAnimationFrame(deltaFrame); flushDelta(); } - setStreaming(false); - abortRef.current = null; + finishStream(controller); + if (abortRef.current === controller) { + abortRef.current = null; + flushRef.current = null; + } } }, [ - appendMessages, collection, - isStreaming, + canUseCredentials, queryClient, - setStreaming, - state.hasOpenAIKey, + finishStream, + startStream, state.model, state.multimodal, state.rerank, diff --git a/app/src/features/collections/collection-settings/allowed-file-types-card.tsx b/app/src/features/collections/collection-settings/allowed-file-types-card.tsx index 1b4d4e9a..f798935b 100644 --- a/app/src/features/collections/collection-settings/allowed-file-types-card.tsx +++ b/app/src/features/collections/collection-settings/allowed-file-types-card.tsx @@ -22,7 +22,8 @@ export const AllowedFileTypesCard = ({ saving: boolean; onSave: () => void; }) => { - const allSelected = allowedTypes.size === ALL_FILE_TYPES.length; + const allSelected = + allowedTypes.size === ALL_FILE_TYPES.length && ALL_FILE_TYPES.every((t) => allowedTypes.has(t)); const noneSelected = allowedTypes.size === 0; return ( @@ -30,7 +31,7 @@ export const AllowedFileTypesCard = ({ Allowed file types - Restrict uploads by extension. Leave all selected to allow the default ingest set. + Filter Studio uploads by extension. Select all to remove the upload filter. @@ -66,8 +67,11 @@ export const AllowedFileTypesCard = ({ ); })} +

+ Select at least one file type, or choose Select all. +

-
diff --git a/app/src/features/collections/collection-settings/collection-settings.tsx b/app/src/features/collections/collection-settings/collection-settings.tsx index f5683112..09c64129 100644 --- a/app/src/features/collections/collection-settings/collection-settings.tsx +++ b/app/src/features/collections/collection-settings/collection-settings.tsx @@ -81,7 +81,12 @@ export const CollectionSettings = ({ name }: { name: string }) => { const saveFileTypes = async () => { const list = draft.fileTypesDraft; - const unrestricted = list.length === ALL_FILE_TYPES.length; + if (list.length === 0) { + toast.error("Select at least one file type, or choose Select all."); + return; + } + const unrestricted = + list.length === ALL_FILE_TYPES.length && ALL_FILE_TYPES.every((type) => list.includes(type)); try { await update.mutateAsync({ metadata: { @@ -91,8 +96,8 @@ export const CollectionSettings = ({ name }: { name: string }) => { }); toast.success( unrestricted - ? "All file types allowed" - : `Restricted to ${list.length} type${list.length === 1 ? "" : "s"}`, + ? "Studio upload filter removed" + : `Studio uploads limited to ${list.length} type${list.length === 1 ? "" : "s"}`, ); } catch (err) { toast.error(err instanceof Error ? err.message : "Save failed"); diff --git a/app/src/features/collections/documents/documents-tab-model.ts b/app/src/features/collections/documents/documents-tab-model.ts index 1508dd7d..92690615 100644 --- a/app/src/features/collections/documents/documents-tab-model.ts +++ b/app/src/features/collections/documents/documents-tab-model.ts @@ -20,7 +20,9 @@ export const defaultDocumentsTabFilters: DocumentsTabFilters = { export const shouldDismissUploadSession = (session: UploadSession) => (session.status === "complete" || session.status === "failed" || session.status === "canceled") && session.active_files === 0 && - session.failed_files === 0; + session.failed_files === 0 && + session.canceled_files === 0 && + session.uploaded_files >= session.total_files; export const getErrorStatus = (error: unknown) => { if (!error || typeof error !== "object") return undefined; diff --git a/app/src/features/collections/documents/documents-tab.tsx b/app/src/features/collections/documents/documents-tab.tsx index c3427438..bf07707c 100644 --- a/app/src/features/collections/documents/documents-tab.tsx +++ b/app/src/features/collections/documents/documents-tab.tsx @@ -22,7 +22,7 @@ import { } from "@/features/collections/documents/upload-session-status-cards"; import { useDocumentUpload } from "@/features/collections/documents/use-document-upload"; import { useDocumentsTabState } from "@/features/collections/documents/use-documents-tab-state"; -import { useUploadSessionStore } from "@/features/collections/upload-session-store"; +import { useActiveUploadSession } from "@/features/collections/use-active-upload-session"; import { WorkerOfflineBanner } from "@/features/workers/worker-status-banner"; import { useCollection } from "@/hooks/use-collections"; import { @@ -44,9 +44,8 @@ type DocumentsTabProps = { export const DocumentsTab = ({ filters, name, onFiltersChange }: DocumentsTabProps) => { const activeFilters = filters ?? defaultDocumentsTabFilters; - const activeSessionId = useUploadSessionStore((state) => state.activeSessionIds[name] ?? null); - const clearActiveSessionId = useUploadSessionStore((state) => state.clearActiveSessionId); - const setActiveSessionId = useUploadSessionStore((state) => state.setActiveSessionId); + const { activeSessionId, clearActiveSessionId, setActiveSessionId } = + useActiveUploadSession(name); const { data: collection } = useCollection(name); const documentFilters: DocumentListFilters = { diff --git a/app/src/features/collections/documents/upload-session-panel.tsx b/app/src/features/collections/documents/upload-session-panel.tsx index 49e17973..afbf05e3 100644 --- a/app/src/features/collections/documents/upload-session-panel.tsx +++ b/app/src/features/collections/documents/upload-session-panel.tsx @@ -34,6 +34,7 @@ const sessionProgress = (session: UploadSession) => { session.completed_files + session.failed_files + session.canceled_files + + (session.closed_at ? Math.max(0, session.total_files - session.uploaded_files) : 0) + session.processing_files * 0.6 + session.queued_files * 0.25; return Math.round(Math.max(0, Math.min(1, weighted / session.total_files)) * 100); @@ -59,11 +60,12 @@ export const UploadSessionPanel = ({ const active = session.recent_items.find( (item) => item.status === "queued" || item.status === "ingesting", ); - const failedItems = session.recent_items.filter((item) => item.status === "failed"); - const statusLabel = - session.status === "complete" && session.failed_files - ? "finished with failures" - : session.status; + const rejectedItems = session.recent_items.filter( + (item) => item.status === "failed" || item.status === "canceled", + ); + const completedWithIssues = + session.status === "complete" && session.failed_files + session.canceled_files > 0; + const statusLabel = completedWithIssues ? "finished with issues" : session.status; return ( @@ -72,7 +74,7 @@ export const UploadSessionPanel = ({
Upload session - + {statusLabel}
@@ -114,7 +116,11 @@ export const UploadSessionPanel = ({ label="Ingesting" value={session.processing_files + session.queued_files} /> - + } label="Failed" @@ -142,11 +148,20 @@ export const UploadSessionPanel = ({
)} - {failedItems.length > 0 && ( + {session.closed_at && remaining > 0 && ( +

+ {remaining} file{remaining === 1 ? " was" : "s were"} not received before the upload + closed. Select those files again to retry. +

+ )} + + {rejectedItems.length > 0 && (
- {failedItems.slice(0, 2).map((item) => ( -
- failed + {rejectedItems.slice(0, 2).map((item) => ( +
+ + {item.status} + {item.filename} {item.error_message ?? "Upload failed"} diff --git a/app/src/features/collections/upload-session-store.ts b/app/src/features/collections/upload-session-store.ts index 44ceb515..b6b07f2a 100644 --- a/app/src/features/collections/upload-session-store.ts +++ b/app/src/features/collections/upload-session-store.ts @@ -2,12 +2,14 @@ import { create } from "zustand"; import { type PersistStorage, persist, type StorageValue } from "zustand/middleware"; type UploadSessionStoreState = { + ownerId: string | null; activeSessionIds: Record; - clearActiveSessionId: (collection: string) => void; + setOwner: (ownerId: string | null) => void; + clearActiveSessionId: (collection: string, sessionId: string | null) => void; setActiveSessionId: (collection: string, sessionId: string) => void; }; -type PersistedUploadSessionState = Pick; +type PersistedUploadSessionState = Pick; const STORAGE_KEY = "bigrag:upload-sessions"; @@ -18,40 +20,55 @@ const getLocalStorage = () => { const uploadSessionStorage: PersistStorage = { getItem: (name) => { - const value = getLocalStorage()?.getItem(name); - return value ? (JSON.parse(value) as StorageValue) : null; + try { + const value = getLocalStorage()?.getItem(name); + return value ? (JSON.parse(value) as StorageValue) : null; + } catch { + return null; + } }, removeItem: (name) => { - getLocalStorage()?.removeItem(name); + try { + getLocalStorage()?.removeItem(name); + } catch {} }, setItem: (name, value) => { - getLocalStorage()?.setItem(name, JSON.stringify(value)); + try { + getLocalStorage()?.setItem(name, JSON.stringify(value)); + } catch {} }, }; export const useUploadSessionStore = create()( persist( (set) => ({ + ownerId: null, activeSessionIds: {}, - clearActiveSessionId: (collection) => + setOwner: (ownerId) => + set((state) => ({ + ownerId, + activeSessionIds: ownerId && ownerId === state.ownerId ? state.activeSessionIds : {}, + })), + clearActiveSessionId: (collection, sessionId) => set((state) => { + if (state.activeSessionIds[collection] !== sessionId) return state; const activeSessionIds = { ...state.activeSessionIds }; delete activeSessionIds[collection]; return { activeSessionIds }; }), setActiveSessionId: (collection, sessionId) => - set((state) => ({ - activeSessionIds: { - ...state.activeSessionIds, - [collection]: sessionId, - }, - })), + set((state) => + state.ownerId + ? { activeSessionIds: { ...state.activeSessionIds, [collection]: sessionId } } + : state, + ), }), { name: STORAGE_KEY, - partialize: (state) => ({ activeSessionIds: state.activeSessionIds }), + partialize: (state) => ({ ownerId: state.ownerId, activeSessionIds: state.activeSessionIds }), + migrate: () => ({ ownerId: null, activeSessionIds: {} }), storage: uploadSessionStorage, - version: 1, + version: 2, }, ), ); diff --git a/app/src/features/collections/use-active-upload-session.ts b/app/src/features/collections/use-active-upload-session.ts new file mode 100644 index 00000000..43c347f6 --- /dev/null +++ b/app/src/features/collections/use-active-upload-session.ts @@ -0,0 +1,27 @@ +import { useQueryClient } from "@tanstack/react-query"; +import { useCallback } from "react"; +import { useUploadSessionStore } from "@/features/collections/upload-session-store"; +import { getSessionState, isCurrentSession } from "@/lib/session-state"; + +export const useActiveUploadSession = (collection: string) => { + const client = useQueryClient(); + const ownerId = useUploadSessionStore((state) => state.ownerId); + const storedId = useUploadSessionStore((state) => state.activeSessionIds[collection] ?? null); + const activeSessionId = + isCurrentSession(client) && ownerId && ownerId === getSessionState().ownerId ? storedId : null; + const setActiveSessionId = useCallback( + (name: string, sessionId: string) => { + if (!isCurrentSession(client)) return; + useUploadSessionStore.getState().setActiveSessionId(name, sessionId); + }, + [client], + ); + const clearActiveSessionId = useCallback( + (name: string) => { + if (!isCurrentSession(client)) return; + useUploadSessionStore.getState().clearActiveSessionId(name, activeSessionId); + }, + [activeSessionId, client], + ); + return { activeSessionId, clearActiveSessionId, setActiveSessionId }; +}; diff --git a/app/src/features/settings/tabs/account-tab.tsx b/app/src/features/settings/tabs/account-tab.tsx index 67f1efd0..5d0149c5 100644 --- a/app/src/features/settings/tabs/account-tab.tsx +++ b/app/src/features/settings/tabs/account-tab.tsx @@ -1,5 +1,4 @@ import { useForm, useStore } from "@tanstack/react-form"; -import { useNavigate } from "@tanstack/react-router"; import { useEffect, useState } from "react"; import { toast } from "sonner"; import { Button } from "@/components/ui/button"; @@ -19,7 +18,6 @@ import { AccountActiveSessionsSection } from "@/features/settings/tabs/account-a import { AccountProfileSection } from "@/features/settings/tabs/account-profile-section"; import { useChangePassword, - useLogout, useLogoutAll, useSession, useUpdateCurrentUserProfile, @@ -27,11 +25,9 @@ import { import { errorText, firstString, submitWith } from "@/lib/form"; export const AccountTab = () => { - const navigate = useNavigate(); const { data: session } = useSession(); const updateProfile = useUpdateCurrentUserProfile(); const changePassword = useChangePassword(); - const logout = useLogout(); const logoutAll = useLogoutAll(); const [profileValues, setProfileValues] = useState(defaultProfileFormValues); const [profileError, setProfileError] = useState(null); @@ -44,9 +40,6 @@ export const AccountTab = () => { onSubmit: async ({ value }) => { try { await changePassword.mutateAsync(passwordBodyFromValues(value)); - toast.success("Password updated"); - await logout.mutateAsync(); - navigate({ to: "/login", replace: true }); } catch (err) { toast.error(err instanceof Error ? err.message : "Failed"); } @@ -97,7 +90,6 @@ export const AccountTab = () => { try { await logoutAll.mutateAsync(); setSignOutAllOpen(false); - navigate({ to: "/login", replace: true }); } catch (err) { toast.error(err instanceof Error ? err.message : "Failed"); } diff --git a/app/src/features/usage/usage-page.tsx b/app/src/features/usage/usage-page.tsx index 34c44d6d..0c84ba68 100644 --- a/app/src/features/usage/usage-page.tsx +++ b/app/src/features/usage/usage-page.tsx @@ -57,7 +57,7 @@ export const UsagePage = () => { @@ -65,16 +65,18 @@ export const UsagePage = () => { Usage and cost - Per-collection document, chunk, query, and embedding-cost totals. Cost is an estimate - from a local rate card; cross-check with your provider's dashboard for the source of - truth. + The query window applies to query counts, latency, and the timeline. Document totals + reflect currently stored data. Document tokens are estimated from text length; cost uses + a local model-name rate card. Unpriced models display $0. Check your provider's + dashboard for actual token usage and spend.
- Window + Query window