Skip to content

feat(reins): port knowledge base UX and index-state work from the Mazda fork (CLEAN-60) - #57

Merged
maksymhryzodub-prog merged 2 commits into
mainfrom
feat/CLEAN-60-knowledge-port
Sep 3, 2026
Merged

feat(reins): port knowledge base UX and index-state work from the Mazda fork (CLEAN-60)#57
maksymhryzodub-prog merged 2 commits into
mainfrom
feat/CLEAN-60-knowledge-port

Conversation

@mykyta-torbynskyi

@mykyta-torbynskyi mykyta-torbynskyi commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

Brings the reins (knowledge) slice of the Mazda fork back upstream, so chore/catch-up-upstream-5 can land on a base that already contains it instead of carrying it as a merge.

Jira: CLEAN-60

What is in it

Knowledge detail screen (admin)

  • Overview tab replaces the two-field General form: counters (total / indexed / processing / failed), what the indexing service is actually running (read from its own /health, so it reflects the container, not Settings), setup checks (URL / model / bucket / reachable), editable name and description.
  • One Add sources button opening a sheet with Files / Archive / Text / URL / Sitemap tabs, replacing three permanently expanded cards above the table.
  • Active tab is highlighted. NuxtLink's active-class appends rather than replaces, so border-primary lost to border-transparent by stylesheet order; the class set is now bound on an explicit route match.
  • Layout no longer runs off the right edge. SidebarInset was a flex item without min-w-0, so one long unbreakable index error widened the whole inset past the viewport. One class, plus a width cap on the Status column and icon row actions.
  • Graph tab no longer freezes: /graph/label/list is unbounded (8,806 entities on a 79-document base) and the old <Select> mounted a node per entry. The picker renders at most 50 rows, filtered as you type.

Sources (api + admin)

  • Server-side paging, filter and search; per-source index status (indexed / pending / failed) with the error text; preview sheet; export the selection (or everything matching the filter) as one zip; archive import with background progress.

Index state (api)

  • indexedAt / indexError on the row are the only proof of success; a lightragDocId alone means "handed to LightRAG", never "searchable".
  • Wait budget scales with content volume (a 1 MB manual is 217 chunks and ~an hour of LLM), capped at four hours.
  • A reconciler confirms documents LightRAG finished after a run stopped waiting, and re-queues a pipeline that lost its queue. LightRAG keeps its queue in the process and document status in its database; a pod move separates the two and every component then behaves correctly forever. POST /documents/reprocess_failed is the only endpoint that re-queues (its name lies: it takes PENDING and PROCESSING too); /documents/scan looks right and is not, it globs only the top level of the input directory and enqueued files have already been moved into __enqueued__.
  • Status polling paced by batch size. A flat 3 s tick over 355 in-flight documents was ~118 requests/second for hours.

How it was assembled

Source of truth is the fork's chore/catch-up-upstream-5, which already reconciles all of this with upstream's instance isolation (#44, #46, #50). Its reins and aws/s3 paths were lifted onto upstream/main, then the fork's two reins commits newer than that merge (pipeline restart, poll pacing) were applied three-way. The two conflicts were the new getPipelineStatus / restartPipeline client methods, which now take the same optional knowledgeId the rest of the client uses, and a dead poll constant.

Also carried: S3Repository.getObjectStream (the source gateway streams file content through it), multer / @types/multer declared explicitly (upstream only had multer transitively via @nestjs/platform-express), and one migration for the Source(knowledgeId, createdAt) index. Every column this slice reads already exists upstream since 20260827135035.

Deliberately not ported: the fork's global APP_GUARD (worth its own PR: 21 of 34 controllers there answered anonymous callers), the analytics and leads slices, Mazda ingress and CORS specifics.

Verification

  • bun run build clean; bunx jest 36/36 suites, 307 tests.
  • api/swagger-spec.json regenerated from the fresh dist, admin client regenerated from it (bun run build:api).
  • Not visually verified in this checkout (no env, no API to boot the consoles against). The same code is live on the Mazda prod admin.

Follow-ups

  • The reconciler's restart addresses the shared instance. With instance isolation switched on, a per-base pipeline that lost its queue needs the same nudge per instance.
  • /documents/pipeline_status could feed the Overview tab, so a stalled pipeline is visible instead of inferred from a counter that stopped moving.

ntorbinskiy and others added 2 commits September 3, 2026 15:23
…hance API schemas

- Added 'worktrees/' to .gitignore to exclude worktree files.
- Bumped package version from 0.3.41 to 0.3.42.
- Extended KnowledgeListItemDtoSchema and related types with new properties: sourceCount, indexedCount, failedCount, and processingCount.
- Introduced SourceDtoSchema, SourcePageDtoSchema, and ImportJobDtoSchema to enhance API data structures.
- Updated KnowledgeSourcesService with new methods for handling source imports and exports.
@maksymhryzodub-prog
maksymhryzodub-prog merged commit d695f7a into main Sep 3, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants