Repository navigation
feat(semantic-layer): metastore scope/ACL support (targeted + org-wide) [AI-3790] - #715
Conversation
There was a problem hiding this comment.
Pull request overview
Adds CLI + service support for the metastore’s new semantic-layer visibility model (PSGO-140): items can be project-private (default), shared with specific projects (targeted grants), or elevated to organization-wide visibility, including new scope management subcommands and scope preservation across DELETE+POST edits.
Changes:
- Add
--scope project|organization|targeted+--target-projectto semantic-layer creation commands, with an interactive picker fallback for targeted scope. - Introduce
semantic-layer scopesubcommands (status/grant/request-elevation/withdraw-elevation/elevate/pending) plus corresponding service/client primitives. - Update metastore envelope schemaVersion to
1.1.0and preserve scope/grants during edit flows.
Reviewed changes
Copilot reviewed 20 out of 20 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/test_semantic_layer_scope.py | Unit tests for scope helpers + service orchestration + edit-preserves-scope regression coverage. |
| tests/test_semantic_layer_scope_cli.py | CLI tests for --scope/--target-project and semantic-layer scope subcommands + permission gating. |
| tests/test_metastore_client.py | Metastore client tests updated for schemaVersion 1.1.0 and new scope/grant endpoints. |
| src/keboola_agent_cli/services/semantic_layer_service.py | Service layer wiring for scope/grants/elevation + passing scope/grants on creates. |
| src/keboola_agent_cli/services/_semantic_layer_scope.py | New helper module: alias→project_id resolution and scope/grant/elevation orchestration logic. |
| src/keboola_agent_cli/services/_semantic_layer_crud.py | Preserve scope/grants across DELETE+POST edits (including rollback). |
| src/keboola_agent_cli/permissions.py | Register permission categories for semantic-layer.scope.* operations. |
| src/keboola_agent_cli/metastore_client.py | Add scope/grant/elevation primitives; bump create envelope schemaVersion to 1.1.0. |
| src/keboola_agent_cli/commands/semantic_layer.py | Wire scope sub-app and add --scope/--target-project to model create. |
| src/keboola_agent_cli/commands/context.py | Update generated context docs for new scope flags and scope subcommands. |
| src/keboola_agent_cli/commands/_semantic_layer_scope.py | New Typer sub-app implementing semantic-layer scope ... CLI surface. |
| src/keboola_agent_cli/commands/_semantic_layer_helpers.py | Add resolve_scope_targets helper with interactive picker / non-interactive fail-fast behavior. |
| src/keboola_agent_cli/commands/_semantic_layer_crud.py | Add --scope/--target-project plumbing to semantic-layer add <kind>. |
| plugins/kbagent/skills/kbagent/SKILL.md | Add command table entries for new semantic-layer scope commands (and sl alias). |
| plugins/kbagent/skills/kbagent/references/semantic-layer-workflow.md | Cross-link to the new scope workflow guidance. |
| plugins/kbagent/skills/kbagent/references/metastore-scope-workflow.md | New workflow doc for sharing/elevation with strong “ask user first” safety rules. |
| plugins/kbagent/skills/kbagent/references/gotchas.md | Add PSGO-140 gotchas (schemaVersion 1.1.0 requirement, replace semantics, 403 vs 404, etc.). |
| plugins/kbagent/skills/kbagent/references/commands-reference.md | Document new scope flags and semantic-layer scope subcommands. |
| plugins/kbagent/agents/keboola-expert.md | Add explicit rule preventing agents from widening scope without user-specified targets. |
| CLAUDE.md | Update command inventory and add scope feature notes + new commands. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
e5c7c64 to
8c8fd94
Compare
Command-line interface: align the new commands with the planned spec#791 (CLI-20) plans one specification for the kbagent command-line interface and a CI check (
CommandsThe draft spec uses a fixed set of verbs:
Options
Permissions
Behavior fixes
Examples# Create with a visibility scope
kbagent semantic-layer model create --project dev --name sales --scope targeted --target-project prod,5678
kbagent semantic-layer add dataset --project dev --model sales --name orders --table-id out.c-sales.orders --scope organization
# Show the scope of one item
kbagent semantic-layer scope get --project dev --type dataset --context-id 7f3c...
# Target projects: add, remove, replace the list, clear the list
kbagent semantic-layer scope add --project dev --type dataset --context-id 7f3c... --target-project prod --target-project 5678
kbagent semantic-layer scope remove --project dev --type dataset --context-id 7f3c... --target-project 5678
kbagent semantic-layer scope set --project dev --type dataset --context-id 7f3c... --target-project prod,5678
kbagent semantic-layer scope set --project dev --type dataset --context-id 7f3c... --clear
# Elevation request (project admin): create, delete
kbagent semantic-layer scope request-create --project dev --type dataset --context-id 7f3c...
kbagent semantic-layer scope request-delete --project dev --type dataset --context-id 7f3c...
# Org admin: list the requests, preview the elevation, elevate
kbagent semantic-layer scope request-list --project org-admin --type dataset --limit 50
kbagent semantic-layer scope set --project org-admin --type dataset --context-id 7f3c... --scope organization --dry-run
kbagent semantic-layer scope set --project org-admin --type dataset --context-id 7f3c... --scope organization --yesFor #791, not for this PR
|
Review: correctness and rebaseThe new metastore calls match the go-monorepo source: verbs, paths, bodies, the 204 on The traceback for an unknown Findings
Rebase and docs
|
…semantic layer Adds kbagent CLI support for PSGO-140's metastore scope model: `--scope project|organization|targeted` and `--target-project` on `model create` / `add <kind>`, plus a new `semantic-layer scope` command group (status/grant/request-elevation/withdraw-elevation/elevate/pending) for managing target-project grants and organization-wide elevation on existing items. Bumps the metastore envelope schema version from 1.0.0 to 1.1.0 -- every semantic-* schema only supports scope="project" at 1.0.0; 1.1.0 is what adds organization/targeted support (purely additive ACL block, verified against go-monorepo). Without this the whole feature would 400 server-side. The DELETE+POST `edit` path now reads and re-applies an item's original scope/target-project grants, so editing an organization/targeted-scope item no longer silently resets it to project scope. Widening visibility always requires an explicit --target-project (or an interactive picker on a real terminal; hard fail in --json/non-interactive) -- never a silent default. keboola-expert.md gets an explicit rule to always ask the user before passing --scope organization|targeted. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…oint response
`scope request-elevation` / `withdraw-elevation` / `elevate` rendered the
mutating endpoint's own response body. That body omits
`meta.targetProjectIds`, so the commands printed `target_project_ids: null`
for an item whose grants were fully intact.
Verified live against metastore.us-east4 (project keboola-ai, model with
scope=targeted, targets=[5024]):
scope status -> targets=[5024]
request-elevation -> targets=None <-- wrong, grants untouched
scope status -> targets=[5024]
An operator reading that output would reasonably conclude that requesting
elevation had just wiped every grant on the object.
`grant_target_projects` already re-read the item after its PUT for exactly
this reason; the three elevation helpers now do the same. Re-verified live:
command output matches `scope status` at every step.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…en for every call (PSGO-282) go-monorepo#596 (PSGO-282) fixed the metastore to accept any valid, non-disabled, non-expired Storage token for reads -- writes still need a project-admin token. Every doc this repo carries about the old behavior (added in #711/#717) still claimed the *whole* semantic-layer family needed a master token, which is now false and would make agents refuse a plain read or hunt for a master token they don't need. Updates CLAUDE.md, keboola-expert.md, commands-reference.md, gotchas.md, semantic-layer-workflow.md and docs/error-codes.md to state the real split (reads: any valid token: writes: project-admin token), and softens metastore_client.py's 401-reclassification message so it no longer overclaims a blanket master-token requirement -- kept as a safety net for a deployment that predates the fix. New gate entries use the vNEXT placeholder per this repo's release convention (check_version_gates.py). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…CLI redesign, serve routes) Correctness (review by Martin Struzsky): - edit, import --overwrite and promote update in place with put_item instead of DELETE+POST: the item keeps its id, scope, grants and pending elevation request, and a failed update changes nothing. Removes delete_then_post. - scope add/remove refuse from a project that does not own the item (the server hides the grants from a non-owner, so a merge would wipe them). - --target-project accepts an alias or numeric project ID (repeatable or comma-separated); an alias must be on the owner's stack. - add <kind> without --scope inherits the model's scope and targets. - schemaVersion 1.1.0 is sent only for non-project scopes. - post_item rejects target_project_ids=[] without targeted scope. CLI spec alignment (#791): - scope verbs: get/add/remove/set/request-create/request-delete/request-list, --id -> --context-id, scope set takes exactly one of --scope organization, --target-project or --clear, with --dry-run/--yes. - --scope/--type are fixed choices; bad values, unknown aliases and --target-project without --scope targeted exit 2. - --scope organization is destructive-class via FLAG_ESCALATIONS. - request-list returns limit/offset/has_more. Serve: 7 scope routes plus scope/target_projects on POST /models and /items/{kind}, with command-map entries and regenerated endpoint docs. Docs: CLAUDE.md, context.py, commands-reference, gotchas, scope workflow, expert prompt, SKILL.md (regenerated + workflow row); PSGO-282 wording. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
8c8fd94 to
ffb8f04
Compare
|
Thanks, all adopted in
|
|
Rebased on
Docs: added the Full suite passes (7346 passed, 189 skipped); Still open until the live run, which I'll do once the PR is green: whether a rename through PUT works, and finding 5. |
keboola-pr-reviewer-bot
left a comment
There was a problem hiding this comment.
Verdict: needs_human (risk 4/5) · profile keboola-mcp-server
Needs human: large semantic-layer ACL feature that also changes existing edit/import/promote to in-place PUT, on a contract the author says is unverified against a live stack.
Impact flags: possible rollback re-introduction — see Check Run summary.
Concerns:
src/keboola_agent_cli/services/_semantic_layer_crud.py: edit/import/promote switched from DELETE+POST-with-rollback to in-place PUT on existing commandssrc/keboola_agent_cli/metastore_client.py: new org-wide/cross-project ACL visibility elevation; one-way, no downgrade endpointPR description: feature not verified against a live stack with PSGO-140; grounded only in go-monorepo source
Re-review of ffb8f04Thanks. I checked the fixes against the code, not only against the reply. These are fixed: the in-place PUT for There are two new blocking items. The first one comes from my own finding 8. Blocking
Non-blocking
Nits
|
…d org scope) Blocking: - Create no longer sends schemaVersion. The metastore stores the version it resolves and checks elevation against the STORED version; pinning 1.0.0 on a plain create made every item created without --scope impossible to elevate. Removes _schema_version() and the version constants. - An inherited organization scope (add <kind> without --scope under an org-level model) is now permission-gated like a typed --scope organization, so --deny-destructive no longer lets it through. A non-org-admin 403 on an inherited scope gets a hint to pass --scope project. Non-blocking / nits: - scope add|remove: ownership is decided by meta.projectId only, so the owner of a targeted item with no grants can add again; an unknown own project ID says to run `kbagent project refresh`. - put_item no longer sends schemaVersion or takes a scope argument; a rename to a taken name maps to ALREADY_EXISTS. - scope set --scope organization --dry-run is gated too (same as sync push --force). - E2E: scope get/request-create/request-delete/dry-run on an item created without --scope, and an edit that keeps the item id. - Remove stale DELETE+POST wording in docstrings, comments and docs. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
|
Thanks for checking the code. Addressed in Blocking
Non-blocking
Nits
Full suite passes (7357 passed, 189 skipped, excluding |
Re-review of 1052004Thanks. Both blocking items are fixed. I checked them against the code:
The full test suite passes on the head, and One small item: The live run is still open: the new E2E block, a rename through PUT, and an elevation of an item created without |
A project admin may create a targeted item, so a 403 on an inherited targeted scope has another cause and the "needs an org-admin token" hint was wrong advice. Only add it when the inherited scope is organization. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
|
Thanks for checking The live run is still open, as you listed: the new E2E block, a rename through PUT, and an elevation of an item created without |
soustruh
left a comment
There was a problem hiding this comment.
Thanks for all the changes! 🙏
…s, destructive deletes Address the review of #778: - default to targeted scope (the schema default); --scope organization is explicit and destructive-class (FLAG_ESCALATIONS, also over REST) - --dialect defaults to and must match the project backend; duplicate principals are refused across the visible policies on the table - --target-project takes alias or ID (#715 resolver); grants go in the create request only; on update they change before the rules - update sends a PATCH of the changed keys and re-reads the result - rls/cls delete are destructive and have --dry-run - own 409 message, owner_project_id in list/detail - command shape per #791: --table-id, --dialect choice, REST table_id / target_projects and PATCH; one router builder for rls and cls - rls setup --json returns a JSON error envelope; typed wizard values Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Summary
Adds kbagent CLI and
kbagent servesupport for the new metastore ACL scope model (PSGO-140): items can now be shared with specific projects or made organization-wide, instead of only ever being project-private.--scope project|organization|targetedand--target-project ALIAS|ID(repeatable or comma-separated) onsemantic-layer model createand everysemantic-layer add <kind>. With--scopeomitted,model createmakes aprojectmodel andadd <kind>inherits its model's scope and target projects, so an org-level model does not show up in consumer projects with no datasets or metrics.semantic-layer scopecommand group, with verbs following the CLI spec (#791):get,add,remove,set,request-create,request-delete,request-list.scope settakes exactly one of--scope organization(elevation, irreversible, org-admin only),--target-project(replaces the whole list) or--clear, with--dry-runand--yes.scope add|removemerge into the grants and are refused (exit 2) from a project that does not own the item, because the server hides the grants from a non-owner and a merge would overwrite them.request-listreturnsitems,limit,offsetandhas_more.--target-projecttakes a registered alias or a numeric project ID, so a target need not be registered. An alias must be on the owner project's stack.--scope/--type,--target-projectwithout--scope targeted, a mixedscope setrequest.--scope targetedand no--target-project, a real terminal runs an interactive picker (projects on the owner's stack only);--json/non-interactive fails fast (exit 2).--scope organizationis destructive-class throughFLAG_ESCALATIONSonscope set(also with--dry-run),model createand everyadd <kind>. Anadd <kind>that would inheritorganizationfrom its model is gated the same way.getandrequest-listare read;add,remove,set,request-createandrequest-deleteare write.kbagent serve: 7 scope routes (GET|PUT /semantic-layer/scope/{context_id},POST|DELETE .../target-projects,PUT|DELETE .../elevation-request,GET /semantic-layer/scope/elevation-requests), plusscopeandtarget_projectsonPOST /semantic-layer/modelsand/items/{kind}.keboola-expert.mdhas an explicit rule: never widen an item's visibility without the user having named the target project(s) first.Bugs fixed along the way
schemaVersion: "1.0.0", which only supportsscope=project. kbagent no longer sendsschemaVersionon create: the server stores the stack's default schema, and a later elevation is checked against that stored version, so a pinned1.0.0would make every item created without--scopeimpossible to elevate.edit,import --overwriteandpromoteused DELETE+POST, which reset an organization/targeted item to project-only visibility, dropped a pending elevation request and changed the item ID. They now update in place withPUT: the item keeps its ID, scope, grants and pending request, and a failed update changes nothing (no rollback is needed any more; therollbackfield is alwaysnull). A rename to a taken name maps toALREADY_EXISTS.PSGO-282 (master-token wording, 6 doc files plus
metastore_client.py): the metastore accepts any valid token for reads and needs a project-admin token only for writes. Docs and the 401 message now say that, and the 401 message keeps the session-aware remedy frommain.Impact analysis
metastore_client.py: scope/grant/elevation primitives;post_itemgainsscope/target_project_idsand sends noschemaVersion;put_itemsends onlynameanddataand maps 409 toALREADY_EXISTS.services/_semantic_layer_scope.py(new),semantic_layer_service.py,_semantic_layer_crud.py,_semantic_layer_internals.py: scope logic, scope inheritance,child_scope(), in-place edit/overwrite paths.commands/_semantic_layer_scope.py(new),_semantic_layer_crud.py,_semantic_layer_helpers.py,semantic_layer.py: scope sub-app, flags, permission gates.server/routers/semantic_layer.py,_serve_command_map.py,docs/web-server-endpoints.md: scope routes.permissions.py:semantic-layer.scope.*entries and the--scope organizationescalations.CLAUDE.md,commands/context.py,commands-reference.md,gotchas.md,SKILL.md(regenerated, plus a workflow-table row),metastore-scope-workflow.md,semantic-layer-workflow.md,keboola-expert.md.add <kind>without--scopenow inherits the model's scope (it used to always beproject); the scope commands are renamed (not released yet, so no deprecation needed).Test plan
tests/test_metastore_client.py,tests/test_semantic_layer_scope.py(target resolution, grant merge with the non-owner guard, elevation, inheritance,child_scope, in-place edit/import/promote),tests/test_semantic_layer_scope_cli.py(flags, exit codes, permission matrix incl. inherited scope and dry-run),tests/test_server_router_calls.py(scope routes and body validation),tests/test_semantic_layer_service.py, and a new block inTestE2ESemanticLayerLifecycle(needs a live stack).pytest tests/gives 7357 passed, 189 skipped (e2e needE2E_API_TOKEN/E2E_URL), 0 failed (excludingtests/test_build_hook.py, which also fails without these changes in my environment).ruff check,ruff format --checkandty check srcare clean;check_command_sync,check_sentinel_guards,check_version_gates,check_error_codesandcheck_file_sizepass.targeted, add a target, create a request, elevate, edit, list the requests, including one item created without--scopeand one created by an earlier kbagent. Open questions that depend on it: whether a rename throughPUTis accepted, whether a create withoutschemaVersionstores the stack default, and whether items pinned to1.0.0by an older kbagent can be elevated.Related issues
Linear: AI-3790
🤖 Generated with Claude Code