Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 17 additions & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -807,7 +807,23 @@ kbagent data-app delete --project NAME --app-id ID [--yes]
kbagent data-app password --project NAME --app-id ID
kbagent data-app logs --project NAME --app-id ID [--lines N] [--since ISO8601]
kbagent data-app runs --project NAME --app-id ID [--limit N]
kbagent data-app secrets-set --project ALIAS --app-id ID --secret '#KEY=VALUE' [--secret ...] [--secrets-file PATH] [--branch ID] [--allow-plaintext-on-encrypt-failure] [--dry-run] [--no-hint-next]
kbagent data-app secrets-set --project ALIAS --app-id ID --secret '#KEY=VALUE'|'KEY=VALUE' [--secret ...] [--secrets-file PATH] [--branch ID] [--allow-plaintext-on-encrypt-failure] [--dry-run] [--no-hint-next]
# secrets-set plaintext entries (since vNEXT, #738): the '#' prefix now DECIDES encryption
# instead of being mandatory, mirroring the UI's "optional encryption" checkbox.
# '#KEY=VALUE' is encrypted under the project KMS (unchanged); 'KEY=VALUE' (no '#') is
# written VERBATIM as a plain env var, so `secrets-list` / `config detail` / the UI show
# the real value instead of <encrypted>. Use it for non-secret runtime settings (a Storage
# Files tag, a feature flag, a log level) -- before vNEXT the only ways were to encrypt a
# non-secret (invisible to everyone) or to `sync pull` + hand-edit _config.yml + `sync push`.
# Both halves of `parameters.dataApp.secrets` were always readable/removable; only WRITING a
# plain key was impossible. A payload with NO '#' key never calls the Encryption API at all,
# so an unreachable Encryption API cannot fail a write that needs no encryption. A `KBC::`
# value is still rejected under EITHER form (double-encryption / a foreign project's
# ciphertext parked where nothing can decrypt it). --allow-plaintext-on-encrypt-failure is
# unrelated and unchanged: it covers a '#' key whose encryption FAILED, never a deliberate
# plain key. New response keys `encrypted_keys` / `plaintext_keys` (env-var names);
# `secrets_set` still lists both. `secrets-list` gains per-entry `encrypted` + `value`
# (`value` is null for encrypted entries -- the CLI never decrypts).
kbagent data-app secrets-list --project ALIAS --app-id ID [--branch ID] [--show-fingerprint]
kbagent data-app secrets-get --project ALIAS --app-id ID --key 'KEY' [--branch ID] # '#' optional; plain values return their value, encrypted return metadata only
kbagent data-app secrets-remove --project ALIAS --app-id ID --key 'KEY' [--key ...] [--branch ID] [--yes] [--dry-run] # '#' optional
Expand Down
2 changes: 1 addition & 1 deletion plugins/kbagent/skills/kbagent/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -149,7 +149,7 @@ When working inside a git repository or project directory, run `kbagent init` (o
| Show the clone URLs of a data app's configured git repository | `kbagent data-app git-repo --project PROJECT --app-id APP-ID` |
| List the credentials of a data app's MANAGED git repository | `kbagent data-app git-credentials --project PROJECT --app-id APP-ID` |
| Create a git credential (SSH key or HTTP token) for a MANAGED repo | `kbagent data-app git-credentials-create --project PROJECT --app-id APP-ID --type CRED-TYPE --permissions PERMISSIONS` |
| Encrypt and write app-runtime secrets to the linked Storage config | `kbagent data-app secrets-set --project PROJECT --app-id APP-ID` |
| Write app-runtime env vars to the linked Storage config | `kbagent data-app secrets-set --project PROJECT --app-id APP-ID` |
| List the keys in parameters.dataApp.secrets, with derived runtime env-var names | `kbagent data-app secrets-list --project PROJECT --app-id APP-ID` |
| Show ONE key from parameters.dataApp.secrets | `kbagent data-app secrets-get --project PROJECT --app-id APP-ID --key KEY` |
| Remove one or more app-runtime secrets. | `kbagent data-app secrets-remove --project PROJECT --app-id APP-ID --key KEY` |
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -266,8 +266,8 @@ Lifecycle for `keboola.data-apps`. Combines Storage API (config body, git block,
- `data-app password --project NAME --app-id ID` -- read the simpleAuth password. Manage token via interactive prompt by default, or `--allow-env-manage-token` + `KBC_MANAGE_API_TOKEN` for CI on 0.29.0+. Auto-generated, not rotatable -- delete + recreate to mint a new one.
- `data-app logs --project NAME --app-id ID [--lines N] [--since ISO8601]` -- tail container logs (Data Science `/apps/{id}/logs/tail`). Plain-text body covering the full spin-up trace ([TIMING] git_clone, Cloning into /app, uv install, supervisord, runtime stack traces). Default `--lines 500`; pass `--lines 0` for the full current buffer (no server-side cap). `--lines` and `--since` are mutually exclusive on the server; `--since` requires a timezone (Z or +00:00). App must be running or recently-stopped — never-started apps return 400 "App X is not running" (recover with `data-app start` or `data-app deploy`). Closes the upstream `keboola-mcp-server` gap where `get_data_apps` hardcodes a 20-line cap; this CLI surface is unconstrained. The log buffer can echo runtime secrets the app printed to stdout/stderr — consider hygiene before piping `--json` output into AI agent context.
- `data-app runs --project NAME --app-id ID [--limit N]` -- list deployment attempts newest-first (Data Science `/apps/{id}/runs`), each with `failure_reason` + `startup_logs`. Captures setup-phase failures (e.g. git-clone errors) that produce NO container logs, so unlike `data-app logs` it works on never-started / failed apps where `data-app logs` returns HTTP 400. This is the way to find WHY a deploy reverted to stopped. Auth: ordinary project storage token only.
- `data-app secrets-set --project ALIAS --app-id ID --secret '#KEY=VALUE' [--secret ...] [--secrets-file PATH] [--branch ID] [--allow-plaintext-on-encrypt-failure] [--dry-run] [--no-hint-next]` -- encrypt and write `#`-prefixed secrets to `parameters.dataApp.secrets`. Per-project KMS encryption, fail-closed. Read-modify-write at the service layer (NOT Storage `merge=True` -- shallow). Runtime exposes each key as an env var with `#` stripped, `-` -> `_`, uppercased. Adding bumps the Storage version; the running container keeps the OLD config until the next `data-app deploy`.
- `data-app secrets-list --project ALIAS --app-id ID [--branch ID] [--show-fingerprint]` -- list secret keys + derived runtime env-var names. Never echoes encrypted ciphertext in full. `--show-fingerprint` opt-in for a short ciphertext fingerprint.
- `data-app secrets-set --project ALIAS --app-id ID --secret '#KEY=VALUE'|'KEY=VALUE' [--secret ...] [--secrets-file PATH] [--branch ID] [--allow-plaintext-on-encrypt-failure] [--dry-run] [--no-hint-next]` -- write env-var entries to `parameters.dataApp.secrets`. **Since vNEXT the `#` prefix DECIDES encryption instead of being mandatory**, mirroring the UI's "optional encryption" checkbox: `'#KEY=VALUE'` is encrypted under the project KMS, `'KEY=VALUE'` is written verbatim as a plain env var whose value stays readable in `secrets-list` / `config detail` / the UI. Use the plain form for non-secret runtime settings (a Storage Files tag, a feature flag, a log level) -- before vNEXT the only options were to encrypt a non-secret (invisible to everyone) or to `sync pull` + hand-edit `_config.yml` + `sync push`; reading and removing plain keys always worked, only writing them did not. A payload with NO `#` key never calls the Encryption API at all. A `KBC::` value is rejected under EITHER form. `--allow-plaintext-on-encrypt-failure` is unrelated and unchanged: it covers a `#` key whose encryption FAILED, never a deliberate plain key. Response adds `encrypted_keys` / `plaintext_keys` beside the combined `secrets_set`. Per-project KMS encryption, fail-closed. Read-modify-write at the service layer (NOT Storage `merge=True` -- shallow). Runtime exposes each key as an env var with `#` stripped, `-` -> `_`, uppercased. Adding bumps the Storage version; the running container keeps the OLD config until the next `data-app deploy`.
- `data-app secrets-list --project ALIAS --app-id ID [--branch ID] [--show-fingerprint]` -- list secret keys + derived runtime env-var names. Never echoes encrypted ciphertext in full. `--show-fingerprint` opt-in for a short ciphertext fingerprint. Since vNEXT each entry carries `encrypted` and `value`: a PLAIN entry shows its literal value (already stored in clear, visible via `config detail`), an ENCRYPTED entry always has `value: null` -- the CLI never decrypts.
- `data-app secrets-get --project ALIAS --app-id ID --key 'KEY' [--branch ID]` -- show ONE key from `parameters.dataApp.secrets`. The leading `#` is OPTIONAL (since v0.43.9); the block holds both encrypted secrets (`#`) and plain unencrypted env-var values, and `secrets-list` enumerates both. For an ENCRYPTED secret it stays metadata-only (`encrypted: true`, `value: null`, fingerprint/prefix) -- the decrypted plaintext is NEVER echoed (Encryption API is one-way). For a PLAIN value it returns the literal value (`encrypted: false`), which is already visible via `config detail`. NOT_FOUND on absent key (exact match, no `#KEY`<->`KEY` fuzzing); never enumerates siblings.
- `data-app secrets-remove --project ALIAS --app-id ID --key 'KEY' [--key ...] [--branch ID] [--yes] [--dry-run]` -- destructive (can break a running app at next deploy). Leading `#` OPTIONAL: removes both encrypted secrets and plain env-var keys. Idempotent: missing keys exit 0 with `removed: 0`.
- `data-app validate-repo --git-repo URL [--git-branch BRANCH] [--git-public/--no-git-public] [--git-pat-env VAR | --git-pat-file PATH] [--type python-js] [--strict]` -- pre-flight Golden-Rule check for a data-app git repo (https://help.keboola.com/data-apps/python-js/). GitHub-only; ≤5 API calls (1 tree + ≤4 contents) regardless of repo size. `--type` restricted to `python-js` in 0.28.0; streamlit / pure-Python / R / Node-only follow-up. `--strict` treats WARNs as failures. Since 0.88.0 (#636) the two `setup.sh` rules (`golden-rule.setup-sh-no-pip`, `golden-rule.setup-sh-uv-sync`) match against **comment-stripped code**: a comment reading `# never pip install` no longer BLOCKS, and a comment merely mentioning `uv sync` no longer satisfies the uv-sync rule
Expand Down
52 changes: 52 additions & 0 deletions plugins/kbagent/skills/kbagent/references/gotchas.md
Original file line number Diff line number Diff line change
Expand Up @@ -4501,6 +4501,58 @@ the web UI.
loudly when it is off. The REST surface mirrors the flag as
`"workspace": true` in the `POST /data-apps/{project}` body.

## `data-app secrets-set` writes plain env vars too -- the `#` decides encryption

*(since vNEXT, #738)*

`parameters.dataApp.secrets` has always held **two** kinds of entry, and the
Keboola UI offers exactly that choice ("optional encryption"): a `#`-prefixed
key is encrypted under the project KMS, a bare key is a plain env var whose
value stays readable. `secrets-list`, `secrets-get` and `secrets-remove`
handled both since 0.43.9. **Only writing a plain key was impossible** --
`secrets-set` demanded the `#` and encrypted unconditionally:

```bash
# before vNEXT
kbagent data-app secrets-set ... --secret 'SCORING_BACKUP_TAG=ppl-assessment-db'
# rejected: entries must be '#KEY=VALUE'
kbagent data-app secrets-set ... --secret '#SCORING_BACKUP_TAG=ppl-assessment-db'
# accepted -- and the tag is now <encrypted> to everyone, including the operator
```

The workaround was `sync pull`, hand-editing `_config.yml`, `sync push`. Now:

```bash
kbagent data-app secrets-set --project P --app-id 74021026 \
--secret '#API_KEY=s3cr3t' \
--secret 'SCORING_BACKUP_TAG=ppl-assessment-db'
```

- **The `#` is the switch, not a syntax requirement.** `#KEY=VALUE` encrypts
(behaviour unchanged); `KEY=VALUE` writes the value verbatim. Put non-secret
runtime configuration -- a Storage Files tag, a feature flag, a log level --
in the plain form so `secrets-list`, `config detail` and the UI show the real
value instead of `<encrypted>`.
- **Encrypting a non-secret is not a safe default.** It is write-only: the
Encryption API has no decrypt endpoint, so once a tag is encrypted **nobody**
-- operator, UI or CLI -- can read back what it was set to.
- **A payload with no `#` key never calls the Encryption API.** An unreachable
Encryption API must not fail a write that needs no encryption.
- **A `KBC::` value is still rejected under EITHER form.** Under `#` it would
double-encrypt; under a bare key it would park a foreign project's ciphertext
where nothing can ever decrypt it.
- **`--allow-plaintext-on-encrypt-failure` is unrelated and unchanged.** It
covers a `#` key whose *encryption failed*, and still warns loudly. It has
nothing to do with a deliberate plain key, and the response's
`plaintext_written` (encryption failures) stays distinct from the new
`plaintext_keys` (deliberate plain entries).
- **New response keys:** `encrypted_keys` and `plaintext_keys` (derived
env-var names); `secrets_set` still lists both, so existing parsers are
unaffected. `secrets-list` entries gain `encrypted` and `value` -- `value` is
`null` for every encrypted entry.
- **`--secrets-file` follows the same rule**: a JSON object whose `#`-prefixed
keys are encrypted and whose bare keys are written in clear.

## `token list --with-last-used`: `never` and `unknown` are not the same answer (since v0.88.0)

`--with-last-used` derives each token's most recent activity so you can tell
Expand Down
58 changes: 43 additions & 15 deletions src/keboola_agent_cli/commands/_data_app_runtime.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,8 @@
Split out of ``data_app.py`` to keep that module under the CONTRIBUTING.md
file-size budget (see "File-size budgets"). These commands manage the
``parameters.dataApp.secrets`` block of a data app's Storage config (the
app-runtime env-var secrets); the business logic stays on ``DataAppService``.
app-runtime env vars -- ``#``-prefixed keys encrypted, bare keys plain); the
business logic stays on ``DataAppService``.
They attach to the existing ``data-app`` Typer sub-app via
:func:`register_secrets_commands`, called at the bottom of ``data_app.py``, so
they still surface as ``kbagent data-app secrets-*`` with identical names,
Expand All @@ -28,19 +29,22 @@


def _parse_secret_arg(arg: str) -> tuple[str, str]:
"""Split ``#KEY=VALUE`` into ``(key, value)``.
"""Split ``#KEY=VALUE`` or ``KEY=VALUE`` into ``(key, value)``.

The value may contain ``=``; only the FIRST ``=`` is the separator.
The ``#`` is kept on the key -- it is what tells the service to encrypt
the value rather than write it in clear. The value may contain ``=``;
only the FIRST ``=`` is the separator.
"""
if "=" not in arg:
raise typer.BadParameter(
f"Expected '#KEY=VALUE'; got {arg!r} (no '=' separator).",
f"Expected '#KEY=VALUE' (encrypted) or 'KEY=VALUE' (plain); "
f"got {arg!r} (no '=' separator).",
param_hint="--secret",
)
key, _, value = arg.partition("=")
if not key:
raise typer.BadParameter(
f"Empty secret key in {arg!r}; expected '#KEY=VALUE'.",
f"Empty key in {arg!r}; expected '#KEY=VALUE' or 'KEY=VALUE'.",
param_hint="--secret",
)
return key, value
Expand All @@ -63,7 +67,8 @@ def _read_secrets_file(path: Path) -> dict[str, str]:
) from exc
if not isinstance(parsed, dict):
raise typer.BadParameter(
f"Secrets file {path} must be a JSON object mapping #KEY -> value.",
f"Secrets file {path} must be a JSON object mapping key -> value "
'("#KEY" is encrypted, a bare "KEY" is written in clear).',
param_hint="--secrets-file",
)
out: dict[str, str] = {}
Expand All @@ -90,14 +95,19 @@ def data_app_secrets_set(
None,
"--secret",
help=(
"One or more '#KEY=VALUE' plaintext entries. Repeatable. "
"One or more entries, repeatable. '#KEY=VALUE' is ENCRYPTED under the "
"project KMS; 'KEY=VALUE' (no '#') is written as a PLAIN env var whose "
"value stays readable in the config -- use it for non-secret settings. "
"Mutually exclusive with --secrets-file."
),
),
secrets_file: Path | None = typer.Option(
None,
"--secrets-file",
help="Path to a JSON file mapping '#KEY' -> 'plaintext value'.",
help=(
"Path to a JSON file mapping key -> value. A '#KEY' is encrypted; "
"a bare 'KEY' is written as a plain env var."
),
exists=True,
readable=True,
dir_okay=False,
Expand Down Expand Up @@ -126,10 +136,15 @@ def data_app_secrets_set(
help="Suppress the 'now run kbagent data-app deploy' hint in the output.",
),
) -> None:
"""Encrypt and write app-runtime secrets to the linked Storage config.
"""Write app-runtime env vars to the linked Storage config.

The '#'-prefix is required on every key (Keboola encryption convention).
The runtime exposes each secret as an env var with '#' stripped, '-'
The '#'-prefix decides encryption, mirroring the UI's "optional
encryption" checkbox: '#KEY=VALUE' is encrypted under the project KMS,
'KEY=VALUE' is written in clear so operators (and `secrets-list`,
`config detail`, the UI) can read it. Put non-secret settings -- a
Storage Files tag, a feature flag, a log level -- in the plain form.

The runtime exposes each entry as an env var with '#' stripped, '-'
replaced with '_', and uppercased ('#my-api-key' -> 'MY_API_KEY').

The command never auto-deploys; the running container keeps the old
Expand All @@ -150,7 +165,10 @@ def data_app_secrets_set(

if not secret and not secrets_file:
formatter.error(
message=("Provide at least one --secret '#KEY=VALUE' or --secrets-file PATH."),
message=(
"Provide at least one --secret '#KEY=VALUE' (encrypted) / "
"'KEY=VALUE' (plain), or --secrets-file PATH."
),
error_code=ErrorCode.MISSING_PARAMETER,
)
raise typer.Exit(code=2) from None
Expand Down Expand Up @@ -250,7 +268,10 @@ def data_app_secrets_list(
) -> None:
"""List the keys in parameters.dataApp.secrets, with derived runtime env-var names.

Never echoes the encrypted ciphertext in full and never decrypts.
Each entry is tagged `encrypted`. A PLAIN (unencrypted) entry shows its
literal value -- it is already stored in clear and visible via
`config detail`. An ENCRYPTED entry shows `<encrypted>`: the ciphertext is
never echoed in full and the CLI never decrypts (the API is one-way).

Reference: https://help.keboola.com/data-apps/python-js/
"""
Expand Down Expand Up @@ -281,17 +302,24 @@ def data_app_secrets_list(
return

if not result["secrets"]:
formatter.console.print("[dim]No secrets set on this data app.[/dim]")
formatter.console.print("[dim]No env-var entries set on this data app.[/dim]")
return
formatter.console.print(
f"\n[bold]{result['count']} secret(s)[/bold] on data app "
f"\n[bold]{result['count']} entr(ies)[/bold] on data app "
f"[cyan]{result['app_id']}[/cyan] in [magenta]{result['project_alias']}[/magenta]:"
)
for entry in result["secrets"]:
marker = (
" [yellow](shadowed by runtime)[/yellow]" if entry.get("shadowed_by_runtime") else ""
)
line = f" [bold]{entry['key']}[/bold] -> env [cyan]{entry['env_var']}[/cyan]{marker}"
# A plain value is stored in clear and readable via `config detail`, so
# printing it leaks nothing; an encrypted one is never decryptable here.
line += (
" = [dim]<encrypted>[/dim]"
if entry.get("encrypted", True)
else f" = {entry.get('value', '')}"
)
if "fingerprint" in entry:
line += f" [dim]fingerprint={entry['fingerprint']} prefix={entry.get('encryption_prefix', '')}[/dim]"
formatter.console.print(line)
Expand Down
Loading