Repository navigation
Conversation
#713) `storage create-table` and `storage swap-tables` -- the two halves of the BigQuery repartition path -- both move real data, yet both inherited the 60s STORAGE_JOB_MAX_WAIT meant for metadata jobs. Measured on an 800 MB BigQuery table: create ~15s, swap ~31s, so a table a few times larger blew the budget and reported STORAGE_JOB_TIMEOUT for a job that was still running and would succeed. Giving up locally never cancels the Storage job, so a short budget buys nothing and costs a false negative on the only in-place repartition path available today (the Storage API's PUT .../definition has no clustering field, so the copy+swap workaround cannot be replaced). - New TABLE_DATA_JOB_MAX_WAIT (300s) default for both commands. - New --timeout SECONDS on both, plus an optional `timeout` on the two REST routes (gt=0, so a bad budget is a 422 at the boundary). - The `None`-sentinel/reject-non-positive guard is now one shared normalize_job_timeout() in services/base.py; workspace_service's private copy is migrated onto it. The client-level default is unchanged (max_wait=None still means 60s), so the raw SDK client and the upload-table auto-create path keep their existing behaviour.
The new test armed a 0.0001s budget and patched out time.sleep, so it depended on real elapsed time between two loop iterations. time.monotonic() advances in ~15.6ms ticks on Windows, so the deadline never registered as expired there: the poller kept polling and the failure surfaced as ErrorCode.TIMEOUT plus unmatched GET /jobs/777 requests. It passed on macOS only because one HTTP roundtrip happens to exceed 100us. Use an already-expired budget (max_wait=-1.0) instead: the deadline is behind the first check, so the timeout branch is reached with no poll, no sleep and no dependence on the clock. Reproduced both shapes under a simulated 15.6ms tick -- old: TIMEOUT after 8 polls; new: STORAGE_JOB_TIMEOUT after 0. Stubbing time.monotonic was rejected as the fix: patching it on the client module patches the global time module, which httpx reads too.
…imeout # Conflicts: # plugins/kbagent/agents/keboola-expert.md # src/keboola_agent_cli/commands/storage.py # src/keboola_agent_cli/services/base.py # src/keboola_agent_cli/services/workspace_service.py # tests/test_server_router_calls.py
…imeout # Conflicts: # plugins/kbagent/skills/kbagent/references/commands-reference.md
…able, name the job
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| if timeout <= 0: | ||
| raise KeboolaApiError( | ||
| message=f"Invalid timeout {timeout}. Must be greater than 0.", | ||
| status_code=0, | ||
| error_code=ErrorCode.INVALID_ARGUMENT, | ||
| ) | ||
| return timeout |
There was a problem hiding this comment.
🔴 Non-finite timeouts leave table waits unbounded
With --timeout nan or --timeout inf, normalize_job_timeout accepts a non-finite budget. The storage poller's deadline never expires, leaving a pending create or swap job polling indefinitely.
Learn more
normalize_job_timeout validates the budget passed by table creation, table swapping, and workspace loading. It rejects zero and negative numbers but accepts NaN and positive infinity. The storage poller compares the current time against time.monotonic() + budget. That comparison never succeeds for NaN or positive infinity, so a job that remains pending is polled without a time limit. The CLI accepts these float spellings, and the new REST table request models also lack the finite-number restriction already used by import requests.
Example: kbagent storage swap-tables ... --timeout nan starts job 777. If job 777 stays in waiting, the poller never raises STORAGE_JOB_TIMEOUT and the command never returns.
Recommended fix: Reject non-finite numbers in normalize_job_timeout using math.isfinite, as validate_wait_timeout does. Add allow_inf_nan=False to the new CreateTable and SwapTables request fields so REST callers receive a 422 before the job starts.
Was this helpful? React with 👍 or 👎 to provide feedback.
keboola-pr-reviewer-bot
left a comment
keboola-pr-reviewer-bot
left a comment
There was a problem hiding this comment.
Verdict: auto_approve (risk 2/5) · profile keboola-mcp-server
Auto-approve: well-tested, opt-in --timeout + larger default budget for data-moving storage jobs, fully conforming to feature-PR conventions.
soustruh
left a comment
soustruh
left a comment
There was a problem hiding this comment.
No blocking findings. One nit inline.
Checked:
--timeouton create-table and swap-tables: default 300 s, a non-positive value is rejected bynormalize_job_timeoutbefore any HTTP (also with--dry-run), and the REST fields usegt=0.workspace_servicenow uses the shared helper with the same behavior.- Wait timeout is
STORAGE_JOB_TIMEOUT,retryable=false,details.job_id, exit 4, and the message namesstorage job-detail --job-id ID --wait. The import path through_wait_keeps_runningbuilds the same message and details as 0.98.0 (#837). - Docs carry
vNEXTtags, andcheck_version_gates.pypasses. No remainingretryable: trueclaim for these commands. - Ran the swap, create-table, base-service, workspace-service and server-router test files with a temp config dir: all pass. The timeout tests assert
retryable is False, exit 4 and the job-detail text, so they would fail on a regression.
The PR description's Testing section still says retryable=True for the exhausted-budget test. The Update section corrects it, so only the old text is stale.
Dismissing prior approval — a new commit was pushed and this review was for an earlier SHA. Run @keboola-pr-reviewer-bot review to get a fresh verdict.
|
New commit on |
keboola-pr-reviewer-bot
left a comment
keboola-pr-reviewer-bot
left a comment
There was a problem hiding this comment.
Verdict: needs_human (risk 3/5) · profile keboola-mcp-server
Well-tested storage job-timeout fix, but it changes a default on a load-bearing data path across CLI/SDK/REST, so a storage owner should confirm.
Concerns:
src/keboola_agent_cli/constants.py: Default job-wait budget raised 60s to 300s for create-table/swap-tablessrc/keboola_agent_cli/client/storage_tables.py: Shared _wait_keeps_running refactor also alters upload-table import timeout pathsrc/keboola_agent_cli/services/base.py: normalize_job_timeout now also rejects NaN/inf for workspace callers (prior check was <=0 only)
zajca
left a comment
zajca
left a comment
There was a problem hiding this comment.
Actionable findings from the automated review.
…inity and NaN with a 422
This branch has not been deployed
4 participants
What
storage create-tableandstorage swap-tablesnow default to a 300s job budget and accept--timeout SECONDS. The matching REST routes take an optionaltimeoutfield.Why
Issue #713 asks for a metadata-only clustering update on an existing BigQuery table. That part is blocked upstream, not here — verified independently against the canonical Storage API PHP client, whose own type for the update endpoint is:
No
clustering, no partitioning.PUT /v2/storage/tables/{id}/definitioncannot express the change, so astorage set-clusteringcommand would have to be written against an endpoint that does not exist. Not implemented here.What is fixable on the kbagent side is the second cost the issue lists: the copy+swap workaround — the only repartition path available today — has a job budget that is too short to survive it.
Both
create-table --source-table-id(a full data copy) andswap-tablesinheritedSTORAGE_JOB_MAX_WAIT = 60s, which exists for metadata jobs. Measured on an 800 MB BigQuery table: create ~15s, swap ~31s. A table a few times larger blows the budget, and because giving up locally never cancels the Storage job, the result is aSTORAGE_JOB_TIMEOUTreported for an operation that is still running and will succeed. A short budget buys nothing and costs a false negative.How
TABLE_DATA_JOB_MAX_WAIT = 300.0inconstants.py— the new default for both commands, with the measurements recorded as the rationale.--timeout SECONDSon both commands;max_waitthreaded through service → client →_wait_for_storage_job.timeout: float | None = Field(default=None, gt=0)on theCreateTable/SwapTablesrequest models, so a bad budget is a 422 at the boundary rather than anINVALID_ARGUMENTfrom the service.None-sentinel guard (reject<= 0, nevertimeout or DEFAULT—0.0is falsy) is now a single sharednormalize_job_timeout()inservices/base.py.workspace_service's private_normalize_timeoutis migrated onto it rather than duplicated.Deliberately unchanged: the client-level default.
max_wait=Nonestill resolves to 60s, so the raw SDK client and theupload-tableauto-create path (an empty-table metadata create) keep their current behaviour. Only the two data-moving commands get the larger budget.Behaviour change
A create/swap that previously failed at 60s now waits up to 300s before reporting a timeout. There is no case where the earlier failure was the desired outcome — it was a false negative on a live job — and
--timeout 60restores the old budget.Testing
make checkgreen (6383 passed, 12 skipped). New coverage:tests/test_base_service.py—normalize_job_timeout:None→ default, positive override wins,0.0/negative rejected asINVALID_ARGUMENT.tests/test_storage_swap.py— default is the data budget (asserted to exceedSTORAGE_JOB_MAX_WAIT), explicit forwarding, non-positive rejected before any HTTP (including on--dry-run), budget reaches the poller, and an exhausted budget raisesSTORAGE_JOB_TIMEOUTwithretryable=Truenaming the still-running job.tests/test_storage_write.py— same matrix forcreate-table, plus CLI forwarding.tests/test_server_router_calls.py— both routes forwardtimeout, omit →None, non-positive → 422 with the service never called.Four pre-existing
assert_called_once_withassertions were updated for the new kwarg.No live-project verification: that needs a BigQuery project and credentials I do not handle.
Docs
All silent-drift surfaces from CLAUDE.md convention #17 updated:
context.py(AGENT_CONTEXT), the CLAUDE.md command list,keboola-expert.md,commands-reference.md,gotchas.md(new entry: aSTORAGE_JOB_TIMEOUTis not a failure — do not blindly re-issue a swap, repeating one that landed swaps the tables back), andstorage-types-workflow.md. Tagged(since vNEXT, #713)per the feature-PR rule; no version bump, no changelog entry.Note: inside CLAUDE.md's command fence every
#line reads as an ATX heading tocheck_version_gates.py, so that blurb is indented past the 3-space allowance. Backticking the placeholder would also have passed, but the release-time residue scan applies the same quotation rule and would then never resolve it.Not addressed
The clustering endpoint itself. #713 should stay open against the Storage API; this PR only makes its documented workaround survive a large table.
Refs #713 — deliberately not
Fixes: the issue's actual ask (a metadata-onlyclustering update) needs a Storage API change that does not exist yet, so merging this
must not auto-close it.
Update 2026-10-10
main(0.98.0).make checkpasses.retryable=True. That conflicts with "do not re-issue the command": an agent that retries onretryable: trueruns the command again. A secondswap-tablesswaps the tables back, and a secondcreate-table --source-table-idstarts a second copy. Now a wait timeout ofcreate-tableandswap-tablesisSTORAGE_JOB_TIMEOUTwithretryable: falseanddetails.job_id, exit 4. The message says that the job keeps running and giveskbagent storage job-detail --project P --job-id ID --wait(from 0.98.0), and forcreate-tablealsostorage table-detail. This follows what 0.98.0 (feat(storage): S3 multipart upload and async table import (#834) #837) did for table imports, and the import path now uses the same helper.upload-tableauto-create step uses the same client call, so its timeout is now not retryable too.storage job-detailinstead of pollingGET /v2/storage/jobs/{id}by hand.create-tabletimeout. The docs note about the timeout moved tostorage-types-workflow.md.STORAGE_JOB_TIMEOUTis still HTTP 502 with nodetailsand noretryablefield, as for imports onmain. Only the message carries the job id. Changing that needs a change in the error handler ofserver/app.pyand is out of scope.create-table,swap-tables,upload-table,load-fileandworkspace loadall check--timeoutwith the same helper. A zero, negative, NaN or infinite value is a usage error, exit 2 withINVALID_ARGUMENT.swap-tableschecks it before the confirmation prompt. For direct SDK callers of the workspace service, a bad timeout is now aValueError, not aKeboolaApiError.timeoutoncreate-table,swap-tablesand the workspace load body rejects Infinity and NaN with a 422. A JSON body withInfinityorNaNused to crash the default 422 handler of FastAPI and give a 500. ARequestValidationErrorhandler inserver/app.pynow writes such input as text. This handler applies to every route.