Skip to content

feat(appkit): migrate analytics to the modular @databricks/sdk-* - #562

Draft
MarioCadenas wants to merge 1 commit into
mainfrom
analytics-migration-sdk
Draft

feat(appkit): migrate analytics to the modular @databricks/sdk-*#562
MarioCadenas wants to merge 1 commit into
mainfrom
analytics-migration-sdk

Conversation

@MarioCadenas

Copy link
Copy Markdown
Collaborator

What

Migrates the analytics stack (SQLWarehouseConnector + type-generator) off the legacy monolithic @databricks/sdk-experimental onto the new modular per-service @databricks/sdk-* SDK (v0.46.0, ESM-only), behind the existing packages/shared/src/workspace-client/ facade seam.

Analytics doesn't touch the SDK directly — it delegates to SQLWarehouseConnector, which uses two services: warehouses and statementExecution. Both move together in this increment; every other service (files/genie/jobs/serving/currentUser/apiClient) still routes through the legacy client behind the same facade (the seam's designed incremental state).

How

  • New boundary module packages/shared/src/workspace-client/modular.ts — the sole importer of @databricks/sdk-* (oxlint no-restricted-imports boundary), mirroring legacy.ts. Builds per-service WarehousesClient / StatementExecutionClient; maps wrapper options → ClientOptions (host scheme-normalization, PAT empty-token privilege-escalation guard, profile); stamps process-global client-info (sanitized, best-effort).
  • Connector + type-generator rewritten to the modular API: method renames (getStatementgetStatementResult, getStatementResultChunkNgetResultData), camelCase response model, CallOptions { signal } for cancellation.
  • pnpm patch (patches/@databricks__sdk-statementexecution@0.46.0.patch) restores the undocumented Reyden attachment response field that the SDK's generated unmarshal transform would otherwise strip. Version-pinned — re-apply on any SDK bump; an upstream fix is preferred long-term.

Runtime bugs found + fixed (caught against live warehouses, not unit tests)

  1. Host scheme — the modular SDK doesn't prepend https://; a bare DATABRICKS_HOSTInvalid URL. Fixed via normalizeHost.
  2. Client-infosetProduct("@databricks/appkit") throws on @//; sanitized + wrapped best-effort.
  3. ApiError.code — the modular ApiError carries the Databricks error code on .code, not .errorCode; the connector now reads either, so the arrow disposition/format capability-rejection fallback (INLINE↔EXTERNAL_LINKS) fires again.
  4. BigInt serialization — the modular SDK types row/byte counts as bigint; Reyden's cached INLINE+ARROW_STREAM result then broke JSON.stringify. Counts are coerced back to number at the connector boundary.

Testing

  • Full appkit + shared suites green (3877 passed / 1 skipped), including new regression tests for the .code read and bigint normalization.
  • pnpm -r typecheck, lint, format clean.
  • Verified live against standard + Reyden serverless warehouses: JSON + arrow (INLINE attachment and EXTERNAL_LINKS), OBO (.obo.sql), warehouse auto-start, metric views.

Notes

  • Metric views run entirely on the migrated services (they compile to SQL); no separate migration needed.
  • Regenerated the workspace-client API docs to match the new types (included).

Migrate the analytics stack (SQLWarehouseConnector + type-generator) off
the legacy monolithic @databricks/sdk-experimental onto the new modular
per-service @databricks/sdk-* SDK (v0.46.0, ESM-only), behind the existing
workspace-client facade seam. The two services analytics depends on —
warehouses and statementExecution — move together; every other service
still routes through the legacy client (mixed state by design).

- New packages/shared/src/workspace-client/modular.ts is the sole importer
  of @databricks/sdk-* (oxlint no-restricted-imports boundary), mirroring
  legacy.ts. Builds per-service WarehousesClient / StatementExecutionClient;
  maps wrapper options -> ClientOptions (host scheme-normalization, PAT
  empty-token guard, profile); stamps process-global client-info
  (sanitized, best-effort).
- Connector + type-generator rewritten to the modular API: method renames
  (getStatement -> getStatementResult, getStatementResultChunkN ->
  getResultData), camelCase response model, CallOptions { signal }.
- statementExecution relies on a pinned pnpm patch that restores the
  undocumented Reyden `attachment` field the SDK's unmarshal transform
  would otherwise strip.
- Coerce the SDK's bigint row/byte counts back to number at the connector
  boundary so INLINE + ARROW_STREAM results stay JSON-serializable
  (cache / SSE frames).
- Read the modular ApiError's `.code` (not only the legacy `.errorCode`) so
  the arrow disposition/format capability-rejection fallback still fires.

Verified against live warehouses (standard + Reyden serverless): JSON and
arrow (INLINE attachment + EXTERNAL_LINKS), OBO, warehouse auto-start, and
metric views. Full appkit + shared suite green (3877 tests).

Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant