Skip to content

[DPS-2427] Document Databricks support in the data quality dashboard - #1966

Merged
Diana Ignatescu (DianaIgnatescu) merged 5 commits into
mainfrom
DPS-2427-dqd-databricks
Oct 15, 2026
Merged

Diana Ignatescu (DianaIgnatescu) merged 5 commits into
mainfrom
DPS-2427-dqd-databricks

Conversation

@DianaIgnatescu

@DianaIgnatescu Diana Ignatescu (DianaIgnatescu) commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Jira: DPS-2427

Summary

  • Add Databricks (via the Iceberg REST catalog) alongside Snowflake and BigQuery on the monitoring page
  • Note that Databricks support in the data quality dashboard is currently AWS only
  • Add a Databricks tab to the missing permissions troubleshooting, covering 31000 (Can use on the SQL warehouse) and 31001 (catalog or schema privileges), with the grant script used in Console
  • Add 32xxx to the query timeout codes, with a Databricks diagnostic query on system.query.history
  • Add a release note dated 2026-10-13

To check

  • The column names in the system.query.history diagnostic query haven't been run against a real Databricks workspace
  • The error descriptions come from the remediation text in the msc-frontend-service tests, so the exact API wording may differ

🤖 Generated with Claude Code

Add Databricks (Iceberg REST catalog) alongside Snowflake and BigQuery on
the monitoring page, note that Databricks support is AWS only, and add
troubleshooting for error codes 31000, 31001, and 32xxx. Add a release note.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@github-actions

github-actions Bot commented Oct 9, 2026

Copy link
Copy Markdown
Contributor

Missing SEO metadata

The following markdown files are missing required metadata fields:

  • release-notes/databricks-support-in-the-data-quality-dashboard/index.md: missing fields: keywords, sidebar_label

Required fields

The file metadata is important for SEO and marketing. All markdown files, except for those with filenames starting with _, should include:

  • title: Full, descriptive page title
  • sidebar_label: Short title for navigation sidebar (can be the same as the main title)
  • description: One to two sentences summarizing the page contents
  • keywords: Array of marketing/SEO keywords

Please add the missing metadata.

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Oct 9, 2026 •

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Preview URL Updated (UTC)
✅ Deployment successful!
View logs
documentation 3c344bc Commit Preview URL

Branch Preview URL
Oct 15 2026, 10:34 AM

@claude

claude Bot commented Oct 9, 2026

Copy link
Copy Markdown

Style review

Overall this PR is well-written and follows repo conventions. The frontmatter, the release note fields (category, components, platforms), the internal links, and the tab structure all look good. A few small points:

1. Avoid "currently"

docs/monitoring/index.md:

Databricks support in the data quality dashboard is currently available only for pipelines running on AWS.

Rule: style guide, Writing style: "Stay in the present tense. Never use 'currently', 'now'…"

Fix: "Databricks support in the data quality dashboard is available only for pipelines running on AWS."

The same applies to the release note: "Databricks support is currently available for pipelines running on AWS." → "Databricks support is available only for pipelines running on AWS." (Adding "only" also makes it match the docs page.)

2. Passive voice in the Databricks permissions section

If the service principal lacks the CAN USE permission on the SQL warehouse, queries cannot be performed:

Rule: CLAUDE.md, Writing style: "Use active voice."

Fix: "If the service principal lacks the CAN USE permission on the SQL warehouse, the dashboard can't run queries and you may receive the following error:"

3. Databricks-specific example in a shared, warehouse-agnostic list

  • Monitor resource usage, and adjust warehouse size as needed, for example by scaling up your Databricks SQL warehouse

This list applies to all warehouses, so a Databricks-only example looks out of place, and the sentence now has a run of trailing clauses. Suggestion: drop the addition, or move it into the Databricks tab above as a separate sentence, for example "To increase capacity, scale up your Databricks SQL warehouse." If you keep it inline, add a comma: "for example, by scaling up…".


External link check: https://docs.databricks.com/aws/en/admin/system-tables/ loads successfully.

@DianaIgnatescu
Diana Ignatescu (DianaIgnatescu) marked this pull request as ready for review October 9, 2026 14:27
Comment thread docs/monitoring/index.md Outdated
Comment thread docs/monitoring/index.md Outdated
Comment thread docs/monitoring/index.md
Comment thread docs/monitoring/index.md Outdated
Scope Databricks support to AWS-hosted workspaces and the Iceberg REST
loader, add a Databricks setup section with the HTTP path and grants,
explain idle SQL warehouse timeouts, and apply style fixes.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@DianaIgnatescu Diana Ignatescu (DianaIgnatescu) added the do not merge Flag to denote a Issue or PR which should not yet be merged (usually pending a release) label Oct 12, 2026

@patofp Patricio (patofp) left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

Just one small correction to a previous suggestion: #1966 (comment)

@DianaIgnatescu
Diana Ignatescu (DianaIgnatescu) merged commit ccf37cc into main Oct 15, 2026
1 check passed
@DianaIgnatescu
Diana Ignatescu (DianaIgnatescu) deleted the DPS-2427-dqd-databricks branch October 15, 2026 10:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

do not merge Flag to denote a Issue or PR which should not yet be merged (usually pending a release)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants