[HWORKS-3370] Provided credentials for data sources (oracle)

[jimdowling]

JIRA: https://hopsworks.atlassian.net/browse/HWORKS-3370

Related PRs: logicalclocks/hopsworks-ee#3434, logicalclocks/hopsworks-api#1244, logicalclocks/hopsworks-front#2147, logicalclocks/loadtest#1115, #687

Merge order: logicalclocks/hopsworks-api#1244 must merge before this PR. The test-docs-build job builds against hopsworks-api main, which does not yet have ExternalFeatureGroup.data_source_access and ExternalFeatureGroup.test_data_source_access, so the strict build fails on those autorefs until #1244 lands. Rerun CI after #1244 merges and before merging this PR.

Documents PROVIDED credentials for Oracle data sources:

• user_guides/fs/data_source/creation/sql.md gets a new section, #data-source-provided-credentials, covering:
  • the credentials mode;
  • setting and validating member credentials from the UI, the SDK and the CLI;
  • the HOPS_DS_* variables;
  • which runtimes receive those variables.
• user_guides/fs/feature_group/create_external.md gets #external-feature-group-data-source, which explains the per-member access status.

markdownlint is clean. The strict build passes with the matching hopsworks-api branch installed, which plain-texts one docstring link to the non-public SqlConnector.

Update

• Trino catalogs on PROVIDED sources: a new section, #data-source-provided-credentials-trino, with a table of which clients send extra credentials.

• Replacing a wallet: a new section, #data-source-provided-credentials-wallet.

• Runtime variables: covers HOPS_DS_<NAME>_CONNECTOR_ID, the Spark wallet fallback, and why deployments do not get the variables.

• Default username: the new name.

• Trino catalogs page: projects/trino/catalogs.md now notes PROVIDED catalogs.

• Review round (51ca5f2): stored-credential steps scoped to Stored with the data source; Yours: incomplete; access checks settle within a few minutes, with member join and the 10-minute reconcile in the trigger list; removing credentials, member removal and data source deletion delete the default-named account entries; the Trino catalog's wallet follows the creator's credentials; shared feature store credentials are added in the owning project; the API box lists the SqlConnector credential methods, linked to the page section since SqlConnector has no API reference entry.

• Private catalogs (7a4cf2c, 26e2d96): a Trino catalog on a provided data source is always a private catalog of its creator; each member (Data Owner or Data Scientist) creates their own, a project-scope request is refused with CREDENTIALS_MODE_NOT_SUPPORTED (270337), and such a catalog cannot be shared. The wallet lives in a mountable secret of the creator's account and follows their credentials. The text on project copies, shared-project copies, unshare cleanup and the wallet hash pin is removed. projects/trino/catalogs.md says the same. Strict build and markdownlint pass.

• Screenshots: not updated yet. sql_creation.png predates the Credentials selector, and there are no images of the My credentials dialog, the Data Source submenu or greyed-out cards.

Loadtests

Loadtests on the final SHAs: 12/12 passed. Run 776b2463 on jim-private-ds, 2026-10-12, test_data_sources_provided_credentials.py (test_01 to test_12) with LOADTEST_TRINO_RESTART=1, laptop runner, 1 worker, 6m18s. It ran on hopsworks-ee 2d961c9ec2 (V118 applied), hopsworks-api 369d7effcc (the same wheel is overlaid on the cluster's Spark and Python job images, so jobs read with this SDK), hopsworks-front 3456cb2df and loadtest bb2573a, against the Oracle Autonomous Database in eu-stockholm-1 over mTLS. That covers the round-1 and round-2 security fixes and the private catalogs for PROVIDED data sources (test_09 creates its catalog as a Data Scientist). The connector uses the wallet's TNS alias with no host. With a host set, flyingduck dials host:port/service over plain TCP and the mTLS listener resets it (DPY-4011). Since then, hopsworks-ee has merged upstream master (502449de93: HWORKS-3384 and HWORKS-3388 proxy tokens, no migrations or REST codes). It compiles but has not been rerun. Not covered by any run: the validate/save rate limit (270343), the cleanup on data source, feature store and project delete, scheduled runs, and a browser pass over the UI. An earlier run, 3af3d0c8 (12/12, 20m39s), was on the round-1 code.

CI: the three references to ExternalFeatureGroup.data_source_access and test_data_source_access link to the section of create_external.md that documents them, so test-docs-build passes strict against hopsworks-api main without waiting for logicalclocks/hopsworks-api#1244.

🤖 Generated with Claude Code

[ErmiasG]

Review against the HWORKS-3370 description. The Oracle provided-credentials section and the Data Source submenu docs are accurate against hopsworks-api#1244, hopsworks-ee#3434 and hopsworks-front#2147 (method names, CLI flags, error codes, UI labels).

Other findings, not anchored to a diff line:

1. [Should Fix] test-docs-build fails in strict mode on three unresolved autorefs: ExternalFeatureGroup.test_data_source_access (sql.md:321 and create_external.md) and ExternalFeatureGroup.data_source_access (create_external.md). CI builds against hopsworks-api main, which doesn't have these members yet. State in the body that hopsworks-api#1244 must merge first, and rerun CI before merging.

2. [Nitpick] sql_creation.png is unchanged, so it presumably lacks the new Credentials selector. There are no images of the My credentials dialog, the Data Source submenu or the greyed-out cards.

[jimdowling]

Body findings from the review, addressed in 51ca5f2:

1. Strict build autorefs (ExternalFeatureGroup.test_data_source_access, ExternalFeatureGroup.data_source_access): the PR body now states that [HWORKS-3370] Provided credentials for data sources (oracle) hopsworks-api#1244 must merge first and that CI must be rerun before this PR merges. Locally, uv run hopsworks-docs check passes with the hopsworks-api branch of #1244 installed, and markdownlint and snakeoil pass.

2. Screenshots: not done. The Oracle database on the test cluster is down and no browser automation was available in this session, so sql_creation.png still lacks the Credentials selector and there are no images of the My credentials dialog, the Data Source submenu or the greyed-out cards. Left open; noted in the PR body.

The same commit also documents contract changes made alongside the review: removing your credentials, removing a member and deleting the data source now delete the default-named env var and secrets (custom-named entries stay); a Trino catalog on a provided data source keeps a copy of the creator's wallet, deleted when the creator removes their credentials or leaves and rebuilt when they save a new wallet; credentials for a feature group read through a shared feature store are added in the project that owns the data source.

[jimdowling]

Round 2 docs at 2bafbcb043: scheduled runs use the creator's credentials (warning), job credentials arrive through a per-run Kubernetes Secret, GET/list no longer return credential values, validate/save are rate limited (HTTP 429), the Trino wallet copy is hash-pinned and removed on catalog, data source, feature store or project delete and on unshare, and only Data owners and Data scientists can add credentials.

uv run hopsworks-docs check (strict) passes with the api#1244 branch installed; markdownlint 0 errors across 282 files; snakeoil leaves the tree clean. The rate-limit error code number will be added once ee#3434's round-2 commit lands.

[jimdowling]

Round 2 cluster verification: test_data_sources_provided_credentials.py passed 12/12.

Run 776b2463 on jim-private-ds, 2026-10-12, laptop runner, 1 worker, LOADTEST_TRINO_RESTART=1, 6m18s, against the Oracle Autonomous Database in eu-stockholm-1 over mTLS.

Repo	Commit
hopsworks-ee	2d961c9ec2 (deployed; V118 schema plus upstream V116/V117 applied)
hopsworks-api	369d7effcc (laptop runner and, as a wheel, in the cluster's spark-feature-pipeline and python-feature-pipeline images)
hopsworks-front	3456cb2df (deployed)
loadtest	bb2573a

What the run covers from round 2:

• Job credentials reach Python and Spark jobs through the per-run <runtime>-dscred Secret (test_05, test_08).
• GET and list of the PROVIDED connector carry no user, password or walletPassword, and the SDK reads the owner's values from the owner's own account entries (test_02).
• Catalogs on the PROVIDED data source are private: owner and reader each create their own and read as themselves; a PROJECT-scope request is refused with 270337 and a share with 403 (test_09); after the reader removes their credentials, reads through their own catalog are refused (test_11).

Defects the cluster runs found and fixed on the way:

• ee be04bf095b: adopt() looked up the runtime with the apiVersion/kind form of genericKubernetesResources, which returned nothing for a live SparkApplication, and then deleted the credentials Secret, so the driver sat in CreateContainerConfigError. It now uses an explicit resource context and keeps the Secret on a miss.
• ee 2d961c9ec2: a Data Scientist could create their private catalog but not fetch its template (403 on catalog/template); the template now applies the create rule.
• loadtest bb2573a: a failing assert printed the module's shared state, which holds the Oracle config, passwords and API keys. The state's repr names no values now, and the credential asserts report field names only.

Not covered: the rate limit on validate/save, the project/feature store delete cleanup, scheduled runs, and a browser pass over the front.

[ErmiasG]

Approved. test-docs-build is expected to stay red until hopsworks-api#1244 is merged.