Skip to content

DRILL-8552: Add Storage Plugin for Apache Accumulo - #3067

Merged
cgivre merged 8 commits into
apache:masterfrom
cgivre:accumulo
Sep 8, 2026
Merged

DRILL-8552: Add Storage Plugin for Apache Accumulo#3067
cgivre merged 8 commits into
apache:masterfrom
cgivre:accumulo

Conversation

@cgivre

@cgivre cgivre commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

DRILL-8552: Add Storage Plugin for Apache Accumulo

Description

This PR introduces a new storage plugin for Apache Accumulo, enabling Drill to query Accumulo tables using standard SQL.

Features:

  • Full SQL query support for Accumulo tables
  • Dynamic schema discovery (column families as MAPs)
  • Filter pushdown (row key ranges to Accumulo Range scans)
  • Projection pushdown (column family/qualifier selection)
  • Limit pushdown (early scan termination)
  • Sort pushdown (ORDER BY row_key uses natural ordering)

Authentication modes:

  • PASSWORD: Username/password authentication
  • KERBEROS + SHARED_USER: Service principal for all queries
  • KERBEROS + USER_IMPERSONATION: Delegation tokens for per-user identity
  • USER_TRANSLATION: Per-user Accumulo credentials from CredentialsProvider

Key components:

  • AccumuloStoragePlugin/Config: Plugin configuration and lifecycle
  • AccumuloConnectionManager: Centralized auth and client management
  • AccumuloGroupScan/SubScan: Distributed scan planning
  • AccumuloRecordReader: Data reading and vector population
  • AccumuloPushFilterIntoScan: Filter pushdown optimizer rule
  • AccumuloPushSortIntoScan: Sort pushdown optimizer rule
  • DelegationTokenInfo: Serializable token wrapper for distributed execution

Documentation

See README.md

Testing

Added extensive unit tests.

@cgivre cgivre self-assigned this Aug 2, 2026
@cgivre cgivre added enhancement PRs that add a new functionality to Drill new-storage New Storage Plugin doc-impacting PRs that affect the documentation labels Aug 2, 2026
Comment thread contrib/storage-accumulo/src/main/resources/drill-module.conf Outdated
@cgivre

cgivre commented Aug 27, 2026

Copy link
Copy Markdown
Contributor Author

@shfshihuafeng Thanks for your review. I believe I addressed all your review comments. Would you please take a look?
@cgivre hi, I took another look and found that the AccumuloClient resource leak I mentioned earlier is still not fixed. do you think this scenario is not an issue that needs to be addressed?

AccumuloClient client = null;
try {
  client = ...;
  // build reader
} catch (Exception e) {
  throw ...;
} finally {
  if (client != null) {
    client.close();
  }
}

@shfshihuafeng shfshihuafeng left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM +1

cgivre and others added 7 commits September 8, 2026 09:52
This PR introduces a new storage plugin for Apache Accumulo, enabling
Drill to query Accumulo tables using standard SQL.

Features:
- Full SQL query support for Accumulo tables
- Dynamic schema discovery (column families as MAPs)
- Filter pushdown (row key ranges to Accumulo Range scans)
- Projection pushdown (column family/qualifier selection)
- Limit pushdown (early scan termination)
- Sort pushdown (ORDER BY row_key uses natural ordering)

Authentication modes:
- PASSWORD: Username/password authentication
- KERBEROS + SHARED_USER: Service principal for all queries
- KERBEROS + USER_IMPERSONATION: Delegation tokens for per-user identity
- USER_TRANSLATION: Per-user Accumulo credentials from CredentialsProvider

Key components:
- AccumuloStoragePlugin/Config: Plugin configuration and lifecycle
- AccumuloConnectionManager: Centralized auth and client management
- AccumuloGroupScan/SubScan: Distributed scan planning
- AccumuloRecordReader: Data reading and vector population
- AccumuloPushFilterIntoScan: Filter pushdown optimizer rule
- AccumuloPushSortIntoScan: Sort pushdown optimizer rule
- DelegationTokenInfo: Serializable token wrapper for distributed execution

Tested with Accumulo 2.1.4 LTS.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Pass the sortDescending flag to the 10-arg AccumuloScanSpec constructor
in createScanSpec() and mergeScanSpecs(), which were still using the
old 9-arg signature and failing compilation.
- Keep the Drill filter in the plan when a row key OR is pushed down. The
  union of two ranges is a superset of the disjunction, so removing the
  filter returned every row between the operands.
- Treat a null bound as unbounded when unioning ranges for OR, so that
  e.g. row_key < 'a' OR row_key > 'z' no longer produces an inverted range.
- Remove the unused exec.accumulo.scan config block from drill-module.conf
  and source the reader's batch size from DrillAccumuloConstants.
- Add integration tests for row key disjunctions.
…lity labels

Address final review comments:

* AccumuloScanBatchCreator leaked the AccumuloClient created from a
  delegation token when the AccumuloRecordReader constructor threw:
  the reader never took ownership, so nothing closed it. Clients we
  create here are now closed on failure; the shared service client is
  left alone since it is owned by the storage plugin.

* Scanners were created with Authorizations.EMPTY, which hides every
  entry carrying a column visibility label — including ones the
  connecting user is entitled to read. Scans now use the connecting
  user's own authorizations, falling back to EMPTY with a warning if
  they cannot be read. Adds AccumuloVisibilityTest, which fails on the
  old behavior, and documents the semantics in the README.
@cgivre
cgivre merged commit 367c14a into apache:master Sep 8, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

doc-impacting PRs that affect the documentation enhancement PRs that add a new functionality to Drill new-storage New Storage Plugin

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants