Skip to content

perf: fetch table indexes for a schema in one query, not one per table - #88

Merged
geekypunk merged 1 commit into
mainfrom
perf/batch-table-index-introspection
Aug 27, 2026
Merged

perf: fetch table indexes for a schema in one query, not one per table#88
geekypunk merged 1 commit into
mainfrom
perf/batch-table-index-introspection

Conversation

@geekypunk

@geekypunk geekypunk commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

Problem

enrichColumnsWithKeyAndIndexMetadata called getTableIndexes once per table inside its loop. On a wide schema that is hundreds of serial round trips, and it re-runs for every caller that misses the databaseObjects cache.

Measured on a 567-table MySQL connection reached over an SSH tunnel, GET /api/connections/{id}/objects:

cache miss cached
before 152.66s / 196.30s (~270ms per table) 0.03–0.48s
after 1.09–1.50s 0.03–0.48s (unchanged)

Both pre-fix requests were abandoned by the client (nginx 499). Because there is no stampede guard, a second caller arriving during the first sweep starts its own full sweep — so retrying made it worse.

Fix

Mirrors what loadForeignKeyColumns already does one line above: fetch the whole schema once and group in memory.

Adds IntrospectionProvider.getAllTableIndexes with a default implementation that loops the existing per-table method, so a provider that does not override it is unchanged, plus overrides for both shipped providers:

  • MySQL — one INFORMATION_SCHEMA.STATISTICS query scoped to a single TABLE_SCHEMA, so it stays bounded on a server hosting many databases.
  • Postgres — the same joins and filters as the per-table query, with t.relname = ANY(?) in place of t.relname = ?.

Behaviour preserved

  • Tables with no indexes map to an empty list, so callers can distinguish "no indexes" from "not scanned" without a per-table fallback.
  • A provider returns no entry for a name it cannot resolve precisely, and the caller falls back to the per-table query. Postgres declines schema-qualified names for that reason; objects qualified with another database are never offered at all.
  • A failed bulk fetch logs and falls back rather than dropping index flags.

Verification

  • The Postgres form was diffed against the per-table form on a live database with EXCEPT ALL in both directions: 744 rows each, zero rows differing either way.
  • Both engines exercised end to end with index flags still populated (MySQL: 760 columns with a single-column index and 375 composite, of 4588; Postgres: 32 and 2, of 128) and no fallback warnings logged.

🤖 Generated with Claude Code

@geekypunk
geekypunk requested a review from a team as a code owner August 27, 2026 20:45
@geekypunk
geekypunk force-pushed the perf/batch-table-index-introspection branch from 4a853b5 to 8f36f5d Compare August 27, 2026 20:56
`enrichColumnsWithKeyAndIndexMetadata` called `getTableIndexes` once per
table inside its loop. On a wide schema that is hundreds of serial round
trips, and it re-runs for every caller that misses the `databaseObjects`
cache.

Measured on a 567-table MySQL connection reached over an SSH tunnel,
`GET /api/connections/{id}/objects`:

    before   152.66s / 196.30s on cache miss (~270ms per table)
    after      1.09s - 1.50s on cache miss
    cached     0.03-0.48s (unchanged)

Both pre-fix requests were abandoned by the client (nginx 499). Because
there is no stampede guard, a second caller arriving during the first
sweep starts its own full sweep, so retrying made it worse. That stampede
is left alone here: at ~1.5s it is no longer material, and adding locking
to this path does not belong in a perf fix.

The fix mirrors what `loadForeignKeyColumns` already does one line above:
fetch the whole schema once and group in memory. Adds
`IntrospectionProvider.getAllTableIndexes` with a default implementation
that loops the existing per-table method, so a provider that does not
override it is unchanged, plus overrides for both shipped providers:

- MySQL: one INFORMATION_SCHEMA.STATISTICS query scoped to a single
  TABLE_SCHEMA, so it stays bounded on a server hosting many databases.
  Enrichment runs before the caller scopes results, so the requested set is
  already the whole schema and nothing extra is fetched.
- Postgres: the same joins and filters as the per-table query, with
  `t.relname = ANY(?)` in place of `t.relname = ?`.

The map distinguishes two outcomes, and this is load-bearing: a present but
empty list means the table was scanned and has no indexes, while an absent
key means the provider declined that name and the caller must fall back to
`getTableIndexes`. Postgres declines schema-qualified names for that reason
— pg_class.relname is bare, so `s.t` matches nothing, and stripping the
qualifier would match that name in every schema and merge their indexes.
The per-table query filters on schema and is the correct answer there.

Behaviour preserved:
- Objects qualified with a database other than the connection's are never
  offered to the bulk path.
- A failed bulk fetch logs and falls back rather than dropping index flags.
- Neither bulk query sets a statement timeout, matching the per-table
  methods they stand in for. A timeout here would be actively harmful: the
  fallback would then re-run the full N+1 on top of the time already spent.

Verification. The Postgres bulk form was diffed against the per-table form
on a live database using EXCEPT ALL in both directions: 744 rows each, zero
rows differing either way, with composite index column order preserved.
Both engines were exercised end to end with index flags still populated
(MySQL: 760 columns with a single-column index and 375 composite, of 4588;
Postgres: 32 and 2, of 128) and no fallback warnings logged.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013dSdoA8UGq7PVyvEwWjPXM
@geekypunk
geekypunk force-pushed the perf/batch-table-index-introspection branch from 8f36f5d to 7ce66b1 Compare August 27, 2026 22:46
@geekypunk
geekypunk merged commit 61e1d57 into main Aug 27, 2026
9 checks passed
@geekypunk
geekypunk deleted the perf/batch-table-index-introspection branch August 27, 2026 22:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant