perf: fetch table indexes for a schema in one query, not one per table - #88
Merged
Merged
Conversation
geekypunk
force-pushed
the
perf/batch-table-index-introspection
branch
from
August 27, 2026 20:56
4a853b5 to
8f36f5d
Compare
`enrichColumnsWithKeyAndIndexMetadata` called `getTableIndexes` once per
table inside its loop. On a wide schema that is hundreds of serial round
trips, and it re-runs for every caller that misses the `databaseObjects`
cache.
Measured on a 567-table MySQL connection reached over an SSH tunnel,
`GET /api/connections/{id}/objects`:
before 152.66s / 196.30s on cache miss (~270ms per table)
after 1.09s - 1.50s on cache miss
cached 0.03-0.48s (unchanged)
Both pre-fix requests were abandoned by the client (nginx 499). Because
there is no stampede guard, a second caller arriving during the first
sweep starts its own full sweep, so retrying made it worse. That stampede
is left alone here: at ~1.5s it is no longer material, and adding locking
to this path does not belong in a perf fix.
The fix mirrors what `loadForeignKeyColumns` already does one line above:
fetch the whole schema once and group in memory. Adds
`IntrospectionProvider.getAllTableIndexes` with a default implementation
that loops the existing per-table method, so a provider that does not
override it is unchanged, plus overrides for both shipped providers:
- MySQL: one INFORMATION_SCHEMA.STATISTICS query scoped to a single
TABLE_SCHEMA, so it stays bounded on a server hosting many databases.
Enrichment runs before the caller scopes results, so the requested set is
already the whole schema and nothing extra is fetched.
- Postgres: the same joins and filters as the per-table query, with
`t.relname = ANY(?)` in place of `t.relname = ?`.
The map distinguishes two outcomes, and this is load-bearing: a present but
empty list means the table was scanned and has no indexes, while an absent
key means the provider declined that name and the caller must fall back to
`getTableIndexes`. Postgres declines schema-qualified names for that reason
— pg_class.relname is bare, so `s.t` matches nothing, and stripping the
qualifier would match that name in every schema and merge their indexes.
The per-table query filters on schema and is the correct answer there.
Behaviour preserved:
- Objects qualified with a database other than the connection's are never
offered to the bulk path.
- A failed bulk fetch logs and falls back rather than dropping index flags.
- Neither bulk query sets a statement timeout, matching the per-table
methods they stand in for. A timeout here would be actively harmful: the
fallback would then re-run the full N+1 on top of the time already spent.
Verification. The Postgres bulk form was diffed against the per-table form
on a live database using EXCEPT ALL in both directions: 744 rows each, zero
rows differing either way, with composite index column order preserved.
Both engines were exercised end to end with index flags still populated
(MySQL: 760 columns with a single-column index and 375 composite, of 4588;
Postgres: 32 and 2, of 128) and no fallback warnings logged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013dSdoA8UGq7PVyvEwWjPXM
geekypunk
force-pushed
the
perf/batch-table-index-introspection
branch
from
August 27, 2026 22:46
8f36f5d to
7ce66b1
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
enrichColumnsWithKeyAndIndexMetadatacalledgetTableIndexesonce per table inside its loop. On a wide schema that is hundreds of serial round trips, and it re-runs for every caller that misses thedatabaseObjectscache.Measured on a 567-table MySQL connection reached over an SSH tunnel,
GET /api/connections/{id}/objects:Both pre-fix requests were abandoned by the client (nginx 499). Because there is no stampede guard, a second caller arriving during the first sweep starts its own full sweep — so retrying made it worse.
Fix
Mirrors what
loadForeignKeyColumnsalready does one line above: fetch the whole schema once and group in memory.Adds
IntrospectionProvider.getAllTableIndexeswith a default implementation that loops the existing per-table method, so a provider that does not override it is unchanged, plus overrides for both shipped providers:INFORMATION_SCHEMA.STATISTICSquery scoped to a singleTABLE_SCHEMA, so it stays bounded on a server hosting many databases.t.relname = ANY(?)in place oft.relname = ?.Behaviour preserved
Verification
EXCEPT ALLin both directions: 744 rows each, zero rows differing either way.🤖 Generated with Claude Code