Repository navigation
make --upstream_sync also sync the gap analysis graph progressively #534
Description
Status
Maintainer-approved design — opportunistic gap-analysis (GA) fetch from production on access, replacing the prior idea of bulk-downloading the full ~9 GB GA graph via --upstream_sync. Implementation spec will land in a follow-up PR; this issue tracks the feature and acceptance criteria.
Supersedes bulk GA download as the primary local-dev path. Prateek's JIT Neo4j compute path remains an advanced fallback documented separately (not the default UI/REST path).
Problem
The gap-analysis cache is very large (~9 GB). Downloading it wholesale during --upstream_sync is unreliable and unnecessary for most local development: contributors typically need only the GA pairs they actually view.
Production (opencre.org) already serves precomputed GA rows from Postgres (cache-only; cache miss → 404). Local instances should reuse that upstream cache opportunistically instead of requiring a full bulk sync.
Proposed solution
When a user requests a GA pair (via UI or local REST GA API — same backend code path):
- If the pair exists in the local GA store, serve it.
- If not, and auto-fetch is enabled, fetch the full material row from prod (read-only HTTP client), upsert into the local GA store, then serve it.
- While a fetch is in flight for a pair, return a "download in progress" state; concurrent requests for the same pair share a single in-flight fetch (frontend polls/waits).
- On prod 404 (pair not in prod cache): show documented instructions for local backfill (links to opencre.org and/or GitHub docs — to be written as part of this work). Do not compute GA in the request path from the UI.
- On prod unreachable or fetch failure: fall through to local cached row if present; otherwise show Try again (not local compute in UI).
GA auto-fetch is independent of --upstream_sync CRE graph sync. Locally cached pairs are preserved and not overwritten by a failed fetch.
Design decisions
- Trigger: Both UI and local REST GA API use the same backend code path.
- Prod URL: Configurable env var (default
https://opencre.org). Prod down / miss behavior:- Prod unreachable (timeout, network error, exhausted retries): serve local cached row if it exists.
- Not in local cache: show link to compute/backfill instructions (no UI compute).
- Do not silently compute on prod-down in the UI/request path. Local compute remains CI/docs-only (advanced JIT Neo4j fallback), not a hidden fallback when prod is down.
- Persist: Fetch the full material row from prod; upsert into the local GA store (
gap_analysis_results/ equivalent). - Concurrency: Single fetch per pair; backend returns "download in progress" while in-flight; frontend polls/waits.
- 404 miss: Exact UX copy plus links to local backfill documentation (opencre.org and/or GitHub). Block local compute from UI for now.
- Instructions: Link to docs to be written as part of this work — do not prescribe specific Makefile targets in this ticket.
- Soft message: When both constituent standards exist locally and prod confirms they are valid GA participants, but the specific pair is missing on prod, show a soft CTA to file a ticket after the backfill instructions. Detection logic (all must hold):
- Both standards exist locally in the CRE graph (nodes present for the pair's two standard names).
- Both are valid GA participants — either:
- the standards list endpoint includes both standards as GA-eligible, or
- individual GET per standard succeeds against prod (or local graph metadata when prod standards list is unavailable).
- GA request for that specific pair returns 404 or an error from prod (after retries exhausted).
- When all three conditions hold, append the soft "file a ticket" CTA; otherwise show only the standard backfill instructions.
- Refresh: Backend owns fetch lifecycle; frontend refresh sees the same state. Log when a local instance pulls a pair from prod.
- Env disable:
CRE_GA_NO_UPSTREAM=1(or similar name) documented in.env.example— disables auto-fetch. On fetch failure: Try again button (not local calc in UI). - Independence: GA auto-fetch is independent of
--upstream_syncCRE graph sync. Locally cached pairs are preserved (not overwritten by failed fetch). - Security: No auth on prod fetch; read-only HTTP client to prod GA endpoints; no writes to prod. State explicitly in implementation spec.
- Configurable prod fetch (env vars): GA upstream HTTP client uses
CRE_GA_UPSTREAM_*env vars, scoped parallel to the existingCRE_UPSTREAM_*pattern from upstream CRE graph sync (feat: retry transient failures during upstream sync #951):CRE_GA_UPSTREAM_TIMEOUT_SECONDS— default 30 (Heroku-typical; mirrorsCRE_UPSTREAM_TIMEOUT_SECONDS)CRE_GA_UPSTREAM_MAX_ATTEMPTS— default 4 (mirrorsCRE_UPSTREAM_MAX_ATTEMPTS)CRE_GA_UPSTREAM_RETRY_BACKOFF_SECONDS— default 2 (mirrorsCRE_UPSTREAM_RETRY_BACKOFF_SECONDS; optional but recommended for parity)- Exhaust retries before surfacing Try again in UI.
Cache policy
- V1: Once cached locally, always serve from local cache — no automatic re-fetch from prod on subsequent requests.
- V2 (future): Add TTL-based re-fetch from prod (e.g. env-controlled staleness window). Document as Version 2; out of scope for v1 implementation.
Open items (TBD — do not block implementation)
| Item | Notes |
|---|---|
| V2 TTL / force-refresh | Deferred to Version 2 (see Cache policy) |
Out of scope
- Bulk download of the full GA graph via
--upstream_syncas the primary path (superseded by this design). - Computing GA on production (prod remains cache-only).
- Computing GA in the UI/request path on cache miss (blocked; backfill via docs/CLI only).
- Prescribing exact Makefile targets in this ticket (docs PR will cover operational steps).
- JIT Neo4j compute as the default path (advanced fallback in docs only).
- V2: TTL-based automatic re-fetch from prod for locally cached pairs (see Cache policy).
Acceptance criteria
- UI and local REST GA API share one backend fetch/cache code path.
- Configurable prod base URL (default opencre.org); read-only prod HTTP client (no auth, no writes).
- On cache miss with auto-fetch enabled: fetch full material row from prod and upsert locally.
- V1 cache policy: once a pair is cached locally, subsequent requests serve from local cache without automatic prod re-fetch.
- On prod unreachable (after
CRE_GA_UPSTREAM_MAX_ATTEMPTSretries): serve local cached row if present; otherwise Try again with compute/backfill instructions link (no UI-triggered compute). - On prod 404 for a pair not in local cache: documented backfill instructions (opencre.org + GitHub links); no local compute from UI.
- Single in-flight fetch per pair; concurrent requests get "download in progress"; frontend polls/waits.
- Soft CTA to file a ticket when soft-message detection (make frontend show Graph #7) conditions are met (both standards local + valid GA participants + prod pair 404/error).
-
CRE_GA_NO_UPSTREAM=1(or agreed name) disables auto-fetch; documented in.env.example. -
CRE_GA_UPSTREAM_TIMEOUT_SECONDSdefault 30;CRE_GA_UPSTREAM_MAX_ATTEMPTSdefault 4; documented in.env.example(and implementation spec). - Failed fetch does not overwrite existing local cached row.
- Logging when local instance pulls a pair from prod.
- Tests: mocked prod HTTP, cache upsert, in-progress lock, 404 UX copy, env disable flag, read-only prod client behavior, logging, env defaults (
CRE_GA_UPSTREAM_TIMEOUT_SECONDS,CRE_GA_UPSTREAM_MAX_ATTEMPTS), soft-message detection logic (positive/negative cases), V1 no re-fetch (cached pair served without second prod call).
Dependencies
- Production GA REST API serving precomputed material rows (existing on opencre.org).
- Local GA store schema (
gap_analysis_results/ equivalent) supporting upsert of material rows. - Documentation PR: local backfill instructions, env vars, advanced JIT Neo4j fallback (not default).
- Detailed implementation spec PR (referenced from this issue).
Note
A detailed implementation spec will land in a follow-up PR linked from this issue. Volunteers should read this body (and the spec PR when available) before claiming the issue.
Activity
kart-u commentedon Oct 11, 2025
hello @northdpole is this issue still open for work , I would love to contribute in it!
I like issues involving optimizations :-)
dhhhhruv commentedon Nov 7, 2025
@northdpole i'd like to work on this issue. could you please assign it to me?
sd2604 commentedon Nov 12, 2025
Hey if this issue is still available, could you pls assign this to me, I'd realy like to work on this issue...
IND-Anshuman commentedon Dec 21, 2025
Hi! @northdpole I'd like to work on this issue. I've analyzed the codebase and have a clear plan to solve the problem.
The Problem I identified: Currently, --upstream_sync only downloads the CRE graph structure but ignores the gap analysis data, which is ~9GB and impractical to download reliably for local development.
My Solution: First, I'll extend the donwload_graph_from_upstream() function in application/cmd/cre_main.py to progressively sync gap analysis data after downloading the CRE structure. It will get the list of standards from upstream, go through standard pairs, and download their gap analysis results incrementally while skipping already-cached data. I can also implement lazy loading by modifying the schedule() function in application/utils/gap_analysis.py to get the gap analysis from upstream on-demand before falling back to expensive local calculations. This means the system will first try to grab pre-calculated results from upstream , only calculating locally if upstream is unavailable or doesn't have the data.
Could you please assign this issue to me? Thanks!
ashutosh-engineer commentedon Jan 19, 2026
@northdpole While exploring and fixing the Issue of #469 I am also exploring the Solution for this issue.
Will Share the report soon.
fix: correct typo in function name donwload_graph_from_upstream -> do…
PRAteek-singHWY commentedon Jan 20, 2026
Hi maintainers @northdpole @robvanderveer
Architectural Analysis: Solving Local Gap Analysis
I have completed a deep trace of the gap analysis subsystem to identify why the local experience diverges from production. My analysis confirms that we do not need to replicate the 9GB production dataset locally. Instead, we can achieve identical functionality through On-Demand Local Computation.
The Core Bottleneck
Currently, the
upstream_syncprocess successfully downloads the CRE hierarchy into SQLite but stops short of populating the Neo4j graph.Key Insight: "Just-in-Time" Architecture
We can solve the root cause without massive downloads by shifting the local architecture from Pre-Computation to On-Demand Computation.
The infrastructure for this already exists (
populate_neo4j_db).We simply need to integrate it into the sync lifecycle. This aligns perfectly with the issue's goal to "load it opportunistically."
Proposed Solution
I propose a complete "Just-in-Time" graph activation strategy:
Strategic Benefits
Next Steps
This architectural fix solves the core issue entirely by reconnecting existing subsystems.
I've been actively working on the MyOpenCRE milestone, recently handling importing logic, validation, and capability flags (e.g., in PRs #700 and #704), which gave me good context here. While we are iterating on the design for Issue #471,
I’d be happy to take this on incrementally and iterate accordingly , if the approach aligns with your expectations.
&
I also encountered this limitation firsthand while trying to run gap analysis locally during #456.
fix: typo in function name donwload_graph_from_upstream → download_gr…
fix: typo in function name donwload_graph_from_upstream → download_gr…
fix: typo in function name donwload_graph_from_upstream → download_gr…
Implement progressive upstream gap-analysis sync and fix local gap-an…
Implement progressive upstream gap-analysis sync and fix local gap-an…
fix: typo in function name donwload_graph_from_upstream → download_gr…
Saksham-official commentedon Apr 10, 2026
Hi, I’d like to work on this issue and have a couple of clarifying questions before I start.
As I understand it, the main problem is that the current gap analysis graph is extremely large (around 9 GB), so downloading it during --upstream_sync is unreliable and not practical for most local setups, especially for GSoC contributors and other developers who want to run OpenCRE locally.
A direction I’m considering is:
Keep the full gap analysis graph in a dedicated, public “large data” location (for example object storage or another dataset service), rather than treating it as a single monolithic artifact to be fetched in one go during --upstream_sync.
Enhance --upstream_sync so that it:
Fetches only the minimal graph metadata/structure needed for MyOpenCRE to function locally.
Then progressively downloads and stores the gap analysis data in smaller chunks (for example per standard or per version) into the local database, either on demand or via a background process.
To align with the existing architecture, could you please clarify:
Where is the gap analysis graph currently stored/generated, and is there already a preferred place to host large public data sets for OpenCRE?
Are there specific constraints on how the gap analysis graph is queried today (e.g., assumptions that everything is present locally and fully loaded) that I should preserve when making the sync progressive?
Would you prefer a design where missing gap analysis data is fetched lazily when a user triggers map analysis, or a design where --upstream_sync eventually brings the full dataset locally but in smaller, resumable steps?
If this approach matches your expectations, I can draft a short design note summarizing the data layout, sync flow, and any migration impact, and share it here before implementing.
/assign @Saksham-official
northdpole commentedon Jul 9, 2026
Design updated per maintainer discussion.
This issue now reflects the approved spec for opportunistic GA fetch from prod on access (replacing bulk ~9GB download via
--upstream_syncas the primary path). A detailed implementation spec will land in a follow-up PR.Refinements (v1 spec):
CRE_GA_UPSTREAM_TIMEOUT_SECONDS=30 /CRE_GA_UPSTREAM_MAX_ATTEMPTS=4 (aligned withCRE_UPSTREAM_*from feat: retry transient failures during upstream sync #951)Volunteers: please read the updated issue body (and the spec PR when it is linked) before claiming this issue.
northdpole commentedon Jul 9, 2026
Full agent-ready spec in PR #972: #972
Covers design decisions,
CRE_GA_UPSTREAM_*env vars (mirroring #951), API/UI fetch flow, in-progress lock, 404/backfill UX, soft CTA detection, V1 cache-always-local policy, phased implementation ladder with gates, and acceptance criteria. Spec-only — implementation follows in separate PRs.skypank-coder commentedon Jul 9, 2026
@northdpole Read #534 update + the RFC in #972.
Direction lines up with what I was tracing (opportunistic fetch beats bulk --upstream_sync, given the :670-673 guardrail on the existing /map_analysis bulk path) — good to have it pinned down.
Given the phased ladder + gates in the RFC, I'd rather take a scoped first slice than claim the whole issue: happy to start on the env var / config scaffolding (CRE_GA_UPSTREAM_TIMEOUT_SECONDS, CRE_GA_UPSTREAM_MAX_ATTEMPTS, mirroring the CRE_UPSTREAM_* pattern from #951) plus the cache-lookup layer (cache-always-local v1, no TTL), gated behind whatever review checkpoint you want before the fetch-on-miss + prod-down fallback (compute-instructions link) lands.
If that's the right first phase, happy to take the assignment on that slice specifically rather than the full issue.
Assign this to me Happy to work on this and deliver it .
docs: add agent-ready RFC for opportunistic local GA fetch (#534)
docs: add agent-ready RFC for opportunistic local GA fetch (OWASP#534)