OIE: scheduled runs (OWASP every 10 min, CRE expansion daily)

[northdpole]

Summary

Stacked on #1127. Adds two scheduled jobs run by python cre.py --run_scheduled {owasp,cre_expansion,all} (make oie-tick), Postgres only.

• owasp (10 min): agent Nest+GitHub sync (chapter repos are metadata-only), then incremental harvest of due project/other/standard repos, capped per tick.
• cre_expansion (daily): noise filter, Librarian, graph filing, then gap analysis only if standard edges were added.
• Graph filing: only Librarian links with confidence >= floor (OIE_FILE_CONFIDENCE_FLOOR, default 0.9, never below the Librarian link threshold) become OWASP/<repo> standard nodes and Automatically linked to edges. Others stay unconsumed for a later Module D. Kill switch: OIE_GRAPH_FILING_ENABLED. Existing edges are never overwritten.
• Run log: oie_run table (migration b8c9d0e1f2a3), slot-derived idempotent run ids, Postgres advisory-lock lease per job, per-stage failure isolation, structured logs carry run_id/job/stage.
• Humans: /admin/oie, /admin/oie/runs, /admin/oie/runs/<id>, /admin/oie/health (existing admin auth), plus make monitor-oie-health.
• repos.yaml now covers all OWASP org repos (generated by scripts/oie_generate_repos_yaml.py from repos.curated.yaml), with per-kind defaults.
• owasp_agent IndexStore ported to SQLAlchemy so it runs on Postgres.
• Harvester fixes found by running against real repos: first run listed files against the git empty tree (unresolvable), docling chunk offsets can run backwards, and the checkpoint advanced before chunks were staged.

Verification

• black --check on all changed files: pass.
• Full unittest suite: 1467 tests; remaining failures are environmental/pre-existing (12 db_test gap-analysis tests need Neo4j; mcp not installed for 2 import tests; 3 failures in cheatsheets_workstream_f_test / import_post_apply_test reproduce on the base branch).
• make mypy (strict) fails repo-wide on the base; new files add only the same no-untyped-call stub noise.
• Real local Postgres: migration upgrade/downgrade/upgrade, alembic guardrail OK, advisory lease (second holder denied, other job allowed, released afterward), real GitHub agent sync, real ASVS+AISVS harvest (695 chunks), noise filter (664 kept), stage isolation when the Librarian embedding dimension mismatched locally.
• Bugbot and security-review passes run; one valid finding (index target mismatch between scheduler and chat/MCP) addressed.

Notes

• If OWASP_AGENT_DB is unset, scheduled runs write the agent index to the app database; set the same value for web/MCP processes.
• Event repos have no markdown parser (covered via Nest events only); filed nodes get no embeddings yet.
• Commits are unsigned (agent cannot sign).

  

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}