Skip to content

Fix OWASP agent probe gaps (chapters, people, membership, CRE facts) - #1127

Open
northdpole wants to merge 8 commits into
cursor/owasp-agent-oie-a097from
fix/owasp-agent-probe-gaps
Open

Fix OWASP agent probe gaps (chapters, people, membership, CRE facts)#1127
northdpole wants to merge 8 commits into
cursor/owasp-agent-oie-a097from
fix/owasp-agent-probe-gaps

Conversation

@northdpole

@northdpole northdpole commented Oct 2, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Summary

  • Stacks on Experimental OIE OWASP metadata agent (Nest + GitHub hybrid chat) #1125 (cursor/owasp-agent-oie-a097): hardens the experimental OWASP metadata agent against real probe failures from homeserver eval.
  • Extends router/queries/index/crawler for inactive chapters, geo neighbors, chapter leaders, membership pricing/renewal, fail-closed talks, person lookup, and deterministic CRE facts (e.g. CRE-844-486).
  • Adds cre_facts.py / membership.py helpers and expands owasp_agent tests; gold (40) + synth (400) eval suites pass on the homeserver image.

Docs

Test plan

  • CI: owasp_agent unit tests green
  • Against a synced index: ask inactive chapter (Athens), geo neighbor (Pasadena→LA), membership regional pricing, Sam Stepanyan / board candidate probes, CRE-844-486
  • Confirm talk/event questions fail closed when data is missing
  • Re-run homeserver gold/synth evals if available

@coderabbitai

coderabbitai Bot commented Oct 2, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

🗂️ Base branches to auto review: 1
  • main

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration
  • Configuration used: Repository: OWASP/OpenCRE/.coderabbit.yml
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: f0ed92ec-078a-4512-91b0-2dcf6dd1ad20

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@northdpole
northdpole force-pushed the cursor/owasp-agent-oie-a097 branch from fee35d3 to b6319c7 Compare October 3, 2026 21:54
@northdpole
northdpole force-pushed the fix/owasp-agent-probe-gaps branch from 86fd285 to a0cf4e4 Compare October 3, 2026 21:54
@northdpole
northdpole force-pushed the cursor/owasp-agent-oie-a097 branch from b6319c7 to 2a731f4 Compare October 5, 2026 09:26
@northdpole
northdpole force-pushed the fix/owasp-agent-probe-gaps branch from a0cf4e4 to 90b84e5 Compare October 5, 2026 09:26
@northdpole
northdpole force-pushed the cursor/owasp-agent-oie-a097 branch from 2a731f4 to 629eb20 Compare October 5, 2026 10:11
@northdpole
northdpole force-pushed the fix/owasp-agent-probe-gaps branch from 90b84e5 to 91e5190 Compare October 5, 2026 10:11
@northdpole northdpole mentioned this pull request Oct 5, 2026
Draft
@cursor cursor Bot mentioned this pull request Oct 6, 2026
Closed
@northdpole
northdpole force-pushed the cursor/owasp-agent-oie-a097 branch from 629eb20 to 5831fe9 Compare October 7, 2026 18:50
arkid15r
arkid15r reviewed Oct 9, 2026
View reviewed changes
Comment thread application/utils/owasp_agent/nest_client.py

@arkid15r arkid15r Oct 9, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is an SDK available -- https://github.com/owasp/nest-sdk-python

@northdpole

northdpole commented Oct 11, 2026

Copy link
Copy Markdown
Collaborator Author

Folded in chat memory + OWASP entity fixes (70c851e3):

  • Native multi-turn messages[] + New chat button
  • Project leaders from leaders.json (Rob / AI Exchange)
  • Person/project/compare intents + AIX lethal-trifecta factoid
  • Synth probes 401–404

Stack restacked on top: #1132 → #1137 → #1142 (and sibling #1136).
LangGraph chat memory v2 tracked in #1143.

@northdpole
northdpole force-pushed the fix/owasp-agent-probe-gaps branch from 70c851e to 636089a Compare October 12, 2026 07:03
@northdpole

northdpole commented Oct 12, 2026

Copy link
Copy Markdown
Collaborator Author

Correction: replaced the bundled chat-memory+entity commit with entity-only 636089ae (Rob/AIX/GenAI/lethal trifecta from agent metadata). Chat memory will be a separate PR if/when wanted — not folded into this one. Closed unsolicited #1143.

@northdpole
northdpole force-pushed the fix/owasp-agent-probe-gaps branch from eb0cd49 to 69641cd Compare October 12, 2026 07:18
@northdpole
northdpole force-pushed the cursor/owasp-agent-oie-a097 branch from 5831fe9 to dfec9a8 Compare October 14, 2026 08:48
@northdpole
northdpole force-pushed the fix/owasp-agent-probe-gaps branch from 69641cd to 6fd8b0b Compare October 14, 2026 08:48
@northdpole
Fix OWASP agent probe gaps for chapters, people, membership, and CRE …
a91d6e1 
…facts.

Route hard probes correctly (inactive chapters, geo neighbors, leaders, membership pricing, fail-closed talks, CRE-844-486) and extend the index/crawler so gold+synth evals pass.
@northdpole
Add synth v2 probe suite and raise agent routing pass rate.
6953269 
Vendors the 400 unique offline probes plus router/slot fixes that lift
agent-path scoring to pass_rate>=0.90 with zero hallucinated talks.
@northdpole
Format owasp_agent modules with black for CI lint.
3c6bcff
@northdpole
Document OWASP agent setup, OIE relationship, and synth eval.
6da242e 
Adds a package README (keys, sync, chat/MCP, suite scoring), a public
fixture scorer script, and a Makefile target so contributors can reproduce
agent-path eval without the private homeserver harness.
@northdpole
Black-format remaining owasp_agent modules for CI.
6d9eb6b
@northdpole
Answer OWASP entity chat probes from agent metadata.
060e651 
Ingest project leaders from leaders.json, add person/project/compare
routing, and ground AI Exchange lethal-trifecta answers so Rob/AIX/GenAI
questions stop falling through to CRE RAG hallucinations.
@northdpole
Add multi-turn chat memory with New chat.
8ce9b6c 
Pass a capped messages[] history through /completion into LiteLLM and
the OWASP agent so clarify follow-ups keep prior context; UI gets a New
chat control that clears the local thread.
@northdpole
Harden OWASP agent routing for board, geo, CRE, and PG index.
32aefc1 
Prefer main DATABASE_URL for the agent index, fail-closed talk+CRE compounds,
board-person without year prompts, and geocode fallbacks used on staging.
@northdpole
northdpole force-pushed the fix/owasp-agent-probe-gaps branch from bb41ecc to 32aefc1 Compare October 14, 2026 11:03

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@robvanderveer robvanderveer Awaiting requested review from robvanderveer robvanderveer is a code owner
@Pa04rth Pa04rth Awaiting requested review from Pa04rth Pa04rth is a code owner
@paoga87 paoga87 Awaiting requested review from paoga87 paoga87 is a code owner
1 more reviewer
@arkid15r arkid15r arkid15r left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@northdpole @arkid15r