🔍 CodeInspectorAI

An autonomous AI code reviewer that reads your pull requests so humans don't have to read them twice.

Fetches a GitHub PR, reviews the diff and hunts for vulnerabilities in parallel, then posts one structured, actionable review comment back on the PR.

Architecture • Sample Output • Quick Start • Engineering Notes • Testing

---

💡 Why this exists

Code review is the highest-leverage quality gate in software — and the first thing that gets skipped when a team is shipping fast. A stale eval() on user input or a string-concatenated SQL query slips through not because reviewers are careless, but because they're reviewing their fourth PR before lunch.

CodeInspectorAI is a multi-agent reviewer that never gets tired, never skips the security pass, and always posts the same structured feedback — in under a minute, automatically, on every push.

---

✨ Features

	Feature	What it means
⚡	Parallel agents	Code-quality review and security scanning run concurrently in the same LangGraph superstep — roughly half the wall-clock of a sequential chain
🛡️	Defence in depth	A deterministic regex sweep for leaked credentials runs alongside the LLM, so an obvious secret is caught even if the model misses it
⚖️	Deterministic verdicts	The LLM finds problems; code decides the outcome. A critical finding always means changes_requested — never a function of the model's mood
🩹	Fail-soft by design	Any node can die without killing the run. Lose the security scan and you still get the review, flagged partial with the error disclosed
🔁	Idempotent comments	Pushing 10 commits updates one comment in place instead of spamming ten
🎯	Smart diff budgeting	Lockfiles, minified bundles and binaries are stripped; what's omitted is named in the prompt so the model knows its view is partial
🔌	Three entry points	GitHub webhook, REST API, or CLI — same graph behind all of them
🔒	Verified webhooks	HMAC-SHA256 signature checking, draft-PR filtering, background execution so GitHub never waits on an LLM

---

🏗 Architecture

Built as a stateful directed graph rather than a linear chain, so the two analysis agents genuinely run in parallel and the workflow can abort early without burning tokens.

flowchart TD
    A["🌐 GitHub Webhook<br/>· REST API · CLI"] --> B["📥 Fetch Context"]
    B -->|"PR fetch failed"| Z["🏁 END"]
    B --> C["📝 Code Review Agent"]
    B --> D["🛡️ Security Agent"]
    C --> E["🧩 Summary"]
    D --> E
    E --> F["💬 Post Review"]
    F --> Z

    style A fill:#1f2937,stroke:#4b5563,color:#f9fafb
    style B fill:#1e3a8a,stroke:#3b82f6,color:#eff6ff
    style C fill:#065f46,stroke:#10b981,color:#ecfdf5
    style D fill:#7f1d1d,stroke:#ef4444,color:#fef2f2
    style E fill:#78350f,stroke:#f59e0b,color:#fffbeb
    style F fill:#4c1d95,stroke:#8b5cf6,color:#f5f3ff
    style Z fill:#1f2937,stroke:#4b5563,color:#f9fafb

Loading

Why the abort edge matters: if the PR can't be fetched, a naive graph would still invoke both LLM agents on an empty diff and post a meaningless review. The conditional route short-circuits straight to END — zero wasted tokens.

Node contracts

Designing the state before writing the nodes kept each one's responsibility honest:

Node	Consumes	Produces
Fetch Context	owner, repo_name, pr_number	raw_diff, changed_files, commit_messages, pr_metadata, pr_intent, parsed_diff, file_summaries, language_detected
Code Review ⇄	parsed_diff, file_summaries, pr_intent	review_comments, code_quality_score, suggestions
Security ⇄	parsed_diff, file_summaries, language_detected	security_issues, severity_levels, vulnerable_lines
Summary	all findings	final_summary, overall_verdict, formatted_comment, key_recommendations
Post Review	formatted_comment	posted_comment_url, review_status

_{⇄ = runs concurrently}

---

📸 Sample output

Given a diff that introduces four classic vulnerabilities, here's what actually lands on the PR:

🤖 CodeInspectorAI Review

🔴 Changes requested  •  Quality score 3.0/10  •  1 file reviewed  •  4 security issues (2 high/critical)

This change adds a user-lookup helper, but the query is assembled by concatenating the raw username into SQL, which is directly exploitable. The run_cmd helper hands unsanitised input to os.system, and the database password is committed in plaintext.

🔐 Security

	Severity	Location	Issue	Recommendation
🔴	Critical	app/db.py:6	SQL injection via string concatenation of user-supplied username	Use a parameterised query: cur.execute("SELECT * FROM users WHERE name = ?", (username,))
🔴	Critical	app/db.py:10	Command injection: user-controlled name concatenated into os.system	Use subprocess.run([...]) with an argument list and validate the host
🟠	High	app/db.py:3	Hardcoded database password in source	Move to an environment variable and rotate the credential
🟡	Medium	app/db.py:13	Reflected XSS: user comment inserted into HTML unescaped	Escape with html.escape() or use the template engine's autoescaping

📝 Code review

• 🔴 Critical app/db.py:6 — the query concatenates the raw username value into the SQL string
  • Suggestion: switch to parameter binding
• 🟡 Warning app/db.py:13 — the comment string is inserted directly into HTML without escaping

---

_{Generated by CodeInspectorAI — an automated reviewer. Treat its findings as suggestions, not gospel.}

And on a clean PR, it says so and gets out of the way — ✅ Approved, no invented nitpicks.

---

🚀 Quick Start

git clone https://github.com/Tharun0127/CodeInspectorAI.git
cd CodeInspectorAI

python -m venv .venv
.venv\Scripts\activate            # Windows
# source .venv/bin/activate       # macOS / Linux

pip install -r requirements.txt
copy .env.example .env            # cp on macOS / Linux

Add your keys to .env, then review any public PR without posting anything:

python -m app.cli pallets/flask 6136 --dry-run

Configuration

Variable	Required	Default	Notes
GROQ_API_KEY	✅	—	console.groq.com/keys
GROQ_MODEL	—	openai/gpt-oss-120b	Any Groq chat model
GITHUB_TOKEN	to post	—	Fine-grained PAT · Pull requests: read & write
GITHUB_WEBHOOK_SECRET	recommended	—	Verifies X-Hub-Signature-256
POST_COMMENTS	—	true	false = run everything, post nothing
MAX_DIFF_CHARS	—	15000	Diff budget per LLM node

⚠️ Groq retires models regularly. If a run fails with a model error, list what your account can use:

curl -H "Authorization: Bearer $GROQ_API_KEY" https://api.groq.com/openai/v1/models

---

🎮 Usage

CLI

python -m app.cli owner/repo 42                 # review and post
python -m app.cli owner/repo 42 --dry-run       # review, print, post nothing
python -m app.cli owner/repo 42 --save out.md   # save the markdown

API server

uvicorn app.main:app --reload --port 8000

Endpoint	Purpose
GET /health	Readiness + which credentials are configured
POST /review	Manual trigger — {"owner", "repo_name", "pr_number"}
POST /webhook	GitHub pull_request webhook receiver
GET /docs	Interactive OpenAPI docs

GitHub webhook

Point a repository webhook at https://<your-host>/webhook:

• Content type → application/json
• Secret → same value as GITHUB_WEBHOOK_SECRET
• Events → Pull requests only

Reviews fire on opened, reopened, synchronize and ready_for_review. Drafts are skipped. The endpoint returns immediately and the review runs in the background, so GitHub's 10-second webhook timeout is never at risk.

---

🧠 Engineering notes

The interesting decisions weren't about prompting — they were about what happens when things go wrong.

⚖️ The LLM finds problems; code decides the verdict

An LLM asked to both find issues and rule on them will happily approve a PR it just told you contains a critical vulnerability. So decide_verdict() in graph/utils.py owns the outcome:

if any(issue.severity in ("high", "critical") for issue in security_issues):
    return "changes_requested"

Same findings in, same verdict out — every time. The model's job is evidence, not judgement.

🩹 Fail-soft nodes: partial results beat no results

Every node catches its own exceptions and appends to a reducer-backed errors list instead of raising. If the security scan dies, the review still posts — with the failure disclosed in a collapsible block and review_status downgraded to partial.

The one exception is a failed PR fetch, which routes straight to END: there is nothing to review, so spending two LLM calls to find that out would be waste.

🔧 Surviving flaky structured output

Open-weight models on Groq intermittently emit an invalid tool call (400 — attempted to call tool 'json'), which killed an entire agent mid-run. This surfaced during live testing, not code review.

app/llm.invoke_structured() now degrades gracefully: tool calling → retry → JSON mode with the schema rendered into the prompt. Severity fields are normalised too, so a model answering "high" where the schema wants "critical" gets mapped instead of throwing away an otherwise valid finding.

🛡️ Regex and LLM, not regex or LLM

Credential patterns are matched deterministically against added lines only — so pre-existing secrets aren't re-reported on every push — then merged with the LLM's findings and deduplicated.

The regexes are tested against real-world shapes (DB_PASSWORD, apiKey, AWS keys, PEM blocks) and against false positives like password = os.environ["DB_PASSWORD"], which must not fire.

📏 Honest context budgeting

Rather than silently truncating a diff at 15,000 characters and letting the model review a fragment while believing it saw everything, build_diff_context() packs whole files and appends an explicit manifest:

--- NOT SHOWN IN THIS DIFF ---
- vendor/bundle.js (too large for the context budget)

The model knows what it doesn't know.

🔁 One comment, not twenty

Every posted review carries an invisible <!-- codeinspector-ai-review --> marker. On the next push the bot finds its own previous comment and edits it in place, so a 20-commit branch has exactly one always-current review instead of a wall of noise.

---

🧪 Testing

python -m pytest          # 101 tests · ~4s · zero network calls

Suite	Covers
test_utils.py	Diff parsing, language detection, verdict rules, budgeting
test_nodes.py	Each node's success and failure path, secret regexes
test_graph.py	Compiled graph end-to-end, parallel fan-out, abort route
test_llm.py	Structured-output retry/fallback, severity normalisation
test_api.py	Endpoints, webhook filtering, HMAC signature verification
test_github.py	Diff fallback, metadata extraction, comment replacement

GitHub and the LLM are both faked, so the suite is fully offline, deterministic and free to run — it needs no API keys and burns no tokens. Failure paths are tested as carefully as happy paths, because a reviewer that crashes on a flaky API is a reviewer nobody trusts.

---

🗂 Project structure

├── app/
│   ├── config.py       Environment/settings, prompt loading
│   ├── llm.py          Groq factory + resilient structured-output helper
│   ├── github.py       PR fetch, diff retrieval, posting, webhook signatures
│   ├── main.py         FastAPI app (webhook + REST trigger)
│   └── cli.py          Command line entry point
├── graph/
│   ├── state.py        ReviewState + Pydantic output schemas
│   ├── utils.py        Diff parsing, language detection, verdict rules
│   ├── graph.py        LangGraph wiring
│   └── nodes/          context · review · security · summary · post_review
└── tests/              101 offline tests

Note: the system prompts in prompts/*.md are loaded at runtime and are not published in this repository. Each node falls back to a built-in default prompt when the file is absent, so the project runs and the full test suite passes without them.

---

🛠 Tech stack

Layer	Choice	Why
Orchestration	LangGraph	Stateful graphs give real parallelism and conditional routing, which a linear chain can't
Inference	Groq · gpt-oss-120b	Sub-second latency keeps a full review under a minute
Validation	Pydantic v2	Schema-bound LLM output — no fragile regex parsing of prose
API	FastAPI	Async, background tasks, free OpenAPI docs
Integration	PyGithub + REST	PyGithub for objects, raw REST for the diff media type it doesn't expose

---

🗺 Roadmap

• Inline review comments anchored to diff positions (not just a summary comment)
• Repository-aware context via embeddings, so the reviewer sees beyond the diff
• Configurable per-repo rules through a .codeinspector.yml
• Review-quality feedback loop from 👍/👎 reactions on posted comments
• Dockerfile + one-click deploy

---

Built by Tharun

If this was useful, a ⭐ is appreciated.