DocuVerify

DocuVerify is a CrewAI-orchestrated multi-agent retrieval system for answering questions over documents with citations and a verification pass. It is inspired by agentic RAG patterns: retrieve evidence, draft an answer, verify every claim against the retrieved sources, and correct unsupported output before returning it.

Unlike a plain chatbot wrapper, DocuVerify emphasizes grounded answers. It uses OpenAI embeddings for semantic retrieval and can use Google Gemini for answer synthesis when GOOGLE_API_KEY is set.

Highlights

• LangChain-powered chunking with RecursiveCharacterTextSplitter.
• Hybrid retrieval that blends LangChain BM25 keyword search with OpenAI vector similarity through EnsembleRetriever.
• CrewAI agent workflow with relevance checking, research, verification, and self-correction agents.
• Source-backed answers with chunk-level citations.
• Hallucination guardrails that flag or remove unsupported claims.
• Optional Google Gemini synthesis and OpenAI text-embedding-3-small embeddings.
• CLI for fast demos and an optional Gradio app for a web UI.
• Unit tests covering retrieval quality and verification behavior.

Architecture

flowchart LR
    A[Documents] --> B[Loader]
    B --> C[Chunker]
    C --> D[Hybrid Retriever]
    D --> E[Relevance Agent]
    E --> F[Research Agent]
    F --> G[Verification Agent]
    G --> H{Supported?}
    H -- yes --> I[Cited Answer]
    H -- no --> J[Self-Correction]
    J --> G

Loading

Quick Start

python3.11 -m venv .venv
source .venv/bin/activate
pip install -e ".[dev,ui,pdf,docx]"
pytest

CrewAI requires Python 3.10 or newer; Python 3.11 is recommended for local setup.

To enable Gemini, create a .env file:

cp .env.example .env

Then add your Gemini API key:

GOOGLE_API_KEY=your_google_api_key_here
GEMINI_MODEL=gemini-2.5-flash

Add your OpenAI embedding key:

OPENAI_API_KEY=your_openai_api_key_here
OPENAI_EMBEDDING_MODEL=text-embedding-3-small

Without GOOGLE_API_KEY, DocuVerify automatically uses its local extractive answer mode for answer drafting. OPENAI_API_KEY is required for vector retrieval.

Ask a question from the sample document:

docuverify ask "What was the Northstar facility uptime?" examples/acme_sustainability.txt

Run the Gradio UI:

docuverify-ui

Example

Question: What was the Northstar facility uptime?

Answer:
The Northstar facility reported 99.93 percent uptime during the 2025 reliability window. [acme_sustainability.txt#chunk-000]

Verification:
supported=true confidence=0.72

Project Structure

docuverify-rag/
  src/docuverify_rag/
    agents.py       # relevance, research, verification, correction agents
    chunking.py     # LangChain recursive text splitting
    cli.py          # command line interface
    crew.py         # CrewAI agent and task workflow
    loaders.py      # text, markdown, PDF, and DOCX loading
    pipeline.py     # end-to-end orchestration
    embeddings.py   # OpenAI embeddings provider
    retrieval.py    # LangChain BM25 + vector hybrid retriever
  tests/
  examples/

Key Capabilities

• Built a CrewAI-orchestrated multi-agent RAG application that combines BM25 and OpenAI vector retrieval to answer document questions with source citations.
• Implemented a verification and self-correction loop that detects unsupported claims before responses are returned to users.
• Packaged the system as a tested Python project with CLI and Gradio interfaces for reproducible demos.

Extending With an LLM

The ResearchAgent uses Gemini through Google's generateContent API when configured. Retrieval uses LangChain BM25 plus vector similarity from OpenAI text-embedding-3-small. The VerificationAgent still checks generated claims against the retrieved evidence before the answer is shown.

License

MIT