research-claim-validator 0.1.12

Python tools for validating research claims against scientific literature.

---

title: Research Finding Validation System description: A multi-agent Graph RAG system for validating scientific claims against research literature from ArXiv and OpenAlex. author: Tolulope Ale ms.date: 2026-04-24 ms.topic: overview keywords:

• graph-rag
• multi-agent
• scientific-validation
• neo4j
• faiss
• langgraph

---

Table of Contents

• Overview
• Getting Started
• Architecture
• Features
• System Components
• Workflows
• Technology Stack
• Project Structure
• Development Notebooks
• Testing and Evaluation

Overview

This system automates the validation of scientific findings by orchestrating multiple specialized agents that search, process, and analyze research papers. The system employs a multi-workflow architecture that separates concerns into paper fetching, embedding/indexing, and retrieval/validation stages.

Key Capabilities

• Multi-source Paper Search: Fetch papers from ArXiv and OpenAlex with semantic ranking
• Intelligent Chunking: Hierarchical text segmentation with section-aware processing
• Knowledge Graph Construction: Neo4j-based graph linking papers, entities, and relationships
• Vector Similarity Search: FAISS-powered semantic search across paper chunks
• Semantic Similarity Linking: Automatic discovery of related papers and content
• LLM-Powered Validation: Claude-based claim validation with structured reasoning
• Modular Workflow Execution: Independent execution of paper fetching, embedding, and validation
• Real-time Streaming: Server-Sent Events for live progress updates
• Comprehensive Observability: Distributed tracing, metrics, and structured logging

Getting Started

Python Package Usage

Install the lightweight package:

pip install research-claim-validator

The base install supports the public API and no-evidence responses without installing the heavier LLM, graph, or vector-search dependencies:

from research_claim_validator import ClaimValidator

validator = ClaimValidator()
result = validator.validate("This research claim is supported by recent literature.")
print(result.to_dict())

For LLM-backed validation against supplied evidence, install the optional LLM dependencies:

pip install "research-claim-validator[llm]"

Set your LLM provider and API key before running validation:

$env:LLM_PROVIDER = "anthropic"
$env:LLM_MODEL = "claude-sonnet-4-6"
$env:ANTHROPIC_API_KEY = "your-api-key"

On macOS or Linux:

export LLM_PROVIDER="anthropic"
export LLM_MODEL="claude-sonnet-4-6"
export ANTHROPIC_API_KEY="your-api-key"

Then pass evidence dictionaries to validate(...). The validator bases its answer only on the evidence you provide:

from research_claim_validator import ClaimValidator

validator = ClaimValidator()
result = validator.validate(
    "Arctic sea ice extent has declined substantially in recent decades.",
    evidence=[
        {
            "title": "Example Arctic evidence",
            "text": (
                "Observations indicate that Arctic sea ice cover has rapidly "
                "retreated in recent decades as global temperatures have increased."
            ),
        }
    ],
)
print(result.to_dict())

In Jupyter notebooks, use the async API because notebooks already run an event loop:

result = await validator.validate_async(
    "Arctic sea ice extent has declined substantially in recent decades.",
    evidence=[
        {
            "title": "Example Arctic evidence",
            "text": (
                "Observations indicate that Arctic sea ice cover has rapidly "
                "retreated in recent decades as global temperatures have increased."
            ),
        }
    ],
)
result.to_dict()

For automatic literature ingestion and corpus-backed validation, install the corpus dependencies:

pip install "research-claim-validator[corpus]"

Corpus mode uses a local Neo4j knowledge graph plus a local FAISS vector index. Neo4j must be running and configured before auto_ingest=True:

$env:LLM_PROVIDER = "anthropic"
$env:LLM_MODEL = "claude-sonnet-4-6"
$env:ANTHROPIC_API_KEY = "your-api-key"
$env:NEO4J_URI = "neo4j://localhost:7687"
$env:NEO4J_USER = "neo4j"
$env:NEO4J_PASSWORD = "your-neo4j-password"

On macOS or Linux:

export LLM_PROVIDER="anthropic"
export LLM_MODEL="claude-sonnet-4-6"
export ANTHROPIC_API_KEY="your-api-key"
export NEO4J_URI="neo4j://localhost:7687"
export NEO4J_USER="neo4j"
export NEO4J_PASSWORD="your-neo4j-password"

Then create a persistent local corpus directory and validate against retrieved paper evidence:

from research_claim_validator import ClaimValidator

validator = ClaimValidator(corpus_dir="./claim-validator-corpus")
result = validator.validate(
    "Arctic sea ice extent has declined substantially in recent decades due to warming temperatures.",
    query="Arctic sea ice extent decline recent decades warming temperatures polar climate change",
    auto_ingest=True,
    max_papers=3,
)
print(result.to_dict())

The first corpus build can take several minutes because papers may be fetched, PDFs processed, embeddings generated, a FAISS index written, and Neo4j metadata populated. Later runs can reuse the same corpus directory and are usually much faster.

See SETUP.md for installation, environment configuration, and local development instructions.

For Docker-based deployment, see DEPLOYMENT.md.

For Neo4j installation and configuration, see NEO4J_SETUP.md.

Architecture

Architectural Design Framework

The system follows a multi-layer agent-based architecture with clear separation of concerns:

┌─────────────────────────────────────────────────────────────┐
│                     Presentation Layer                      │
│  ┌─────────────────────┐         ┌────────────────────┐   │
│  │  Streamlit UI       │         │  FastAPI Endpoints │   │
│  │  - Real-time SSE    │         │  - RESTful API     │   │
│  │  - Workflow Control │         │  - OpenAPI Docs    │   │
│  └─────────────────────┘         └────────────────────┘   │
└─────────────────────────────────────────────────────────────┘
                              │
┌─────────────────────────────────────────────────────────────┐
│                    Orchestration Layer                      │
│  ┌──────────────────┐  ┌──────────────┐  ┌──────────────┐ │
│  │ Paper Fetching   │  │  Embedding   │  │  Retrieval   │ │
│  │   Workflow       │  │   Workflow   │  │   Workflow   │ │
│  └──────────────────┘  └──────────────┘  └──────────────┘ │
│                    (LangGraph-based)                        │
└─────────────────────────────────────────────────────────────┘
                              │
┌─────────────────────────────────────────────────────────────┐
│                       Agent Layer                           │
│  ┌────────────────────────────────────────────────────────┐│
│  │ Ingestion Agents                                       ││
│  │  • QueryGenerator  • ArxivScout  • PaperFetcher       ││
│  └────────────────────────────────────────────────────────┘│
│  ┌────────────────────────────────────────────────────────┐│
│  │ Processing Agents                                      ││
│  │  • ChunkAgent  • EntityAgent  • GraphWriter           ││
│  │  • VectorIndexer  • SimilarityLinker                  ││
│  └────────────────────────────────────────────────────────┘│
│  ┌────────────────────────────────────────────────────────┐│
│  │ Query Agents                                           ││
│  │  • Planner  • Retriever  • Neo4jEnricher              ││
│  │  • EvidenceBuilder  • Validator  • Critic             ││
│  └────────────────────────────────────────────────────────┘│
└─────────────────────────────────────────────────────────────┘
                              │
┌─────────────────────────────────────────────────────────────┐
│                      Service Layer                          │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────────┐ │
│  │ Paper Source │  │     LLM      │  │  Embedding Gen   │ │
│  │  (ArXiv,     │  │  (Claude,    │  │  (Nomic, OpenAI) │ │
│  │   OpenAlex)  │  │   GPT-4)     │  │                  │ │
│  └──────────────┘  └──────────────┘  └──────────────────┘ │
└─────────────────────────────────────────────────────────────┘
                              │
┌─────────────────────────────────────────────────────────────┐
│                      Storage Layer                          │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────────┐ │
│  │    Neo4j     │  │    FAISS     │  │  File Cache      │ │
│  │ (Knowledge   │  │  (Vector     │  │  (Embeddings,    │ │
│  │   Graph)     │  │   Store)     │  │   PDFs, ArXiv)   │ │
│  └──────────────┘  └──────────────┘  └──────────────────┘ │
└─────────────────────────────────────────────────────────────┘
                              │
┌─────────────────────────────────────────────────────────────┐
│                   Observability Layer                       │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────────┐ │
│  │   Jaeger     │  │  Prometheus  │  │  Structured      │ │
│  │  (Tracing)   │  │  (Metrics)   │  │   Logging        │ │
│  └──────────────┘  └──────────────┘  └──────────────────┘ │
└─────────────────────────────────────────────────────────────┘

Design Principles

1. Separation of Concerns: Three independent workflows (paper fetching, embedding, validation) can run separately or sequentially
2. Agent Modularity: Each agent is self-contained with clear inputs/outputs
3. Stateful Orchestration: LangGraph manages workflow state and agent transitions
4. Async-First: All I/O operations use async/await for optimal performance
5. Caching Strategy: Multi-level caching (embeddings, API responses, PDFs) reduces latency
6. Observability: Comprehensive tracing, metrics, and logging across all components
7. Type Safety: Pydantic models enforce data contracts throughout the system

Data Flow

User Input → Query Generation → Paper Search → Paper Fetching
                                                      ↓
                                            PDF Processing → Chunking
                                                      ↓
                                            Entity Extraction → Graph Writing
                                                      ↓
                                            Embedding Generation → Vector Indexing
                                                      ↓
                                            Similarity Linking (Papers + Chunks)
                                                      ↓
Query Planning → Semantic Retrieval → Graph Enrichment → Evidence Building
                                                      ↓
                                            Validation → Confidence Evaluation → Critic
                                                      ↓
                                            Structured Result → User

Features

Paper Acquisition

• Multi-Source Support: ArXiv and OpenAlex integration
• Intelligent Search: LLM-generated queries or manual query input
• Semantic Ranking: Relevance scoring and filtering
• Date Filtering: Publication year range constraints
• PDF Download: Automatic full-text acquisition with fallback to abstracts
• Rate Limiting: Compliant with API rate limits
• Caching: Persistent storage of search results and PDFs

Text Processing

• Hierarchical Chunking: Section-aware segmentation with 7-level priority system
• Entity Extraction: Named Entity Recognition using spaCy
• Relation Extraction: Dependency-based relationship identification
• Citation Tracking: Inter-paper citation graph construction

Knowledge Representation

• Graph Database: Neo4j stores papers, chunks, entities, and relationships
• Vector Store: FAISS indexes semantic embeddings for similarity search
• Similarity Linking:
  • Paper-level similarity (title + abstract)
  • Chunk-level similarity (K-NN semantic search)
  • Configurable thresholds and limits

Retrieval and Validation

• Two-Stage Retrieval:

  1. Coarse semantic search (FAISS)
  2. Priority-based reranking with diversity control

• Graph Enrichment: Neo4j traversal for entity and citation context

• Smart Context Building: Token-aware evidence assembly (8000 token budget)

• LLM Validation: Claude-powered claim analysis with structured reasoning

• Confidence Scoring: Multi-factor confidence evaluation

• Quality Assurance: Critic agent reviews validation quality

Observability

• Distributed Tracing: Jaeger integration for request flow visualization
• Metrics Collection: Prometheus-compatible metrics export
• Structured Logging: JSON logs with correlation IDs
• Real-time Monitoring: SSE streams for live workflow progress

System Components

Agents (17 Total)

Ingestion Agents (3)

• QueryGenerator: LLM-powered query formulation from findings
• ArxivScout: Paper search across ArXiv/OpenAlex with ranking
• PaperFetcher: PDF download and metadata extraction

Processing Agents (5)

• ChunkAgent: Hierarchical text segmentation
• EntityAgent: Named entity and relation extraction
• GraphWriter: Neo4j graph population
• VectorIndexer: FAISS index construction
• ParallelWriter: Concurrent graph and vector writes
• SimilarityLinker: Semantic similarity relationship creation

Query Agents (7)

• DatabaseChecker: Validates existing data availability
• Planner: Query planning and search strategy
• Retriever: Two-stage semantic retrieval
• Neo4jEnricher: Graph-based context enrichment
• EvidenceBuilder: Evidence compilation and ranking
• Validator: LLM-based claim validation
• ConfidenceEvaluator: Multi-factor confidence scoring
• Critic: Quality assurance and reasoning review

Services

• Paper Sources: ArXiv and OpenAlex API clients
• LLM Interface: Anthropic Claude and OpenAI GPT-4 integration
• Embedding Generator: Nomic embedding models
• Graph Manager: Neo4j driver with connection pooling
• Vector Store: FAISS index with metadata management
• Cache Manager: Multi-level caching system

Infrastructure

• FastAPI Backend: Async REST API with SSE streaming
• Streamlit Frontend: Interactive web UI with real-time updates
• Neo4j Database: Graph database for knowledge representation
• Jaeger: Distributed tracing system
• Prometheus: Metrics collection and monitoring

Workflows

1. Paper Fetching Workflow

Purpose: Discover and download relevant research papers

Flow: QueryGenerator (optional) → ArxivScout → PaperFetcher → END

Inputs:

• Scientific finding (for LLM query generation) OR
• Manual query string
• Paper source (arxiv/openalex)
• Max papers count
• Date range filters

Outputs:

• Paper artifacts (metadata + PDFs)
• Session state for downstream workflows

2. Embedding Workflow

Purpose: Process papers into searchable knowledge graph and vector index

Flow: ChunkAgent → EntityAgent → ParallelWriter(GraphWriter || VectorIndexer) → SimilarityLinker → END

Inputs:

• Paper artifacts from fetching workflow OR
• Auto-discovery from pdf_cache/cache directories

Outputs:

• Neo4j knowledge graph
• FAISS vector index
• Similarity relationships

3. Retrieval Workflow

Purpose: Validate scientific claims using retrieved evidence

Flow: Planner → Retriever → Neo4jEnricher → EvidenceBuilder → Validator → ConfidenceEvaluator → Critic → END

Inputs:

• Scientific claim to validate
• Optional finding optimization
• Existing knowledge graph + vector index

Outputs:

• Validation result (supported/not supported/partial)
• Confidence score
• Evidence chunks with citations
• Reasoning and limitations

Technology Stack

Component	Technology	Purpose
Orchestration	LangGraph 0.0.20	Stateful workflow management
Backend	FastAPI 0.104+	Async REST API with SSE
Frontend	Streamlit 1.28+	Interactive web UI
Knowledge Graph	Neo4j 5.15	Graph database
Vector Store	FAISS (IndexFlatIP)	Similarity search
LLM	Claude	Validation and reasoning
Embeddings	Nomic-AI nomic-embed-text-v1.5	Semantic embeddings (768-dim)
NLP	spaCy en_core_web_md	Entity extraction
Tracing	Jaeger 1.52	Distributed tracing
Metrics	Prometheus	System monitoring
Caching	File-based + Memory	Multi-level caching
PDF Processing	PyPDF2	Text extraction
Paper APIs	ArXiv API, OpenAlex	Paper metadata

Project Structure

Groundtruth_validation_search/
├── research_claim_validator/
│   ├── agents/                    # 17 specialized agents
│   │   ├── arxiv_scout.py
│   │   ├── chunk_agent.py
│   │   ├── entity_agent.py
│   │   ├── graph_writer.py
│   │   ├── vector_indexer.py
│   │   ├── similarity_linker.py
│   │   ├── retriever.py
│   │   ├── validator.py
│   │   └── ...
│   ├── api/                       # FastAPI application
│   │   ├── main.py
│   │   ├── routers/
│   │   │   └── validation.py      # API endpoints
│   │   └── middleware/            # Rate limiting, tracing
│   ├── frontend/                  # Streamlit UI
│   │   ├── app_modular.py
│   │   ├── sse_client.py
│   │   └── components/            # UI components
│   ├── workflow/                  # Workflow orchestration
│   │   ├── orchestrator.py
│   │   ├── paper_fetching_workflow.py
│   │   ├── embedding_workflow.py
│   │   ├── retrieval_workflow.py
│   │   ├── event_emitter.py
│   │   └── state_manager.py
│   ├── services/                  # External integrations
│   │   ├── papers/                # ArXiv, OpenAlex
│   │   ├── llm/                   # Claude, GPT-4
│   │   ├── graph/                 # Neo4j manager
│   │   └── embeddings/            # Vector generation
│   ├── models/                    # Data models
│   │   ├── artifacts.py
│   │   ├── events.py
│   │   └── workflow_state.py
│   ├── config/
│   │   └── settings.py            # Configuration
│   ├── utils/                     # Utilities
│   │   ├── cache_manager.py
│   │   └── latency_tracker.py
│   └── observability/             # Tracing, metrics
│       ├── tracing.py
│       └── metrics.py
├── tests/                         # Test suite
│   ├── unit/
│   ├── integration/
│   ├── conftest.py
│   └── run_tests.py
├── cache/                         # Cached data
│   ├── arxiv_*.json              # ArXiv search results
│   └── emb_*.npy                 # Cached embeddings
├── pdf_cache/                     # Downloaded PDFs
├── vector_index/                  # FAISS index files
│   ├── faiss.index
│   └── metadata.json
├── logs/                          # Application logs
├── docker-compose.yml             # Service orchestration
├── Dockerfile                     # Backend container
├── Dockerfile.frontend            # Frontend container
├── requirements.txt               # Python dependencies
├── .env.example                   # Configuration template
├── README.md                      # This file
├── SETUP.md                       # Setup and installation guide
├── DEPLOYMENT.md                  # Deployment guide
├── NEO4J_SETUP.md                 # Neo4j setup guide
├── multiagent_rag.ipynb          # Development notebook
└── evaluation_ragas.ipynb        # Evaluation notebook

Development Notebooks

multiagent_rag.ipynb and knowledge_graph_optimized.ipynb

The primary development notebook demonstrating the complete multi-agent RAG system. This notebook served as the prototype for the production implementation and includes:

• End-to-end workflow examples
• Agent initialization and testing
• Knowledge graph construction
• Vector index building
• Validation examples with real papers
• Performance benchmarking

Use cases:

• Understanding system architecture
• Testing individual agents
• Prototyping new features
• Educational reference

evaluation_ragas.ipynb

Comprehensive evaluation notebook using the RAGAS framework to assess system performance:

Metrics Evaluated:

• Faithfulness: Factual consistency of generated answers
• Answer Relevancy: Pertinence to the input question
• Context Precision: Relevance of retrieved context
• Context Recall: Proportion of relevant context retrieved

Features:

• Automated evaluation pipeline
• Ground truth dataset construction
• Comparison across configurations
• Performance visualization
• Metric aggregation and reporting

Use cases:

• System evaluation and benchmarking
• A/B testing configuration changes
• Quality assurance validation
• Research and publication

Testing and Evaluation

Unit Tests

# Run all unit tests
pytest tests/unit/

# With coverage
pytest tests/unit/ --cov=research_claim_validator --cov-report=html

Integration Tests

# Run integration tests (requires Neo4j)
pytest tests/integration/

# Specific test
pytest tests/integration/test_workflow.py

End-to-End Testing

# Run complete system test
python run_tests.py

Evaluation Metrics

Run RAGAS evaluation (see evaluation_ragas.ipynb):

from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy

# Load test dataset
dataset = ...

# Run evaluation
results = evaluate(
    dataset=dataset,
    metrics=[faithfulness, answer_relevancy]
)

print(results)

Last Updated: April 2026

---

Designed and Developed by Tolulope Ale

Multi-Agent GraphRAG System for Research Finding Validation