scinr

scinr is an AI-powered document knowledge library for the life sciences industry, built by Scinr AI.

📖 Documentation Site  |  🤖 llms.txt for Agents (Full Context)

Its newton module (scinr.newton) is the document ingestion engine: a complete 5-stage LLM pipeline that converts raw documents (PDF, DOCX, XLSX, PPTX, HTML, CSV, JSON, XML) into structured Neo4j knowledge graphs.

About Scinr

SCINR is the first agentic memory platform purpose-built for life sciences domains. Learn more at scinr.com.

scinr is the open library layer of the Newton platform, providing AI-driven extraction of knowledge from complex datasets and regulatory documents. It's newton module is the structured and unstructured ingestion foundation.

Table of Contents

1. Features
2. Architecture
3. Installation
4. Quick Start
5. Configuration
6. LLM Providers
7. Pipeline Stages
8. Extraction Domain Models
9. Project Structure
10. Further Reading
11. Contributing
12. License

Features

• Multi-format ingestion — PDF (Mistral OCR), DOCX, XLSX, PPTX, CSV, HTML, TXT, MD, XML/SOAP APIs, JSON APIs — all normalized to the same intermediate format
• Provider-agnostic — OpenAI, AWS Bedrock, Ollama, Anthropic, or any LangChain-compatible model
• LLM-powered extraction — sliding-window 2-phase extraction with an automatic repair loop; crash-safe intermediate writes after every chunk
• Knowledge graph output — idempotent Neo4j writes (MERGE throughout); safe to re-run at any stage
• Two independent LangGraph agents — Annotation agent assigns an extraction model to each structural node; Entity Extraction agent pulls typed Pydantic entities from text
• Auto-discovery of extraction domains — ThemeRegistry scans models/*/catalog.py at startup; no registration code needed when a new domain is added
• Dynamic schema composition — per-node composite Pydantic schemas built at runtime from the annotation decision (primary + complementary models + supplementary fields)
• Global entity deduplication — LabeledEntity singletons keyed by (label, normalized_value) enable cross-document dedup in the graph
• Cross-document model linking — instance_relationships create typed edges between ModelInstance nodes across different document sections; forward-reference shell nodes are merged when the target model is later extracted
• Versioning & folder hierarchy — full document version chain in Neo4j; folder structure mirrored as IS_COMPOSED_OF relationships
• Document lifecycle — delete_document() removes a document and its whole cascade; freeze_document() archives its subgraph to a snapshot and leaves a stub in the graph, and restore_document() rebuilds it with no LLM calls. All three are tenant-scoped and work in bounded transactions, so they fit in Neo4j's transaction memory whatever the size of the folder. See the Document Freezing guide
• Read-back navigation API — scinr.newton.navigation: a read-only, async, engine-abstracted layer of ~90 typed methods over the graph (documents, structure nodes, model instances, entities, schema introspection) — no Cypher required. See the Graph Navigation guide
• Tabular bypass pipeline — direct CSV/XLSX → Neo4j without LLM extraction stages; only 3 LLM calls per sheet (classify, decide model, map columns)
• Parallel processing — parallel_docs=N for concurrent document handling at every stage
• Live progress reporting — run_pipeline(callbacks=[...]) emits an event at every stage boundary of every document; use the built-in ProgressTracker (snapshot().percent) or your own sync/async callbacks to update a job in your database. See the Progress Tracking guide
• Prompt caching — Bedrock cachePoint support for ~90% reduction in repeated token costs
• Optional storage layer — MongoDB backend for raw file + page storage; pipeline runs without it
• Two retry layers — bedrock_retry (exponential backoff for throttling) + neo4j_retry (deadlock-safe writes)

Architecture

 Raw Files
 (PDF, DOCX, XLSX, PPTX, CSV, HTML, XML, TXT …)
        │
        ▼
┌──────────────────────┐
│  Stage 0             │  converters/
│  PREPROCESS          │  Mistral OCR (PDF) · python-docx · openpyxl · …
│                      │  → data/json/{filename}.json
└─────────┬────────────┘
          │  {"pages": [{"index": N, "markdown": "…"}]}
          │
          │         CSV / XLSX ──────────────────────────────────────────────┐
          ▼                                                                  │
┌──────────────────────┐                                              ┌──────▼──────────────┐
│  Stage 1             │  extraction/                                 │  Tabular Bypass     │
│  EXTRACT             │  any LangChain BaseChatModel                 │  tabular/           │
│                      │  Sliding window → Document tree              │  3 LLM calls / sheet│
│                      │  → data/output/extract-{filename}.json       └─────────────────────┘
└─────────┬────────────┘                                                    
          │  Document + StructureNode + InfoUnit                             
          ▼                                                                  
┌──────────────────────┐                                                     
│  Stage 2             │  ingest/                                            
│  INGEST              │  Neo4j MERGE writes (idempotent)    
│                      │  Versioning · folder hierarchy
└─────────┬────────────┘
          │  (:Document)-[:HAS_STRUCTURE]→(:StructureNode)
          ▼
┌──────────────────────┐
│  Stage 3             │  annotation/
│  ANNOTATE            │  LangGraph agent
│                      │  classify_theme → decide_model → format_decision
│                      │  → (:StructureNode)-[:HAS_MODEL_DECISION]→(:ModelDecision)
└─────────┬────────────┘
          │
          ▼
┌──────────────────────┐
│  Stage 4             │  entity_extraction/
│  ENTITY EXTRACT      │  LangGraph agent
│                      │  Dynamic schema composition → typed entity nodes
│                      │  → (:StructureNode)-[:HAS_EXTRACTION]→(:ExtractionResult)
└──────────────────────┘
                              → (:LabeledEntity) global cross-document singletons

Installation

Requirements: Python 3.11+, a running Neo4j instance (5.x+)

pip install scinr

Install with your preferred LLM provider:

pip install 'scinr[openai]'    # OpenAI / Azure OpenAI
pip install 'scinr[bedrock]'   # AWS Bedrock (Claude)
pip install 'scinr[ollama]'    # Ollama (local models)
pip install 'scinr[mongodb]'   # Optional MongoDB storage layer

For development:

git clone https://github.com/Scinr-AI/scinr.git
cd scinr
uv sync --all-extras

Quick Start

Configuration in code

from scinr.newton import configure, run_pipeline
from langchain_openai import ChatOpenAI
import asyncio

# Configure once at startup
configure(
    llm=ChatOpenAI(model="gpt-4o"),
    neo4j_uri="bolt://localhost:7687",
    neo4j_user="neo4j",
    neo4j_password="your-password",
)

# Run the full pipeline
result = asyncio.run(run_pipeline(input_raw="files/"))
print(f"Processed {result.total_documents} documents")

AWS Bedrock variant:

from langchain_aws import ChatBedrockConverse
from scinr.newton import configure, run_pipeline

configure(
    llm=ChatBedrockConverse(
        model="anthropic.claude-3-5-sonnet-20241022-v2:0",
        region_name="us-east-1",
    ),
    neo4j_uri="bolt://localhost:7687",
    neo4j_user="neo4j",
    neo4j_password="your-password",
)

Configuration from environment variables

Every configure() parameter you omit is read from the environment. Copy .env.example to .env and fill in the required values:

cp .env.example .env

# LLM — configured in code via configure(llm=...), NOT here.
# Provider SDK settings still come from the environment:
AWS_DEFAULT_REGION=us-east-1

# Required — Neo4j
NEO4J_URI=bolt://localhost:7687
NEO4J_USER=neo4j
NEO4J_PASSWORD=your_password
NEO4J_DATABASE=neo4j

# Required to ingest any PDF (PDF conversion is Mistral OCR only)
MISTRAL_API_KEY=your_mistral_key

# Optional — MongoDB storage backend
STORAGE_BACKEND=mongodb
MONGODB_URI=mongodb://localhost:27017
MONGODB_DATABASE=scinr

Then pass only the LLM in code and run the pipeline:

configure(llm=ChatOpenAI(model="gpt-4o"))  # Neo4j, Mistral and storage come from the environment

result = asyncio.run(run_pipeline(input_raw="files/", parallel_docs=4))

Configuration

Library mode — configure() key parameters

For the full parameter reference, see the module README.

Environment variables

configure() reads these when the matching parameter is omitted. The LLM is not an environment variable — build a LangChain chat model and pass it to configure(llm=...). Provider SDK settings (AWS_DEFAULT_REGION, OPENAI_API_KEY, …) come from the environment as each SDK expects.

AWS region & model ID (when using ChatBedrockConverse): the model ID prefix must match the region group. us.anthropic.claude-sonnet-4-6 needs AWS_DEFAULT_REGION in us-east-1 / us-west-2; Europe uses the eu. prefix with eu-central-1; Asia Pacific uses ap. with ap-northeast-1. Set region_name= on the model.

LLM Providers

scinr accepts any LangChain BaseChatModel via configure(llm=...).

You can set a cheaper repair_llm model for JSON repair retries:

configure(
    llm=ChatOpenAI(model="gpt-4o"),
    repair_llm=ChatOpenAI(model="gpt-4o-mini"),
)

Pipeline Stages

Run them all with run_pipeline(), a subset with run_pipeline(stages=[...]), or one at a time with run_preprocess(), run_extraction(), run_ingestion(), run_annotation(), run_entity_extraction() and run_tabular_pipeline(). For every parameter, see the Running the Pipeline guide; for per-stage examples, input/output formats, and agent diagrams, see the module README.

Extraction Domain Models

Model system overview

Extraction models are Pydantic classes that define the typed entities the pipeline should extract from a structural node. They are organised into themes (extraction domains) under models/. ThemeRegistry scans models/*/catalog.py at startup using Python's importlib — no registration code is needed when a new domain is added.

Open-source models

How to add an extraction domain

1. Create models/<your_theme>/ with __init__.py and catalog.py.
2. Export THEME_DESCRIPTION: str and SELECTABLE_MODELS: list[type[ExtractionModel]] from catalog.py.
3. Write Pydantic model classes that inherit from ExtractionModel.
4. No other changes required — ThemeRegistry picks up the new domain automatically on next startup.

See src/scinr/newton/model-creation/README.md for the full guide, including worked examples, entity label conventions, nested model patterns, field_relationships syntax (entity-level edges within a model instance), and instance_relationships syntax (cross-document model linking via ModelInstance nodes).

Project Structure

scinr/
├── README.md
├── LICENSE
├── pyproject.toml
├── .env.example
├── CHANGELOG.md
├── files/                          ← sample input files
├── tests/
└── src/
    └── scinr/
        └── newton/                 ← scinr.newton package
            ├── __init__.py         ← Public API
            ├── config.py           ← ScinrConfig + configure()
            ├── pipeline.py         ← run_pipeline()
            ├── results.py          ← Result dataclasses
            ├── exceptions.py       ← Exception hierarchy
            ├── stages/             ← Stage runner functions
            ├── converters/         ← Stage 0: file → paged JSON
            ├── extraction/         ← Stage 1: paged JSON → Document tree
            ├── ingest/             ← Stage 2: Document tree → Neo4j; delete / freeze / restore
            ├── annotation/         ← Stage 3: LangGraph annotation agent
            ├── entity_extraction/  ← Stage 4: LangGraph entity extraction
            ├── tabular/            ← Tabular bypass pipeline
            ├── storage/            ← Optional MongoDB storage layer
            ├── freeze/             ← Snapshot backend for document freezing
            ├── models/             ← Extraction domain models
            │   └── default/        ← Built-in open-source models
            ├── utils/              ← Shared utilities
            └── model-creation/     ← Developer guide (not a Python package)
                └── README.md

Further Reading

• src/scinr/newton/README.md — Public API reference (configure(), run_pipeline(), result types, exceptions), detailed stage documentation, Neo4j schema, storage layer configuration, and extension guides.
• src/scinr/newton/model-creation/README.md — Step-by-step guide for authoring new extraction domain models.

Contributing

Contributions are welcome! Please follow these steps:

1. Fork the repository and create a feature branch from main.
2. Install dependencies with uv sync --all-extras.
3. Make your changes following the existing code style (async functions, Pydantic v2 models, explicit error handling, no magic values).
4. Test your changes against a local Neo4j instance with a small set of sample documents.
5. Open a pull request with a clear description of what changed and why.

Areas where contributions are especially welcome

• New file format converters (converters/)
• New open-source extraction domain models (models/)
• Additional storage backends (storage/)
• Performance improvements to the extraction or entity extraction agents
• Documentation and examples

Reporting issues

Please open a GitHub issue with:

• A minimal reproducible example (document type, the run_pipeline() / run_*() call used)
• The full error traceback
• Your Python version, OS, and Neo4j version

License