Skip to main content

CrossRef Local (crossref-local)

SciTeX

Local CrossRef database with 167M+ scholarly works, full-text search, and impact factor calculation

Demo

CrossRef Local Demo

# Search 167M papers locally — no API rate limits, ~22 ms full-text query
crossref-local search "epilepsy seizure prediction"

# Resolve a DOI to full record (title, abstract, citations, journal IF)
crossref-local search-by-doi 10.1038/nature11247

# Drive from MCP / Claude Code
crossref-local mcp serve

The image is a live capture against the local DB; the <details> block below has a 6m55s MCP-driven demo video.

Architecture

┌──────────────────────────┐    ┌──────────────────────────┐
│ CrossRef public dump     │    │ JCR / OpenAlex IF tables │
│ (~100 GB compressed)     │    │                          │
└──────────────┬───────────┘    └──────────────┬───────────┘
               │ dois2sqlite                   │
               ▼                               ▼
       ┌─────────────────┐               ┌──────────────┐
       │ crossref.db     │ ◀── joins ──▶ │ impact-factor│
       │ (SQLite + FTS5) │               │ table        │
       └────────┬────────┘               └──────────────┘
                │
                ▼
   ┌──────────────────────────────────┐
   │ crossref-local — Python / CLI / MCP │
   │   search · search-by-doi · cache    │
   │   stats · check-citations · relay   │
   └──────────────────────────────────┘

The DB lives entirely on disk; crossref-local is a thin facade over SQLite + FTS5 + a small impact-factor table. No network calls during queries; rebuild scripts under make fts-build-screen / citations-build-screen are the only producers of state.

PyPI version Documentation Tests Coverage Python License

MCP Demo Video

Demo Video Thumbnail

Live demonstration of MCP server integration with Claude Code for epilepsy seizure prediction literature review:

  • Full-text search on title, abstracts, and keywords across 167M papers (22ms response)

📄 Full demo documentation | 📊 Generated diagrams

Why CrossRef Local?

Built for the LLM era - features that matter for AI research assistants:

Feature Benefit
📝 Abstracts Full text for semantic understanding
📊 Impact Factor Filter by journal quality
🔗 Citations Prioritize influential papers
⚡ Speed 167M records in ms, no rate limits

Perfect for: RAG systems, research assistants, literature review automation.

Installation
pip install crossref-local

From source:

git clone https://github.com/ywatanabe1989/crossref-local
cd crossref-local && make install

Database setup (1.5 TB, ~2 weeks to build):

# 1. Download CrossRef data (~100GB compressed)
aria2c "https://academictorrents.com/details/..."

# 2. Build SQLite database (~days)
pip install dois2sqlite
dois2sqlite build /path/to/crossref-data ./data/crossref.db

# 3. Build FTS5 index (~60 hours) & citations table (~days)
make fts-build-screen
make citations-build-screen
Python API
from crossref_local import search, get, count

# Full-text search (22ms for 541 matches across 167M records)
results = search("hippocampal sharp wave ripples")
for work in results:
    print(f"{work.title} ({work.year})")

# Get by DOI
work = get("10.1126/science.aax0758")
print(work.citation())

# Count matches
n = count("machine learning")  # 477,922 matches

Async API:

from crossref_local import aio

async def main():
    counts = await aio.count_many(["CRISPR", "neural network", "climate"])
    results = await aio.search("machine learning")
CLI
crossref-local search "CRISPR genome editing" -n 5
crossref-local search-by-doi 10.1038/nature12373
crossref-local status  # Configuration and database stats

With abstracts (-a flag):

$ crossref-local search "RS-1 enhances CRISPR" -n 1 -a

Found 4 matches in 128.4ms

1. RS-1 enhances CRISPR/Cas9- and TALEN-mediated knock-in efficiency (2016)
   DOI: 10.1038/ncomms10548
   Journal: Nature Communications
   Abstract: Zinc-finger nuclease, transcription activator-like effector nuclease
   and CRISPR/Cas9 are becoming major tools for genome editing...
HTTP API

Start the FastAPI server:

crossref-local relay --host 0.0.0.0 --port 31291

Endpoints:

# Search works (FTS5)
curl "http://localhost:31291/works?q=CRISPR&limit=10"

# Get by DOI
curl "http://localhost:31291/works/10.1038/nature12373"

# Batch DOI lookup
curl -X POST "http://localhost:31291/works/batch" \
  -H "Content-Type: application/json" \
  -d '{"dois": ["10.1038/nature12373", "10.1126/science.aax0758"]}'

# Citation endpoints
curl "http://localhost:31291/citations/10.1038/nature12373/citing"
curl "http://localhost:31291/citations/10.1038/nature12373/cited"
curl "http://localhost:31291/citations/10.1038/nature12373/count"

# Collection endpoints
curl "http://localhost:31291/collections"
curl -X POST "http://localhost:31291/collections" \
  -H "Content-Type: application/json" \
  -d '{"name": "my_papers", "query": "CRISPR", "limit": 100}'
curl "http://localhost:31291/collections/my_papers/download?format=bibtex"

# Database info
curl "http://localhost:31291/info"

HTTP mode (connect to running server):

# On local machine (if server is remote)
ssh -L 31291:127.0.0.1:31291 your-server

# Python client
from crossref_local import configure_http
configure_http("http://localhost:31291")

# Or via CLI
crossref-local --http search "CRISPR"
MCP Server

Run as MCP (Model Context Protocol) server:

crossref-local mcp start

Local MCP client configuration:

{
  "mcpServers": {
    "crossref-local": {
      "command": "crossref-local",
      "args": ["mcp", "start"],
      "env": {
        "CROSSREF_LOCAL_DB": "/path/to/crossref.db"
      }
    }
  }
}

Remote MCP via HTTP (recommended):

# On server: start persistent MCP server
crossref-local mcp start -t http --host 0.0.0.0 --port 8082
{
  "mcpServers": {
    "crossref-remote": {
      "url": "http://your-server:8082/mcp"
    }
  }
}

Diagnose setup:

crossref-local mcp doctor        # Check dependencies and database
crossref-local mcp list-tools    # Show available MCP tools
crossref-local mcp installation  # Show client config examples

See docs/remote-deployment.md for systemd and Docker setup.

Available tools:

  • search - Full-text search across 167M+ papers
  • search_by_doi - Get paper by DOI
  • enrich_dois - Add citation counts and references to DOIs
  • status - Database statistics
  • cache_* - Paper collection management
Impact Factor
from crossref_local.impact_factor import ImpactFactorCalculator

with ImpactFactorCalculator() as calc:
    result = calc.calculate_impact_factor("Nature", target_year=2023)
    print(f"IF: {result['impact_factor']:.3f}")  # 54.067
Journal IF 2023
Nature 54.07
Science 46.17
Cell 54.01
PLOS ONE 3.37
Citation Network
from crossref_local import get_citing, get_cited, CitationNetwork

citing = get_citing("10.1038/nature12373")  # 1539 papers
cited = get_cited("10.1038/nature12373")

# Build visualization (like Connected Papers)
network = CitationNetwork("10.1038/nature12373", depth=2)
network.save_html("citation_network.html")  # requires: pip install crossref-local[viz]
Performance
Query Matches Time
hippocampal sharp wave ripples 541 22ms
machine learning 477,922 113ms
CRISPR genome editing 12,170 257ms

Searching 167M records in milliseconds via FTS5.

Related Projects

openalex-local - Sister project with OpenAlex data:

Feature crossref-local openalex-local
Works 167M 284M
Abstracts ~21% ~45-60%
Update frequency Real-time Monthly
DOI authority ✓ (source) Uses CrossRef
Citations Raw references Linked works
Concepts/Topics ❌ ✓
Author IDs ❌ ✓
Best for DOI lookup, raw refs Semantic search

When to use CrossRef: Real-time DOI updates, raw reference parsing, authoritative metadata. When to use OpenAlex: Semantic search, citation analysis, topic discovery.

Installation

pip install crossref-local              # core
pip install crossref-local[mcp]         # + MCP server

From source:

git clone https://github.com/ywatanabe1989/crossref-local
cd crossref-local && make install

4 Interfaces

Python API
from crossref_local import search, get, count

# Full-text search (22ms for 541 matches across 167M records)
results = search("hippocampal sharp wave ripples")
for work in results:
    print(f"{work.title} ({work.year})")

# Get by DOI
work = get("10.1126/science.aax0758")
print(work.citation())

# Count matches
n = count("machine learning")  # 477,922 matches
CLI
crossref-local search "CRISPR genome editing" -n 5
crossref-local search-by-doi 10.1038/nature12373
crossref-local status  # Configuration and database stats
HTTP API

See the HTTP API section above for all endpoints.

MCP Server

See the MCP Server section above for configuration.

Part of SciTeX

crossref-local is part of SciTeX.

Four Freedoms for Research

  1. The freedom to run your research anywhere — your machine, your terms.
  2. The freedom to study how every step works — from raw data to final manuscript.
  3. The freedom to redistribute your workflows, not just your papers.
  4. The freedom to modify any module and share improvements with the community.

AGPL-3.0 — because we believe research infrastructure deserves the same freedoms as the software it runs on.


SciTeX

Metadata

Release files for crossref-local 0.8.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for crossref-local 0.8.1
File Size Uploaded Provenance
crossref_local-0.8.1.tar.gz 4.8 MB Without provenance Details

Built distribution (wheel)

Table of built distributions (wheels) for crossref-local 0.8.1
File Interpreter ABI Platform
crossref_local-0.8.1-py3-none-any.whl Python 3 none any Details

Total release size: 9.6 MB

Release files / crossref_local-0.8.1.tar.gz

Download URL crossref_local-0.8.1.tar.gz
Size 4.8 MB
Tags Source
SHA-256 checksum
How to use checksums
54659517a97d7264d93d72ab0448aab74c5da687d58bcc8e24a6ca2401b1b1d6
BLAKE2b-256 checksum
How to use checksums
15329b07578e90f6656fbae6a15773fd6eb401b04780572da17ea1ecd3593b6c
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/6.2.0 CPython/3.12.13

Release files / crossref_local-0.8.1-py3-none-any.whl

Download URL crossref_local-0.8.1-py3-none-any.whl
Size 4.8 MB
Tags Python 3
SHA-256 checksum
How to use checksums
99cdc411940451845d2650c3cb576e6a55185832312486fbbd5053146d25292f
BLAKE2b-256 checksum
How to use checksums
1cf41ed36a1f6dd46cd4117545e241d7fa1ca2ad2a8b5d52392c382126331492
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/6.2.0 CPython/3.12.13

Security for crossref-local 0.8.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page