litdiscover 2.0.0

Queue-driven, adaptive literature discovery engine — bidirectional citation traversal, LLM screening, thematic synthesis

LitDiscover

Queue-driven, adaptive literature discovery. Give it seed papers and inclusion criteria; LitDiscover traverses the citation graph bidirectionally, surfaces semantically similar papers from outside the graph, screens candidates with an LLM, and writes a thematic narrative review with numbered citations.

This is the active development repository. The empirical paper validating the approach (Robust Literature Discovery from Minimal Seeds) is maintained separately.

Looking for the older v1 pipeline? See automated-lit-reviews (archived).

---

Quick start

# 1. Install
git clone https://github.com/davidcagoh/automated-lit-reviews-v2.git
cd automated-lit-reviews-v2
pip install -e .
cp .env.example .env   # fill in keys (see Environment variables)

# 2. Apply DB schema (once, in Supabase SQL editor)
#    paste contents of supabase/schema.sql

# 3. Initialise a project from a folder of seed PDFs
litdiscover init projects/my-review/project.toml

# 4. Run to stability
litdiscover run projects/my-review/project.toml

# 5. Extract structured fields, then synthesise
litdiscover extract my-review
litdiscover synthesize my-review

Output: projects/my-review/my-review_review_numbered.md — a thematic narrative review with sequential [N] citations and a ## References section.

---

Environment variables

SUPABASE_URL=https://<project>.supabase.co
SUPABASE_SERVICE_KEY=<service-role key>

SEMANTIC_SCHOLAR_API_KEY=<key>   # required for S2 traversal + embeddings
GEMINI_API_KEY=<key>             # screening, extraction, synthesis (default)
GROQ_API_KEY=<key>               # alternative screening backend (free, fast)

---

CLI reference

litdiscover init   <project.toml>    # create DB record, ingest seeds
litdiscover run    <project.toml>    # main loop: traverse → screen → repeat
litdiscover status <slug>            # corpus stats
litdiscover reset  <slug>            # reset paper queue, keep criteria history
litdiscover extract   <slug>         # structured extraction (post-stability)
litdiscover synthesize <slug>        # thematic narrative review

---

Architecture

LitDiscover is a queue-driven state machine (litdiscover/core/loop.py). All state is persisted in Supabase, making runs fully restartable.

SEED papers
    │
    ▼
SEARCH (S2 keyword query)
    │
    ▼
SCREEN (LLM batch, up to 20 papers)
    │
  included?
    │── yes → TRAVERSE (forward + backward citation graph)
    │          PDF worker (background reference extraction)
    │          SEARCH (escape-hatch keyword queries when yield is low)
    │          → new candidates added to queue
    │
    └── cycle yield < threshold for 2 rounds → STOP
    │
    ▼
EXTRACT  (structured fields per included paper, Gemini)
    │
    ▼
SYNTHESIZE (k-means cluster → per-theme narrative → numbered bibliography)

Key design choices:

• Backward traversal is never hub-filtered (survey papers with many references are the most valuable traversal targets).
• Forward traversal applies a Pareto-80 hub filter by default to avoid flooding the queue with citing papers of landmark works.
• An escape-hatch keyword search fires when cycle yield drops below threshold, recovering papers not citation-connected to the seed cluster.
• Background PDF workers run concurrently with screening, extracting reference lists to feed the next traversal cycle.

---

Relationship to robust-literature-discovery

Repository	Role
automated-lit-reviews-v2 (this repo)	Deployed engine: queue loop, DB integrations, LLM screening, extraction, synthesis
robust-literature-discovery	Empirical paper: APS benchmark, publication figures, manuscript (Robust Literature Discovery from Minimal Seeds)

The paper repository is self-contained — you can reproduce all benchmark results without the full application.

---

Tests

python -m pytest tests/ -v

All tests are mocked (no live DB or API connections required).

---

Requirements

• Python ≥ 3.11
• A Supabase project (free tier works)
• Gemini API key (default backend) or Groq API key (free screening alternative)
• Semantic Scholar API key (for traversal and embeddings)