publiminer 0.2.0

Publication Literature Miner — PubMed API, embedding-based clustering, and LLM structured-output extraction

PubLiMiner

Publication Literature Miner — fetch, parse, deduplicate, embed, cluster, and extract structured data from PubMed at scale.

PubLiMiner is a modular Python pipeline for mining biomedical literature from PubMed. It is designed for 200K+ paper corpora with monthly incremental updates, a single Parquet file as the source of truth, and pluggable steps for embedding, clustering, and LLM-based structured extraction.

Features

• PubMed retrieval — date-batched fetcher with rate limiting, retry, and SQLite response caching
• XML parsing — extracts title, abstract, authors, journal, year, DOI, MeSH, keywords, grants, publication type, and more
• Deduplication — 4-layer: PMID exact → DOI exact → fuzzy title (year-grouped) → retracted-paper removal
• Single source of truth — every step reads/writes columns to one papers.parquet file
• Streamlit UI — visual config editor, live progress, status panel, and an Explore tab with lazy-scan filters (year slider, publication-status, language) and sampling modes (first-N / random-N / stride), plus XLSX/JSON download
• Resumable — atomic writes, idempotent imports, crash-safe
• Legacy import — bulk-import existing JSON batches without re-downloading
• CLI + library — use as a Typer CLI or as a Python package

Pipeline overview

fetch → parse → deduplicate → embed → reduce (optional)
                                ├→ cluster → sample → extract → score → trend
                                ├→ rag
                                └→ export ← trend, patent

Currently implemented: fetch, parse, deduplicate. The remaining steps are scaffolded in the architecture and being added incrementally.

Architecture

Full architecture diagrams: docs/architecture.md

graph TB
    subgraph User Interface
        CLI["CLI: publiminer run/status/inspect/ui"]
        UI["Streamlit UI: publiminer ui"]
    end

    subgraph Pipeline
        FETCH["FetchStep<br/>PubMed API → raw XML"]
        PARSE["ParseStep<br/>XML → structured fields"]
        DEDUP["DeduplicateStep<br/>4-layer dedup"]
    end

    subgraph Data Layer
        SPINE["Spine"]
        PARQUET["papers.parquet<br/>(source of truth)"]
        STAGING["papers.parquet.staging<br/>(crash checkpoint)"]
        CACHE["SQLite cache<br/>(raw API responses)"]
    end

    CLI --> FETCH --> PARSE --> DEDUP
    UI --> CLI
    FETCH --> SPINE
    PARSE --> SPINE
    DEDUP --> SPINE
    SPINE --> PARQUET
    SPINE --> STAGING
    FETCH --> CACHE

Key design decisions:

• Single Parquet file is the source of truth — every step reads columns, adds columns, writes back
• Staging checkpoint makes fetch crash-safe — resume from where you left off
• Incremental parse — only processes rows where title IS NULL
• Memory-bounded — fetch streams in 5 MB batches, merge uses pyarrow row groups (~50 MB cap)
• Binary bisection + PMID-list fallback — handles PubMed's 10k pagination limit automatically

Getting started

One line gets you installed and into the UI — same text on macOS, Linux, and Windows:

uv tool install "publiminer[ui]" && publiminer ui

The first time you run it, an interactive wizard walks you through email + NCBI key capture and scaffolds a starter config. From then on, publiminer ui (or publiminer run) just launches.

🔧 Install uv first (one-time, per machine)

uv is a fast, cross-platform Python package manager from Astral. If you already have it, skip this.

macOS / Linux:

curl -LsSf https://astral.sh/uv/install.sh | sh

Windows (PowerShell):

powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"

Close and reopen your terminal so uv is on PATH, then run the one-liner above.

🔑 How to get an NCBI API key (optional, takes ~2 min)

An API key raises your rate limit from 3 → 10 requests per second (≈3× faster on large corpora).

1. Go to https://account.ncbi.nlm.nih.gov/settings/
2. Sign in with Google, ORCID, or your NCBI username
3. Scroll to API Key Management → click Create an API Key
4. Copy the key (looks like abc123def456…) and paste it into the wizard when prompted

The wizard writes it to .env in your working directory and adds .env to .gitignore automatically. You can also skip and add it later by running publiminer setup again.

🖱️ No-code users — what the UI gives you

After publiminer ui launches and you finish the wizard, the browser opens a 4-tab interface:

1. Configure — edit your PubMed query, date range, and pipeline steps. Click Save YAML
2. Run — execute the pipeline; watch the live progress bar
3. Explore — filter by year / language / publication status, sample papers (first-N, random, or every Xth), download as XLSX or JSON
4. Status — total papers in corpus, file size, schema

Re-run the wizard any time with publiminer setup --force to change credentials or re-scaffold.

⌨️ Developers — CLI + Python API

publiminer setup                                      # interactive first-run wizard
publiminer run --config publiminer.yaml               # full pipeline
publiminer run --steps parse,deduplicate              # specific steps
publiminer status --output output                     # corpus summary
publiminer inspect parse --output output              # step metadata
publiminer import-legacy /path/to/batches --output output   # idempotent import

Skip the auto-wizard in scripts with --no-setup or PUBLIMINER_NO_WIZARD=1.

Use as a library:

from publiminer import FetchStep, ParseStep, DeduplicateStep, Spine, GlobalConfig

cfg = GlobalConfig()                      # loads publiminer.yaml + defaults
spine = Spine("output")
print(spine.count(), "papers currently in spine")

From source (for contributors):

git clone https://github.com/sdamirsa/PubLiMiner.git
cd PubLiMiner
uv sync --all-extras
uv run pytest                             # run tests
uv run publiminer ui                      # launch UI from the checkout

🧪 Alternate install paths (pip, pipx, uvx)

If you don't want to adopt uv, the same package works via pip and pipx. The wizard runs on first launch regardless.

# pip in your current venv — simplest, works anywhere Python exists
pip install "publiminer[ui]" && publiminer ui

# pipx — isolates into its own managed env
pipx install "publiminer[ui]" && publiminer ui

# uvx — ephemeral one-shot run, no persistent install (good for demos)
uvx --from "publiminer[ui]" publiminer ui

Project layout

src/publiminer/
├── core/         # Spine (Parquet), cache (SQLite), config, models, base step
├── steps/        # Pipeline steps — each self-contained (step.py, schema.py, default.yaml)
│   ├── fetch/
│   ├── parse/
│   └── deduplicate/
├── utils/        # Logger, rate limiter, env loader, batching, progress, legacy import
├── ui/           # Streamlit UI
├── cli.py        # Typer CLI
└── pipeline.py   # Full run orchestrator

Configuration

PubLiMiner uses a single publiminer.yaml file. Each step has its own section with sane defaults — you only need to override what you want to change.

general:
  output_dir: output
  log_level: INFO
fetch:
  query: "diabetes AND machine learning"
  start_date: "2024/01/01"
  end_date: "2024/12/31"
  email: ""           # or set PUBMED_EMAIL env var
  api_key: ""         # or set NCBI_API_KEY env var
  max_results: 0      # 0 = no cap (use date partitioning for large queries)
parse:
  prepare_llm_input: true
  flag_exclusions: true
deduplicate:
  fuzzy_threshold: 90
  remove_retracted: true

Secrets (NCBI_API_KEY, OPENROUTER_API_KEY, PATENT_API_KEY) should always come from environment variables, never the YAML.

Resumable nightly runs

PubLiMiner is designed to be re-run nightly without redoing any work:

• Streaming fetch: every batch is flushed to a papers.parquet.staging checkpoint, so a crash mid-run loses nothing. The next run merges the staging file before continuing. Memory stays bounded (~50 MB) regardless of corpus size.
• Incremental parse: only rows without a title column are parsed. After the first sweep, subsequent runs only touch newly-fetched papers.
• Auto date resume: set fetch.start_date: "auto" to resume from (latest fetch_date − 7 days). The 7-day overlap covers PubMed back-dating and is harmless because PMID-level dedup skips anything already on disk.
• run_nightly.bat: Windows wrapper that logs to nightly.log. Schedule via Task Scheduler:

  schtasks /create /tn "PubLiMiner Nightly" /tr "C:\path\to\PubLiMiner\run_nightly.bat" /sc daily /st 02:00
  

Performance notes

• 400K papers: ~1.0–1.2 GB parquet, ~30–60 min total runtime with an NCBI API key
• PubMed WebEnv limit: a single esearch+efetch session caps at 9,999 records — use date partitioning (start_date / end_date) for larger queries
• Memory: parse streams in 5K-row batches (~1.5 GB peak on a 500K-row corpus); dedup reads only the columns it needs per layer (~1 GB peak). The only step that loads the full parquet is the final add_columns write after parse, which spikes to ~3–5 GB transiently depending on corpus size
• Parquet format: files are written with zstd-3 compression and 50K-row groups — required for pyarrow.iter_batches to actually stream. If you have a pre-v0.1.x parquet written with snappy and large row groups, run uv run python scripts/migrate_parquet.py once to re-chunk it
• Disk: keep at least 2× your final parquet size free for atomic writes

Development

See CLAUDE.md for the developer guide (architecture, conventions, how to add a new step).

pip install -e ".[dev]"
pytest                          # run the test suite
ruff check src/                 # lint
ruff format src/                # format
mypy src/publiminer/            # type-check

License

MIT — see LICENSE.

Citation

If you use PubLiMiner in academic work, please cite the repository:

@software{publiminer,
  author = {Safavi-Naini, Seyed Amir Ahmad},
  title = {PubLiMiner: Publication Literature Miner},
  url = {https://github.com/sdamirsa/PubLiMiner},
  year = {2026}
}