pubmed-markdown 0.1.0

Convert PubMed articles (PMIDs or PMCIDs) to clean, structured markdown with full text, abstracts, and supplementary materials

PubMed Downloader

Convert PubMed articles to clean, structured markdown. Handles the full pipeline: PMID resolution, full-text extraction via PubMed Central, HTML-to-markdown conversion, and supplementary material retrieval.

Articles without open-access full text automatically fall back to abstract-only download.

Installation

pip install git+https://github.com/shloknatarajan/PubMedDownloader.git

Setup

Set your email for NCBI API identification (optional but recommended):

export NCBI_EMAIL=[e-mail]

Or create a .env file in your working directory:

NCBI_EMAIL=[e-mail]

Usage

Python API

Single article (returns markdown string, no files created):

from pubmed_downloader import PubMedDownloader

downloader = PubMedDownloader()

# From PMID (resolves to PMCID automatically, falls back to abstract if not open access)
markdown = downloader.single_pmid_to_markdown("12895196")

# From PMCID directly
markdown = downloader.single_pmcid_to_markdown("PMC1884285")

Batch processing (saves HTML and markdown files to disk):

from pubmed_downloader import PubMedDownloader

downloader = PubMedDownloader()
pmids = ["12895196", "17872605", "25051018"]
downloader.pmids_to_markdown(pmids, save_dir="data")

This creates:

data/
├── html/          # Raw HTML from PMC
├── markdown/      # Converted markdown files
├── cache/         # PMID-to-PMCID mapping cache
└── pmcids.txt     # Resolved PMCIDs

Add supplementary materials to existing markdown files:

downloader.add_supplements_to_existing(save_dir="data")

Individual utility functions:

from pubmed_downloader import (
    get_pmcid_from_pmid,
    get_html_from_pmcid,
    get_abstract_markdown_from_pmid,
    fetch_bioc_supplement,
)

# Resolve PMIDs to PMCIDs
mapping = get_pmcid_from_pmid(["12895196", "17872605"])

# Fetch raw HTML from PMC
html = get_html_from_pmcid("PMC1884285")

# Get abstract for non-open-access articles
abstract_md = get_abstract_markdown_from_pmid("12345678")

# Get supplementary material text
supplement = fetch_bioc_supplement("PMC6435416")

Command Line

# Convert PMIDs from a file (one PMID per line)
pubmed-download --file_path=pmids.txt --save_dir=data

# Add supplementary materials to existing markdown
pubmed-download --add_supplements --save_dir=data

# Clear all caches
pubmed-download --clear_caches

API Reference

Method	Creates Files	Returns	Use Case
single_pmid_to_markdown()	No	Markdown string	Single article, programmatic use
single_pmcid_to_markdown()	No	Markdown string	Direct PMCID conversion
pmids_to_markdown()	Yes	None	Batch processing, building datasets
local_html_to_markdown()	Yes	None	Re-convert existing HTML files
add_supplements_to_existing()	Yes	None	Append supplements to existing markdown

PharmGKB Integration

Extract PMIDs from PharmGKB variant annotations for pharmacogenomics research:

from pubmed_downloader.pharmgkb_annotations import get_pmid_list
from pubmed_downloader import PubMedDownloader

# Download PharmGKB annotations and extract PMIDs
pmids = get_pmid_list(save_dir="data")

# Convert to markdown
downloader = PubMedDownloader()
downloader.pmids_to_markdown([str(p) for p in pmids], save_dir="data")

How It Works

1. PMID to PMCID -- Uses NCBI's ID Converter API with batching, caching (30-day expiry), and rate limiting
2. HTML extraction -- Fetches full article HTML from PubMed Central
3. Markdown conversion -- Converts HTML to structured markdown preserving tables, figures, citations, and references
4. Supplementary materials -- Fetches pre-processed supplement text via NCBI's BioC API
5. Abstract fallback -- Articles not in PMC Open Access get abstract + metadata via NCBI E-Fetch

Configuration

Environment Variable	Default	Description
NCBI_EMAIL	None	Email for NCBI API identification
PMID_CACHE_DIR	data/cache	Cache directory path
PMID_CACHE_FILE	pmid_to_pmcid.json	Cache filename

License

MIT