pubmed-stream

From a PubMed search to a knowledge graph of the literature. pubmed-stream downloads full-text articles from PubMed Central, maps the evidence around key papers, and links everything a study collects into an Obsidian knowledge graph, where papers connect through citations, similarity and MeSH topics. It is built on NCBI's official E-utilities, works from the command line or Python, and needs nothing beyond requests.

Documentation: melhzy.github.io/pubmed-stream, the tutorials, guides and reference in one searchable website, with every notebook shown with its results.

A study made with it: the Aβ*56 retraction impact page, how a retracted Alzheimer's paper spread through the literature and how much of it language models learned, as an interactive page where every number comes from the study's files; see Studies.

New to programming? Start with tutorial 0, which takes a research question to a map of its literature in an afternoon. Click Open in Colab to run it in your web browser, with nothing to install; you only type your question into one settings cell.

flowchart LR
    S["PubMed searches"] -->|"download"| D["Full-text articles<br/>metadata, references, text"]
    K["Key papers"] -->|"neighborhood"| N["Similar, citing and cited papers<br/>flagged reviews, retractions, NIH funding"]
    D --> G["Knowledge graph<br/>kg_topic, an Obsidian vault"]
    N --> G
    G --> O["Explore, write, refresh"]

Contents

• Quick start
• Tutorials
• Studies
• Installation and configuration
• 1. Download full-text articles
• 2. Write searches that find the right papers
• 3. Map the evidence around a paper
• 4. Build the knowledge graph of a study
• 5. Work with AI assistants, local models and Obsidian
• Python API
• Troubleshooting
• How it works
• Contributing
• FAQ
• Citation, acknowledgments and license

Quick start

pip install pubmed-stream

# Size a search and preview its newest titles, without downloading: it counts PMC and all of PubMed
pubmed-stream search '"gut microbiota"[Title] AND frailty[Title/Abstract]'

# Download the full text of up to 50 articles, add the abstracts of those whose full text is not
# free, then read them compactly
pubmed-stream download '"gut microbiota"[Title] AND frailty[Title/Abstract]' --max-results 50
pubmed-stream abstracts '"gut microbiota"[Title] AND frailty[Title/Abstract]' --max-results 200
pubmed-stream list publications --abstracts

# Export them as citations for Zotero, EndNote or Mendeley
pubmed-stream export publications -o frailty.bib

# Map the evidence around one paper: similar, citing and cited papers, with retractions flagged
pubmed-stream neighborhood 31319564

# Build a study's knowledge graph in ./kg_gut_microbiome_frailty, then open it in Obsidian
pubmed-stream kg "gut microbiome frailty" -q '"gut microbiota"[Title] AND frailty[Title/Abstract]' --seed 31319564
pubmed-stream open kg_gut_microbiome_frailty

# Summarize the 10 newest abstracts and save the summaries in the knowledge graph. --model picks who
# summarizes: a model on your own computer, in Ollama (https://ollama.com), so nothing leaves it, or the
# model of an AI assistant installed below: claude, codex or gemini. Section 5 helps you choose
ollama pull qwen3.5:4b
pubmed-stream review "Is the gut microbiota linked to frailty?" --limit 10 --model qwen3.5:4b --vault kg_gut_microbiome_frailty
pubmed-stream review "Is the gut microbiota linked to frailty?" --limit 10 --model claude --vault kg_gut_microbiome_frailty

# Or ask in plain English and let Claude Code, Codex or Gemini CLI run all of this: see below

With an AI assistant: Claude Code, Codex or Gemini CLI

An AI assistant can do all of the above for you. Install one: Claude Code needs a paid Claude plan, Codex a paid ChatGPT plan, and Gemini CLI only a Google account. These commands need Node.js 22 or later; tutorial 7 shows the other installers and walks through a session.

npm install -g @anthropic-ai/claude-code   # Claude Code
npm install -g @openai/codex               # Codex
npm install -g @google/gemini-cli          # Gemini CLI

# Set up a research folder for all three, then start your assistant in it
mkdir gut_frailty
cd gut_frailty
pubmed-stream init
claude        # or: codex, or: gemini

The first time, the assistant asks you to sign in. Claude Code then asks to approve the pubmed-stream server, and Gemini CLI asks whether you trust the folder: say yes to both, or the assistant will not have pubmed-stream's tools. Then ask in plain language:

Is the gut microbiota linked to frailty? Size the search first, download the clinical studies, and summarize the evidence with PMIDs.

Claude Code and Codex can also carry pubmed-stream's tools into every folder as a plugin; section 5 compares the two ways.

In Python

The first steps of the quick start, in Python:

from pubmed_stream import create_kg, get_neighborhood, search_and_download

stats = search_and_download('"gut microbiota"[Title] AND frailty[Title/Abstract]', max_results=50)
hood = get_neighborhood("31319564")
result = create_kg("gut microbiome frailty", queries=['"gut microbiota"[Title] AND frailty[Title/Abstract]'],
                   seeds=["31319564"])
print(stats.successful, hood.totals, result.kg_dir)

Tutorials

The notebooks in tutorials/ are written for researchers, not programmers. Each follows a real research question. The notebooks have a single Your settings cell for your own question, run in Google Colab with nothing to install, and are saved with their results so you can read them first, on GitHub or on the documentation site; tutorial 7 happens in a terminal, with an AI assistant.

#	Tutorial	Use case
0	Start here: a new research topic in an afternoon	From a question to a map of its literature: size the field, find the paper everyone cites, build a knowledge graph, read its themes, evidence and gaps, export a reading list
1	Getting started	Download a search, read the files, choose a format, resume, fetch one known article, rescue a search that finds nothing, read downloads compactly
2	Designing searches	Measure the size, precision and availability of five versions of one search, and combine words with MeSH headings
3	A literature-review corpus	Run a keyword list, deduplicate, screen by theme, export to CSV and BibTeX
4	The evidence around one paper	Map a paper's neighborhood and screen it for retractions
5	The knowledge graph of a study	Build a graph and explain it with twelve charts and tables
6	Text mining and RAG preparation	Split articles into sections and chunk them for embeddings
7	Your literature search with an AI assistant	For medical students and biologists: install Claude Code or Codex, choose a model, ask in plain English, check the evidence, keep it in Obsidian, or use a local model through Ollama
8	Measuring the impact of a retracted paper	Follow the retracted 2006 Nature paper on Aβ*56 through the papers citing it and the papers citing those: when and how they cite it, who still cites it after the retraction, and a knowledge graph of its impact

Five short guides support them:

Guide	For
How to run the tutorials	Colab, installing Python on Windows or Mac, running a notebook, a free NCBI API key
Reading a knowledge graph in Obsidian	Opening, exploring, searching and writing in a graph
Glossary	MeSH, PMC, stub, seed and every other term, in plain language
What pubmed-stream can and cannot do	Coverage, blind spots, and why it is not a systematic review
Troubleshooting for beginners	Setup, notebook, Colab and Obsidian problems

The tutorials README lists everything, including how programmers can run the notebooks locally.

Studies

Research done with pubmed-stream lives in studies/, one folder per study with its frozen data, scripts and write-up, and a web page where the study has one, published with the documentation site.

Study	Page	Question
Aβ*56 retraction impact	Open the page	Continues tutorial 8. How far the retracted 2006 Nature paper on Aβ*56 spread through the literature: the 1,240 papers citing it and the 44,957 papers citing those, frozen as tables; a sourced account of what the paper claimed and exactly what was wrong; counts of the claim and its correction in two open language-model training corpora; and probe questions for language models, with the first answers

The studies README explains the layout of a study and how to start one.

Installation and configuration

pip install pubmed-stream                  # the library and the pubmed-stream command
pip install "pubmed-stream[tutorials]"     # plus jupyter, matplotlib, networkx and pandas for the tutorials

The AI-assistant, local-model and Obsidian features of section 5 need version 0.3.0 or later, PubMed abstracts and citation export 0.4.0, reviews by Claude Code, Codex or Gemini CLI (review --model claude) 0.4.1, and pubmed_stream.impact, which tutorial 8 imports, and saving your NCBI API key with init 0.4.2: pubmed-stream --version shows yours, and pip install --upgrade pubmed-stream updates it. From a clone, pip install -e ".[dev]" installs it in editable mode with pytest and ruff. pubmed-stream needs Python 3.9 or later; its only runtime dependency is requests.

Set an NCBI API key. NCBI allows 3 requests per second without a key and 10 with one, so a key makes everything about three times faster. Keys are free: create one under API Key Management in the settings of an NCBI account. Save it once, for every folder:

pubmed-stream init --ncbi-api-key "your-key"   # kept in ~/.pubmed-stream.json; init also asks for it while none is saved

init checks the key with NCBI before saving it, and if NCBI ever rejects a saved key, pubmed-stream says so and continues without it rather than failing every request. Without a key, a download that a key would have shortened by ten seconds or more ends by saying how much time it would have saved.

Claude Code asks for it when the pubmed-stream plugin is installed. For scripts and servers, the environment works too, with an optional contact email, and takes precedence:

export NCBI_API_KEY="your-key"        # PowerShell: $env:NCBI_API_KEY = "your-key"
export NCBI_EMAIL="you@example.org"   # PowerShell: $env:NCBI_EMAIL = "you@example.org"

Every command also accepts --api-key and --email. pubmed-stream sets its request rate from the key automatically, retries failed requests three times with back-off, and never writes the key to its logs. --rate-limit SECONDS slows it down further if needed.

1. Download full-text articles

pubmed-stream download searches PubMed Central directly, so every result is an article with a PMC record, then fetches each one's full-text XML in parallel and saves it as one JSON file. Some publishers do not allow their articles to be downloaded as XML; those are reported as unavailable and skipped, rather than saved half-empty.

pubmed-stream download "gut microbiome" --max-results 50

from pubmed_stream import search_and_download

stats = search_and_download("gut microbiome", max_results=50)
print(f"{stats.successful} saved, {stats.skipped} already there, {stats.unavailable} unavailable "
      f"of {stats.total_found} matches, in {stats.duration_seconds:.0f} s")

Each search gets its own folder, named after the query; a name longer than 100 characters is shortened and ends with a short hash of the query. Running a search again skips the files that already exist, so an interrupted download resumes and a repeated search only fetches what is new.

To see what you downloaded without opening every file, list prints one line per article, newest first, and each article once however many searches found it:

pubmed-stream list publications                       # identifiers, year, title, journal, length
pubmed-stream list publications --id 31319564 --abstracts   # the abstracts of chosen articles
pubmed-stream list publications --cited               # the works the downloads cite most
pubmed-stream list publications --json --abstracts > corpus.jsonl

Articles without free full text. PMC holds the free full text of only part of the literature: search counts both, and for the frailty search of the quick start PubMed has 95 articles where PMC has 66. abstracts saves the PubMed record of each match, with its abstract, MeSH terms and publication types, in the same folder as the full texts. list, review and the knowledge graph read both; list marks the abstracts [abstract only], and an article saved both ways is shown with its full text.

pubmed-stream abstracts '"gut microbiota"[Title] AND frailty[Title/Abstract]' --max-results 200
pubmed-stream kg "gut microbiome frailty" -q '"gut microbiota"[Title] AND frailty[Title/Abstract]' --abstracts

from pubmed_stream import fetch_abstracts

stats = fetch_abstracts('"gut microbiota"[Title] AND frailty[Title/Abstract]', max_results=200)
print(f"{stats.saved} abstracts saved of {stats.total_found} in PubMed; {stats.in_pmc} are also free in PMC")

Citations. export writes the articles, full texts and abstracts alike, as BibTeX or RIS, which Zotero (File > Import), EndNote, Mendeley and LaTeX read. Entries have the title, authors, journal, year, DOI, PubMed and PMC IDs, a link and the abstract. The first word of each author name is taken as the surname, so check multi-word surnames such as "van der Berg" after importing.

pubmed-stream export publications -o review.bib
pubmed-stream export publications --format ris --id 38686220 41113141 -o chosen.ris

publications/
└── gut_microbiome/
    ├── query.json          the original search, recorded for later tools
    ├── PMC12345678.json    a full text from PMC
    ├── PMC12345679.json
    └── PMID41361501.json   an abstract from PubMed, saved by 'abstracts'

Every article file has the same shape. metadata.references lists the cited works that have a PubMed ID or DOI; those become citation links in a knowledge graph.

{
  "pmcid": "PMC13279599",
  "source": "PMC",
  "download_date": "2026-10-18T13:20:45",
  "metadata": {
    "pmid": "42060019",
    "pmcid": "PMC13279599",
    "doi": "10.1007/s40520-026-03403-6",
    "title": "Association between gut microbiota and sarcopenia in older adults: ...",
    "journal": "Aging Clinical and Experimental Research",
    "year": "2026", "month": "4", "day": "30",
    "authors": ["Ramezani Kashal Fatemeh", "Sharifi Farshad", "..."],
    "abstract": "Introduction Investigating gut microbiota has emerged as a novel approach ...",
    "keywords": ["Sarcopenia", "Gut microbiota", "Aging", "Muscle metabolism"],
    "references": [
      {"pmid": "32033882", "doi": "10.1016/j.jamda.2019.12.012",
       "title": "Asian Working Group for Sarcopenia: 2019 Consensus Update ...", "year": "2020"}
    ],
    "linked_discoveries_url": "https://linkeddiscoveries.ncbi.nlm.nih.gov/42060019/"
  },
  "text": "Association between gut microbiota and sarcopenia ...\nAbstract\nIntroduction ..."
}

The format decides what is kept beside the metadata:

--format	Adds	Best for
text (default)	text: title, abstract and body, one paragraph per line	Reading, search, embeddings, most NLP
xml	xml: the raw JATS XML	Section-level parsing, tables, figures
both	text and xml	Keeping every option open

Option	Meaning	Default
QUERY	PubMed search; field tags such as [Title] work	required
--max-results N	Articles to download, newest first	100
--format	text, xml or both	text
--exclude-text	Leave out the text field to save space	included
--output-dir, -o	Base folder	./publications
--workers N, --sequential	Parallel downloads, or one at a time for unstable networks	5 workers
--topic, --vault	Also add the results to a knowledge graph, see section 4	off
--api-key, --email, --rate-limit, --user-agent, -v	NCBI settings and verbose logging	environment
--quiet	Print only the summary, without progress	progress shown

download exits with 0 when articles were saved or already present, 1 when the search found nothing, and 2 when every download failed.

2. Write searches that find the right papers

PMC searches the full text of every article, including its reference list, so a plain search such as gut microbiome frailty matches thousands of articles that only mention the words in passing. Field tags fix this. In tutorial 2, five versions of one question range from 7,587 matches with almost no relevant recent results to 66 matches where most are on topic.

Technique	Example	Effect
Put the core concept in the title	"gut microbiota"[Title]	The largest gain in precision
Restrict the rest to title and abstract	frailty[Title/Abstract]	Drops passing mentions in the body
Quote multi-word terms, and cover synonyms with OR	("gut microbiome" OR "gut microbiota")	Keeps phrases together without losing recall
Restrict to open access	AND "open access"[filter]	Every result can be downloaded
Limit the years	AND 2020:2026[pdat]	Reaches past the newest results, or focuses on them

PMC returns the newest matches first, so --max-results takes the most recent articles. Check a search before downloading: pubmed-stream search QUERY prints its size and newest titles without downloading anything, and --show 0 prints the count only.

$ pubmed-stream search '"gut microbiota"[Title] AND sarcopenia[Title/Abstract]' --show 2
80 articles in PMC for: "gut microbiota"[Title] AND sarcopenia[Title/Abstract]
Newest 2:
  2026  PMC13668947  PMID 42875678  Correspondence for the article: "Association between gut microbiota and sarcopenia ...
  2026  PMC13641649  PMID 41879259  Alterations in the Gut Microbiota of Cirrhotic Patients With Sarcopenia and PH After TIPS. ...

3. Map the evidence around a paper

A search finds papers by their words. An evidence neighborhood starts from one key paper and follows PubMed's links instead: the papers PubMed rates as similar, the papers citing it, and the papers it cites. Each is flagged as a review, retracted or NIH-funded. NLM's Linked Discoveries web tool shows the same kind of neighborhood but has no API; pubmed-stream builds one from documented E-utilities.

pubmed-stream neighborhood 9500320

Seed: PMID 9500320 (1998) Ileal-lymphoid-nodular hyperplasia, non-specific colitis, and ...
Similar       100            (4 reviews, 0 retracted, 0 NIH-funded, 18 in PMC)
Cited by      100 of 489     (145 reviews, 0 retracted, 52 NIH-funded, 421 in PMC)
References      0            (0 reviews, 0 retracted, 0 NIH-funded, 0 in PMC)
Neighborhood  200 unique articles fetched
[RETRACTED] PMID 9500320 (1998) Ileal-lymphoid-nodular hyperplasia, non-specific colitis, and ...
Saved: publications/neighborhood_9500320/neighborhood.json
Explore the graph: https://linkeddiscoveries.ncbi.nlm.nih.gov/9500320/

100 of 489 means 489 PubMed articles cite the seed and the newest 100 were fetched in detail; --max-per-relation changes that cap. The counts in brackets always cover all 489, counted with PubMed searches rather than by downloading every record. [RETRACTED] lists every retracted article among those fetched, so check them before citing or building on them.

Flag	Meaning
review	Publication type Review, Systematic Review or Scoping Review; narrower than PubMed's Review filter, which also counts guidelines and meta-analyses
retracted	Publication type Retracted Publication, or a linked retraction notice
NIH-funded	An NIH grant in the grant list, or an N.I.H. research-support publication type
in PMC	The article has a PMC ID, so its full text may be downloadable

The neighborhood is saved as publications/neighborhood_<PMID>/neighborhood.json. --download also saves the PMC full text of the seed and every neighbor that has one, beside it. In Python:

from pubmed_stream import get_neighborhood, save_neighborhood, download_neighborhood

hood = get_neighborhood("31319564", max_per_relation=50)
print(hood.totals)                      # {"similar": 384, "cited_by": 239, "references": 139}
print(hood.flag_counts["cited_by"])     # reviews, retracted, NIH-funded and in-PMC counts across all of them

retracted = [a for a in hood.articles() if a.is_retracted]          # seed and neighbors
cites_retracted = [a for a in hood.references if a.is_retracted]    # does the paper cite retracted work?

save_neighborhood(hood)                 # publications/neighborhood_31319564/neighborhood.json
download_neighborhood(hood)             # PMC full text of the seed and its neighbors

Each neighbor is a RelatedArticle with pmid, title, journal, year, authors, pub_types, is_review, is_retracted, nih_funded, pmcid, doi, mesh_terms, and, for similar articles, a similarity score. A neighborhood costs two requests for up to 200 articles, plus five for each relation cut short by the cap.

Compared with Linked Discoveries: similarity comes from PubMed's Similar Articles algorithm rather than a BiomedBERT model, so the neighbors can differ; there are no condition, gene or chemical filters; and citation links exist only where reference lists were deposited, mostly for PMC articles. Like the web tool, the flags describe connections; they do not judge whether a study is sound. Tutorial 4 charts a neighborhood and walks through retraction screening.

neighborhood exits with 0 on success, 1 when the PMID is invalid or not in PubMed, and 2 when NCBI could not be reached. It takes a PubMed ID, the number in a pubmed.ncbi.nlm.nih.gov/<PMID>/ link or a downloaded article's metadata.pmid, not a PMC ID.

4. Build the knowledge graph of a study

Downloads and neighborhoods come together in a knowledge graph. A study has a topic, the searches that collect its literature, and optionally key papers whose neighborhoods belong in it. pubmed-stream kg runs them all and links the results in an Obsidian vault named kg_<topic>: a folder of Markdown notes, one per paper, MeSH concept and collection. Open it in Obsidian with Open folder as vault, and the graph view shows which papers cite each other and which topics they share.

pubmed-stream kg "gut microbiome frailty" \
    -q '"gut microbiota"[Title] AND frailty[Title/Abstract]' \
    -q '"gut microbiota"[Title] AND sarcopenia[Title/Abstract]' \
    --seed 31319564 --max-results 25 --max-per-relation 30

That study gives a graph of 127 papers, 42 with full text and 85 from the seed's neighborhood, linked to 309 MeSH concepts: 440 notes in all. The study is remembered, so it keeps growing and stays current with only its topic:

pubmed-stream kg "gut microbiome frailty" -q "microbiome inflammaging"   # add a search
pubmed-stream kg "gut microbiome frailty" --add ./publications/old_query # add earlier downloads
pubmed-stream kg "gut microbiome frailty" --refresh                      # rerun everything, add new papers
pubmed-stream kg "gut microbiome frailty"                                # rebuild the notes only

download and neighborhood can join a study too: with --topic "gut microbiome frailty", their search or seed is recorded like one made with kg. A bare --vault names the graph after the query or the neighborhood instead, and --vault DIR or --kg-dir DIR picks any folder. Obsidian links notes only inside one vault, so give every search of a study the same topic.

What the graph contains

Note	Location	Links to
Paper	papers/PMID<id>.md	Papers it cites, papers citing it, similar papers, its MeSH concepts and collections
Concept	concepts/<MeSH heading>.md	Papers indexed with that heading
Collection	collections/Search - <query>.md, collections/Neighborhood - PMID <id>.md	Its papers and top concepts
Home	PubMed Stream.md	Collections, most connected concepts, retracted papers

Here is a slice of the example study's graph. Every link in it is real: two 2026 studies from different searches cite the same 2021 study, which PubMed also rates as similar to the seed review. Connections like this are what a list of search results hides.

flowchart LR
    C1["Search -<br/>gut microbiota AND sarcopenia"] --> P1["PMID 42060019<br/>gut microbiota and sarcopenia<br/>in older adults, 2026"]
    C2["Search -<br/>gut microbiota AND frailty"] --> P4["PMID 41601864<br/>gut microbiota and<br/>cognitive frailty, 2026"]
    C3["Neighborhood -<br/>PMID 31319564"] --> P2["PMID 31319564<br/>seed review, 2019"]
    C3 --> P3["PMID 33633246<br/>microbiota in patients<br/>with sarcopenia, 2021"]
    P1 -->|cites| P3
    P4 -->|cites| P3
    P2 -.->|similar| P3
    P1 --> M1(["Sarcopenia"])
    P1 --> M2(["Gastrointestinal Microbiome"])
    P3 --> M1
    P3 --> M2
    P2 --> M1
    classDef collection fill:#8E4EC6,stroke:#8E4EC6,color:#fff
    classDef stub fill:#8B8D98,stroke:#8B8D98,color:#fff
    classDef concept fill:#12A594,stroke:#12A594,color:#fff
    class C1,C2,C3 collection
    class P2,P3 stub
    class M1,M2 concept

Where the links come from:

• Citations come from the reference lists of downloaded articles, for references with a PMID, and from neighborhood citing and cited papers. References with only a DOI are listed under Other references.
• Similarity comes from neighborhood similar articles, with PubMed's score.
• Concepts are MeSH headings fetched from PubMed, once per 200 papers, and cached in the graph. Check tags such as Humans or Aged stay in the paper's properties but get no note, because they would link almost every paper.
• Stubs are neighborhood papers that were not downloaded. Their notes carry PubMed details and links, and are tagged pubmed/stub.

A paper note holds Obsidian properties, then a generated block, then space for your own notes:

---
type: "paper"
title: "Association between gut microbiota and sarcopenia in older adults: ..."
pmid: "42060019"
year: 2026
journal: "Aging Clinical and Experimental Research"
mesh:
  - "Sarcopenia"
  - "Gastrointestinal Microbiome"
status: "full-text"
tags:
  - "pubmed/paper"
---
%% pubmed-stream:start - generated; edits inside this block are overwritten %%
# Association between gut microbiota and sarcopenia in older adults: ...

**Published:** *Aging Clinical and Experimental Research* · 2026 · [PubMed](...) · [PMC](...) · [DOI](...)

## Abstract
## MeSH concepts
- [[Sarcopenia]] (major topic)
## Cites (39)
- [[PMID33633246|Alterations in intestinal microbiota diversity, composition, and function in patients with sarcop... (2021)]]
## Cited by (1)
%% pubmed-stream:end %%

## Notes

Your own notes go here and survive every rebuild.

Rebuilding is safe. A rebuild replaces only the generated block and the properties the tool owns. Text outside the block, and properties, tags and aliases you add in Obsidian, are kept; unchanged notes are not rewritten. Do not rename or move generated notes, or a rebuild will recreate them. Properties work in Obsidian search, Bases and the Dataview plugin, and on the first build the graph view gets colours for retracted papers, reviews, stubs, concepts and collections.

Tutorial 5 builds this study and explains the graph with charts: where the papers came from, which concepts travel together, which papers the study relies on most, and which ones bridge its parts.

kg option	Meaning	Default
TOPIC	Study topic; the graph is ./kg_<topic>	required
--query, -q	Search to download into the study; repeatable	none
--seed, -s	PMID whose neighborhood joins the study; repeatable	none
--add DIR	Folder or file downloaded earlier; repeatable	none
--refresh	Run every remembered search and seed again	only new ones
--max-results, --max-per-relation	Articles per search; neighbors per relation	100, 100
--download-neighbors	Download neighbors' full text instead of stub notes	stubs
--fulltext	Put each article's full text into its note	abstract only
--offline	Use cached MeSH terms only	fetch missing ones
--kg-dir DIR	Graph folder instead of ./kg_<topic>	./kg_<topic>

kg exits with 0 when everything worked, 1 when there is nothing to build or an option is invalid, and 2 when a search or neighborhood failed; the graph is still built from everything else.

Limitations to know about:

• Some papers lack MeSH terms. Indexing takes months, and papers in journals outside MEDLINE are never indexed, so such papers have no concept links. Rebuild later to pick up newly indexed papers; the others stay unlinked to concepts but keep their citation links.
• Downloads from pubmed-stream 0.1.1 or earlier have no reference lists, unless saved with --format xml or both. Download them again to add citation links.
• Many references have no PMID. Conference papers, preprints and books appear under Other references, not in the graph.
• Notes are never deleted. Removing a source leaves its notes in the graph.

5. Work with AI assistants, local models and Obsidian

pubmed-stream connects to the tools researchers now work with: AI assistants that run in a terminal, local models that keep everything on your computer, and Obsidian, where the knowledge graph lives.

flowchart LR
    A["Claude Code, Codex,<br/>Gemini CLI"] -->|"MCP tools"| P["pubmed-stream"]
    P --> K["kg_topic<br/>Obsidian vault"]
    P -->|"pubmed-stream review"| O["Ollama<br/>local model"]
    O -->|"review notes"| K
    A -->|"obsidian CLI"| K

AI assistants. Claude Code, Codex and Gemini CLI call pubmed-stream directly through its MCP server: nine tools, from sizing a search to building the knowledge graph, each returning compact text, so the assistant can scan 64 papers in about 3,000 tokens instead of reading 4.3 MB of full text. The instructions that come with the tools make it size searches before downloading, cite only papers it downloaded and read in the session, say what kind of evidence each is, report its searches with their counts, and name what it could not read. Set it up in one of two ways:

	The plugin	pubmed-stream init
Gives the tools to	Claude Code or Codex, in every folder	any of the three assistants, in one research folder
Needs	uv; the plugin fetches pubmed-stream itself	pip install pubmed-stream
Carries the instructions as	the skill literature-search	AGENTS.md in the folder, which CLAUDE.md and GEMINI.md import
Asks for your NCBI API key	at install (Claude Code); the server then saves it for the command line too	in the terminal, once
Also		chooses the folder's models: init --claude-model opus, init --review-model <ollama model>

# The plugin, once
claude plugin install pubmed-stream --marketplace melhzy/pubmed-stream   # Claude Code
codex plugin marketplace add melhzy/pubmed-stream                         # Codex: then /plugins, Install plugin

# Or one folder
pubmed-stream init

Then start the assistant in your research folder, claude, codex or gemini, and ask a research question in plain language:

Does fecal microbiota transplantation help people with Parkinson's disease? Find the clinical evidence and summarize it, with PMIDs.

The two ways do not conflict: where the plugin is installed, init leaves its server out of the folder, so the tools are not listed twice, and running it again in a folder set up earlier removes the extra one. The plugin, in plugins/pubmed-stream, runs the pubmed-stream release of its own version through uvx and saves files in the folder you start the assistant in. Without uv, register the server you installed with pip: claude mcp add pubmed-stream -- pubmed-stream mcp, or the same with codex mcp add.

Local models. For questions you would rather not send to a company, pubmed-stream review has a model running in Ollama on your own computer summarize the abstracts you chose; only the PubMed searches leave your computer. The model reads one abstract at a time, at temperature 0, and never sees your question, so it cannot present the question as a finding. pubmed-stream attaches each PubMed ID itself and flags numbers and citations the abstracts do not support.

1. Install Ollama from ollama.com/download, or on Windows with winget install Ollama.Ollama. It runs in the background; ollama list checks that it works.

2. Choose a model your computer can run. Any model in Ollama's library works, open-weight models such as Qwen, Gemma or Llama included; pubmed-stream turns off the "thinking" step of models that have one, so it does not use up the summary's length. Larger models make fewer mistakes but need more memory: the model's download size should be at most about two-thirds of your computer's memory, and a model that fits in a graphics card's memory runs much faster.

Your computer	Model size	For example (download size)
Any laptop, 8 GB of memory	1 to 4 billion parameters	llama3.2:3b (2.0 GB), qwen3.5:4b (4.0 GB)
16 GB of memory, or a graphics card with 8 GB	8 to 12 billion	qwen3.5:9b (7.6 GB), gemma4:12b (7.7 GB)
32 GB of memory, or a graphics card with 16 to 24 GB	26 to 35 billion	gemma4:26b (16 GB), qwen3.5:27b (20 GB)

Without --model, review uses Llama 3.2 1B (807 MB), which runs on almost any computer but makes more mistakes: on seven Parkinson's papers it called a review of eleven studies a randomized controlled trial, which Qwen3.5 2B did not.

3. Download the model and ask:

ollama pull qwen3.5:9b
pubmed-stream review "Does FMT help people with Parkinson's disease?" --id 38686220 41826284 41113141 --model qwen3.5:9b --vault kg_fmt_parkinson

--vault also saves the summaries as a note in the knowledge graph's reviews folder, named after the question, the date and the model, so reviews by several models sit side by side. pubmed-stream init --review-model qwen3.5:9b makes the model the folder's default, so --model can be left out. --answer adds one answer drawn from all the summaries; check it against them, as smaller models credit findings to the wrong paper. If Ollama runs on another computer, such as a lab server, point to it with --ollama-host http://<server>:11434.

Or an AI assistant's model. --model claude, --model codex or --model gemini has Claude Code, Codex or Gemini CLI summarize instead, if it is installed and signed in on your computer (see the quick start). Their models are much larger than a laptop can run, so they make fewer mistakes: on the seven Parkinson's papers above, Claude Code's summaries were all accurate, including the review of eleven studies that Llama 3.2 1B called a trial. In exchange, the abstracts go to Anthropic, OpenAI or Google, and each run uses some of your plan. pubmed-stream runs the assistant in an empty folder with its tools turned off, so it only answers; everything else works as with Ollama: one abstract at a time, without the question, the same checks, --answer and --vault. Add the assistant's own model after a colon, as in --model claude:opus.

pubmed-stream review "Does FMT help people with Parkinson's disease?" --id 38686220 41826284 41113141 --model codex --vault kg_fmt_parkinson

--model	Who summarizes	Where the abstracts go
an Ollama model, such as qwen3.5:9b	a model on your computer	nowhere
claude, claude:opus	Claude Code	Anthropic
codex, codex:<model>	Codex	OpenAI
gemini, gemini:<model>	Gemini CLI	Google

Obsidian. Every knowledge graph has property types for Obsidian and Papers.base, a Bases table of all papers with views for full texts, reviews, retracted papers and your #to-read list. pubmed-stream open kg_<topic> opens it, and with Obsidian's command line switched on, an assistant can query it: obsidian vault=kg_<topic> base:query path=Papers.base view="Reading list".

Tutorial 7 walks through real sessions: an assistant going from 4,863 search results to a cited report, Claude Code and Codex using the MCP tools, and what a 1-billion-parameter local model gets right and wrong. A model that small summarizes well but cannot run the searches itself.

Python API

Everything the commands do is available from pubmed_stream:

Function	Purpose	Returns
search_and_download(query, max_results, fmt="text", ...)	Search PMC and download full text	DownloadStats
get_neighborhood(pmid, max_per_relation=100, ...)	Similar, citing and cited papers with flags	Neighborhood
save_neighborhood(hood, output_dir=None)	Write neighborhood.json	path
download_neighborhood(hood, ...)	Full text of the seed and neighbors	DownloadStats
create_kg(topic, queries, seeds, sources, refresh=False, ...)	Run a study and build its knowledge graph	KGResult
load_study(kg_dir), record_study(kg_dir, topic, queries, seeds)	Read, or add to, a graph's recorded study	dict
build_vault(sources, vault_dir, ...)	Build graph notes from download folders only	VaultStats
topic_vault_dir(topic)	The kg_<topic> folder of a topic	path
search_pmc(query, max_results=10, ...)	Count a search and list its newest matches, without downloading	SearchResult
count_pubmed(query)	The number of PubMed articles matching a search	int
fetch_abstracts(query, max_results=100, ...)	Save PubMed abstracts, including articles without free full text	AbstractStats
list_articles(path)	Compact records of downloaded articles and abstracts, newest first, each once	list of dicts
export_citations(path, fmt="bibtex", ids=None), to_bibtex(records), to_ris(records)	Citations for Zotero, EndNote or Mendeley	(text, count, missing), str
most_cited(path, n=10)	The works the downloaded articles cite most	list of dicts
review(question, path, ids, model, answer=False, ...)	Summaries by a local model through Ollama, or by claude, codex or gemini, with checks	Review
init_folder(folder)	Instructions and MCP settings for Claude Code, Codex and Gemini CLI	list of actions
open_vault(kg_dir)	Open a knowledge graph in Obsidian	(opened, message)
esearch_pmc(term, max_results, api_key, session, rate_limiter)	One PMC search: IDs and total count	(ids, total)
esummary_pmc(pmcids, api_key, session, rate_limiter)	Titles, journals, years and identifiers of PMC articles	list of dicts
efetch_pmc(pmcid, out_dir, fmt, api_key, session, rate_limiter)	Download one article	(ok, status)
extract_metadata_from_pmc_xml(xml), extract_text_from_pmc_xml(xml)	Parse JATS XML you already have	dict, str
create_session(user_agent), RateLimiter(seconds)	Building blocks for the low-level functions

Functions that call NCBI accept api_key, email, user_agent, rate_limit and a session to reuse, and fall back to the environment variables. The low-level functions take an explicit session and rate limiter:

from pathlib import Path
from pubmed_stream import RateLimiter, create_session, efetch_pmc, esearch_pmc

session, limiter = create_session("my-project/1.0"), RateLimiter(0.34)
ids, total = esearch_pmc('"gut microbiota"[Title] AND frailty[Title/Abstract]', 20, None, session, limiter)
for pmc_id in ids:
    ok, status = efetch_pmc(pmc_id, Path("downloads"), "both", None, session, limiter)   # success, unavailable, exists or error

How a paper spread through the literature is measured by a module of its own, from pubmed_stream import impact. impact.citers_of(pmids) asks PubMed for the articles citing each of many papers at once, impact.count_by_year(pmids, first, last) and impact.count_flags(pmids) count them by year, or as reviews, retracted or NIH-funded, without downloading their records, and impact.corrections(pmid) lists a paper's retraction notice, errata and comments. impact.citation_contexts(folder, pmid, doi) finds the sentence in which each downloaded full text cites the paper, impact.semantic_scholar_contexts(pmid) adds the sentences Semantic Scholar has for papers whose full text is not free, and impact.classify(sentences) sorts them: discussing the retraction or doubts about the paper, naming its finding, or citing it in general. impact.annotate(kg_dir, properties) writes the results into the notes of a knowledge graph. Tutorial 8 follows a retracted paper through two hops of citing papers with them.

Troubleshooting

Symptom	Cause and fix
A quoted phrase is ignored on Windows	Windows PowerShell 5.1 drops inner double quotes, and Command Prompt does not understand single quotes. In PowerShell write '\"gut microbiota\"[Title] AND frailty[Title/Abstract]'; in Command Prompt, "\"gut microbiota\"[Title] AND frailty[Title/Abstract]". PowerShell 7.3 and later, macOS and Linux take the examples as written
0 saved but many matches	The files already exist and were skipped, which counts as success; stats.skipped shows them
Many unavailable articles	Their publishers do not allow XML download. Add AND "open access"[filter] to the search
HTTP 429, Too Many Requests	Requests are retried automatically. Add your NCBI API key (pubmed-stream init --ncbi-api-key), or slow down with --rate-limit 0.5
NCBI rejected the API key	The saved key has a typo or was revoked; pubmed-stream continues without it. Create a key in your NCBI account settings and save it again with pubmed-stream init --ncbi-api-key
Timeouts or a flaky network	Use --sequential, or fewer --workers
The assistant lists every pubmed-stream tool twice	The folder was set up with init before the plugin was installed. Run pubmed-stream init again in it: with the plugin installed, it removes the folder's own server
Results unrelated to the question	The search matched text anywhere in the article; see section 2
'PMC…' is a PMC ID	neighborhood and --seed need the PubMed ID, which downloaded files have in metadata.pmid
Few references or citations in a neighborhood	NCBI has citation links only where reference lists were deposited, mostly for PMC articles
Papers without concept links in a graph	PubMed has not indexed them with MeSH yet; rebuild later

Problems with installing Python, Jupyter, Colab or Obsidian are covered in the troubleshooting guide for beginners.

For details, add -v to any command, or enable debug logging for this package only, since the HTTP library's debug output would log request URLs with your API key:

import logging
logging.basicConfig(level=logging.INFO)
logging.getLogger("pubmed_stream").setLevel(logging.DEBUG)

How it works

pubmed-stream is a stack of five small modules. Each one uses only those below it, from the command line down to the HTTP client:

Module	Role
cli.py	The pubmed-stream command: init, search, download, abstracts, list, export, neighborhood, kg, review, open and mcp
corpus.py	Reads downloaded articles and abstracts back as compact records, and finds the works they cite most
export.py	BibTeX and RIS citations
abstracts.py	PubMed search and abstracts, for articles without free full text
mcp_server.py	The MCP server that gives AI assistants pubmed-stream's tools
review.py	Summaries by a local model through Ollama or an AI assistant, with number and citation checks
obsidian.py, assistants.py	Opening graphs in Obsidian; setting up folders for AI assistants
kg.py	Studies: run searches and seeds, record them, refresh them
vault.py	Knowledge-graph notes: merge every source, render notes, rebuild safely
neighborhood.py	Similar, citing and cited papers, with flags and MeSH terms
downloader.py	The E-utilities client, PMC search and download, JATS parsing

The architecture guide explains the whole repository with diagrams, tables and figures: every NCBI request, the request flows, how knowledge graphs are merged and safely rebuilt, the data model, and where to extend the code.

pubmed-stream/
├── pubmed_stream/      the package
├── tests/              140 offline tests; NCBI is faked with canned responses
├── tutorials/          seven notebooks, the AI assistant tutorial and instructions, guides, kg_study.py, keyword lists, tools
├── docs/               the architecture guide and its figures
├── pyproject.toml      package metadata and dependencies
├── studies/            research done with pubmed-stream, with its frozen data; see studies/README.md
├── CHANGELOG.md        changes in each version
├── CITATION.cff        how to cite pubmed-stream
└── LICENSE             MIT

Contributing

Contributions are welcome. Fork the repository, create a feature branch, and open a pull request. Add tests for new behaviour, and make sure pytest and ruff check . pass; GitHub Actions runs both on every pull request. The tests run offline in a few seconds:

pip install -e ".[dev]"
pytest

The documentation site is built from the README, tutorials/, docs/ and the changelog, and published by GitHub Actions on every push to main. Preview it at http://127.0.0.1:8000 with:

pip install -r docs/requirements.txt
mkdocs serve

Ideas on the roadmap: PubMed abstracts for articles outside PMC, a search query builder, progress bars, and export to BibTeX or RIS from the command line.

Releasing to PyPI

Maintainers publish new versions by hand. Replace X.Y.Z with the new version.

1. Bump the version in pubmed_stream/__init__.py (__version__ = "X.Y.Z"), which pyproject.toml reads, and CITATION.cff (version and date-released). In the plugin, plugins/pubmed-stream, bump it in both plugin.json files (version), in the release its server runs (pubmed-stream==X.Y.Z, in .mcp.json and in .claude-plugin/plugin.json) and in the skill's note, twice. Turn the changelog's Unreleased heading into X.Y.Z - YYYY-MM-DD. The tests name any copy left behind.

2. Run the tests: pytest.

3. Build and check the distributions; dist/ is git-ignored:

   pip install --upgrade build twine
   rm -rf dist build *.egg-info
   python -m build
   python -m twine check dist/*
   

4. Upload to PyPI with an API token from https://pypi.org/manage/account/token/, scoped to the pubmed-stream project. The username is literally __token__; a PyPI password will not work. A 403 Forbidden means the token is wrong, expired or not scoped to this project. PyPI never accepts the same version twice, so fix a broken release with a new patch version. InvalidDistribution: Metadata is missing required fields: Name, Version means twine is too old to read the package's metadata (version 2.4): use twine 6.1 or later. In a shared environment such as Anaconda's, upgrading twine also upgrades packages that other tools there depend on, so run it from a separate virtual environment.

   python -m twine upload -u __token__ dist/*
   

   Paste the token when twine asks for it; typed there, it stays out of your shell's history. Upload before you push the release: the plugin on GitHub runs the release its pubmed-stream==X.Y.Z names, so until PyPI has that release, the plugin's server cannot start.

5. Commit, tag and push the release:

   git commit -am "chore: Release X.Y.Z"
   git tag X.Y.Z
   git push origin main X.Y.Z
   

   If the tag already exists on the wrong commit, move it and force-push only that tag, never main: git tag -f X.Y.Z <commit>, then git push --force origin refs/tags/X.Y.Z.

6. Create the GitHub release. In the web interface, open https://github.com/melhzy/pubmed-stream/releases/new?tag=X.Y.Z, set the title to X.Y.Z, paste the version's section of CHANGELOG.md as the description, attach the two files from dist/, and publish. With the GitHub CLI, after a one-time gh auth login: gh release create X.Y.Z --title "X.Y.Z" --notes "<the changelog section>" dist/*.

7. Verify: pip install --upgrade pubmed-stream, then pubmed-stream --version.

FAQ

What is the difference between PubMed and PMC? PubMed indexes the citations and abstracts of over 35 million articles. PubMed Central (PMC) is NCBI's archive of full-text articles, a subset of them. pubmed-stream downloads full text from PMC, and uses PubMed for neighborhoods, MeSH terms and the abstracts of articles whose full text is not free.

Does it comply with NCBI's terms? Yes. It uses the official E-utilities, keeps within NCBI's rate limits, identifies itself with a User-Agent, and sends your contact email when you set one.

Why one JSON file per article? One file keeps an article's metadata, references and text together, in every format, and makes corpora easy to filter, merge and resume.

Can I resume an interrupted download? Yes. Run the same command again; existing files are skipped.

Can I use it for a systematic review? For the first steps: scoping the field, testing and recording searches, and finding key papers. A systematic review also needs other databases, screening by two reviewers and risk-of-bias assessment; see what pubmed-stream can and cannot do.

Why are some important papers missing? Full text comes only from PMC, and searches return the newest matches first. abstracts saves the PubMed abstracts of the rest, and older and subscription-only papers also enter a knowledge graph through key papers' neighborhoods, as stubs, or as links from the papers that cite them.

Do I need Obsidian? Only to explore a knowledge graph visually. The notes are plain Markdown, and tutorial 5 analyses one in Python.

Can I use it commercially? Yes, under the MIT license, while respecting NCBI's usage policies.

Citation, acknowledgments and license

pubmed-stream was developed to support biomedical research at the University of Massachusetts by the Haran Lab, the Bucci Lab and the Microbiology & Microbiome Dynamics AI HUB. Thanks to NCBI for the E-utilities and open access to PubMed Central. If you use it in your research, please cite it. GitHub's Cite this repository button, on the repository's main page, gives the citation in APA and BibTeX from CITATION.cff:

@software{pubmed_stream_2026,
  title = {pubmed-stream: From PubMed Searches to Knowledge Graphs of the Biomedical Literature},
  author = {Haran Lab and Bucci Lab and Microbiology \& Microbiome Dynamics AI HUB},
  organization = {University of Massachusetts},
  year = {2026},
  url = {https://github.com/melhzy/pubmed-stream},
  note = {Built on NCBI E-utilities}
}

pubmed-stream is free software under the MIT License. Changes in each version are listed in the changelog.

Questions and bug reports: open an issue. Package: pypi.org/project/pubmed-stream.