GeneResearcher

GeneResearcher searches biomedical literature for gene/function/background relationships, retrieves article text, filters evidence, synthesizes results, and writes a Markdown report.

Choose how to use it

Use case	Install	Run
Use a built package	Install the wheel, then run python -m generesearcher install-spacy-model.	Import ResearchConfig and run_analysis in your Python code.
Run or modify the source	From the project source directory, run python -m pip install -e ., then python -m generesearcher install-spacy-model.	Create a study YAML file and run python -m generesearcher run --config study.yaml.

Both routes use the same ResearchConfig and pipeline. Users provide their own .env credentials. The SciSpaCy language model is a separate download, installed with install-spacy-model after the package dependencies are installed. Install the published package from PyPI with python -m pip install generesearcher; add the dashscope extra when using DashScope embeddings.

How a study is organized

• One project_folder represents one research background.
• A project can contain multiple function analyses. Each run analyzes one function_name for one or more genes and writes to its own analysis_<function> directory.
• Re-running the same function clears that function directory's Milvus database and generated exports before starting. The Python API emits a warning if a database already exists; the pipeline also logs the directory it clears.
• CSV and TXT files are display exports. Pipeline recovery and later steps use the database as their source of truth.

The six pipeline stages run in this order:

1. Retrieve articles and their available text.
2. Embed article text and recall relevant sentences.
3. Filter recalled evidence for relevance.
4. Synthesize per-article and cross-article results.
5. Build gene-level and study-level reporting results.
6. Generate the final Markdown report.

Install and run from source

Use Python 3.10–3.12 in an isolated environment. SciSpaCy 0.6.x requires NumPy 1.x, while many existing data-science environments now use NumPy 2.x. Installing GeneResearcher into a shared environment can therefore downgrade NumPy and conflict with its other packages. A fresh virtual environment keeps those dependencies separate. SciSpaCy 0.6.x does not support Python 3.13. The clean installation has been verified on macOS arm64 with Python 3.11. From this directory:

python -m venv .venv
# macOS/Linux
source .venv/bin/activate
# Windows PowerShell: .venv\Scripts\Activate.ps1
python -m pip install -e .
python -m generesearcher install-spacy-model
cp .env.example .env

In Jupyter or Colab, installing packages into a running kernel can leave an older NumPy binary loaded in memory even after pip replaces the files. Restart the kernel after such an installation. The supplied Colab notebook uses a separate virtual environment and subprocess so it does not alter the notebook kernel's packages.

Edit .env with the required service values. Do not commit it. The default .env is read from the current working directory; --env-file can point to another file.

Step 2 uses a SciSpaCy model to split article text into sentences. The default is en_core_sci_lg; you may instead select en_core_sci_sm, en_core_sci_md, or en_core_sci_scibert. The four choices are SciSpaCy's general biomedical pipelines; GeneResearcher does not use the separate NER-only models. After installing GeneResearcher, download and validate the default model with:

python -m generesearcher install-spacy-model

To use another model, install it and set the same name in study.yaml under scispacy_model:

python -m generesearcher install-spacy-model --model en_core_sci_sm

scispacy_model: en_core_sci_sm

The installer downloads the selected model archive from the official SciSpaCy model list. It does not retain the archive in pip's cache. The default en_core_sci_lg model uses about 620 MB after installation; choose en_core_sci_sm if you need a smaller model. GeneResearcher checks required dependencies and loads the selected model before it starts a run or clears an existing analysis database; a missing or incompatible model produces an actionable error. Model names are also saved in params.json with the analysis parameters.

The default install supports the configured Gitee-compatible embedding endpoint. DashScope and local Transformers are optional extras. Install the matching extra with the package:

# Source checkout
python -m pip install -e '.[dashscope]'

# Or a built wheel
python -m pip install 'path/to/generesearcher-0.1.3-py3-none-any.whl[dashscope]'

For a local Transformers embedding model, use .[local-embedding] instead and set EMBEDDING_PROVIDER=local plus EMBEDDING_MODEL_NAME in .env. PyTorch can be a large download, so install that extra only when using local embeddings.

Copy and edit the example study file, then run:

cp study.example.yaml study.yaml
python -m generesearcher run --config study.yaml

You can specify an environment file and see detailed logs with:

python -m generesearcher run --config study.yaml --env-file .env --log-level DEBUG
python -m generesearcher run --config study.yaml --quiet

The log levels follow one rule across the package:

• DEBUG: cache decisions, retry attempts, and low-level diagnostics.
• INFO: stage starts and completions, durations, paths, and record counts.
• WARNING: recoverable omissions, fallbacks, and partial results.
• ERROR: failures that stop a stage or the full run.

The default is INFO. Use --log-level DEBUG (or --verbose) for detail, --log-level WARNING to reduce routine progress messages, --quiet to show errors only, or --log-level OFF to disable logs. Fatal run failures include a traceback at the Python API boundary. Progress bars appear at INFO or DEBUG only when output is attached to an interactive terminal; captured output such as Colab notebooks stays on concise log lines. Application logs do not include prompts or article text. The Python warning about clearing an existing analysis database remains visible even when logs are disabled.

Relative project_folder paths in YAML are resolved from the YAML file's directory. The CLI looks for .env beside the YAML file, then falls back to .env in the current working directory.

Use as an installed Python package

To create an installable wheel from this checkout:

python -m pip install build
python -m build
python -m pip install dist/generesearcher-0.1.3-py3-none-any.whl
python -m generesearcher install-spacy-model

Users can install the current release directly from PyPI:

Create and activate a fresh environment using the commands above before running these install commands; this keeps GeneResearcher's NumPy/SciSpaCy stack separate from other projects.

python -m pip install 'generesearcher==0.1.3'
# For DashScope embeddings:
python -m pip install 'generesearcher[dashscope]==0.1.3'

People working from the source checkout can use pip install . or pip install -e ..

Synchronous Python:

import logging
import os
from generesearcher import ResearchConfig, run_analysis

# Python API users control package log output with Python's logging settings.
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(name)s: %(message)s")
# To suppress GeneResearcher logs in this process, use:
# logging.getLogger("generesearcher").setLevel(logging.CRITICAL + 1)

# Service credentials can be set in this process. Values already present in
# the process environment take precedence over values loaded from .env.
os.environ["ENTREZ_EMAIL"] = "your-email@example.org"
os.environ["EMBEDDING_PROVIDER"] = "dashscope"
os.environ["EMBEDDING_API_KEY"] = "your-embedding-api-key"
os.environ["LLM_MODEL_NAME"] = "your-model-name"
os.environ["LLM_API_KEY"] = "your-llm-api-key"
os.environ["LLM_BASE_URL"] = "https://your-llm-endpoint/v1"

config = ResearchConfig(
    genes=["EP300", "CREBBP"],
    function_name="histone acetylation",
    research_background_template=(
        "What is the role of {gene} in {function_name}, especially in cancer development?"
    ),
    project_folder="./results/example_study",
    keywords=["cancer", "tumor"],
    gene_aliases={"EP300": ["p300"], "CREBBP": ["CBP"]},
    scispacy_model="en_core_sci_lg",
)
result = run_analysis(config)
print(result.report_path)

The values above are examples; do not commit real credentials in a Python script. You can instead set the variables in the shell, a deployment secret store, or a local .env file. GeneResearcher reads process environment variables directly and does not overwrite them when it loads .env. The .env file is searched in the current working directory unless you pass env_file explicitly.

For an already-running async application, use await run_analysis_async(config, env_file=".env").

Save and restore an analysis

The usual workflow needs no explicit save or load call. run_analysis() returns an AnalysisResult in memory. If you want to keep a separate copy or restore a previous result later, use the optional helpers:

from generesearcher import load_analysis, save_analysis

# Save the completed function analysis to a new directory.
saved_directory = save_analysis(result, "./backups/ep300_histone_acetylation")

# Restore the parameters and result tables into memory later.
restored_result = load_analysis(saved_directory)
print(restored_result.analysis_data.raw_papers_df.head())
print(restored_result.report_path)

save_analysis() copies the function's complete analysis directory, including its Milvus Lite database, parameters, and generated files. It does not serialize through CSV. The destination directory must not already exist. load_analysis() restores from the database and does not call literature, embedding, or language-model services. It returns the saved result for inspection and downstream Python use; it does not rerun the pipeline. Keep the saved directory separate from the active project_folder, because a new full run clears the active function analysis before starting.

ResearchConfig, params.json, and Milvus use the same canonical field names. PMID is the exclusion identifier; PMCID is used only as the retrieval/full-text identifier and as the rare fallback when an article has no PMID.

Configuration fields

The YAML file and ResearchConfig accept the same fields:

Field	Required	Meaning
genes	Yes	One or more gene symbols
function_name	Yes	The function or pathway analyzed in this run
research_background_template	Yes	Background text; {gene} and {function_name} are replaced during retrieval and analysis
project_folder	No	Root folder shared by runs for the same research background
keywords	No	Literature search terms
function_query	No	Optional search wording for the function
exclude_pmids	No	PMID values to exclude
publication_date_before	No	Inclusive search cutoff in the format supported by the search code
gene_aliases	No	Additional search aliases keyed by gene symbol
max_papers_per_gene	No	Maximum search results per gene; defaults to 10
scispacy_model	No	SciSpaCy sentence-processing model; defaults to en_core_sci_lg

Credentials and provider settings belong in environment variables or .env, not in the study YAML. Existing process environment variables take precedence over .env values. Use EMBEDDING_API_KEY for the embedding provider key, including DashScope.

Naming and persisted data

Public Python entry points, configuration fields, persisted JSON keys, and Milvus fields use descriptive snake_case names. The project uses the canonical first-version format and does not translate alternate spellings. Research background templates use {gene} and {function_name} placeholders.

Only the generesearcher package is included in built distributions. Notebook checkpoints, bytecode caches, and historical duplicate scripts at the repository root are not runtime package modules.

License

GeneResearcher is distributed under the BSD-2-Clause License. See LICENSE for the full terms.