sirna-data-grabber

A standalone siRNA knockdown-efficacy dataset: the raw data files, full provenance/license documentation, and a small reusable Python package (sirna_data) for loading it -- and, since pip install sirna-data-grabber alone can't ship most of this non-commercial data, a bundled sirna-data-fetch command that re-fetches it from its original sources. Any project that wants this dataset can depend on this repo (or just the PyPI package) rather than vendoring a copy of the data or the loading code.

Currently: 18,077 siRNA records across 116 genes (load_records() default). Every source is individually toggleable via its own include_<data-source> flag -- see data/DATA_SOURCES.md for the per-source breakdown and audit.

Data sources at a glance

"Published" is the year of the paper/database each source comes from, not when it was added here -- see data/DATA_SOURCES.md for full citations, license terms, and how each source's data was verified. "Genes" is how many distinct genes/reporters that source contributes to this dataset; some genes (e.g. APP, MAPT) are covered by more than one source, so the per-source counts don't sum to the 116 total. CMsiRNAdb's count combines its PCSK9 subset and the other-12-genes addition (same underlying paper) -- see the gene-level table below for the split. Davis 2025 and Monopoli 2023 are from the same lab and cover the same 4 genes (APP/MAPT/BACE1/SNCA) -- Davis 2025's 966 is already net of deduping against Monopoli 2023 and CMsiRNAdb (full) by exact sequence.

Genes in this dataset

All 116 genes currently in load_records()'s default output, the source dataset(s) each came from, how many siRNA records target that gene, and the length of the real mRNA/GenBank transcript its target sites were located in, and the cell line(s) it was tested in. Computed directly from the fetched data/raw/ files, not hand-maintained -- for genes with more than one distinct transcript accession across records (marked ²), the length shown is for the one used by the most records.

License

The code in this repo (sirna_data, tests/) is MIT licensed — see LICENSE. Use it, modify it, ship it commercially, whatever you want.

The data in data/raw/ is NOT covered by that license. It's redistributed under each original source's own terms, and most of those sources are non-commercial only (CC BY-NC / CC BY-NC-ND). Loading the data with this permissively-licensed code does not lift those restrictions — you still have to comply with them separately. See NOTICE.md for the per-source summary and data/DATA_SOURCES.md for full terms before using the data itself, especially commercially.

The same per-source table is available in code (sirna_data.list_sources()), and load_records(licenses=[...]) loads only the sources carrying the licenses you name -- see Loading only the licenses you can use.

What's here

LICENSE                    MIT license -- covers the code only, not data/raw/
NOTICE.md                  per-source data license summary (see License section above)
data/
  raw/                       fetched CSVs + FASTA transcripts (the actual dataset)
  DATA_SOURCES.md            full provenance, license terms, and trainable-data
                              audit for every source (including CMsiRNAdb full-
                              database retrieval)
  data_source_ledger.csv     machine-readable companion to DATA_SOURCES.md
src/sirna_data/
  raw_loader.py               load + merge every source into SiRNARecord rows
  genes.py                     list_genes / describe_genes -- what genes are available to load
  licenses.py                  machine-readable per-source data-license table (load_records(licenses=...))
  ncbi_fetch.py                fetch a gene's RefSeq mRNA transcript by symbol
  sequence_utils.py            DNA/RNA sequence helpers (to_rna, to_dna, transcribe_template_to_mrna)
  splitting.py                 train_test_split / leave_n_genes_out dataset splitters
  evaluation.py                evaluate_predictions + PredictionMetrics/GeneCorrelation
  rank_confidence.py           probability/confidence model for "how many top-K predictions to check"
  rank_confidence_cli.py       `sirna-rank-confidence` entry point ([project.scripts])
  rank_confidence_plot.py      optional matplotlib plotting for rank_confidence (requires [plot] extra)
  rank_confidence_plot_cli.py  `sirna-rank-confidence-plot` entry point ([project.scripts], requires [plot] extra)
  __init__.py                  public API
  fetch/                       sirna-data-fetch CLI + per-source fetchers (see Install below)
    cli.py                       `sirna-data-fetch` entry point ([project.scripts])
    sirna_efficacy.py            siRNAEfficacyDB + NCBI -> sirna_efficacy.csv, mrna_transcripts.fasta
    monopoli.py                  Monopoli et al. 2023 supplementary data -> monopoli_*
    shabalina.py                 Shabalina et al. 2006 supplementary data -> shabalina_*
    cmsirnadb.py                 CMsiRNAdb + NCBI -> cmsirnadb_full_raw.tsv, cmsirnadb*_transcripts.fasta
tests/
  test_raw_loader.py          unit tests for raw_loader.py (fixtures, no real data needed)
  test_genes.py               unit tests for genes.py
  test_licenses.py            unit tests for licenses.py (incl. registry/loader sync check)
  test_sciabola2013.py        unit tests for the Sciabola 2013 loader
  test_harborth2003.py        unit tests for the Harborth 2003 loader + its supersede rule
  test_ncbi_fetch.py          unit tests for ncbi_fetch.py (mocked HTTP calls)
  test_fetch_cli.py           unit tests for fetch/cli.py
  test_sequence_utils.py      unit tests for sequence_utils.py
  test_splitting.py           unit tests for splitting.py
  test_evaluation.py          unit tests for evaluation.py
  test_rank_confidence.py     unit tests for rank_confidence.py
  test_rank_confidence_cli.py unit tests for rank_confidence_cli.py
  test_rank_confidence_plot.py unit tests for rank_confidence_plot.py (skipped without the [plot] extra)
  test_rank_confidence_plot_cli.py unit tests for rank_confidence_plot_cli.py (skipped without the [plot] extra)
  conftest.py                 shared pytest fixtures

Start with data/DATA_SOURCES.md for what's in the dataset, where it came from, and the bottom-line audit (7,510 trainable records across 107 genes, 7 sources — 18,077 records / 116 genes if the optional CMsiRNAdb full-database retrieval and Davis2025 are also included). The largest single source is siRNAEfficacyDB (Zhang et al. 2024, CC BY-NC); see the docs for the rest and their individual license terms before reusing this data outside this project.

Install

sirna-data-grabber is on PyPI, so most users just need:

pip install sirna-data-grabber

That installs the sirna_data package plus the sirna-data-fetch command (no extras needed). Since the PyPI package can't ship most of this non-commercial data, use sirna-data-fetch to reconstruct it from its original sources into a local directory:

sirna-data-fetch --dest ./my_data

Then point sirna_data at that directory. Two equivalent ways to do this -- pass it directly, no env var needed:

from sirna_data import load_records
records = load_records(data_dir="./my_data")

or export it once as SIRNA_DATA_DIR and call load_records() with no arguments:

export SIRNA_DATA_DIR=./my_data

sirna-data-fetch --only sirna_efficacy monopoli fetches a subset instead of all nine sources; see sirna-data-fetch --help. Every source load_records() reads has a fetcher, so a bare pip install plus one fetch reconstructs the whole dataset quoted at the top of this file -- nothing here depends on having the git checkout. tests/test_fetch_coverage.py enforces that: adding a loadable source without a fetcher fails the suite.

From a git checkout

If you're working from this repo instead (e.g. to browse data/raw/ and the provenance docs alongside the code, or to contribute):

python3 -m venv .venv && source .venv/bin/activate
pip install -e .

This installs sirna_data in editable mode, so it resolves data/raw/ relative to the checkout automatically -- no sirna-data-fetch, SIRNA_DATA_DIR, or data_dir needed if data/raw/ already has the files. If you copy the data/ folder somewhere else, point at it with either data_dir or SIRNA_DATA_DIR as shown above.

Usage

from sirna_data import load_records, fetch_mrna_by_gene

records = load_records()  # reads from data_dir / SIRNA_DATA_DIR / default data/raw/, in that order
print(len(records), "records across", len({r.gene for r in records}), "genes")

r = records[0]
r.guide_seq       # siRNA antisense strand
r.mrna_window      # local mRNA context around the real target site
r.label            # experimental %knockdown / %inhibition
r.source           # provenance, e.g. "siRNAEfficacyDB"

# Chemical modification (most records are standard/unmodified; a minority
# -- currently CMsiRNAdb, Monopoli2023, Martinelli_sirna_reproduction, and
# Davis2025 -- are chemically modified):
r.is_modified              # bool
r.modification_chemistry   # short summary, e.g. "2'-OMe/2'-F/PS-backbone (per-position, CMsiRNAdb)"
r.sense_modifications       # per-position modified-nucleoside name or None; CMsiRNAdb only
r.antisense_modifications   # same, for the guide strand

# Look up any gene's RefSeq transcript live from NCBI:
transcript = fetch_mrna_by_gene("TP53")
transcript.accession, transcript.sequence

load_records() takes include_sirna_efficacy / include_monopoli / include_shabalina / include_martinelli / include_harborth2003 / include_sciabola2013 / include_cmsirnadb / include_cmsirnadb_full / include_davis2025 flags (all default True) to include or exclude any individual source.

data_dir (a Path or str) points every source at a specific directory of fetched files, as a plain function argument -- no SIRNA_DATA_DIR export required. It falls back to SIRNA_DATA_DIR if set, then the package's default relative data/raw/ location, in that order.

Listing the genes available to load

from sirna_data import list_genes, describe_genes

list_genes()          # ['ACP5', 'AGT', 'AKT1', ...] -- all 116, sorted
len(list_genes())     # 116

# same thing with per-gene detail:
for info in describe_genes():
    info.gene         # 'APP'
    info.n_records    # 1244
    info.sources      # ('CMsiRNAdb_full', 'Davis2025', 'Monopoli2023')
    info.licenses     # ('CC BY 4.0', 'CC BY-NC-ND 4.0')  -- mixed: see below
    info.accessions   # ('NM_000484', 'NM_001198823', ...)

Both are computed from the actual fetched data files rather than a hardcoded table, so they can't go stale, and both accept every load_records() argument -- list_genes(licenses=["CC BY 4.0"]), list_genes(include_cmsirnadb_full=False), list_genes(data_dir="./my_data") -- so you can ask what a given subset contains before loading it. Since that means a full load under the hood, pass records you already have to skip it: list_genes(records).

Gene strings are the raw sources' own, not normalized -- "EGFP" and "EGFP " stay two entries, for the reason in the gene-table footnote above.

Loading only the licenses you can use

The data's licenses vary per source and most are non-commercial (see License below). That table is also available in code, so "load only what my project is allowed to use" doesn't mean maintaining your own list of include_* flags:

from sirna_data import load_records, list_licenses, list_sources, license_for_source

list_licenses()
# ['CC BY 2.0', 'CC BY 4.0', 'CC BY-NC', 'CC BY-NC 2.0 UK', 'CC BY-NC 3.0', 'CC BY-NC 4.0', 'CC BY-NC-ND 4.0', 'unresolved']

# only the permissively-licensed sources (Shabalina 2006, Monopoli 2023, Davis 2025):
records = load_records(licenses=["CC BY 4.0", "CC BY 2.0"])
len(records), len({r.gene for r in records})   # 1255, 45

for source in list_sources():
    source.license_id                   # 'CC BY-NC-ND 4.0'
    source.commercial_use               # False  (None == unresolved, NOT "probably fine")
    source.derivatives_redistributable  # False  (the "ND" term)
    source.notes                        # the caveat worth reading

license_for_source(records[0].source).license_id   # per-record provenance -> license

License ids are matched case- and punctuation-insensitively ("cc-by-4.0" works) but are version-specific, since two sources here state two different things ("CC BY-NC" vs "CC BY-NC 4.0"). An id no source carries raises ValueError listing the valid ones, rather than quietly loading nothing. licenses= intersects with the include_* flags, and the two unresolved-license sources are never selected by a CC license -- only by asking for "unresolved" explicitly.

This registry is a machine-readable copy of NOTICE.md's table, not legal advice; read the real terms before relying on it.

Splitting into train/test

from sirna_data import load_records, train_test_split, leave_n_genes_out

records = load_records()

# sklearn-style train_test_split, but grouped by gene by default so no gene
# straddles both splits (see by_gene below for why this matters):
train, test = train_test_split(records, test_size=0.2, random_state=0)

# leave-N-genes-out cross-validation: a generator yielding one (train, test)
# fold per group of N genes, until every gene has been held out exactly once
for train, test in leave_n_genes_out(records, n=5, random_state=0):
    ...  # train + evaluate a model on this fold

train_test_split mirrors sklearn.model_selection.train_test_split's name and parameters (test_size, random_state) -- the one addition is by_gene (default True), which sklearn has no equivalent for. With by_gene=True, every record for a given gene goes entirely into train or entirely into test, so a model can't partly "solve" a test siRNA just by having seen another siRNA against the same gene during training. Pass by_gene=False for a plain per-record random split with no regard for gene.

leave_n_genes_out(records, n, random_state=None) generalizes leave-one- gene-out cross-validation: it shuffles the distinct genes once, partitions them into consecutive groups of n, and yields one (train, test) fold per group -- so every gene appears in exactly one test fold across the full iteration (n=1 reproduces classic leave-one-gene-out CV). If the gene count isn't evenly divisible by n, the last fold holds out fewer than n genes.

Rank confidence: how many top predictions do you need to check?

from sirna_data import min_top_k_for_confidence, probability_true_top_in_predicted_top_k

# Given only a correlation between a model's predicted and true rankings of
# 4561 candidate items, how many of the top-predicted items do you need to
# check to be 95% confident the true best one is among them?
min_top_k_for_confidence(n_items=4561, confidence=0.95, pcc=0.3686)

# Or ask it the other way: given you check the top 50, how confident can
# you be that the true best item is in there?
probability_true_top_in_predicted_top_k(50, 4561, pcc=0.3686)

Both take the correlation as either pcc (Pearson's r, used directly) or spcc (Spearman's rho, converted internally) -- exactly one of the two. top_n (default 1) generalizes the question from "is the single true best item captured" to "is at least one of the true top top_n items captured" -- pass e.g. top_n=10 to ask about catching any of the top 10, which needs a smaller K for the same confidence. See sirna_data.rank_confidence's module docstring for the full model and its caveats -- this is a planning heuristic (generally conservative), not a certified statistical bound.

Also installed: the sirna-rank-confidence CLI -- sirna-rank-confidence --pcc 0.3686 --n-items 4561 --confidence 0.99 0.95 0.9.

Comparing multiple models at once

from sirna_data import min_top_k_for_confidence_multi, probability_curves_for_pccs

pccs = [0.2, 0.4, 0.6]  # one Pearson correlation per model to compare

# {pcc: min top-K needed for 95% confidence}, one entry per model
min_top_k_for_confidence_multi(pccs, n_items=4561, confidence=0.95)

# {pcc: [probability at each K in a default spread of K's]}, one entry per model
probability_curves_for_pccs(pccs, n_items=4561)

Both run the single-model function above once per PCC in the list -- min_top_k_for_confidence_multi for a straight side-by-side "tests needed" comparison, probability_curves_for_pccs for the full probability-vs-K curve each model traces out (this is what the plotting function below draws). Pass k_values to either the fixed set of K's you want the comparison at instead of the default spread.

Plotting probability vs. number of tests

from sirna_data.rank_confidence_plot import plot_probability_vs_num_tests

plot_probability_vs_num_tests(pccs, n_items=4561, save_path="curves.png")

One curve per PCC, x-axis is K (number of top-predicted items checked), y-axis is the probability of capturing at least one true top-top_n item at that K -- lets you see at a glance how the number of tests needed relates to each model's correlation. Requires the optional plot extra (pip install sirna-data-grabber[plot]) for matplotlib -- not installed by the core package, and this function lives in its own sirna_data.rank_confidence_plot module (not sirna_data's top-level import) specifically so nothing else in this package needs matplotlib. Returns the matplotlib.axes.Axes for further customization; pass an existing ax= to draw on it instead of creating a new figure.

Per-point markers, marker size, and line style are all configurable via marker / markersize / linestyle (each forwarded straight to Axes.plot) -- e.g. marker=None for plain lines with no dots, useful once k_values gets dense enough that individual markers just clutter the curve:

# Dots (default):
plot_probability_vs_num_tests(pccs, n_items=4561, save_path="curves.png")

# Plain lines, no per-point markers:
plot_probability_vs_num_tests(pccs, n_items=4561, marker=None, save_path="curves.png")

# Dashed lines with square markers:
plot_probability_vs_num_tests(
    pccs, n_items=4561, marker="s", linestyle="--", save_path="curves.png"
)

Also installed: the sirna-rank-confidence-plot CLI, a thin wrapper around the same function that writes straight to a file --

sirna-rank-confidence-plot --pcc 0.2 0.4 0.6 --n-items 4561 \
    --marker none --save-path curves.png

--marker/--marker-size/--linestyle mirror the Python function's marker/markersize/linestyle (pass --marker none or --linestyle none for no markers / no connecting line, respectively); --labels sets the legend text per --pcc value; --k-max/--num-points/--k-values control which K's get plotted. Run sirna-rank-confidence-plot --help for the full option list.

Using this from another project

Install as a sibling checkout in editable mode:

pip install -e ../sirna-data-grabber

That gives you import sirna_data with no other coupling — this repo only depends on pandas and requests, and knows nothing about any particular downstream model or feature-engineering pipeline.

Tests

pip install -e ".[test]"
pytest

Tests run entirely against small in-memory/tmp-dir fixtures (see tests/conftest.py) and mocked HTTP calls, so they don't touch the real dataset or the network.

Linting and type checking

pip install -e ".[lint]"
ruff check .
mypy

Both run in CI on every pull request (.github/workflows/tests.yml), alongside the test matrix.