sirna-data-grabber

A standalone siRNA knockdown-efficacy dataset: the raw data files, full provenance/license documentation, and a small reusable Python package (sirna_data) for loading it -- and, since pip install sirna-data-grabber alone can't ship most of this non-commercial data, a bundled sirna-data-fetch command that re-fetches it from its original sources. Any project that wants this dataset can depend on this repo (or just the PyPI package) rather than vendoring a copy of the data or the loading code.

Currently: 16,439 siRNA records across 105 genes (load_records() default). Every source is individually toggleable via its own include_* flag -- see data/DATA_SOURCE_LEDGER.md for the per-source breakdown.

Data sources at a glance

"Published" is the year of the paper/database each source comes from, not when it was added here -- see data/DATA_SOURCES.md for full citations, license terms, and how each source's data was verified. "Genes" is how many distinct genes/reporters that source contributes to this dataset; some genes (e.g. APP, MAPT) are covered by more than one source, so the per-source counts don't sum to the 105 total. CMsiRNAdb's count combines its PCSK9 subset and the other-12-genes addition (same underlying paper) -- see the gene-level table below for the split.

Genes in this dataset

All 105 genes currently in load_records()'s default output, the source dataset(s) each came from, how many siRNA records target that gene, and the length of the real mRNA/GenBank transcript its target sites were located in. Computed directly from the fetched data/raw/ files, not hand-maintained -- for genes with more than one distinct transcript accession across records (marked ¹), the length shown is for the one used by the most records.

License

The code in this repo (sirna_data, tests/) is MIT licensed — see LICENSE. Use it, modify it, ship it commercially, whatever you want.

The data in data/raw/ is NOT covered by that license. It's redistributed under each original source's own terms, and most of those sources are non-commercial only (CC BY-NC / CC BY-NC-ND). Loading the data with this permissively-licensed code does not lift those restrictions — you still have to comply with them separately. See NOTICE.md for the per-source summary and data/DATA_SOURCES.md for full terms before using the data itself, especially commercially.

What's here

LICENSE                    MIT license -- covers the code only, not data/raw/
NOTICE.md                  per-source data license summary (see License section above)
data/
  raw/                       fetched CSVs + FASTA transcripts (the actual dataset)
  DATA_SOURCES.md            full provenance + license terms for every source
  DATA_SOURCE_LEDGER.md      audit: what's trainable, what's not, and why
  CMSIRNADB_FULL_RETRIEVAL.md   detail on the CMsiRNAdb full-database retrieval
  DEMETER2_README.txt        upstream release notes for DepMap DEMETER2 (investigated, not included -- see FUNCTIONAL_GENOMICS_SCREENS.md)
  FUNCTIONAL_GENOMICS_SCREENS.md   notes on functional-genomics screen sources considered
  POTENTIAL_DATA_SOURCES.md  landscape of sources investigated
  sirecords_overlap_analysis.md    siRecords overlap/dedup analysis
  data_source_ledger.csv     machine-readable companion to DATA_SOURCE_LEDGER.md
  *.png                      figures referenced by the docs above
src/sirna_data/
  raw_loader.py               load + merge every source into SiRNARecord rows
  ncbi_fetch.py                fetch a gene's RefSeq mRNA transcript by symbol
  sequence_utils.py            DNA/RNA sequence helpers (to_rna, to_dna, transcribe_template_to_mrna)
  splitting.py                 train_test_split / leave_n_genes_out dataset splitters
  evaluation.py                evaluate_predictions + PredictionMetrics/GeneCorrelation
  rank_confidence.py           probability/confidence model for "how many top-K predictions to check"
  rank_confidence_cli.py       `sirna-rank-confidence` entry point ([project.scripts])
  rank_confidence_plot.py      optional matplotlib plotting for rank_confidence (requires [plot] extra)
  rank_confidence_plot_cli.py  `sirna-rank-confidence-plot` entry point ([project.scripts], requires [plot] extra)
  __init__.py                  public API
  fetch/                       sirna-data-fetch CLI + per-source fetchers (see Install below)
    cli.py                       `sirna-data-fetch` entry point ([project.scripts])
    sirna_efficacy.py            siRNAEfficacyDB + NCBI -> sirna_efficacy.csv, mrna_transcripts.fasta
    monopoli.py                  Monopoli et al. 2023 supplementary data -> monopoli_*
    shabalina.py                 Shabalina et al. 2006 supplementary data -> shabalina_*
    cmsirnadb.py                 CMsiRNAdb + NCBI -> cmsirnadb_full_raw.tsv, cmsirnadb*_transcripts.fasta
tests/
  test_raw_loader.py          unit tests for raw_loader.py (fixtures, no real data needed)
  test_ncbi_fetch.py          unit tests for ncbi_fetch.py (mocked HTTP calls)
  test_fetch_cli.py           unit tests for fetch/cli.py
  test_sequence_utils.py      unit tests for sequence_utils.py
  test_splitting.py           unit tests for splitting.py
  test_evaluation.py          unit tests for evaluation.py
  test_rank_confidence.py     unit tests for rank_confidence.py
  test_rank_confidence_cli.py unit tests for rank_confidence_cli.py
  test_rank_confidence_plot.py unit tests for rank_confidence_plot.py (skipped without the [plot] extra)
  test_rank_confidence_plot_cli.py unit tests for rank_confidence_plot_cli.py (skipped without the [plot] extra)
  conftest.py                 shared pytest fixtures

Start with data/DATA_SOURCES.md for what's in the dataset and where it came from; data/DATA_SOURCE_LEDGER.md for the bottom-line audit (6,838 trainable records across 95 genes, 6 sources — 16,439 records / 105 genes if the optional CMsiRNAdb full-database retrieval is also included). Primary source is siRNAEfficacyDB (Zhang et al. 2024, CC BY-NC); see the docs for the rest and their individual license terms before reusing this data outside this project.

Install

sirna-data-grabber is on PyPI, so most users just need:

pip install sirna-data-grabber

That installs the sirna_data package plus the sirna-data-fetch command (no extras needed). Since the PyPI package can't ship most of this non-commercial data, use sirna-data-fetch to reconstruct it from its original sources into a local directory:

sirna-data-fetch --dest ./my_data

Then point sirna_data at that directory. Two equivalent ways to do this -- pass it directly, no env var needed:

from sirna_data import load_records
records = load_records(data_dir="./my_data")

or export it once as SIRNA_DATA_DIR and call load_records() with no arguments:

export SIRNA_DATA_DIR=./my_data

sirna-data-fetch --only sirna_efficacy monopoli fetches a subset instead of all four sources; see sirna-data-fetch --help.

From a git checkout

If you're working from this repo instead (e.g. to browse data/raw/ and the provenance docs alongside the code, or to contribute):

python3 -m venv .venv && source .venv/bin/activate
pip install -e .

This installs sirna_data in editable mode, so it resolves data/raw/ relative to the checkout automatically -- no sirna-data-fetch, SIRNA_DATA_DIR, or data_dir needed if data/raw/ already has the files. If you copy the data/ folder somewhere else, point at it with either data_dir or SIRNA_DATA_DIR as shown above.

Usage

from sirna_data import load_records, fetch_mrna_by_gene

records = load_records()  # reads from data_dir / SIRNA_DATA_DIR / default data/raw/, in that order
print(len(records), "records across", len({r.gene for r in records}), "genes")

r = records[0]
r.guide_seq       # siRNA antisense strand
r.mrna_window      # local mRNA context around the real target site
r.label            # experimental %knockdown / %inhibition
r.source           # provenance, e.g. "siRNAEfficacyDB"

# Chemical modification (most records are standard/unmodified; a minority
# -- currently CMsiRNAdb, Monopoli2023, and Martinelli_sirna_repro -- are
# chemically modified):
r.is_modified              # bool
r.modification_chemistry   # short summary, e.g. "2'-OMe/2'-F/PS-backbone (per-position, CMsiRNAdb)"
r.sense_modifications       # per-position modified-nucleoside name or None; CMsiRNAdb only
r.antisense_modifications   # same, for the guide strand

# Look up any gene's RefSeq transcript live from NCBI:
transcript = fetch_mrna_by_gene("TP53")
transcript.accession, transcript.sequence

load_records() takes include_sirna_efficacy / include_monopoli / include_pdcd1 / include_shabalina / include_martinelli / include_cmsirnadb / include_cmsirnadb_full flags (all default True) to include or exclude any individual source, including the primary siRNAEfficacyDB set -- no source is loaded unconditionally.

data_dir (a Path or str) points every source at a specific directory of fetched files, as a plain function argument -- no SIRNA_DATA_DIR export required. It falls back to SIRNA_DATA_DIR if set, then the package's default relative data/raw/ location, in that order.

Splitting into train/test

from sirna_data import load_records, train_test_split, leave_n_genes_out

records = load_records()

# sklearn-style train_test_split, but grouped by gene by default so no gene
# straddles both splits (see by_gene below for why this matters):
train, test = train_test_split(records, test_size=0.2, random_state=0)

# leave-N-genes-out cross-validation: a generator yielding one (train, test)
# fold per group of N genes, until every gene has been held out exactly once
for train, test in leave_n_genes_out(records, n=5, random_state=0):
    ...  # train + evaluate a model on this fold

train_test_split mirrors sklearn.model_selection.train_test_split's name and parameters (test_size, random_state) -- the one addition is by_gene (default True), which sklearn has no equivalent for. With by_gene=True, every record for a given gene goes entirely into train or entirely into test, so a model can't partly "solve" a test siRNA just by having seen another siRNA against the same gene during training. Pass by_gene=False for a plain per-record random split with no regard for gene.

leave_n_genes_out(records, n, random_state=None) generalizes leave-one- gene-out cross-validation: it shuffles the distinct genes once, partitions them into consecutive groups of n, and yields one (train, test) fold per group -- so every gene appears in exactly one test fold across the full iteration (n=1 reproduces classic leave-one-gene-out CV). If the gene count isn't evenly divisible by n, the last fold holds out fewer than n genes.

Rank confidence: how many top predictions do you need to check?

from sirna_data import min_top_k_for_confidence, probability_true_top_in_predicted_top_k

# Given only a correlation between a model's predicted and true rankings of
# 4561 candidate items, how many of the top-predicted items do you need to
# check to be 95% confident the true best one is among them?
min_top_k_for_confidence(n_items=4561, confidence=0.95, pcc=0.3686)

# Or ask it the other way: given you check the top 50, how confident can
# you be that the true best item is in there?
probability_true_top_in_predicted_top_k(50, 4561, pcc=0.3686)

Both take the correlation as either pcc (Pearson's r, used directly) or spcc (Spearman's rho, converted internally) -- exactly one of the two. top_n (default 1) generalizes the question from "is the single true best item captured" to "is at least one of the true top top_n items captured" -- pass e.g. top_n=10 to ask about catching any of the top 10, which needs a smaller K for the same confidence. See sirna_data.rank_confidence's module docstring for the full model and its caveats -- this is a planning heuristic (generally conservative), not a certified statistical bound.

Also installed: the sirna-rank-confidence CLI -- sirna-rank-confidence --pcc 0.3686 --n-items 4561 --confidence 0.99 0.95 0.9.

Comparing multiple models at once

from sirna_data import min_top_k_for_confidence_multi, probability_curves_for_pccs

pccs = [0.2, 0.4, 0.6]  # one Pearson correlation per model to compare

# {pcc: min top-K needed for 95% confidence}, one entry per model
min_top_k_for_confidence_multi(pccs, n_items=4561, confidence=0.95)

# {pcc: [probability at each K in a default spread of K's]}, one entry per model
probability_curves_for_pccs(pccs, n_items=4561)

Both run the single-model function above once per PCC in the list -- min_top_k_for_confidence_multi for a straight side-by-side "tests needed" comparison, probability_curves_for_pccs for the full probability-vs-K curve each model traces out (this is what the plotting function below draws). Pass k_values to either the fixed set of K's you want the comparison at instead of the default spread.

Plotting probability vs. number of tests

from sirna_data.rank_confidence_plot import plot_probability_vs_num_tests

plot_probability_vs_num_tests(pccs, n_items=4561, save_path="curves.png")

One curve per PCC, x-axis is K (number of top-predicted items checked), y-axis is the probability of capturing at least one true top-top_n item at that K -- lets you see at a glance how the number of tests needed relates to each model's correlation. Requires the optional plot extra (pip install sirna-data-grabber[plot]) for matplotlib -- not installed by the core package, and this function lives in its own sirna_data.rank_confidence_plot module (not sirna_data's top-level import) specifically so nothing else in this package needs matplotlib. Returns the matplotlib.axes.Axes for further customization; pass an existing ax= to draw on it instead of creating a new figure.

Per-point markers, marker size, and line style are all configurable via marker / markersize / linestyle (each forwarded straight to Axes.plot) -- e.g. marker=None for plain lines with no dots, useful once k_values gets dense enough that individual markers just clutter the curve:

# Dots (default):
plot_probability_vs_num_tests(pccs, n_items=4561, save_path="curves.png")

# Plain lines, no per-point markers:
plot_probability_vs_num_tests(pccs, n_items=4561, marker=None, save_path="curves.png")

# Dashed lines with square markers:
plot_probability_vs_num_tests(
    pccs, n_items=4561, marker="s", linestyle="--", save_path="curves.png"
)

Also installed: the sirna-rank-confidence-plot CLI, a thin wrapper around the same function that writes straight to a file --

sirna-rank-confidence-plot --pcc 0.2 0.4 0.6 --n-items 4561 \
    --marker none --save-path curves.png

--marker/--marker-size/--linestyle mirror the Python function's marker/markersize/linestyle (pass --marker none or --linestyle none for no markers / no connecting line, respectively); --labels sets the legend text per --pcc value; --k-max/--num-points/--k-values control which K's get plotted. Run sirna-rank-confidence-plot --help for the full option list.

Using this from another project

Install as a sibling checkout in editable mode:

pip install -e ../sirna-data-grabber

That gives you import sirna_data with no other coupling — this repo only depends on pandas and requests, and knows nothing about any particular downstream model or feature-engineering pipeline.

Tests

pip install -e ".[test]"
pytest

Tests run entirely against small in-memory/tmp-dir fixtures (see tests/conftest.py) and mocked HTTP calls, so they don't touch the real dataset or the network.

Linting and type checking

pip install -e ".[lint]"
ruff check .
mypy

Both run in CI on every pull request (.github/workflows/tests.yml), alongside the test matrix.