bio-scraper 0.1.0

Scrape GEO dataset IDs from ExomiRHub and download datasets from NCBI GEO.

biomedical_scraper

A Python module to extract GEO dataset IDs from ExomiRHub and download the corresponding datasets from NCBI GEO.

The site is a Vue.js SPA with an ag-Grid table — Selenium is required.

---

Requirements

• Python 3.10+
• Google Chrome installed

---

Installation

From PyPI:

pip install bio_scraper

From source:

git clone <repo-url>
cd biomedical_scraper
pip install -e .

---

Usage

As a CLI tool

# Fetch all GEO IDs, save to CSV, download datasets 0–4
bio-scraper --save-metadata --start 0 --end 5

# Download only the first dataset
bio-scraper --start 0 --end 1

# Run with a visible browser window (useful for debugging)
bio-scraper --no-headless --start 0 --end 1

Flag	Default	Description
--start	0	Start index of GEO IDs to download
--end	5	End index (exclusive)
--save-metadata	off	Save all GEO IDs to geo_ids.csv
--headless	True	Run Chrome headlessly

---

As a Python module

Fetch all GEO IDs

from biomedical_scraper import BiomedicalScraper

with BiomedicalScraper(headless=True) as scraper:
    geo_ids = scraper.get_geo_ids()
    print(geo_ids)
# ['GSE144777', 'GSE115572', 'GSE115114', ...]

Get a download link for a specific dataset

from biomedical_scraper import BiomedicalScraper

with BiomedicalScraper() as scraper:
    url = scraper.get_download_link("GSE144777")
    print(url)
# https://ftp.ncbi.nlm.nih.gov/geo/series/GSE144nnn/GSE144777/...

Fetch IDs + download datasets

from biomedical_scraper import BiomedicalScraper
from biomedical_scraper.downloader import GEODownloader
from biomedical_scraper.utils import select
from biomedical_scraper.metadata import save_metadata

with BiomedicalScraper(headless=True) as scraper:
    geo_ids = scraper.get_geo_ids()       # all 191 IDs
    save_metadata(geo_ids)                # writes geo_ids.csv

    subset = select(geo_ids, start=0, end=5)
    downloader = GEODownloader()          # saves to data/raw/geo/

    for geo_id in subset:
        url = scraper.get_download_link(geo_id)
        if url:
            downloader.download_from_url(url, geo_id)

Save metadata only (no download)

from biomedical_scraper import BiomedicalScraper
from biomedical_scraper.metadata import save_metadata

with BiomedicalScraper() as scraper:
    geo_ids = scraper.get_geo_ids()

save_metadata(geo_ids, path="my_geo_ids.csv")

---

Output

File	Description
geo_ids.csv	All GEO IDs scraped from ExomiRHub
data/raw/geo/<ID>.tar	Downloaded dataset archives
debug_screenshot.png	Auto-saved on timeout (debugging only)
debug_page.html	Auto-saved on timeout (debugging only)

---

Project structure

biomedical_scraper/
├── biomedical_scraper/
│   ├── scraper.py      # Selenium scraper (get_geo_ids, get_download_link)
│   ├── downloader.py   # Resumable file downloader
│   ├── cli.py          # bio-scraper entry point
│   ├── config.py       # URLs, timeouts, paths
│   ├── metadata.py     # CSV export
│   └── utils.py        # ID selection helpers
└── setup.py