sci-etl-cli

sci-etl runs sci-etl-core extraction pipelines from a single YAML file, with no wiring code. It searches arXiv, asks an LLM which papers are relevant, extracts structured entities from their full text, and upserts them into a CSV. Runs resume where the last one stopped.

$ sci-etl init hot-jupiters
$ sci-etl validate hot-jupiters/config.yaml
$ sci-etl search hot-jupiters/config.yaml --limit 5
$ sci-etl run hot-jupiters/config.yaml --limit 20
$ sci-etl status hot-jupiters/config.yaml

• Installation
• Quick start
• Commands
• Config file
• Exit codes
• Development
• License

Installation

Python 3.10 or newer is required.

pip install sci-etl-cli

This installs the sci-etl command along with sci-etl-core[async,llm,pdf] 0.1.2 or newer, click, and rich. pipx install sci-etl-cli keeps it in an environment of its own.

Quick start

1. Create a project. sci-etl init hot-jupiters writes a working example that collects hot Jupiter measurements: config.yaml, a relevance prompt, an extraction prompt, and .env.example.
2. Add your API key. Copy .env.example to .env beside the config and set LLM_API_KEY. The key is read from the environment or that file, never printed, and never written to the config.
3. Check the project. sci-etl validate hot-jupiters/config.yaml checks the config, the prompts, the key, and the installed packages. Add --online to also send one arXiv request and one LLM request.
4. Tune the query. sci-etl search hot-jupiters/config.yaml lists a page of matching papers without calling the LLM.
5. Run it. sci-etl run hot-jupiters/config.yaml exports entities to out/planets.csv and records progress under state/. Run the same command again to continue; add --rescan to pick up papers submitted since.

To use the CLI for another subject, change pipeline.search_query, both prompts, and the export columns. The extraction prompt has to ask for JSON containing result_key, key_column, and every value column; validate checks this.

Commands

Every command accepts --help. Logs and errors go to stderr; search --json, status --json, and parse print their results to stdout, so they can be piped.

Stopping a run. Press Ctrl+C once: records in flight are cancelled and left unmarked, state is saved, and the command exits with 130. The next run retries those records. A second Ctrl+C exits immediately.

Resuming. A run starts at the listing offset saved by the previous one. arXiv lists the newest submissions first, so new papers push older ones to higher offsets; --rescan starts again at offset 0 and skips processed papers by id, which costs listing requests but no LLM calls.

Config file

A config holds the library's sections (llm, http, pipeline) and the CLI's own (prompts, export, state, logging). Unknown keys are rejected, so a typo fails validation instead of being ignored. Relative paths resolve from the config file's folder, so a project behaves the same whichever directory it is run from.

llm:
  base_url: https://api.openai.com/v1
  model: gpt-4o-mini
  timeout: 120
  api_key_env: LLM_API_KEY

http:
  user_agent: "sci-etl-project/0.1 (mailto:you@example.org)"
  max_retries: 4
  backoff_factor: 5.0
  timeout: 25

pipeline:
  search_query: 'cat:astro-ph.EP AND abs:"hot Jupiter"'
  max_records: 20
  page_size: 100
  search_delay: 3.0
  sleep_between: 5.0
  max_workers: 4

prompts:
  relevance: prompts/relevance.txt
  extraction: prompts/extraction.txt
  result_key: planets

export:
  destination: out/planets.csv
  key_column: planet_name
  value_columns: [orbital_period_days, mass_jupiter, radius_jupiter]
  numeric_clip:
    mass_jupiter: [0.0, 80.0]

state:
  backend: file
  processed_ids: state/processed_ids.txt
  metadata: state/metadata.json

logging:
  file: logs/run.log
  level: INFO

Exit codes

Development

git clone https://github.com/xueromll/sci-etl-cli.git
cd sci-etl-cli
python -m venv .venv
source .venv/bin/activate        # Windows: .venv\Scripts\activate
pip install -e ".[dev]"
pytest --cov=sci_etl_cli --cov-report=term-missing

To work against a local sci-etl-core checkout, install it first with pip install -e "../sci-etl-core[async,llm,pdf]".

The suite runs offline: arXiv is served by an httpx.MockTransport and the LLM by a scripted client, so run is exercised end to end against real CSV and state files. Coverage must stay at 100%.

License

Released under the MIT License. See LICENSE.