sci-etl-cli

sci-etl runs sci-etl-core extraction pipelines from a single YAML file, with no wiring code. It searches arXiv, asks an LLM which papers are relevant, extracts structured entities from their full text, and upserts them into a CSV. Runs resume where the last one stopped.

$ sci-etl init hot-jupiters
$ sci-etl validate hot-jupiters/config.yaml
$ sci-etl search hot-jupiters/config.yaml --limit 5
$ sci-etl run hot-jupiters/config.yaml --limit 20
$ sci-etl status hot-jupiters/config.yaml

• Installation
• Quick start
• Commands
• Config file
• Plug-ins
• Exit codes
• Development
• License

Installation

Python 3.10 or newer is required.

pip install sci-etl-cli

This installs the sci-etl command along with sci-etl-core[async,llm,pdf] 0.2, click, and rich. pipx install sci-etl-cli keeps it in an environment of its own.

Quick start

1. Create a project. sci-etl init hot-jupiters writes a working example that collects hot Jupiter measurements: config.yaml, a relevance prompt, an extraction prompt, and .env.example.
2. Add your API key. Copy .env.example to .env beside the config and set LLM_API_KEY. The key is read from the environment or that file, never printed, and never written to the config.
3. Check the project. sci-etl validate hot-jupiters/config.yaml checks the config, the prompts, the key, and the installed packages. Add --online to also send one arXiv request and one LLM request.
4. Tune the query. sci-etl search hot-jupiters/config.yaml lists a page of matching papers without calling the LLM.
5. Run it. sci-etl run hot-jupiters/config.yaml exports entities to out/planets.csv and records progress under state/. Run the same command again to continue; add --rescan to pick up papers submitted since.

To use the CLI for another subject, change pipeline.search_query, both prompts, and the export columns. The extraction prompt has to ask for JSON containing result_key, key_column, and every value column; validate checks this.

Commands

Every command accepts --help. Logs and errors go to stderr; search --json, status --json, and parse print their results to stdout, so they can be piped.

Stopping a run. Press Ctrl+C once: records in flight are cancelled and left unmarked, state is saved, and the command exits with 130. The next run retries those records. A second Ctrl+C exits immediately.

Resuming. A run starts at the listing offset saved by the previous one. arXiv lists the newest submissions first, so new papers push older ones to higher offsets; --rescan starts again at offset 0 and skips processed papers by id, which costs listing requests but no LLM calls.

Throttling. When arXiv or the LLM provider answers 429 with a Retry-After header, the request waits as long as it asks, up to 60 seconds, and each retry is logged.

Token usage. When a run finishes, aborts, or is interrupted, run logs how many LLM requests it made and the prompt and completion tokens they used. Set llm.input_cost_per_million and llm.output_cost_per_million to your provider's prices to add an estimated cost, in the currency of those prices:

LLM usage: 42 requests, 1,204,388 prompt and 9,112 completion tokens, estimated cost 0.1861

Config file

A config holds the library's sections (llm, http, pipeline) and the CLI's own (prompts, export, state, logging). Unknown keys are rejected, so a typo fails validation instead of being ignored. Relative paths resolve from the config file's folder, so a project behaves the same whichever directory it is run from.

llm:
  base_url: https://api.openai.com/v1
  model: gpt-4o-mini
  timeout: 120
  api_key_env: LLM_API_KEY

http:
  user_agent: "sci-etl-project/0.1 (mailto:you@example.org)"
  max_retries: 4
  backoff_factor: 5.0
  timeout: 25

pipeline:
  search_query: 'cat:astro-ph.EP AND abs:"hot Jupiter"'
  max_records: 20
  page_size: 100
  search_delay: 3.0
  sleep_between: 5.0
  max_workers: 4

prompts:
  relevance: prompts/relevance.txt
  extraction: prompts/extraction.txt
  result_key: planets

export:
  destination: out/planets.csv
  key_column: planet_name
  value_columns: [orbital_period_days, mass_jupiter, radius_jupiter]
  numeric_clip:
    mass_jupiter: [0.0, 80.0]

state:
  backend: file
  processed_ids: state/processed_ids.txt
  metadata: state/metadata.json

logging:
  file: logs/run.log
  level: INFO

Plug-ins

Domain rules that don't fit in a config file plug in as Python code. Point export.normalizer at a KeyNormalizer and export.validators at one or more RecordValidators, each written as module:attribute:

export:
  destination: out/planets.csv
  key_column: planet_name
  value_columns: [orbital_period_days, mass_jupiter, radius_jupiter]
  normalizer: planet_rules:DesignationNormalizer
  validators:
    - planet_rules:short_period_planets

With planet_rules.py saved beside config.yaml:

from sci_etl_core.processors import DefaultKeyNormalizer, KeyNormalizer, NumericRangeValidator


class DesignationNormalizer(KeyNormalizer):
    def normalize(self, raw_value):
        key = DefaultKeyNormalizer().normalize(raw_value)
        return "wasp" + key[len("superwasp"):] if key.startswith("superwasp") else key


def short_period_planets():
    return NumericRangeValidator({"orbital_period_days": (0.0, 10.0)})

• Where modules are found. The config file's folder is searched first, so a module or package beside config.yaml works without being installed. Installed packages work too.
• Classes or factories. The attribute is called with no arguments, so it can be a class or a function that builds the object. The result must be a KeyNormalizer or a RecordValidator, as the key requires.
• What they do. The normalizer turns each key_column value into the key that decides which CSV rows are the same entity; here SuperWASP-12 b and WASP-12 b share one row. Validators see every extracted entity before it is exported, and an entity any validator rejects is dropped and logged.
• Catching mistakes early. sci-etl validate imports and builds every plug-in, and run does so before any network request, so a broken plug-in exits with code 3 without spending tokens.

Exit codes

Development

git clone https://github.com/xueromll/sci-etl-cli.git
cd sci-etl-cli
python -m venv .venv
source .venv/bin/activate        # Windows: .venv\Scripts\activate
pip install -e ".[dev,lint]"
pytest --cov=sci_etl_cli --cov-report=term-missing
ruff check .
mypy

To work against a local sci-etl-core checkout, install it first with pip install -e "../sci-etl-core[async,llm,pdf]".

The suite runs offline: arXiv is served by an httpx.MockTransport and the LLM by a scripted client, so run is exercised end to end against real CSV and state files. Coverage must stay at 100%.

License

Released under the MIT License. See LICENSE.