praw-cli 0.7.0

Composable Reddit crawler CLI with lazy pipelines, powerful filtering, reproducible datasets, and flexible output formats.

praw-cli

A composable Reddit crawler for the command line. Fetch posts and comments from any source, shape the data through a lazy filter pipeline, and emit it in any format — all without writing a single line of Python.

# stream the top 500 posts from r/MachineLearning as JSONL
praw-cli posts r/MachineLearning --sort top --time year --limit 500 --format jsonl

# search across r/programming, keep only high-signal posts, project to four fields
praw-cli search "New web framework" --sub r/programming \
  --filter "score>=100" --filter "num_comments>=10" \
  --fields id,title,score,url --format csv --output results.csv

# full comment tree of a submission, depth-first, minimum score 5
praw-cli comments https://reddit.com/r/Python/comments/xyz/ \
  --depth 10 --min-score 5 --format jsonl

# fetch a user's recent posts as a terminal table
praw-cli user spez --mode posts --limit 20 --format table

# re-process a saved dataset offline — no API, no credentials needed
praw-cli input posts.jsonl --filter "score>=500" --format csv --output filtered.csv

Key Features & How They Solve Your Problems

Smart Memory & Reliability

• Constant Memory Footprint (Lazy Pipelines): Typical scrapers buffer huge arrays in memory, causing out-of-memory crashes on large crawls. praw-cli processes data lazily item-by-item (Iterator[Record]). Streaming 100,000 posts uses the same constant memory as streaming 10.
• Resumable Extractions (Checkpointing): Long-running scrapes often fail halfway due to network drops or API limits. praw-cli checkpoints progress, letting you --resume interrupted sessions without refetching from scratch.
• Offline Re-processing (input): Apply new filters, change output formats, or extract field subsets from any previously saved .jsonl, .json, or .csv file — without touching the API or needing credentials. Pipe from stdin too.

Zero-Code Data Preparation

• Data Science-Focused DSL: Stop writing custom Python scripts just to filter text. Chain --filter conditions directly in the CLI:
  • NLP Cleaning: Keep long-form content using selftext len>= 500.
  • Temporal Sorting: Restrict dates natively using ISO-8601 strings, like created_utc >= 2024-01-01.
  • Targeted Mining: Target specific topics with keyword groupings (has, has_all) or regular expressions (title ~= \bbot\b).
• Field Projections: Keep output datasets lightweight and clean by extracting only the schema columns you need (e.g., --fields id,title,score,author).

Instant Pandas & R Integrations

• Diverse Output Formats: Stream outputs directly to jsonl (preferred for streaming/Pandas), standard json, csv (for R/Excel), or rich console tables and Markdown reports.
• Multi-Sink Pipeline: Write the raw data to a .jsonl database while simultaneously writing a preview to a .csv summary in a single pass.

Built-in Scientific Rigor

• Crawl Manifests: Every execution automatically generates a .manifest.json detailing exact parameters, versioning, records filtered, and a config fingerprint, preventing configuration drift in research environments.
• Deterministic Sampling: Extract reproducible subsets of huge subreddits using systematic or randomized sampling (e.g., Bernoulli trial at rate = 0.1 with a fixed seed).

Installation

Requires Python 3.12 or later and a Reddit API application (free, read-only access is sufficient for most use cases).

Via PyPI (Recommended)

Install the package directly using pip or pipx:

pip install praw-cli

From Source (Using uv)

If you want to run it locally or contribute to development:

git clone https://github.com/othonhugo/praw-cli
cd praw-cli
uv sync
source .venv/bin/activate

Documentation

Full documentation is available in the docs/ directory.

• Getting Started: Installation and Authentication
• Usage Guide:
  • Commands
  • Pipelines & Filters
  • Formatters & Outputs
  • Scientific Use & Reproducibility
• Development:
  • System Architecture
  • Decision Log (ADR)
  • Contributing Guide