redditdumps 0.1.0

Lightweight utilities for processing Reddit data dumps in ZST format

redditdumps

Lightweight Python utilities for processing Reddit data dumps in ZST format.

These dumps are commonly found on Academic Torrents (Pushshift archives) and contain newline-delimited JSON compressed with Zstandard.

Installation

uv add redditdumps

Or with pip:

pip install redditdumps

Usage

Read a ZST file into a DataFrame

import redditdumps as rd

# Read entire file
df = rd.read_zst("RC_2024-01.zst")

# Filter by subreddit
df = rd.read_zst("RC_2024-01.zst", subreddit="science")

# Filter by multiple subreddits
df = rd.read_zst("RC_2024-01.zst", subreddit=["science", "askscience"])

# Select specific columns
df = rd.read_zst("RC_2024-01.zst", columns=["author", "body", "score"])

# Combine filters
df = rd.read_zst(
    "RC_2024-01.zst",
    subreddit="python",
    columns=rd.MINIMAL_COMMENT_COLUMNS,
    max_lines=100000,
)

Inspect file schema

# Discover columns in a file
schema = rd.inspect_schema("RC_2024-01.zst", sample_size=1000)
for col, info in schema.items():
    print(f"{col}: {info['type']} ({info['count']} records)")

Built-in column schemas

# Common column sets for convenience
rd.COMMENT_COLUMNS       # All standard comment fields
rd.SUBMISSION_COLUMNS    # All standard submission fields
rd.MINIMAL_COMMENT_COLUMNS   # Lightweight subset for comments
rd.MINIMAL_SUBMISSION_COLUMNS  # Lightweight subset for submissions

File naming conventions

• RC_*.zst - Reddit Comments
• RS_*.zst - Reddit Submissions