Skip to main content

redditdumps

Lightweight Python utilities for processing Reddit data dumps in ZST format.

These dumps are commonly found on Academic Torrents (Pushshift archives) and contain newline-delimited JSON compressed with Zstandard.

Installation

uv add redditdumps

Or with pip:

pip install redditdumps

Usage

Read a ZST file into a DataFrame

import redditdumps as rd

# Read entire file
df = rd.read_zst("RC_2024-01.zst")

# Filter by subreddit
df = rd.read_zst("RC_2024-01.zst", subreddit="science")

# Filter by multiple subreddits
df = rd.read_zst("RC_2024-01.zst", subreddit=["science", "askscience"])

# Select specific columns
df = rd.read_zst("RC_2024-01.zst", columns=["author", "body", "score"])

# Combine filters
df = rd.read_zst(
    "RC_2024-01.zst",
    subreddit="python",
    columns=rd.MINIMAL_COMMENT_COLUMNS,
    max_lines=100000,
)

Stream records with bounded memory

Use iter_zst when you want one decoded record at a time:

for record in rd.iter_zst("RC_2024-01.zst", subreddit="science"):
    process(record)

Use read_zst_batches to receive bounded pandas DataFrames. Exact filters and the optional predicate run before records enter a batch.

start_utc = 1704067200
end_utc = 1706745600
stats = rd.ReadStats()

for batch in rd.read_zst_batches(
    "RC_2024-01.zst",
    subreddit="science",
    record_filter=lambda record: (
        start_utc <= int(record["created_utc"]) < end_utc
    ),
    batch_size=100_000,
    stats=stats,
):
    process(batch)

print(stats)

read_zst remains available when the complete result fits in memory.

Inspect file schema

# Discover columns in a file
schema = rd.inspect_schema("RC_2024-01.zst", sample_size=1000)
for col, info in schema.items():
    print(f"{col}: {info['type']} ({info['count']} records)")

Built-in column schemas

# Common column sets for convenience
rd.COMMENT_COLUMNS       # All standard comment fields
rd.SUBMISSION_COLUMNS    # All standard submission fields
rd.MINIMAL_COMMENT_COLUMNS   # Lightweight subset for comments
rd.MINIMAL_SUBMISSION_COLUMNS  # Lightweight subset for submissions

File naming conventions

  • RC_*.zst - Reddit Comments
  • RS_*.zst - Reddit Submissions

Metadata

Release files for redditdumps 0.2.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for redditdumps 0.2.0
File Size Uploaded Provenance Actions
redditdumps-0.2.0.tar.gz 11.2 kB Without provenance Details for redditdumps-0.2.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for redditdumps 0.2.0
File Interpreter ABI Platform Actions
redditdumps-0.2.0-py3-none-any.whl Python 3 none any Details for redditdumps-0.2.0-py3-none-any.whl

Total release size: 21.8 kB

Release files / redditdumps-0.2.0.tar.gz

Download URL redditdumps-0.2.0.tar.gz
Size 11.2 kB
Tags Source
SHA-256 checksum
How to use checksums
4e31d0b0424a18c57e32b4531f39fc1fcf19d8430acd177ad3ee4c7b0062216c
BLAKE2b-256 checksum
How to use checksums
ac20c09713e9a18048a2862c9ee337e146900defa0713eea3265b55200e890f4
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.11.23 {"installer":{"name":"uv","version":"0.11.23","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}

Release files / redditdumps-0.2.0-py3-none-any.whl

Download URL redditdumps-0.2.0-py3-none-any.whl
Size 10.6 kB
Tags Python 3
SHA-256 checksum
How to use checksums
fa6f69a6f4a4039dc15635d3b3f8ffac53bec704b52d3f3cfd86cecb0a11e4da
BLAKE2b-256 checksum
How to use checksums
82d8ca34685ffb7273ab78d663d89e2647f46392ae8f6295dd54bd0bb49d3dd9
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.11.23 {"installer":{"name":"uv","version":"0.11.23","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}

Security for redditdumps 0.2.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.2.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page