Skip to main content

reddit-to-sqlite

Save data from Reddit to SQLite. Dogsheep-based.

Inserts records of posts and comments into a SQLite database. Can be run repeatedly safely; will refresh already-saved results (see Reload, below). Creates posts and comments tables, plus an items view with a unified view.

Usage

reddit-to-sqlite r/python
reddit-to-sqlite u/catherinedevlin 
reddit-to-sqlite --help 

By default, writes to a local reddit.db database (change with --db).

Authorizing

reddit-to-sqlite will look for a file of authorization info (location determined by --auth, defaults to ~/.config/reddit-to-sqlite.json) and, if not found, will query the user and then save the info there. You will need a Reddit username and password, and you will need to register your app with Reddit to get a client_id and client_secret. (More instructions)

Limits

Whether used for users or for subreddits, can't guarantee getting all posts or comments, because

  • Reddit's API only supplies the last 1000 items for each API call, and does not support pagination;
  • Comments nested under a single post won't be retrieved if they are deeply nested in long comment chains (see replace_more)

Reload

reddit_to_sql can be run repeatedly for a given user or subreddit, replacing previously saved results each time. However, to save excessive API use, it works backward through time and stops after it reaches the timestamp of the last saved post, plus an overlap period (default 7 days). That way, recent changes (scores, new comments) will be recorded, but older ones won't unless --post_reload is increased. If posts keep getting comments of interest long after they are posted, you can increase --post_reload.

When loading an individual user's comments, by default reddit_to_sql stops just 1 day after reaching the most recent comment that is already recorded in the database. However, if you're interested in comment scores, you may want to impose a longer --comment_reload, since scores may keep changing for longer than a single day after the comment is posted.

Notes

  • author is saved in case-sensitive form, so case-insensitive searching with LIKE may be helpful.

Metadata

Release files for reddit-to-sqlite 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for reddit-to-sqlite 0.1.0
File Size Uploaded Provenance Actions
reddit-to-sqlite-0.1.0.tar.gz 6.5 kB Without provenance Details for reddit-to-sqlite-0.1.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for reddit-to-sqlite 0.1.0
File Interpreter ABI Platform Actions
reddit_to_sqlite-0.1.0-py3-none-any.whl Python 3 none any Details for reddit_to_sqlite-0.1.0-py3-none-any.whl

Total release size: 13.8 kB

Release files / reddit-to-sqlite-0.1.0.tar.gz

Download URL reddit-to-sqlite-0.1.0.tar.gz
Size 6.5 kB
Tags Source
SHA-256 checksum
How to use checksums
7707dd5e41fc1e9e72be968873ce02a9bf3cd983dbf7b364c53eaa0b4ae5894e
BLAKE2b-256 checksum
How to use checksums
99b1d0af3e21c49ef59878558d2b803514296d4973acd3cdb3f8af8bbc3fc43a
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via poetry/1.1.6 CPython/3.9.1 Darwin/18.7.0

Release files / reddit_to_sqlite-0.1.0-py3-none-any.whl

Download URL reddit_to_sqlite-0.1.0-py3-none-any.whl
Size 7.3 kB
Tags Python 3
SHA-256 checksum
How to use checksums
bcf8141212b6b650c94dbb0165d536ac5a01ad8c06b41e6ef97ff13af0b8cfea
BLAKE2b-256 checksum
How to use checksums
1cfc460dbd06447796497ff4ae89e3a35bb5e875569ca793e74439c97bcce082
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via poetry/1.1.6 CPython/3.9.1 Darwin/18.7.0

Security for reddit-to-sqlite 0.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page