Fetch public posts from Pastebin.com for archival
Project description
Pastebin Archiver
What is this?
This app retrieves new posts made on Pastebin.com and stores them offline in a database. You can see the latest public posts it will retrieve here.
Why?
Some of the pastes posted to Pastebin contain interesting or sensitive data, and sometimes pastes are deleted by their poster or Pastebin staff. Running an instance of this archiver lets you retrieve deleted pastes and build a large dataset to run queries against.
Pastebin API info
Important: This archiver uses the Pastebin Scraping API which requires a whitelisted IP address and a Lifetime Pro account to use. More info here.
Installation
Install from PyPI (recommended)
- Ensure you have Python 3.7+ installed.
- Run
pip install pastebin_archiver
- Done! Jump down to the Usage section to get started.
Install from source
- Ensure you have Python 3.7+ and pipenv installed
$ python --version Python 3.7.4 $ pipenv --version pipenv, version 2018.11.26
- Clone the git repository
git clone https://gitlab.com/jonpavelich/pastebin-archiver.git
- Install the dependencies
$ cd pastebin-archiver $ pipenv install --dev
- Install the local package
$ pipenv shell $ pip install -e .
- Run it
$ python -m pastebin_archiver
Usage
Command line usage
If you installed the package using pip, then you can simply run pastebin-archiver
:
$ pastebin-archiver # Run with default settings
$ pastebin-archiver --help # Print available command line options
Python usage
If you'd prefer to use the package in your own code, you can do so like this:
# Import the package
from pastebin_archiver import PastebinArchiver
# (Optional) configure logging
logging.basicConfig(level=logging.DEBUG)
# Run the application
app = PastebinArchiver()
app.main()
Important: app.main()
does not return, it runs forever looking for new pastes to fetch.
Configuration
Database
By default, the fetched data will be saved to a SQLite database file in your working directory called pastebin.db
. You can change this behaviour by passing in a database connection string using the --database
option. For example:
$ pastebin-archiver --database 'postgresql://user:pass@localhost/mydatabase'
For detailed info on connection strings, see the SQLAlchemy documentation.
Contributing
If you find any bugs or have any suggestions to improve the project, please open an issue on GitLab. I'm not accepting merge requests for the project at this time, but you're always welcome to fork the project and work on it yourself.
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for pastebin_archiver-0.1.0-py2.py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 26d36e863b5f317da26af33f4cced47090a95a32243d5b33d538325b9724c4f3 |
|
MD5 | faf5ee396103cb42afc2953c04b15181 |
|
BLAKE2b-256 | fe02bec3262c19b0e06d51d4b48df84b63e4a0df80504b8752ad373104adda99 |