Skip to main content

log archiver for pypi cdn logs

Project description

# PyPI CDN Log Archiver

  1. Get raw logs from S3 like object store, written by the cdn

  2. Archive and compress them

  3. Upload archive back to S3 like object store, in a sane structure

  4. Remove raw logs that were archived

# Usage

You’ll need to export some env variables to get started:

$ export ACCESS_KEY=<your_access_key> $ export SECRET_KEY=<your_access_key> $ export PYPI_LOG_BUCKET=<bucket_name> $ export S3_HOST=<s3_like_host> (e.g. s3.amazonaws.com, objects.dreamhost.com)

Optionally you can set the date to archive like so:

$ export PYPI_LOG_DATE=2014-02-11

You can enable some debug output with:

$ export DEBUG=10

If you don’t set PYPI_LOG_DATE, the script will default to looking for yesterdays date.

Now you should be able to run it:

$ python pypi-cdn-log-archiver.py 112373 bytes written to testpypi-cdn-logs/archive/2014/2

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

pypi-cdn-log-archiver-0.1.5.tar.gz (3.0 kB view details)

Uploaded Source

Built Distribution

pypi_cdn_log_archiver-0.1.5-py2.7.egg (3.1 kB view details)

Uploaded Egg

File details

Details for the file pypi-cdn-log-archiver-0.1.5.tar.gz.

File metadata

File hashes

Hashes for pypi-cdn-log-archiver-0.1.5.tar.gz
Algorithm Hash digest
SHA256 bdb2eedd1e8ee5f76b7ef708e5942f0572a3e0f6211917697d60873ca9bb7455
MD5 a53c2ccf1dbef568412692dddd0510c1
BLAKE2b-256 5416f080d195a6ab5541775f7ee772699f42c5d7d7dd531627b702fed7c20f9b

See more details on using hashes here.

File details

Details for the file pypi_cdn_log_archiver-0.1.5-py2.7.egg.

File metadata

File hashes

Hashes for pypi_cdn_log_archiver-0.1.5-py2.7.egg
Algorithm Hash digest
SHA256 eaae5496b34393a5cda68488bbdb895a2ac1062e99c9d03b46a1e196d9468eb8
MD5 45379cf87de0e27bd4ff9e8538dc3e29
BLAKE2b-256 05f85c0fb05f76cbd931b7a01603429597afed2dc71eb032fb9e16548d0f146f

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page