Skip to main content

Python convenience function for opening compressed URLs and files.

Project description

Python convenience function canopener(filename, mode='r') for opening files.

Local files behave identically to open():

>>> canopener('local_file.txt')

URLs can also be passed as the filename and opened for reading. urllib2.urlopen() is used under the covers, so it has equivalent support:

>>> canopener('http://remote/file.txt')

S3 URLs can also be read if the boto module is installed and the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables are set. The file is downloaded to a temporary local file on open:

>>> os.environ['AWS_ACCESS_KEY_ID'] = 'key'
>>> os.environ['AWS_SECRET_ACCESS_KEY'] = 'secret'
>>> canopener('s3://bucket/file.txt')

Any paths with “.gz” or “.bz2” extensions are transparently decompressed:

>>> canopener('local_file.txt.gz')
>>> canopener('local_file.txt.bz2')
>>> canopener('http://remote/file.txt.gz')
>>> canopener('s3://bucket/file.txt.gz')

There’s also transparent compression when writing to local files:

>>> canopener('local_file.txt.gz', 'w')

Project details


Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page