Skip to main content

Python library for opening URIs as streaming file-like objects

Project description


Python library for opening URIs as streaming file-like objects

This library provides the function with an interface very similar to that of the built in Python open function. The difference is that while Python's open operates only on local filesystem paths, omnio accepts URIs as well.

It currently supports file, http, and s3 URIs, though it may be expanded to support additional schemes in the future.

In addition, it supports compression and decompression of streams with gzip or bz2.


  • API is a superset of Python 3's built-in open() function
  • Based on Python 3 io module
  • Gzip and bzip2 support both for reading and writing
  • Local file support using standard library
  • HTTP support using requests
  • S3 support using boto3


Read a local binary file:

>>> with'example.bin', 'r') as f:
>>>     data =

Read a local text file:

>>> with'example.txt', 'rt') as f:
>>>     text =

Read a text file from the web:

>>> with'', 'rt') as f:
>>>     text =

Read a gzipped text file from the web, uncompressing on the fly:

>>> with'', 'rtz') as f:
>>>     text =

Read a text file from s3:

>>> with's3://my-bucket/my-key', 'rt') as f:
>>>     text =

Write a megabyte of random data to s3, compressing with bzip2:

>>> import os
>>> data = os.urandom(1024**2)
>>> with's3://my-bucket/my-key', 'wbj') as f:
>>>     f.write(data)

Read a bzip2 compressed csv file into a list of data rows:

>>> import csv
>>> with'data/example_data.csv.bz2', 'rtj') as f:
>>>     reader = csv.reader(f)
>>>     data = list(reader)


The top-level API provides of a single function intended to be referenced as This function API is designed to mimic Python 3's built-in open() as much as possible, and should normally be able to be used as a drop-in replacement.

Signature:, mode='rb', encoding=None, errors=None, newline=None)


A file-like object whose type depends on the scheme and the mode.


  • uri -- URI or local path. Supported URI schemes are file, http, and s3. Local paths may be specified by as ordinary path strings.

  • mode -- Optional string that specifies the mode in which the file is opened. It defaults to 'rb' which means open for reading in binary mode. Supported modes are documented below.


Character Meaning
'r' open for reading (default)
'w' open for writing, truncating the file first
'b' binary mode (default)
't' text mode
'z' use gzip compression
'j' use bzip2 compression

These characters can be composed into valid modes. Binary mode is always the default, so some mode specifications are equivalent. The complete list of supported modes are as follows:

Mode Meaning
'r', 'rb' read bytes
'rt' read and decode to unicode
'rz', 'rbz' read, decompressing gzip to bytes
'rj', 'rbj' read, decompressing bzip2 to bytes
'rtz' read, decompress gzip to bytes, decode to unicode
'rtj' read, decompress bzip2 to bytes, decode to unicode
'w', 'wb' write bytes
'wt' write unicode, encoding to bytes
'wz', 'wbz' write bytes, compress with gzip
'wj', 'wbj' write bytes, compress with bzip2
'wtz' write unicode, encode to bytes, compress with gzip
'wtj' write unicode, encode to bytes, compress with bzip2

Some keyword arguments may be applicable to only certain modes. For example, encoding only applies to 't' (text) modes.

Some schemes may not support some modes. For example, the http scheme currently does not support any 'w' (write) modes.


The glob submodule is intended to be a drop-in replacement for the standard library glob module. It provides the functions escape(), glob(), and iglob() with the signature of each matching that of the original.

The difference is that glob() and iglob() optionally accept URIs. Currently, local paths, file://, and s3:// URIs are supported.


The function accepts an optional config parameter. This allows for specifying scheme-specific configuration.

The default_config() method returns a config dictionary with all supported keys defined along with their default values.

>>> import omnio
>>> omnio.default_config()
{'file': {}, 'http': {'content_iter_chunk_size': 512}, 's3': {'upload_part_size': 5242880, 'boto_client_config_args': [], 'boto_client_config_kwargs': {}}}

To specify alternate values for these parameters, instantiate a default config, update the dict with the desired values and pass it as a keyword arg to the function.

>>> config = omnio.default_config()
>>> config["s3"]["boto_client_config_kwargs"] = {"read_timeout": 600}
>>> with"s3://my-bucket/my-key", "rt", config=config) as fd:

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

omnio-1.3.0.tar.gz (10.0 kB view hashes)

Uploaded source

Built Distribution

omnio-1.3.0-py3-none-any.whl (13.2 kB view hashes)

Uploaded py3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page