omnio

Python library for opening URIs as streaming file-like objects

This library provides the omnio.open() function with an interface very similar to that of the built in Python open function. The difference is that while Python's open operates only on local filesystem paths, omnio accepts URIs as well.

It currently supports file, http, and s3 URIs, though it may be expanded to support additional schemes in the future.

In addition, it supports compression and decompression of streams with gzip or bz2.

Features

• API is a superset of Python 3's built-in open() function
• Based on Python 3 io module
• Gzip and bzip2 support both for reading and writing
• Local file support using standard library
• HTTP support using requests
• S3 support using boto3

Examples

Read a local binary file:

>>> with omnio.open('example.bin', 'r') as f:
>>>     data = f.read()

Read a local text file:

>>> with omnio.open('example.txt', 'rt') as f:
>>>     text = f.read()

Read a text file from the web:

>>> with omnio.open('https://example.com/example.txt', 'rt') as f:
>>>     text = f.read()

Read a gzipped text file from the web, uncompressing on the fly:

>>> with omnio.open('https://example.com/example.txt.gz', 'rtz') as f:
>>>     text = f.read()

Read a text file from s3:

>>> with omnio.open('s3://my-bucket/my-key', 'rt') as f:
>>>     text = f.read()

Write a megabyte of random data to s3, compressing with bzip2:

>>> import os
>>> data = os.urandom(1024**2)
>>> with omnio.open('s3://my-bucket/my-key', 'wbj') as f:
>>>     f.write(data)

Read a bzip2 compressed csv file into a list of data rows:

>>> import csv
>>> with omnio.open('data/example_data.csv.bz2', 'rtj') as f:
>>>     reader = csv.reader(f)
>>>     data = list(reader)

API

omnio.open()

The top-level API provides of a single function intended to be referenced as omnio.open(). This function API is designed to mimic Python 3's built-in open() as much as possible, and should normally be able to be used as a drop-in replacement.

Signature:

omnio.open(uri, mode='rb', encoding=None, errors=None, newline=None)

Returns:

A file-like object whose type depends on the scheme and the mode.

Parameters:

• uri -- URI or local path. Supported URI schemes are file, http, and s3. Local paths may be specified by as ordinary path strings.

• mode -- Optional string that specifies the mode in which the file is opened. It defaults to 'rb' which means open for reading in binary mode. Supported modes are documented below.

Modes:

These characters can be composed into valid modes. Binary mode is always the default, so some mode specifications are equivalent. The complete list of supported modes are as follows:

Some keyword arguments may be applicable to only certain modes. For example, encoding only applies to 't' (text) modes.

Some schemes may not support some modes. For example, the http scheme currently does not support any 'w' (write) modes.

omnio.glob

The glob submodule is intended to be a drop-in replacement for the standard library glob module. It provides the functions escape(), glob(), and iglob() with the signature of each matching that of the original.

The difference is that glob() and iglob() optionally accept URIs. Currently, local paths, file://, and s3:// URIs are supported.

Configuration

The omnio.open function accepts an optional config parameter. This allows for specifying scheme-specific configuration.

The default_config() method returns a config dictionary with all supported keys defined along with their default values.

>>> import omnio
>>> omnio.default_config()
{'file': {}, 'http': {'content_iter_chunk_size': 512}, 's3': {'upload_part_size': 5242880, 'boto_client_config_args': [], 'boto_client_config_kwargs': {}}}

To specify alternate values for these parameters, instantiate a default config, update the dict with the desired values and pass it as a keyword arg to the omnio.open() function.

>>> config = omnio.default_config()
>>> config["s3"]["boto_client_config_kwargs"] = {"read_timeout": 600}
>>> with omnio.open("s3://my-bucket/my-key", "rt", config=config) as fd:
    fd.read()