Skip to main content

Consistent iterable API for reading and writing to external datasets

Project description is a Pythonic library for consuming (input), iterating over, and generating (output) external data resources in various formats. facilitates interoperability between the wq framework and other systems and formats. is designed to be customized, with a base class and modular mixin classes that handle loading, parsing, and mapping external data to a convenient API.

Latest PyPI Release Release Notes Documentation License GitHub Stars GitHub Forks GitHub Issues

Travis Build Status Python Support

Somewhat coincidentally, is also the URL for the website describing the wq framework as a whole. The documentation for (the library) is available on (the website) at

Getting Started

# Recommended: create virtual environment
# python3 -m venv venv
# . venv/bin/activate

# Install entire wq suite (recommended)
pip install wq

# Install only
pip install

# To enable's GIS support
pip install geopandas # includes Shapely & Fiona

# To enable's Excel write support
pip install xlwt # xls support
pip install xlsxwriter # xlsx support
# (xls/xlsx read support is enabled by default)

See the wq documentation for more information.

Features provides a general purpose API for loading, iterating over, and writing tabular datasets. The basic idea is to avoid needing to remember the unique usage of e.g. csv, xlrd, or xml.etree every time one needs to work with external data. Instead, abstracts these libraries into a consistent interface that works as an iterable of namedtuples. Whenever possible, the field names for a dataset are automatically determined from the source file, e.g. the column headers in an Excel spreadsheet.

from import ExcelFileIO
data = ExcelFileIO(filename='example.xls')
for row in data:
    print(, provides a number of built-in classes like the above, including a CsvFileIO, XmlFileIO, and JsonFileIO. There is also a convenience function, load_file(), that attempts to automatically determine which class to use for a given file.

from import load_file
data = load_file('example.csv')
for row in data:

All of the included *FileIO classes support both reading and writing to external files, though write support for Excel files requires additional libraries (xlwt and xlsxwriter) that aren’t listed as dependencies.

Network Client also provides network-capable equivalents of each of the above classes, to facilitate loading data from third party webservices.

from import JsonNetIO
class WebServiceIO(JsonNetIO):
    url = ""

data = WebServiceIO(params={'type': 'all'})
for row in data:
    print(row.timestamp, row.value)

The powerful requests library is used internally to load data over HTTP.

Pandas Analysis

When Pandas is installed, the as_dataframe() method on classes can be used to create a DataFrame, enabling more extensive analysis possibilities.

instance = WebServiceIO(params={'type': 'all'})
df = instance.as_dataframe()

GIS Support

When Fiona and Shapely are installed, can also open and create shapefiles and other OGR-compatible geographic data formats.

from import ShapeIO
data = ShapeIO(filename='sites.shp')
for id, site in data.items():
    print(id, site.geometry.wkt)


Each IO class is composed of mixin classes (loaders, parsers, and mappers) that handle the various steps of the process. By extending these mixin or the pre-mixed classes above, it is straightforward to extend to support arbitrary formats. The climata library provides a number of examples of custom IO classes for loading climate and hydrology data.

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for, version 1.1.0
Filename, size File type Python version Upload date Hashes
Filename, size (17.5 kB) File type Source Python version None Upload date Hashes View

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring DigiCert DigiCert EV certificate Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page