Skip to main content

Fast streaming I/O of numeric matrices

Project description

fmio - Fast stream I/O for numeric matrices

If you have ever piped large CSV or TSV numeric data between
scripts, you might realize just how much time is taken parsing
strings rather than performing actual computations.

``fmio`` is a simple compressed, binary format and Python library
to read and write matrices -- defined as 2D numeric data with row
and column names, analogous to pandas DataFrames that only accept
numeric data.


The only dependencies are numpy and pandas.

.. code-block:: bash
$ pip install fmio


From the command-line, you can serialize and deserialize fmio

.. code-block:: bash

$ fmio < in.tsv > out.fmio
$ fmio -dc < out.fmio
<same as input>

The real purpose is to perform fast reads from within Python:

.. code-block:: python

import fmio, sys
with fmio.Reader(sys.stdin) as h:
for r in h:
print(, r.sum())

.. code-block:: bash

$ python < out.fmio


The file format is in machine-native format. Although almost all
modern processors are "little-endian", these files may not be
completely portable.

The library is still in development. The file format is mostly
stable but still subject to change. Don't use this for long-term
data storage.



Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Filename, size & hash SHA256 hash help File type Python version Upload date
fmio-1.0-beta.tar.gz (3.7 kB) Copy SHA256 hash SHA256 Source None

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page