This is a pre-production deployment of Warehouse. Changes made here affect the production instance of PyPI (
Help us improve Python packaging - Donate today!
Project Description

Efficient storage of same-type, uneven-size arrays

Jagged is an ongoing amateur project exploring the storage panorama for datasets containing (large amounts of) arrays with the same type and number of columns, but varying number of rows. Examples of such datasets for which jagged has been used are collections of multivariate timeseries (short animal behaviour snippets) and collections of molecules (represented as varying length strings).

Jagged aims to help analyzing data in the laptop and the cluster, in batch or interactively, providing a very lightweight store. Jagged provides fast retrieval of array subsets for many-GB datasets containing millions of rows.

By-design constraints

Focus is on fast retrieval of arbitrary batch queries.

Jagged stores are append only.

There is no transaction, replication or distribution. It is all files in your local or network disks.

Not important efforts have been given yet to optimize (although some backends work quite smoothly).

At the moment, everything is simple algorithms implemented in pure python.


It should suffice to use pip:

pip install jagged

Jagged stores builds on top of several high quality python libraries: numpy, blosc, bloscpack, bcolz and joblib. It also needs whatami and python-future. Testing relies on pytest (you need to install all dependencies to test at the moment, this will change soon).


Using jagged is simple. There are different implementations that provide two basic methods: append adds a new array to the store, get retrieves collections of arrays identified by their insertion order in the store.

import os.path as op
import numpy as np
from jagged.mmap_backend import JaggedByMemmap

# A Jagged instance is all you need
jagged = JaggedByMemmap(op.expanduser(path='~/jagged-example/mmap'))
# You can drop here any you want to

# Generate a random dataset
rng = np.random.RandomState(0)
max_length = 2000
num_arrays = 100
originals = [rng.randn(rng.randint(0, max_length), 50)
             for _ in range(num_arrays))

# Add these to the store (context is usually optional but recommended)
with jagged:
    indices = map(jagged.append, originals)

# What do we have in store?
print('Number of arrays: %d, number of rows: %d' % (jbmm.narrays, jbmm.nrows))
print('Jagged shape=%r, dtype=%r, order=%r' %
      (jagged.shape, jagged.dtype, jagged.order))

# Check roundtrip
roundtripped = jagged.get(indices)
print('The store has %d arrays')

# Jagged stores self-identified themselves (using whatami)

# Jagged stores can be iterated in chunks
# See iter

# Jagged stores can be populated from other jagged stores

# Some jagged stores allow to retrieve arbitrary rows as fast
# as arbitrary arrays.


Although rapidly changing, jagged already provides the following storage backends that can be considered as working and stable. Other backends are planned.

Backend comp chunk column mmap lin lazy cont
JaggedByBlosc X     X      
JaggedByCarray X X     X   X
JaggedByH5Py X X     X X X
JaggedByJoblib X X          
JaggedByMemMap       X X X X
JaggedByBloscpack X            
JaggedByPickle X X          
  • comp: can be compressed
  • chunk: can be chunked
  • column: stores columns of the array contiguously (can be easily implemented by using a store per column)
  • mmap: can open a memmap to the data
  • lin: can retrieve any row without the need to retrieve the whole
    array it contains it
  • lazy: the arrays are not fetched immediatly; this can mean also that they can be managed
    as virtual-memory by the OS (JaggedByMemMap only)
  • cont: retrieved arrays can be forced to lie in contiguous memory segments


What backend and parameters work best depends on whether your data is compressible or not and the sizes of the arrays. We have a good idea of what works best for our data and are working at providing a benchmarking framework. Find here a preview.

Release History

Release History


This version

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

Download Files

Download Files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

File Name & Checksum SHA256 Checksum Help Version File Type Upload Date
jagged-0.1.0.tar.gz (23.7 kB) Copy SHA256 Checksum SHA256 Source Aug 29, 2015

Supported By

WebFaction WebFaction Technical Writing Elastic Elastic Search Pingdom Pingdom Monitoring Dyn Dyn DNS Sentry Sentry Error Logging CloudAMQP CloudAMQP RabbitMQ Heroku Heroku PaaS Kabu Creative Kabu Creative UX & Design Fastly Fastly CDN DigiCert DigiCert EV Certificate Rackspace Rackspace Cloud Servers DreamHost DreamHost Log Hosting