This is a pre-production deployment of Warehouse, however changes made here WILL affect the production instance of PyPI.
Help us improve Python packaging - Donate today!
Project Description

# Recon File Format

The goal of the recon file format (or more precisely, suite of file formats) is to provide a compact format for storing (typically time-series oriented) simulation data in a network friendly way. Along the way, we’ve added support for embedding metadata at the file, table, object and signal level as well.

Obviously compactness is important to keep the size of files down. Disk space is less an issue these days than it has been in the past. Nevertheless, one our requirements was to be competitive in terms of how much disk space is required to store signals.

In trying to develop a network friendly format, we quickly found ourselves facing two conflicting goals. On the one hand, we wanted the ability to stream data from simulations to files. This meant, essentially, having a format that was “append friendly”. On the other hand, we wanted a format where data for an individual signal could be accomplished with a “single” read (i.e. single request over the network). Note, our definition of “single” has some caveats.

We recognized that we could not accomplish this goal with a single format (at least not without making the implementation extremely complicated and involving lots of extra writes). So we opted to specify to file formats that could easily be converted.

## Dependencies

This package requires an implementation of msgpack. However, there are several implementation in Python. During development, the msgpack-python package was used. It is possible that other packages might work as well.

## The “Wall” Format

The wall format is the “append friendly” format. You can think of the wall format as a series of “bricks” being laid down. Each “brick” represents some data being added to the file. One of the nice advantages of this format is that it allows concurrent writing to multiple tables of data. So in a simulation where different results are reported at different intervals, this file format can be used to append different results to different tables. In other words, it supports appending to multiple tables, not just one.

## The “Meld” Format

The meld format is mainly an archival format. It rearranges the data so that individual signals can be easily extracted. This is what enables data to be extracted with a minimal number of “reads” from the data source. The key issue with reads is the case where the data is being read over a network.

As simulation moves to cloud based systems, it will be come increasingly cumbersome to move entire files back and forth between the cloud and the desktop/browser. Having a format that supports “pulling” just the information that is required on demand facilitates utilizing cloud/remote storage solutions which will lead to more responsive interfaces and better data management practices and capacity. The meld format is designed for this use case.

Release History

Release History


This version

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

Download Files

Download Files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

File Name & Checksum SHA256 Checksum Help Version File Type Upload Date
pyrecon-0.3.0.tar.gz (14.1 kB) Copy SHA256 Checksum SHA256 Source Mar 4, 2014

Supported By

WebFaction WebFaction Technical Writing Elastic Elastic Search Pingdom Pingdom Monitoring Dyn Dyn DNS Sentry Sentry Error Logging CloudAMQP CloudAMQP RabbitMQ Heroku Heroku PaaS Kabu Creative Kabu Creative UX & Design Fastly Fastly CDN DigiCert DigiCert EV Certificate Rackspace Rackspace Cloud Servers DreamHost DreamHost Log Hosting