corintick

Column-based datastore for historical timeseries data. Corintick is designed mainly to store pandas DataFrames that represent timeseries.

Instalation

In order to use Corintick you need MongoDB. See installation instructions here.

Corintick itself can be installed with pip:

$ pip install corintick

Quickstart

Initialize Corintick:

from corintick import Corintick
corin = Corintick()

Now we need a DataFrame to insert into Corintick. For demonstration purposes, we will get data from Quandl:

import quandl
df1 = quandl.get('TSE/7203')

Here, df1 looks like this:

              Open    High     Low   Close      Volume
Date
2012-08-23  3240.0  3270.0  3220.0  3260.0   4652200.0
2012-08-24  3225.0  3245.0  3210.0  3235.0   3659600.0
2012-08-27  3250.0  3280.0  3215.0  3220.0   3614600.0
2012-08-28  3235.0  3260.0  3150.0  3180.0   6759100.0
2012-08-29  3180.0  3195.0  3160.0  3175.0   2614800.0
2012-08-30  3180.0  3190.0  3160.0  3170.0   3291700.0
2012-08-31  3135.0  3155.0  3095.0  3095.0   5663800.0
...

Writing

Inserting df1 into Corintick is simple:

corin.write('7203.T', df1, source='Quandl', country='Japan')

The first argument passed to corintick.write is an UID (universal identifier) and must be unique for each timeseries inserted in a given collection. The second argument is the dataframe to be inserted. The remaining keyword arguments are optional metadata tags that can be attached to the dataframe/document for querying.

Reading

Reading from Corintick is also straightforward:

df2 = corin.read('7203.T')

You can also specify start and end as ISO-8601 datetime string…

df2 = corin.read('7203.T', start='2014-01-01', end='2014-12-31')

              Open    High     Low   Close      Volume
2014-01-06  6360.0  6400.0  6280.0  6300.0  12249300.0
2014-01-07  6270.0  6340.0  6260.0  6270.0   7891400.0
2014-01-08  6310.0  6320.0  6260.0  6300.0   7184100.0
2014-01-09  6310.0  6340.0  6260.0  6270.0   8653000.0
2014-01-10  6260.0  6310.0  6250.0  6290.0   7815900.0
...
2014-12-24  7645.0  7687.0  7639.0  7657.0  9287900.0
2014-12-25  7600.0  7655.0  7597.0  7611.0  5362700.0
2014-12-26  7629.0  7700.0  7615.0  7696.0  6069100.0
2014-12-29  7740.0  7746.0  7565.0  7662.0  9942800.0
2014-12-30  7652.0  7674.0  7558.0  7558.0  7821200.0

…and which columns you want retrieved:

df2 = corin.read('7203.T', columns=['Close', 'Volume'], start='2017-05-10')

             Close      Volume
2017-05-10  6081.0   7823700.0
2017-05-11  6123.0  13511900.0
2017-05-12  6047.0   8216600.0
2017-05-15  6009.0   5925200.0
2017-05-16  6093.0   6449300.0
...

Configuration

By default, Corintick tries to use a MongoDB instance running at localhost:27017. This can be changed through the host and port arguments of the Corintick initializer. Similarly, the database to be used by Corintick defaults to corintick and can also be changed using the db parameter. All the data in the db database is assumed to be Corintick data. Avoid having any other process/application reading/writing data to that database.

In case your MongoDB setup requires authentication, you can use the username and password arguments.

See Corintick.__init__ for details.

Collections

Corintick can use multiple collections to better organize data. A Corintick collection is the same as a MongoDB collection. In each collection, only a single dataframe/document can exist for a given UID for a given time period.

In case you need to store two different types of data for a same UID over an overlapping time frame (i.e. trade data and order book data for a given stock), you should separate the two different types of data into different collections.

By default, data is written to the corintick collection. This default collection can be changed by assigning a string to Corintick.default_collection.

>>> corin.collection = 'another_collection'

Collections can also be specified on a method call basis:

df = corin.read('7203.T', collection='orderbook')

corin.write(df, collection='another_collection')

Corintick mechanics

During writing, Corintick does the following:

Takes the input DataFrame and splits into columns
Serializes/compresses each using the LZ4 compression algorithm
Generates a MongoDB document containing the binary blobs corresponding to each column and other metadata

During reading, the opposite takes places:

Documents are fetched
Data is decompressed and converted back to numpy arrays
DataFrame is reconstructed and returned to the user

Background

Corintick was inspired by and aims to be a simplified version of Man AHL’s Arctic.

Differences from Arctic

Corintick has a single storage engine, which is column-based and not versioned, similar to Arctic’s TickStore. However, differently from TickStore, it does support non-numerical object dtype columns by parsing them into MessagePack string objects

Naming

Corintick aimed from the beginning to be a column-based data storage. “Corintick” is a blend of “Corinthan” (style of Roman columns) and “tick”.

Benchmarks

TODO

vs InfluxDB
vs vanila MongoDB
vs MySQL
vs KDB+ (32-bit)

Contributing

To contribute, fork the repository on GitHub, make your changes and submit a pull request.

Corintick is not a mature project yet, so just simply raising issues is also greatly appreciated :)

Metadata

Release files for corintick 0.2.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for corintick 0.2.0
File	Size	Uploaded	Provenance
corintick-0.2.0.tar.gz	9.1 kB	Aug 7, 2018	Without provenance	Details

Built distribution (wheel)

Table of built distributions (wheels) for corintick 0.2.0
File	Interpreter	ABI	Platform	Reset
corintick-0.2.0-py3-none-any.whl 9.4 kB Aug 7, 2018 Without provenance	Python 3	none	any	Details

Total release size: 18.5 kB

Release files / corintick-0.2.0.tar.gz

Download URL	corintick-0.2.0.tar.gz
Size	9.1 kB
Tags	Source
SHA-256 checksum How to use checksums	`72ac41692339cb43bf3aefae7062a2f98ded9d74ec857cbb6847ba3375fc4e06`
BLAKE2b-256 checksum How to use checksums	`740f66460fe68466887d624ec9d7341364e6cd94e81dd2ed8ec19c13eebaca91`
Upload date	Aug 7, 2018
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/1.11.0 pkginfo/1.4.2 requests/2.18.4 setuptools/39.0.1 requests-toolbelt/0.8.0 tqdm/4.24.0 CPython/3.6.5`

Release files / corintick-0.2.0-py3-none-any.whl

Download URL	corintick-0.2.0-py3-none-any.whl
Size	9.4 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`0017fc157bd0fdf2be1c7bf0028ebf7736f19fb012043c3e9b36ecff8822f43b`
BLAKE2b-256 checksum How to use checksums	`c58d3380674228520046992d179751e3ddd5c42c6f232b64e5ef3ec44567e891`
Upload date	Aug 7, 2018
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/1.11.0 pkginfo/1.4.2 requests/2.18.4 setuptools/39.0.1 requests-toolbelt/0.8.0 tqdm/4.24.0 CPython/3.6.5`

Security for corintick 0.2.0

Report project as malware

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.2.0 This release

Aug 7, 2018 2 release files No provenance

corintick 0.2.0