CCXT Download Utility

A lightweight wrapper to conventiently and efficiently download cryptocurrency data using CCXT.

About ccxt-download

What is it for? Downloading cryptocurrency data accessible via CCXT, but with less effort.

Why is this necessary? Many times I have found myself needing some data, only to spend some time writing a quick and dirty script to download said data with no thought about storing it for later. Add multiple symbols across different time periods to the mix, and things just get worse. Then when I want to load that data later, its so badly organised (or not at all) that it is easier to write another quick and dirty script and repeat the cycle. With this package, the above issues are no longer issues. For me anyway.

What makes this useful? The following features:

• asynchronous downloading (download data in parallel)
• intelligent file management (won't re-download data if it already exists)
  • if data is downloaded for the current day, it will be marked as incomplete, which will signal that it should be updated in future downloads
• efficient data storage (using Apache Parquet)
• helpful utilities for loading and processing data

Usage

Below are some brief examples illustrating the basic functionality of ccxt-download. For more, see the examples.

Downloading data

from ccxt_download import public, CANDLES

# Download candles
public.download(
    exchange="bybit",
    data_types=[CANDLES],
    symbols=[
        "ETH/USDT:USDT",
        "BTC/USDT:USDT",
    ],
    start_date="2023-09-01",
    end_date="2023-09-05",
)

Data will be downloaded between the dates specified. If the data already exists, it will not be re-downloaded. The exception to this is when a dataset has been marked as incomplete, for example when you download data for the current day (which has not ended yet). When this is detected, that incomplete dataset will be updated. If it can be completed, the incomplete marking will be removed.

Reading downloaded data

from ccxt_download import CANDLES
from ccxt_download.utilities import load_data

df = load_data(
    exchange="bybit",
    data_type=CANDLES,
    data_type_id="1m",
    symbols=["ETH/USDT:USDT"],
    start_date="2023-09-01",
    end_date="2023-09-04",
)

Installation

pip install ccxt-download

Notes and future work

• Support for private downloads to assist in accounting, account tracking and analysis, etc.

Contributing

Contributions to ccxt-download are welcomed. However, please try follow the guidelines below.

Seek early feedback

Please open an issue before a pull request to discuss any changes you wish to make.

Code style

The code in this project is formatted using Black. The Black package is included in the dev dependencies of ccxt-download. As mentioned below, please run black . before opening a pull request.

Setting up for Development

Fork this repository, then install from source using an editable installation. Additionally, install with the optional dev dependencies.

pip install -e .[dev]

If you want to make sure you have formatted the code before commiting, install the pre-commit hook using the command below. This will check that the code is formatted whenever you try to make a commit.

pre-commit install

If you get a message saying the code isn't formatted correctly, simply run the command below and try again.

black .