bc-utils

Barchart.com allows paid subscribers to download historic futures contract prices in CSV format. Individual contracts must be downloaded separately, which is laborious and slow. This script automates the process for daily and hourly data.

Quickstart

import os
import logging
from bcutils.bc_utils import get_barchart_downloads, create_bc_session

logging.basicConfig(level=logging.INFO)

CONTRACTS = {
    "AUD" : {"code": "A6", "cycle": "HMUZ", "exchange": "CME"},
    "GOLD" : {"code": "GC", "cycle": "GJMQVZ", "exchange": "COMEX"},
}

session = create_bc_session(
    config_obj=dict(
        barchart_username="user@domain.com",
        barchart_password="s3cr3t_321",
    )
)

get_barchart_downloads(
    session,
    contract_map=CONTRACTS,
    instr_list=["AUD", "GOLD"],
    save_dir=os.getcwd(),
    start_year=2020,
    end_year=2021,
)

The code above would:

• for the CME Australian Dollar future, get OHLCV price data for the Mar, Jun, Sep and Dec 2020 contracts
• download in CSV format
• save with filenames like Hour_AUD_20200300.csv, Day_AUD_20200600.csv into the specified directory
• for COMEX Gold, get Feb, Apr, Jun, Aug, Oct, and Dec data, with filenames like Hour_GOLD_20200200.csv etc

Features:

• Designed to be run once a day by a scheduler
• the script skips contracts already downloaded
• by default gets 120 days of data per contract, override possible per instrument
• dry run mode to check setup
• allows updates to previously downloaded files
• you must have a Barchart subscription. Premier users get 250 downloads a day, Plus users get 5

Browser requirements

Since version 0.1.8, the code in this project drives a real browser (via patchright) rather than making plain HTTP requests. Install the browser once with patchright install chromium (or uv run patchright install chromium in this repo).

By default, the browser runs headed (headless=False). On a server or CI runner with no display, run under a virtual display such as Xvfb (e.g. xvfb-run -a python your_script.py), or pass headless=True.

Login/session state persists across runs in auth_dir (defaults to ~/.bc_utils/auth) so you don't have to log in again every run. Treat this directory like a browser profile - don't commit it or share it.

For pysystemtrade users

This project was originally created to make it easier to populate pysystemtrade (PST) with futures prices from Barchart, so setup is straightforward. Steps:

1. Clone the bc-utils repo, or your own fork. The remaining steps assume the location ~/bc-utils

2. Have a look at ~/bc-utils/bcutils/config.py. This file already contains config items for many futures instruments, with their matching PST symbols. For example,

CONTRACT_MAP = {
    ...
    "GOLD": {"code": "GC", "cycle": "GJMQVZ", "exchange": "COMEX"},
    ...
}

indicates that instrument with PST code GOLD has the Barchart symbol GC, months GJMQVZ, and exchange COMEX. It also contains date config for various Futures exchanges. For example,

EXCHANGES = {
    ...
    "COMEX": {"tick_date": "2008-05-04", "eod_date": "1978-02-27"},
    ...
}

That indicates that futures instruments with exchange COMEX have daily price data from 27 Feb 1978, and hourly from 4 May 2008. Those date attributes are provided by Barchart, and turn out to be inaccurate. Previous versions of this library would waste valuable allowance by attempting to download data that was not there. Newer versions handle this much better. If you use this library to download prices that are not in the config file, please consider contributing with a PR.

3. Have a look at the sample snippets in ~bc-utils/sample/pst.py. There are examples for use with an external config file. Use the sample config ~bc-utils/sample/private_config_sample.yaml, copy and rename to the top level of the ~bc-utils dir. Update with your credentials and save path etc

Alternatively, paste the contents of the sample config into your PST private config, and do something like

def download_with_pst_config():
    config = load_config("<path to your PST private config>")
    get_barchart_downloads(
        create_bc_session(config),
        instr_list=config['barchart_download_list'],
        start_year=config['barchart_start_year'],
        end_year=config['barchart_end_year'],
        save_dir=config['barchart_path'],
        do_daily=config['barchart_do_daily'],
        dry_run=config['barchart_dry_run'],
    )

4. add bc-utils to your crontab. Since a headless server has no display, run under xvfb-run (see Browser requirements above):

00 08 * * 1-7 . $HOME/.profile; cd ~/bc-utils; xvfb-run -a python3 bcutils/sample/pst.py >> $ECHO_PATH/barchart_download.txt 2>&1

You could also add another entry to run the updater once a week.

5. To import the prices into PST (see below):

from sysdata.config.production_config import get_production_config
from syscore.fileutils import resolve_path_and_filename_for_package
from sysdata.csv.csv_futures_contract_prices import ConfigCsvFuturesPrices
from sysinit.futures.contract_prices_from_split_freq_csv_to_db import (
    init_db_with_split_freq_csv_prices_for_code,
)

BARCHART_CONFIG = ConfigCsvFuturesPrices(
    input_date_index_name="Time",
    input_skiprows=0,
    input_skipfooter=0,
    input_date_format="%Y-%m-%dT%H:%M:%S",
    input_column_mapping=dict(
        OPEN="Open", HIGH="High", LOW="Low", FINAL="Close", VOLUME="Volume"
    ),
)

# assuming bc-utils config pasted into private
datapath = resolve_path_and_filename_for_package(
    get_production_config().get_element_or_default("barchart_path", None)
)

# import prices for a single instrument
init_db_with_split_freq_csv_prices_for_code("GOLD", datapath=datapath, csv_config=BARCHART_CONFIG)