Fast Python Betfair historic data file parser
Project description
Betfair Data
Betfair Data is a very fast, Rust based, Betfair historical data parsing library for python. It supports both the official Betfair's historic data and self recorded stream files.
Installation
pip install betfair_data
Note: requires Python >= 3.7.
Example
import betfair_data as bfd
paths = [
"data/2021_12_DecRacingAUPro.tar",
"data/2021_10_OctRacingAUPro.tar",
"data/2021_11_NovRacingAUPro.tar",
]
market_count = 0
update_count = 0
for file in bfd.Files(paths):
market_count += 1
for market in file:
update_count += 1
print(f"Markets {market_count} Updates {update_count}", end='\r')
print(f"Markets {market_count} Updates {update_count}")
Loading Files
You can read in files quickly in a worker thread using the provided Files
utility. It supports reading in bz2, gzip or uncompressed (.json) stream files or .tar or .zip archives containing such files.
paths = [
"data/2021_10_OctRacingAUPro.tar",
"data/2021_11_NovRacingAUPro.zip",
"data/self_recorded_market.gz",
"data/uncompressed_market.json",
]
files = bfd.Files(paths)
Or you can use the glob library to find and select all the paths automatically.
import glob
paths = glob.glob("data/betfair_official/*.tar") + glob.glob("data/self_recorded/*.gz")
files = bfd.Files(paths)
You can also load the file through any other means and pass the raw bytes and name into the File object constructor.
# generator to read in files
def load_files(paths: str):
for path in glob.glob(paths, recursive=True):
with open(path, "rb") as file:
yield bfd.File(path, file.read())
for file in load_files("markets/*.json"):
for market in file:
pass
Benchmarks
Running over 3 months of Australian racing data on a 2021 M1 Macbook Pro.
betfair_data (mutable) | betfair_data (immutable/bflw compat) | betfairlightweight lightweight=True | betfairlightweight lightweight=False |
---|---|---|---|
5m 12sec | 6m 50sec | 1hour 1min 45sec | 3hours 46mins 39sec |
~70 markets/sec | ~53.5 markets/sec | ~6 markets/sec | ~1.62 markets/sec |
~534,200 updates/sec | ~406,500 updates/sec | ~45,500 updates/sec | ~12,250 updates/sec |
Types
IDE's should automatically detect the types and provide checking and auto complete. See the pyi stub file for a comprehensive view of the types and method available.
Betfairlightweight
We also support a format that is a drop in replacement for betfairlightweight
objects. We have rigorously tested it against betfairlightweight to ensure a complete match of its structure, any differences should be submitted as issues with the corresponding differences and the data used to create them.
from betfair_data import bflw
files = bflw.Files(paths)
from betfair_data import bflw
file_bytes = ...
file = bflw.File("file_name", file_bytes)
Logging
Logging can be enabled and warnings are emitted for IO and JSON errors.
import logging
logging.basicConfig(level=logging.WARN, format='%(levelname)s %(name)s %(message)s')
Example logged errors
WARNING betfair_data file: data/2021_10_OctRacingAUPro.tar/PRO/2021/Oct/4/30970292/1.188542184.bz2 err: (JSON Parse Error) expected value at line 1480 column 1
WARNING betfair_data file: data/2021_10_OctRacingAUPro.tar/PRO/2021/Oct/8/30985584/1.188739324.bz2 err: (JSON Parse Error) expected `:` at line 1 column 909
WARNING betfair_data file: data/2021_10_OctRacingAUPro.tar/PRO/2021/Oct/8/30985584/1.188739325.bz2 err: (JSON Parse Error) expected `:` at line 1 column 904
WARNING betfair_data file: data/2021_10_OctRacingAUPro.tar/PRO/2021/Oct/15/31001342/1.189124831.bz2 err: (JSON Parse Error) expected value at line 1335 column 1
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for betfair_data-0.3.4-cp37-abi3-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c9889065f410ce378fccce96cda3835ba73feeda4ae0bf48b9b5411892299641 |
|
MD5 | f8ca1e79d145bd43a0376f7db761b5ff |
|
BLAKE2b-256 | 765887cfd8162e68e89876c189c60f7949c7eb7199fc91b31b4896718647f545 |
Hashes for betfair_data-0.3.4-cp37-abi3-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6139cb0ca0b472379162f993185405a700d52c86b0ad0c1ad5bfbb8da5dfe2a3 |
|
MD5 | 906105c92b6b411ff714b7f16ef7b248 |
|
BLAKE2b-256 | a23db53ed4f40d9df8727fa81d01edcc558925a840e96098d62c56577e858445 |
Hashes for betfair_data-0.3.4-cp37-abi3-manylinux_2_17_s390x.manylinux2014_s390x.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a6d5feee62ae91a446395e9d9d7eaa88fafc13e28ab3f52fdc38f7908292c0a0 |
|
MD5 | c785027f93741c710d0727246de02609 |
|
BLAKE2b-256 | 96ac9e399d71428c4a7de789aa65aa8bbb8e79a38b0ecda6615fd003bc77331f |
Hashes for betfair_data-0.3.4-cp37-abi3-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 012d228d3e2ca8c63520913f51d90fe701f0e81f04f3d8fce6968197d1f9f970 |
|
MD5 | 2191cc4470115fa44d8fe363eba7b271 |
|
BLAKE2b-256 | b97071f5b68a93ac6eacc4db4975e582a0077fecdde973e848dd98633e107289 |
Hashes for betfair_data-0.3.4-cp37-abi3-manylinux_2_17_ppc64.manylinux2014_ppc64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5664584d6e78fe804283562824abc6309fbf8f528a9e5ca260736677cdd22bf2 |
|
MD5 | 6445e0ca543fe6357d8bd031b2a32090 |
|
BLAKE2b-256 | 3171377cbd792ca6d99d195ffeab9acc0dbff32df54e5c0df49d4accd6ff3d1e |
Hashes for betfair_data-0.3.4-cp37-abi3-manylinux_2_17_armv7l.manylinux2014_armv7l.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5f026572fb5f449f9dd13321e747d28d6d562ff17983cecdef20871277a058d6 |
|
MD5 | 9a845cdaac696a071fa3ff16a343c4dc |
|
BLAKE2b-256 | 78edbe0247682c0ea2ee07e8f849632f092b4d86086ef32b8f228fd707953be8 |
Hashes for betfair_data-0.3.4-cp37-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4ff0fdbcfaa53897b20e177f5dafce31c89d40f70cd9f41620b542eb55085a26 |
|
MD5 | f40b0e972ae8b504af5bea26ea95d219 |
|
BLAKE2b-256 | 44a27a2ac682834a8cb9b1a0d8cad89c27b60e1c034c93be30084fceba4fc161 |
Hashes for betfair_data-0.3.4-cp37-abi3-manylinux_2_5_x86_64.manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 658cd657280d147f415845e7997765638194115909b47dacafd3e8250a9e7a94 |
|
MD5 | 3f2b16c50a32a589fabcdfe687a63741 |
|
BLAKE2b-256 | 6a5e71e6ea64bc103c9d069ac28d81f89d2bddb96a6228b18bd3d1f057499242 |
Hashes for betfair_data-0.3.4-cp37-abi3-manylinux_2_5_i686.manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6bb6afdcc7f20fc3d10de3c1a126118e0b5224e1fcbf39dfe43ff295902df43c |
|
MD5 | 1ca09c01d75ee26533b85633c06eaf94 |
|
BLAKE2b-256 | d524192628614dab2412e6c326b985b8dac546340f122c95dbb09e0e695c004a |
Hashes for betfair_data-0.3.4-cp37-abi3-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 614f90f5dff0a8925f5564c49268340cc5263f76a64e3d375a8bad506ae06dd1 |
|
MD5 | 73cf359fc9fa8c0755d0f89e7f483170 |
|
BLAKE2b-256 | 69ef36d2078bb254bc6e4696e8d5298c9c0bddf1f6c3eaf0c47099d2d914f113 |
Hashes for betfair_data-0.3.4-cp37-abi3-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 767d259fb38ec72025ddb3a076c85e7cc38a1f0be9529c98face616879b7778c |
|
MD5 | 4e1e0ecfd5d568ee8d4e5e80a845f62d |
|
BLAKE2b-256 | 27414ff22d6a7a9a19f85bf7f6ac2d789f7342b21b83b0fbcbba35175b79083f |