Fast Python Betfair historic data file parser
Project description
Betfair Data
Betfair Data is a very fast Betfair historical data file parsing library for python. It currently supports tar archives containing BZ2 compressed NLJSON files (the standard format provided by Betfair's historic data portal).
The library is written in Rust and uses advanced performance enhancing techniques, like in place json deserialization and decompressing Bz2 encoded data on worker threads and is ideal for parsing large quantities of historic data that could otherwise take hours or days to parse.
Installation
pip install betfair_data
Note: requires Python >= 3.6.
Example
import betfair_data
paths = [
"data/2021_12_DecRacingAUPro.tar",
"data/2021_10_OctRacingAUPro.tar",
"data/2021_11_NovRacingAUPro.tar",
]
market_count = 0
update_count = 0
for market in betfair_data.TarBz2(paths):
market_count += 1
update_count += 1
while market.update():
update_count += 1
print(f"Markets {market_count} Updates {update_count}", end='\r')
print(f"Markets {market_count} Updates {update_count}")
Types
IDE's should automatically detect the types and provide checking and auto complete. See the pyi stub file for a comprehensive view of the types and method available.
Benchmarks
Betfair Data (this) | Betfairlightweight |
---|---|
3m 37sec | 1hour 1min 45sec |
~101 markets/sec | ~6 markets/sec |
~768,000 updates/sec | ~45,500 updates/sec |
Benchmarks were run against 3 months of Australian racing markets comprising roughly 22,000 markets. Benchmarks were run on a M1 Macbook Pro with 32GB ram.
These results should only be used as a rough comparison, different machines, different sports and even different months can effect the performance and overall markets/updates per second.
No disrespect is intended towards betfairlightweight, which remains an amazing library and a top choice for working with the Betfair API. Every effort was made to have its benchmark below run as fast as possible, and any improvements are welcome.
Betfair_Data benchmark show in the example above.
Betfairlightweight Benchmark
from typing import Sequence
import unittest.mock
import tarfile
import bz2
import betfairlightweight
trading = betfairlightweight.APIClient("username", "password", "appkey")
listener = betfairlightweight.StreamListener(
max_latency=None, lightweight=True, update_clk=False, output_queue=None, cumulative_runner_tv=True, calculate_market_tv=True
)
paths = [
"data/2021_10_OctRacingAUPro.tar",
"data/2021_11_NovRacingAUPro.tar",
"data/2021_12_DecRacingAUPro.tar"
]
def load_tar(file_paths: Sequence[str]):
for file_path in file_paths:
with tarfile.TarFile(file_path) as archive:
for file in archive:
yield bz2.open(archive.extractfile(file))
return None
market_count = 0
update_count = 0
for file_obj in load_tar(paths):
with unittest.mock.patch("builtins.open", lambda f, _: f):
stream = trading.streaming.create_historical_generator_stream(
file_path=file_obj,
listener=listener,
)
gen = stream.get_generator()
market_count += 1
for market_books in gen():
for market_book in market_books:
update_count += 1
print(f"Markets {market_count} Updates {update_count}", end='\r')
print(f"Markets {market_count} Updates {update_count}")
Logging
Logging can be enabled and warnings are emitted for IO and JSON errors
import logging
logging.basicConfig(level=logging.WARN, format='%(levelname)s %(name)s %(message)s')
Example logged errors
WARNING betfair_data source: data/2021_10_OctRacingAUPro.tar file: PRO/2021/Oct/4/30970292/1.188542184.bz2 err: (JSON Parse Error) expected value at line 1480 column 1
WARNING betfair_data source: data/2021_10_OctRacingAUPro.tar file: PRO/2021/Oct/8/30985584/1.188739324.bz2 err: (JSON Parse Error) expected `:` at line 1 column 909
WARNING betfair_data source: data/2021_10_OctRacingAUPro.tar file: PRO/2021/Oct/8/30985584/1.188739325.bz2 err: (JSON Parse Error) expected `:` at line 1 column 904
WARNING betfair_data source: data/2021_10_OctRacingAUPro.tar file: PRO/2021/Oct/15/31001342/1.189124831.bz2 err: (JSON Parse Error) expected value at line 1335 column 1
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for betfair_data-0.1.8-cp36-abi3-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b5e9322dce428473a49e28cc2592d500183836260b39e9e3133c74e50aa1ac6f |
|
MD5 | f95d55e4f0ebddcb1e9843dd291dd39a |
|
BLAKE2b-256 | ffa2a28e8048cf17b959fe8729aa60944f850522a6596bd62a9d499d43b9ec27 |
Hashes for betfair_data-0.1.8-cp36-abi3-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b1a7ac10e4d0913846d4f55d92d6b07f421994c43008f1b8d3caf7328f230c2a |
|
MD5 | 3b9a43e8bcad71e77754f0b77ccadcea |
|
BLAKE2b-256 | 0181856cb60a9d188f60da0a0926327567441bbe5107947dceef1edd7cb4c731 |
Hashes for betfair_data-0.1.8-cp36-abi3-manylinux_2_17_s390x.manylinux2014_s390x.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e21cac6fc40a1cffdcadfec34b72da81b86b7cdd436dd5b438eadfa540404006 |
|
MD5 | bb104522f81bcd83dfeefaf342cf764b |
|
BLAKE2b-256 | 6b0a2bc616bb913cedd79aebd3979f6c73361def33f67537211d2df670209dc7 |
Hashes for betfair_data-0.1.8-cp36-abi3-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 00476cc9c88a2223ae6e74c983ffb36feb612b4548facfe20771aaf2e5e2f728 |
|
MD5 | b95408b12efb15940c69b144866af9e0 |
|
BLAKE2b-256 | 53d3677cf3fec169afb202ad9ec3292b08c809c7f7643732bf0d10eae53b1a64 |
Hashes for betfair_data-0.1.8-cp36-abi3-manylinux_2_17_ppc64.manylinux2014_ppc64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e37c763e7b04058c5dbcb994403e9d64d4969ee7eae6d08c00b1ffec13338d55 |
|
MD5 | a755ac34011c85564eb89318ffd351b7 |
|
BLAKE2b-256 | f429484a7477d0f42f0770a3f3909799590b61bde14d90e9f5054c1fae355f91 |
Hashes for betfair_data-0.1.8-cp36-abi3-manylinux_2_17_armv7l.manylinux2014_armv7l.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8b2b1f578f026f974be7f88caf5c62d8ce940a6a0cf329f29ba723de42736dd8 |
|
MD5 | 411a908b2469b7e2a50c77553aaa11aa |
|
BLAKE2b-256 | 87df12328dbffc455a383e99f526a80c382917f439209e9c608ef84a433addea |
Hashes for betfair_data-0.1.8-cp36-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8bdbe47c4741f46592cf38493017646e0a7b3ef4d3810e2eac2972d997d8016f |
|
MD5 | 3b715b54b5d2c36bd8a46dfe3b6d0a90 |
|
BLAKE2b-256 | 5eb25def6eeb3892bac23dc5f28609ad8e8b0c0c94155de0015b815c2935d8c1 |
Hashes for betfair_data-0.1.8-cp36-abi3-manylinux_2_5_x86_64.manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | cf50fed7f77a70d84a1c96215f4e13d473151a33d4e442305c85c17c26470bcd |
|
MD5 | 8160c2523b395cf43c8507a0857be053 |
|
BLAKE2b-256 | e00f808272c6aa139ff7b7905d549fef2d6a402d5ef5f86f73a0dc2958febfd5 |
Hashes for betfair_data-0.1.8-cp36-abi3-manylinux_2_5_i686.manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e2435f08e259a5544b4ee00ac8ae777f25f546567caf69e775d4834a16bca5a3 |
|
MD5 | c85d648910aa157a95366bf23fa38c80 |
|
BLAKE2b-256 | dc25f85a1c06f598c059eb2c7978f18bf97a65941b4951002d6d9a7fc6a95af4 |
Hashes for betfair_data-0.1.8-cp36-abi3-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 42fe59b9e58c5ccf93040d276c8df41cac1c6fd75843d5f1f7ac9af1f970693a |
|
MD5 | 6c4a753039180d6c5fab0accfbe18f2b |
|
BLAKE2b-256 | 6b0382482990d593d70c4ee7b713c09c8f7cebb49db8f5517a3e6cfaf6a47748 |
Hashes for betfair_data-0.1.8-cp36-abi3-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a7a525b3358ccebe10033aa31806e8248381e6c019243006aca83311cc7dd86d |
|
MD5 | 35e702e5cc248ad9dfe3cb0b2f4beec9 |
|
BLAKE2b-256 | 78c1e07124708a622893439f38e15a970437add1bfe971c6d56b65a7ca6644e0 |