ibstract 1.0.0a2

Asynchronous financial trading data management

ibstract is a Python 3 package for trading data acquiring and management. Thanks to Python’s asyncio library, it can accesses Interactive Brokers API for concurrent remote data downloading, and a MySQL database as local cache for concurrent data archiving and offline query. Classes in the package also combine, transform, and maintain trading data, and provide organized and aggregated data or signals for algorithmic trading. ibstract users can focus on trading algorithms without worrying about the hassels of handling a broker API or the tedious and error-prone trading data management.

Features

• Concurrent data acquiring and processing with asynchronous access to remote IB API server and local MySQL database, powered by async/await syntax of asyncio module in Python 3.6+ and 3rd-party aio-libs.

• Automatically analyze and split a user’s historical data request, and dispatch data acquiring tasks to local MySQL database (preferred) or remote IB API server. In this way much downloading efforts could be saved for repeating requests for the same data pieces.

• MarketDataBlock class manages and merges historical data pieces with different symbols, types, durations and date/time in an organized and standardized way. Data time zone is region-based using pytz, and automatically converted and maintained.

Planned Features:

• Asynchronously generating technical signals from user-specified historical data.

• Concurrent real-time market data streaming and real-time trading signal generating.

• Background order submission, status monitoring, and logging.

Installation

pip3 install -U ibstract

Requirements

• Python 3.6+ (Anaconda 4.4.0+)

• Interactive Brokers API 9.73.2+

• IB gateway latest 967+

• ib_insync 0.8.5+

• aiomysql 0.0.9+

• sqlalchemy 1.1.9+

• pandas 0.20.1+

• tzlocal 1.4+

Documentation

Full Documentation

Examples

For full explanation and detailed examples, please take a look at the example notebooks:

• Historical data operations example notebook

• MarketDataBlock class example notebook

• IB class example notebook

Example 1: Concurrently acquire data from local MySql database and remote IB API server.

A user coroutine requests wider range of historical data than those existing in MySQL. The data pieces existing in MySQL will not be downloaded, but will be queried and combined with those downloaded. A request could be split into multiple downloading tasks and perfored concurrently and asynchronously, as well as inserting the downloaded data to MySQL in the background.

Data pre-existing in MySQL database:

                                                   opening    high     low  closing  volume  barcount average
Symbol DataType BarSize TickerTime
GS     TRADES   1d      2017-08-31 00:00:00-04:00   223.25  224.49  222.58   223.74   15491     10053 223.764
                        2017-09-01 00:00:00-04:00   224.55  227.56  223.53   225.88   16940     11739 226.350
                        2017-09-05 00:00:00-04:00   223.85  224.00  217.30   217.78   45499     28392 218.901

Request for wider range of data:

async def user_coro(req, broker, mysql):
    blk_ret = await get_hist_data(req, broker, mysql)
    return blk_ret

# Request daily data of 8 days, from 8/29 - 9/8.
# Data from 8/31 - 9/5 exist in local database and will not be downloaded.
req = HistDataReq('Stock', 'GS', '1d', '8d', dtest(2017, 9, 9))
broker = IB('127.0.0.1', 4002)
db_info = {'host': '127.0.0.1', 'user': 'root', 'password': 'ibstract',
           'db': 'ibstract_test'}

loop = asyncio.get_event_loop()
mysql={**db_info, 'loop': loop}
blk_ret = loop.run_until_complete(user_coro(req, broker, mysql))
blk_ret.df

Output data is the combination of those in database and downloaded:

                                                   opening    high     low  closing  volume  barcount     average
Symbol DataType BarSize TickerTime
GS     TRADES   1d      2017-08-29 00:00:00-04:00   217.27  220.14  215.75   219.96   18795     12617    218.7545
                        2017-08-30 00:00:00-04:00   220.25  224.22  220.09   222.42   18580     12085    222.7730
                        2017-08-31 00:00:00-04:00   223.25  224.49  222.58   223.74   15491     10053    223.7635
                        2017-09-01 00:00:00-04:00   224.55  227.56  223.53   225.88   16940     11739    226.3505
                        2017-09-05 00:00:00-04:00   223.85  224.00  217.30   217.78   45499     28392    218.9010
                        2017-09-06 00:00:00-04:00   218.98  221.02  217.61   218.83   26158     15960    219.5335
                        2017-09-07 00:00:00-04:00   218.73  218.81  214.64   215.84   27963     17892    215.7020
                        2017-09-08 00:00:00-04:00   215.51  219.28  215.40   217.21   23250     15562    217.5120

Example 2: Create, update and combine MarketDataBlock instances.

Input pandas.DataFrames having different columns, symbols, barsize, and dates/times:

print(df_gs1)
print(df_gs2)
print(df_fb5m)
print(df_fb1m)
print(df_amzn)

  symbol  barsize                        date   close
0     GS    5 min   2016-07-12 10:35:00-07:00  140.05
1     GS    5 min   2016-07-12 11:20:00-07:00  141.34

  symbol  barSize                    datetime   close   volume
0     GS    5 min   2016-07-12 10:35:00-07:00  140.05   344428

                  time       c     vol
0  2016-07-21 09:30:00  120.05  234242
1  2016-07-21 09:35:00  120.32  410842

                  time       c     vol
0  2016-07-25 09:40:00  120.47  579638
1  2016-07-25 09:41:00  120.82  192476

   symb     bar         date   close   volume
0  AMZN   1 day   2016-07-21  749.22    27917
1  AMZN   1 day   2016-07-22  738.87    36662
2  AMZN   1 day   2016-07-23  727.23     8766

MarketDatablock organizes DataFrames together:

import pytz
from ibstract import MarketDataBlock

east = pytz.timezone('US/Eastern')

blk = MarketDataBlock(df_gs1, datatype='TRADES', tz=east)
blk.update(df_gs2, datatype='TRADES', tz=east)
blk.update(df_fb5m, symbol='FB', datatype='TRADES', barsize='5m', tz=east)
blk.update(df_fb1m, symbol='FB', datatype='TRADES', barsize='1m', tz=east)
blk_amzn = MarketDataBlock(df_amzn, datatype='TRADES', tz=east)
blk.combine(blk_amzn)

Output MarketDataBlock:

                                                   closing  volume
Symbol DataType BarSize TickerTime
AMZN   TRADES   1d      2016-07-21 00:00:00-04:00   749.22   27917
                        2016-07-22 00:00:00-04:00   738.87   36662
                        2016-07-23 00:00:00-04:00   727.23    8766
FB     TRADES   1m      2016-07-25 09:40:00-04:00   120.47  579638
                        2016-07-25 09:41:00-04:00   120.82  192476
                5m      2016-07-21 09:30:00-04:00   120.05  234242
                        2016-07-21 09:35:00-04:00   120.32  410842
GS     TRADES   5m      2016-07-12 13:35:00-04:00   140.05  344428
                        2016-07-12 14:20:00-04:00   141.34      -1

References

• Interactive Brokers API Documentation

• Interactive Brokers API User Group

Changelog

Version 1.0.0

• Migrated to native Python IB API.

• Asynchronous operations based on asyncio and aio-libs.

• New structures and features.

• Added documentation and test cases.

Version 0.1.0 (Deprecated)

• This experimental version was developed based on IB API v9.72 or older, using swigibpy v0.5.0.