KDB-X Python

Introduction

KDB-X Python is a Python first interface to the worlds fastest time-series database kdb+ and it's underlying vector programming language q. KDB-X Python takes a Python first approach to integrating q/kdb+ with Python following 10+ years of integrations between these two languages. Fundamentally it provides users with the ability to efficiently query and analyze huge amounts of in-memory and on-disk time-series data.

This interface exposes q as a domain-specific language (DSL) embedded within Python, taking the approach that q should principally be used for data processing and management of databases. This approach does not diminish the ability for users familiar with q or those wishing to learn more about it from making the most of advanced analytics and database management functionality but rather empowers those who want to make use of the power of kdb+/q who lack this expertise to get up and running fast.

KDB-X Python supports three principal use cases:

• It allows users to store, query, manipulate and use q objects within a Python process.
• It allows users to query external q processes via an IPC interface.
• It allows users to embed Python functionality within a native q session using it's under q functionality.

Users wishing to install the library can do so following the instructions here.

Once you have the library installed you can get up and running with KDB-X Python following the quickstart guide here.

What is q/kdb+?

Mentioned throughout the documentation q and kdb+ are respectively a highly efficient vector programming language and highly optimised time-series database used to analyse streaming, real-time and historical data. Used throughout the financial sector for 25+ years this technology has been a cornerstone of modern financial markets providing a storage mechanism for historical market data and tooling to make the analysis of this vast data performant.

Kdb+ is a high-performance column-oriented database designed to process and store large amounts of data. Commonly accessed data is available in RAM which makes it faster to access than disk stored data. Operating with temporal data types as a first class entity the use of q and it's query language qsql against this database creates a highly performant time-series analysis tool.

q is the vector programming language which is used for all interactions with kdb+ databases and which is known both for its speed and expressiveness.

For more information on using q/kdb+ and getting started with see the following links:

• An introduction to q/kdb+
• Tutorial videos introducing kdb+/q

Installation

Installing KDB-X Python using pip

Ensure you have a recent version of pip:

pip install --upgrade pip

Then install the latest version of KDB-X Python with the following command:

pip install --upgrade pykx

To install a specific version of KDB-X Python run the following command replacing <INSERT_VERSION> with a specific released semver version of the interface

pip install pykx==<INSERT_VERSION>

Warning: Python packages should typically be installed in a virtual environment. This can be done with the venv package from the standard library.

KDB-X Python License access and enablement

Installation of KDB-X Python via pip provides users with access to the library with limited functional scope, full details of these limitations can be found here. To access the full functionality of KDB-X Python you must first download and install a license.

If you do not have a license go to the KX Developer Center to obtain a KDB-X license.

Full instructions on license installation are covered on the Installing page.

Supported Environments

KX only officially supports versions of KDB-X Python built by KX, i.e. versions of KDB-X Python installed from wheel files. Support for user-built installations of KDB-X Python (e.g. built from the source distribution) is only provided on a best-effort basis. Currently, KDB-X Python provides wheels for the following environments:

• Linux
  • (manylinux2014_x86_64, manylinux2014_aarch64) with CPython 3.9-3.11
  • (manylinux_2_28_x86_64, manylinux_2_28_aarch64) with CPython 3.12-3.14
• macOS (macosx_10_15_x86_64, macosx_10_15_arm64) with CPython 3.9-3.14
• Windows (win_amd64) with CPython 3.9-3.14

Dependencies

Python Dependencies

KDB-X Python depends on the following third-party Python packages:

• pandas>=1.2; python_version>'3.8'
• numpy>=1.22; python_version<'3.11'
• numpy>=1.23; python_version=='3.11'
• numpy>=1.26; python_version>'3.11'
• pytz>=2022.1
• toml~=0.10.2
• dill>=0.2.0
• requests>=2.25.0

They are installed automatically by pip when KDB-X Python is installed.

KDB-X Python also has an optional Python dependency of pyarrow>=3.0.0, which can be included by installing the pyarrow extra, e.g. pip install pykx[pyarrow]

When using KDB-X Python with KX Dashboards users will be required to install ast2json~=0.3 this can be installed using the dashboards extra, e.g. pip install pykx[dashboards]

When using KDB-X Python Streaming users may require the ability to stop processes initialized in a now unavailable process to facilitate this KDB-X Python can make use of psutil this can be installed using the streaming extra, e.g. pip install pykx[streaming]

When using Streamlit users will be required to install streamlit~=1.28 this can be installed using the streamlit extra, e.g. pip install pykx[streamlit]

When attempting to convert data to/from PyTorch users will be required to install torch>2.1 this can be installed using the torch extra, e.g. pip install pykx[torch]

Warning: Trying to use the pa conversion methods of pykx.K objects or the pykx.toq.from_arrow method when PyArrow is not installed (or could not be imported without error) will raise a pykx.PyArrowUnavailable exception.

Optional Non-Python Dependencies

• libssl for TLS on IPC connections.
• libpthread on Linux/MacOS when using the PYKX_THREADING environment variable.

Building from source

Installing Dependencies

The full list of supported environments is detailed here. Installation of dependencies will vary on different platforms.

apt example:

apt-install python3 python3-venv build-essential python3-dev

yum example:

yum install python3 gcc gcc-c++ python3-devel.x86_64

Windows:

• Python
• Build Tools for Visual Studio.
• dlfcn-win32. Can be installed using Vcpkg.

To install the above dependencies, you can run the w64_install.ps1 script as an administrator:

cd pykx
.\w64_install.ps1

Building

Using a Python virtual environment is recommended:

python3 -m venv pykx-dev
source pykx-dev/bin/activate

Build and install KDB-X Python:

cd pykx
pip3 install -U '.[all]'

To run in licensed mode see here.

Now you can run/test KDB-X Python:

(pykx-dev) /data/pykx$ python
Python 3.10.6 (main, May 29 2023, 11:10:38) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pykx
>>> pykx.q('1+1')
pykx.LongAtom(pykx.q('2'))

Testing

Contributions to the project must pass a linting check:

pflake8

Contributions to the project must include tests. To run tests:

export PATH="$PATH:/location/of/your/q/l64" # q must be on PATH for tests
export QHOME=/location/of/your/q #q needs QHOME available
python -m pytest -vvv -n 0 --no-cov --junitxml=report.xml

KDB-X Python Licenses

This work is dual licensed under Apache 2.0 and the Software License for q.so and users are required to abide by the terms of both licenses in their entirety.

Community Help

If you have any issues or questions you can post them to community.kx.com. Also available on Stack Overflow are the tags pykx and kdb.

Customer Support

• Inquires or feedback: pykx@kx.com
• Support for Licensed Subscribers: support.kx.com