Skip to main content

An interface between Python and q

Project description

PyKX

Introduction

PyKX is a Python first interface to the worlds fastest time-series database kdb+ and it's underlying vector programming language q. PyKX takes a Python first approach to integrating q/kdb+ with Python following 10+ years of integrations between these two languages. Fundamentally it provides users with the ability to efficiently query and analyze huge amounts of in-memory and on-disk time-series data.

This interface exposes q as a domain-specific language (DSL) embedded within Python, taking the approach that q should principally be used for data processing and management of databases. This approach does not diminish the ability for users familiar with q or those wishing to learn more about it from making the most of advanced analytics and database management functionality but rather empowers those who want to make use of the power of kdb+/q who lack this expertise to get up and running fast.

PyKX supports three principal use cases:

  • It allows users to store, query, manipulate and use q objects within a Python process.
  • It allows users to query external q processes via an IPC interface.
  • It allows users to embed Python functionality within a native q session using it's under q functionality.

Users wishing to install the library can do so following the instructions here.

Once you have the library installed you can get up and running with PyKX following the quickstart guide here.

What is q/kdb+?

Mentioned throughout the documentation q and kdb+ are respectively a highly efficient vector programming language and highly optimised time-series database used to analyse streaming, real-time and historical data. Used throughout the financial sector for 25+ years this technology has been a cornerstone of modern financial markets providing a storage mechanism for historical market data and tooling to make the analysis of this vast data performant.

Kdb+ is a high-performance column-oriented database designed to process and store large amounts of data. Commonly accessed data is available in RAM which makes it faster to access than disk stored data. Operating with temporal data types as a first class entity the use of q and it's query language qsql against this database creates a highly performant time-series analysis tool.

q is the vector programming language which is used for all interactions with kdb+ databases and which is known both for its speed and expressiveness.

For more information on using q/kdb+ and getting started with see the following links:

Installation

Installing PyKX using pip

Ensure you have a recent version of pip:

pip install --upgrade pip

Then install the latest version of PyKX with the following command:

pip install pykx

To install a specific version of PyKX run the following command replacing <INSERT_VERSION> with a specific released semver version of the interface

pip install pykx==<INSERT_VERSION>

Warning: Python packages should typically be installed in a virtual environment. This can be done with the venv package from the standard library.

PyKX License access and enablement

Installation of PyKX via pip provides users with access to the library with limited functional scope, full details of these limitations can be found here. To access the full functionality of PyKX you must first download and install a kdb+ license, this can be achieved either through use of a personal evaluation license or receipt of a commercial license.

Personal Evaluation License

The following steps outline the process by which a user can gain access to an install a kdb Insights license which provides access to PyKX

  1. Visit https://kx.com/kdb-insights-personal-edition-license-download/ and fill in the attached form following the instructions provided.
  2. On receipt of an email from KX providing access to your license download this file and save to a secure location on your computer.
  3. Set an environment variable on your computer pointing to the folder containing the license file (instructions for setting environment variables on PyKX supported operating systems can be found here.
    • Variable Name: QLIC
    • Variable Value: /user/path/to/folder

Commercial Evaluation License

The following steps outline the process by which a user can gain access to an install a kdb Insights license which provides access to PyKX

  1. Contact you KX sales representative or sales@kx.com requesting a trial license for PyKX evaluation. Alternately apply through https://kx.com/book-demo.
  2. On receipt of an email from KX providing access to your license download this file and save to a secure location on your computer.
  3. Set an environment variable on your computer pointing to the folder containing the license file (instructions for setting environment variables on PyKX supported operating systems can be found here.
    • Variable Name: QLIC
    • Variable Value: /user/path/to/folder

Note: PyKX will not operate with a vanilla or legacy kdb+ license which does not have access to specific feature flags embedded within the license. In the absence of a license with appropriate feature flags PyKX will fail to initialise with full feature functionality.

Supported Environments

KX only officially supports versions of PyKX built by KX, i.e. versions of PyKX installed from wheel files. Support for user-built installations of PyKX (e.g. built from the source distribution) is only provided on a best-effort basis. Currently, PyKX provides wheels for the following environments:

  • Linux (manylinux_2_17_x86_64) with CPython 3.8-3.11
  • macOS (macosx_10_10_x86_64) with CPython 3.8-3.11
  • Windows (win_amd64) with CPython 3.8-3.11

Dependencies

Python Dependencies

PyKX depends on the following third-party Python packages:

  • pandas>=1.2, <2.0; python_version=='3.8'
  • pandas>=1.2, <=2.2.3; python_version>'3.8'
  • numpy~=1.22, <2.0; python_version<'3.11'
  • numpy~=1.23, <2.0; python_version=='3.11'
  • numpy~=1.26, <2.0; python_version=='3.12'
  • pytz>=2022.1
  • toml~=0.10.2

They are installed automatically by pip when PyKX is installed.

PyKX also has an optional Python dependency of pyarrow>=3.0.0, which can be included by installing the pyarrow extra, e.g. pip install pykx[pyarrow]

When using PyKX with KX Dashboards users will be required to install ast2json~=0.3 this can be installed using the dashboards extra, e.g. pip install pykx[dashboards]

When using PyKX Beta features users will be required to install dill>=0.2.0 this can be installed using the beta extra, e.g. pip install pykx[beta]

When using Streamlit users will be required to install streamlit~=1.28 this can be installed using the streamlit extra, e.g. pip install pykx[streamlit]

Warning: Trying to use the pa conversion methods of pykx.K objects or the pykx.toq.from_arrow method when PyArrow is not installed (or could not be imported without error) will raise a pykx.PyArrowUnavailable exception. pyarrow is supported Python 3.8-3.10 but remains in Beta for Python 3.11.

Optional Non-Python Dependencies

  • libssl for TLS on IPC connections.
  • libpthread on Linux/MacOS when using the PYKX_THREADING environment variable.

Windows Dependencies

To run q or PyKX on Windows, msvcr100.dll must be installed. It is included in the Microsoft Visual C++ 2010 Redistributable.

Building from source

Installing Dependencies

The full list of supported environments is detailed here. Installation of dependencies will vary on different platforms.

apt example:

apt-install python3 python3-venv build-essential python3-dev

yum example:

yum install python3 gcc gcc-c++ python3-devel.x86_64

Windows:

To install the above dependencies, you can run the w64_install.ps1 script as an administrator:

cd pykx
.\w64_install.ps1

Building

Using a Python virtual environment is recommended:

python3 -m venv pykx-dev
source pykx-dev/bin/activate

Build and install PyKX:

cd pykx
pip3 install -U '.[all]'

To run PyKX in licensed mode ensure to follow the steps to receive a Personal Evaluation License

Now you can run/test PyKX:

(pykx-dev) /data/pykx$ python
Python 3.10.6 (main, May 29 2023, 11:10:38) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pykx
>>> pykx.q('1+1')
pykx.LongAtom(pykx.q('2'))

Testing

Contributions to the project must pass a linting check:

pflake8

Contributions to the project must include tests. To run tests:

export PATH="$PATH:/location/of/your/q/l64" # q must be on PATH for tests
export QHOME=/location/of/your/q #q needs QHOME available
python -m pytest -vvv -n 0 --no-cov --junitxml=report.xml

PyKX License access and enablement

This work is dual licensed under Apache 2.0 and the Software License for q.so and users are required to abide by the terms of both licenses in their entirety.

Community Help

If you have any issues or questions you can post them to community.kx.com. Also available on Stack Overflow are the tags pykx and kdb.

Customer Support

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release.See tutorial on generating distribution archives.

Built Distributions

pykx-2.5.5-cp312-cp312-win_amd64.whl (6.8 MB view details)

Uploaded CPython 3.12 Windows x86-64

pykx-2.5.5-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (10.0 MB view details)

Uploaded CPython 3.12 manylinux: glibc 2.17+ x86-64

pykx-2.5.5-cp312-cp312-manylinux2014_aarch64.whl (9.8 MB view details)

Uploaded CPython 3.12

pykx-2.5.5-cp311-cp311-win_amd64.whl (6.9 MB view details)

Uploaded CPython 3.11 Windows x86-64

pykx-2.5.5-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (10.6 MB view details)

Uploaded CPython 3.11 manylinux: glibc 2.17+ x86-64

pykx-2.5.5-cp311-cp311-manylinux2014_aarch64.whl (10.5 MB view details)

Uploaded CPython 3.11

pykx-2.5.5-cp310-cp310-win_amd64.whl (6.9 MB view details)

Uploaded CPython 3.10 Windows x86-64

pykx-2.5.5-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (9.7 MB view details)

Uploaded CPython 3.10 manylinux: glibc 2.17+ x86-64

pykx-2.5.5-cp310-cp310-manylinux2014_aarch64.whl (9.6 MB view details)

Uploaded CPython 3.10

pykx-2.5.5-cp39-cp39-win_amd64.whl (6.9 MB view details)

Uploaded CPython 3.9 Windows x86-64

pykx-2.5.5-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (9.7 MB view details)

Uploaded CPython 3.9 manylinux: glibc 2.17+ x86-64

pykx-2.5.5-cp39-cp39-manylinux2014_aarch64.whl (9.6 MB view details)

Uploaded CPython 3.9

pykx-2.5.5-cp38-cp38-win_amd64.whl (6.9 MB view details)

Uploaded CPython 3.8 Windows x86-64

pykx-2.5.5-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (9.8 MB view details)

Uploaded CPython 3.8 manylinux: glibc 2.17+ x86-64

pykx-2.5.5-cp38-cp38-manylinux2014_aarch64.whl (9.7 MB view details)

Uploaded CPython 3.8

File details

Details for the file pykx-2.5.5-cp312-cp312-win_amd64.whl.

File metadata

  • Download URL: pykx-2.5.5-cp312-cp312-win_amd64.whl
  • Upload date:
  • Size: 6.8 MB
  • Tags: CPython 3.12, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.1.1 CPython/3.10.15

File hashes

Hashes for pykx-2.5.5-cp312-cp312-win_amd64.whl
Algorithm Hash digest
SHA256 3a9667af2d20a1f62811fb8908f37af7779dbf16a7f4541ad9de5798738e8be6
MD5 31d1fb5bf7f4b6e742d1f8b0e24ba4ab
BLAKE2b-256 fc10007c074bae6934b4a825909ba2614dd6bf599aba80b3e6202220fbe528dd

See more details on using hashes here.

File details

Details for the file pykx-2.5.5-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for pykx-2.5.5-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 0f1abd1f277f203f09284fa556aef5adec575041b1f5c7162cd38254d93168a5
MD5 e92d82a662857597857249be77d6be9b
BLAKE2b-256 5c1107c3a7942844c04e12db6ff0f9b4dcfcf542bbbd9d1bde5ffcb7e01ff394

See more details on using hashes here.

File details

Details for the file pykx-2.5.5-cp312-cp312-manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for pykx-2.5.5-cp312-cp312-manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 8369927530c5b01a55d0695c9f9ca48ac9c56d83d09596df550ac8860e506a00
MD5 0db1b62a3e3f8f0f233061c3798a2e87
BLAKE2b-256 e8e75be714dca1eacc77bb3b84096e2094131c83d6ec552f633fd631321b68d3

See more details on using hashes here.

File details

Details for the file pykx-2.5.5-cp311-cp311-win_amd64.whl.

File metadata

  • Download URL: pykx-2.5.5-cp311-cp311-win_amd64.whl
  • Upload date:
  • Size: 6.9 MB
  • Tags: CPython 3.11, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.1.1 CPython/3.10.15

File hashes

Hashes for pykx-2.5.5-cp311-cp311-win_amd64.whl
Algorithm Hash digest
SHA256 c3bb222b6cab3d2dc84eb9f7ba1ac263bd884853303da85524aa95e7b3ff94ab
MD5 6b90c5895c91687b25f7b59fc5c8fa3f
BLAKE2b-256 7292db729bfe9083adb58b92f71f7c65391afc69085f54520df3711205b969ff

See more details on using hashes here.

File details

Details for the file pykx-2.5.5-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for pykx-2.5.5-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 a3837509188d3b8e6759bb1eb70bd14316c0b7491575f1946ed155b4db410666
MD5 1c66e00ce83e4b08b2fd613eb3dbf17b
BLAKE2b-256 adc37c6893ac9bb6380f0184ff79f87ad94450a4ce8ee0dfc837b4efa8957644

See more details on using hashes here.

File details

Details for the file pykx-2.5.5-cp311-cp311-manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for pykx-2.5.5-cp311-cp311-manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 2e5e251da7e3313c9173dc29395ba334cfe397c95d46cdc7839ba42fc9840447
MD5 07cb714af0dfa0d6af090f89551a98d6
BLAKE2b-256 2bb3590264727df3724e07332faf38ee605d2b7f63cbb2e094bdeddc2d1ef610

See more details on using hashes here.

File details

Details for the file pykx-2.5.5-cp310-cp310-win_amd64.whl.

File metadata

  • Download URL: pykx-2.5.5-cp310-cp310-win_amd64.whl
  • Upload date:
  • Size: 6.9 MB
  • Tags: CPython 3.10, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.1.1 CPython/3.10.15

File hashes

Hashes for pykx-2.5.5-cp310-cp310-win_amd64.whl
Algorithm Hash digest
SHA256 2e8ee7adfa3729fbb0cfb2ccd4928bd4744880496e58376175cf261df3121019
MD5 4c131456883c35099af24b3aa8364a3f
BLAKE2b-256 22703f5cf01562b01d97f9d868c83608753645636d2c6d7a1c91b6899b976b2b

See more details on using hashes here.

File details

Details for the file pykx-2.5.5-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for pykx-2.5.5-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 efc4c95503d563a7c868851b22a727c8a0317caef9a69c8a5123a9dcd6ed2c0a
MD5 0d07b9825ce3dc5de72216a6367056ae
BLAKE2b-256 eaf7e705f150e2ff19fe60ba0e3988c587891990b34f2629b7bf8548f693ab88

See more details on using hashes here.

File details

Details for the file pykx-2.5.5-cp310-cp310-manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for pykx-2.5.5-cp310-cp310-manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 2b8c3f577321198e8f7a02039d167699c3dc47b83d5cc1e9024f9131af13fedf
MD5 739280802cf39f6484d9f6847b345b6d
BLAKE2b-256 902bcd3b9d5fe3538683d1bf8582fc3241d7bad018a71535fbb3722a47974a70

See more details on using hashes here.

File details

Details for the file pykx-2.5.5-cp39-cp39-win_amd64.whl.

File metadata

  • Download URL: pykx-2.5.5-cp39-cp39-win_amd64.whl
  • Upload date:
  • Size: 6.9 MB
  • Tags: CPython 3.9, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.1.1 CPython/3.10.15

File hashes

Hashes for pykx-2.5.5-cp39-cp39-win_amd64.whl
Algorithm Hash digest
SHA256 b23c9dd1ea8790f0772635436bba7e5fabd4d05c3cd3ac6f949c8d76213ae349
MD5 bc3143be060646f23eb82dc35cabd03e
BLAKE2b-256 8031b8f40df6690b0f55c479704ed58902e107eaff4b554eeef052617b794c4f

See more details on using hashes here.

File details

Details for the file pykx-2.5.5-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for pykx-2.5.5-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 9bf15a7bd7ecdd589824ab0b03d94ec2a14788c344d2fd3e18f9dd1856398aea
MD5 5ed0076a4b2bd87da64ba674157f3531
BLAKE2b-256 4672d07ba9ad9ed070e2f0978cb1b87f206f1f057b1953a0afc1cd69388c0f38

See more details on using hashes here.

File details

Details for the file pykx-2.5.5-cp39-cp39-manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for pykx-2.5.5-cp39-cp39-manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 18db8b5b95ea6d04ba891dc47086041036bdab6e9c043b6cb700293feddfe8db
MD5 78bd8cbf744c2afabc0f03a1c277339f
BLAKE2b-256 b850151617232e577ea11ea5c0473de680c670ec51e843084806fd33ef17bf49

See more details on using hashes here.

File details

Details for the file pykx-2.5.5-cp38-cp38-win_amd64.whl.

File metadata

  • Download URL: pykx-2.5.5-cp38-cp38-win_amd64.whl
  • Upload date:
  • Size: 6.9 MB
  • Tags: CPython 3.8, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.1.1 CPython/3.10.15

File hashes

Hashes for pykx-2.5.5-cp38-cp38-win_amd64.whl
Algorithm Hash digest
SHA256 0fe510b247e3bdcb95f3cb900c19dbc123f22e74cfe74289e55eb206f1ba11d1
MD5 b9be39dfe6c0e55d2aecaa662e0f95e7
BLAKE2b-256 d9b1e63c6adbae1e55ba703295500eedd685356a0bfe4910021d1eb1b14f83bd

See more details on using hashes here.

File details

Details for the file pykx-2.5.5-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for pykx-2.5.5-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 b6ad9e3112fba8c954a897db667d5fd7f7be731c5bb5eb3f9d658fbe619fb10d
MD5 b9604ace519942158a519c8c7342df03
BLAKE2b-256 bfbbc99f9764ee35a8f82877b385aa533feadad2cdac43e773ff6b4dd55974f9

See more details on using hashes here.

File details

Details for the file pykx-2.5.5-cp38-cp38-manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for pykx-2.5.5-cp38-cp38-manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 f2d20d80eb7ca094b062b93bf664542164584354a21f44feeb448dd1777325b0
MD5 cc06bbfdee6e730f49a7368727e146b6
BLAKE2b-256 037a3d48b0593b393efd821bd96d899182d8d62298ddd8e528fd605f3a0fa29e

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page