a Python Polars interface to kdb+/q
Project description
kola
a Python Polars Interface to kdb+/q
Basic Data Type Map
Deserialization
Atom
k type | n | size | python type | note |
---|---|---|---|---|
boolean |
1 | 1 | bool |
|
guid |
2 | 16 | str |
|
byte |
4 | 1 | int |
|
short |
5 | 2 | int |
|
int |
6 | 4 | int |
|
long |
7 | 8 | int |
|
real |
8 | 4 | float |
|
float |
9 | 8 | float |
|
char |
10 | 1 | str |
|
string |
10 | 1 | str |
|
symbol |
11 | * | str |
|
timestamp |
12 | 8 | datetime |
|
month |
13 | 4 | - |
|
date |
14 | 4 | date |
0001.01.01 - 9999.12.31 |
datetime |
15 | 8 | datetime |
|
timespan |
16 | 8 | timedelta |
|
minute |
17 | 4 | time |
00:00 - 23:59 |
second |
18 | 4 | time |
00:00:00 - 23:59:59 |
time |
19 | 4 | time |
00:00:00.000 - 23:59:59.999 |
Composite Data Type
k type | n | size | python type |
---|---|---|---|
boolean list |
1 | 1 | pl.Boolean |
guid list |
2 | 16 | pl.List(pl.Binary(16)) |
byte list |
4 | 1 | pl.Uint8 |
short list |
5 | 2 | pl.Int16 |
int list |
6 | 4 | pl.Int32 |
long list |
7 | 8 | pl.Int64 |
real list |
8 | 4 | pl.Float32 |
float list |
9 | 8 | pl.Float64 |
char list |
10 | 1 | pl.Utf8 |
string list |
10 | 1 | pl.Utf8 |
symbol list |
11 | * | pl.Categorical |
timestamp list |
12 | 8 | pl.Datetime |
month list |
13 | 4 | - |
date list |
14 | 4 | pl.Date |
datetime list |
15 | 8 | pl.Datetime |
timespan list |
16 | 8 | pl.Duration |
minute list |
17 | 4 | pl.Time |
second list |
18 | 4 | pl.Time |
time list |
19 | 4 | pl.Time |
table |
98 | * | pl.DataFrame |
dictionary |
99 | * | - |
keyed table |
99 | * | pl.DataFrame |
performance is impacted by converting guid to string, deserialize the uuid to 16 fixed binary list, use .hex() to convert binary to string if required
real/float 0n is mapped to Polars null not NaN
short/int/long 0Nh/i/j, 0Wh/i/j and -0Wh/i/j are mapped to null
df.with_columns([
(pl.col("uuid").apply(lambda u: u.hex()))
])
Serialization
Basic Data Type
python type | k type | note |
---|---|---|
bool |
boolean |
|
int |
long |
|
float |
float |
|
str |
symbol |
|
bytes |
string |
|
datetime |
timestamp |
|
date |
date |
0001.01.01 - 9999.12.31 |
datetime |
datetime |
|
timedelta |
timespan |
|
time |
time |
00:00:00.000 - 23:59:59.999 |
Dictionary, Series and DataFrame
python type | k type |
---|---|
dict |
dict |
pl.Boolean |
boolean |
pl.List(pl.Binary(16)) |
guid |
pl.Uint8 |
byte |
pl.Int16 |
short |
pl.Int32 |
int |
pl.Int64 |
long |
pl.Float32 |
real |
pl.Float64 |
float |
pl.Utf8 |
char |
pl.Categorical |
symbol |
pl.Datetime |
timestamp |
pl.Date |
date |
pl.Datetime |
datetime |
pl.Duration |
timespan |
pl.Time |
time |
pl.DataFrame |
table |
Limited Support for dictionary as arguments, requires
string
as keys.
Quick Start
Create a Connection
import polars as pl
import kola
q = kola.Q('localhost', 1800)
# with retries for IO Errors, 1s, 2s, 4s ...
q = kola.Q('localhost', 1800, retries=3)
# with read timeout error, 2s, "Resource temporarily unavailable"
q = kola.Q('localhost', 1800, retries=3, timeout=2)
Connect(Optional)
Automatically connect when querying q process
q.connect()
Disconnect
Automatically disconnect if any IO error
q.disconnect()
String Query
q.sync("select from trade where date=last date")
Lambda Query
When the first string starts with {
and ends with }
, it is treated as a lambda.
d = {"a": 1, "b": 2}
q.sync("{key x}", d)
Functional Query
For functional query, kola
supports Python Basic Data Type, pl.Series
, pl.DataFrame
and Python Dictionary with string keys and Python Basic Data Type and pl.Series
values.
from datetime import date, time
q.sync(
".gw.query",
"table",
{
"date": date(2023, 11, 21),
"syms": pl.Series("", ["sym0", "sym1"], pl.Categorical),
# 09:00
"startTime": time(9),
# 11:30
"endTime": time(11, 30),
},
)
Send DataFrame
# pl_df is a Polars DataFrame
q.sync("upsert", "table", pl_df)
# pd_df is a Pandas DataFrame, use pl.DateFrame to cast Pandas DataFrame
q.sync("upsert", "table", pl.DataFrame(pd_df))
Async Query
# pl_df is a Polars DataFrame
q.asyn("upsert", "table", pl_df)
Subscribe
from kola import QType
q.sync(".u.sub", pl.Series("", ["table1", "table2"], QType.Symbol), "")
# specify symbol filter
q.sync(
".u.sub",
pl.Series("", ["table1", "table2"], QType.Symbol),
pl.Series("", ["sym1", "sym2"], QType.Symbol),
)
while true:
# ("upd", "table", pl.Dataframe)
upd = self.q.receive()
print(upd)
Generate IPC
import polars as pl
from kola import generate_ipc
df = pl.DataFrame(
{
"sym": pl.Series("sym", ["a", "b", "c"], pl.Categorical),
"price": [1, 2, 3],
}
)
# without compression
buffer = generate_ipc("sync", False, ["upd", "table", df])
# with compression
buffer = generate_ipc("sync", True, ["upd", "table", df])
Polars Documentations
Refer to
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distributions
Hashes for kola-1.5.0-cp312-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9a99f6530709be3e4c71179684ddb8c6a5ec571ac4d905f945fb55a36aebb025 |
|
MD5 | afa4dba4ed9a09d597449750709e86d9 |
|
BLAKE2b-256 | d39185f196c7d17451b3fae5d949b76c36d34931a0f40dd4a287e21b69ad04dd |
Hashes for kola-1.5.0-cp312-cp312-manylinux_2_34_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 93f3e68377a0e1606400760d2e9864e71f9993c93bb8ec632d558625b32fcadf |
|
MD5 | bcf86b2540a8ab997df10152581bb73e |
|
BLAKE2b-256 | 20f7e2fd5547829cc22bbd8cd9ce2c35dde9a356c9b7f353aa1e8ac382a4b592 |
Hashes for kola-1.5.0-cp312-cp312-manylinux_2_31_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5806bb87d0db65e8a3ed86c6f716a8aa82ad9e11707f71ef7e0f619acae5a89c |
|
MD5 | 42d64c32673f91dc606bbd1d10078351 |
|
BLAKE2b-256 | e81acd888501b98633bdb4661052fa469914256cac5ef2870b3018693ff9de6f |
Hashes for kola-1.5.0-cp312-cp312-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d4120afb9524c809a88b0e119de551fa3a5718f1cb9e4ecbc7769a2ed8a6b228 |
|
MD5 | dcda70efc8467c8feb716b0d441f0528 |
|
BLAKE2b-256 | d921a75d13cfa735a11032fd985a2c75c074b8ffb7dcb3f77e70e0d9d637cac5 |
Hashes for kola-1.5.0-cp311-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 96666c6104bdb09d42974307844d87df7316f7bef133466639b9ec6a62027c7b |
|
MD5 | 55c6cf4b0ff76c802cc119d0def1ed1d |
|
BLAKE2b-256 | fd94230556a45cd59c6cfa520864cec53e1b5a0cf4ec726a137050f8da6039a6 |
Hashes for kola-1.5.0-cp311-cp311-manylinux_2_34_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e544b7d2aba2442fdad6ac8ce5d27e46c365b86ae6c4ff617443732ec2f5fa81 |
|
MD5 | a55ae7c14791679376cb12a8478e726d |
|
BLAKE2b-256 | 2477ec52fe704213c9f314c97e272fa99cfa8fd0f80f2b1c412adf2f129377a3 |
Hashes for kola-1.5.0-cp311-cp311-manylinux_2_31_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 47722b827088d3ecaa1f670ec53b37150ca73664d74fd81f1b98156816f6c3cf |
|
MD5 | 97a7935a64ead0ad724e2980f61ea046 |
|
BLAKE2b-256 | abecde4a87f20eacc1f1601b410bf21757c39fc6931d0600fa09ac48864bab95 |
Hashes for kola-1.5.0-cp311-cp311-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0aa89302a64ae0281e76cc72d81b46f2ddb2cbc49f85b837e033b5ad7bdf5399 |
|
MD5 | 1384a8ae654c744b7a864e0891a6e297 |
|
BLAKE2b-256 | c90f48ba0f1258d43093b9cf53c0c052491d579a8278c8185a722f9285d72210 |
Hashes for kola-1.5.0-cp310-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 08c82d7a866f65c4cc6cf13d2971ad0c4616f40a6598e2f77425f4e9552855d3 |
|
MD5 | 775e8c7af4fe53e95b6f9ef4feb9ca3f |
|
BLAKE2b-256 | 2ab6f9588ad8671fc9f3f504bc3ac2b0fd51d77a950f629bd2d43057da2865ec |
Hashes for kola-1.5.0-cp310-cp310-manylinux_2_31_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b17835f1a52064a44839f4eec95867c2b3e30cc7272977280753b54db83dbdfa |
|
MD5 | 865fc3dbe40b7e75b69004e131aa78cb |
|
BLAKE2b-256 | 54b84345eb5d4caa92baeb761ebf252efea88bfb3bad945f9e9828fd34d30aae |
Hashes for kola-1.5.0-cp310-cp310-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ac72a36745d4d94aa601a1c39f5426d987997697c1b5987b189bdfc30432774e |
|
MD5 | aea5222e83bef10fa0ef6abaac611022 |
|
BLAKE2b-256 | ec7b76bf74e61613a346257a9b9b8a5a01367bfe2377e6c60d923330fdbb054f |
Hashes for kola-1.5.0-cp39-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a70a47f26fe0adffa55909556c46ff32de78d71ca63bceae5221a4ba8a83519b |
|
MD5 | 95f039797c61687e8b64c95bf35438e3 |
|
BLAKE2b-256 | b76669d263c9a7d5bfc3baade0f17068494bcd6762e5e054dd13aedaa8bb0e2f |
Hashes for kola-1.5.0-cp39-cp39-manylinux_2_31_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | bc439356b1f62780abbbbc690944fd6073365bcf520430c81d5b407faf24c27f |
|
MD5 | 71c6f9740ce53b86f340fe9cbff48bc0 |
|
BLAKE2b-256 | 65b673f6cdf5697038784026918234c4c05f2c7e47c2902aa88cb12129b6b9ee |
Hashes for kola-1.5.0-cp39-cp39-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 23ab3090be478b6ce2628b904b1962a589a3bc1397aef18d40687aaf8ed499ad |
|
MD5 | 01125c9e81ac5194cfd6910ceb1777d1 |
|
BLAKE2b-256 | e85046b6d279fd5ca8b87cd7de59ca37414c74da4871ecbd26441bc50650fb9d |
Hashes for kola-1.5.0-cp38-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 67c0d75ee0b99abc744d895be8aae846f5b66563ae6ae0e787841d1bed8ed463 |
|
MD5 | 8afe75ada8888dc22db4be476a8102d5 |
|
BLAKE2b-256 | 1d0e0ec8bc46581568070f140de05ea2d815356acad7c95984a881d7354fec8d |
Hashes for kola-1.5.0-cp38-cp38-manylinux_2_31_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5a1ce9cf600e882f3e14e3e31e898f758d67560b38a0242da42daa5427090dde |
|
MD5 | fb4f80ce70782db91f2558d27c1a8dd1 |
|
BLAKE2b-256 | cafb479de206b3414f8b5c3d52e318c44e4f5e0bc3f3c31b560cbcef4cf7b51b |
Hashes for kola-1.5.0-cp38-cp38-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a3454a8590c9a20bc6a4ce73a5b0d25b84b30afbb03308db9eae64087892da8a |
|
MD5 | 9961ac58706a5b30b7a86278d6744a04 |
|
BLAKE2b-256 | 42a84257c3661fd16f2c56cb182a0b3e3ba93afb467b081989a63933b610e67a |