Skip to main content

KORE FileFormat — Python

Version 1.7.30 | PyPI | GitHub

World's fastest human-readable columnar format. .kore v3 opens in Notepad AND reads 12x faster than CSV.

Install

pip install kore-fileformat

.kore v3 — One Format, Everything

KORE2 offset=0000000455      ← jump straight to data
# KORE Format v3.0
# Rows: 100,000  Columns: 3
# Compressed: 28,500 bytes (Rust ZSTD/LZ4)
# Schema:
#   price                F64
#   qty                  I64
# Preview (first 5 rows):
#   [price=10.5 | qty=100]
[binary compressed data — 10x smaller than JSON]

Quick Start

import kore_fileformat as kore

# Write — human-readable header + compressed binary
block = kore.DataBlock()
block.add_column('price', kore.DataType.F64, [10.5, 20.0, 30.75])
block.add_column('qty',   kore.DataType.I64, [100,  200,  300])
kore.write_file('data.kore', block)

# Read — returns array.array (no Python object overhead)
result = kore.read_file('data.kore')
print(result.num_rows, result.num_columns)

# Inspect without loading data
kore.inspect_kore('data.kore')           # prints header
header = kore.kore_header('data.kore')   # returns string
stats  = kore.kore_stats('data.kore')    # {'total_kb', 'overhead_pct', ...}

CLI (installed automatically)

kore inspect data.kore            # show schema + preview (no full read)
kore stats   data.kore            # file size breakdown
kore convert src.kore dst.hkore   # convert formats
kore bench                        # write/read speed benchmark
kore version                      # version string

Benchmark

Format Read Write Size
KORE .kore 79 ns/row 255 ns/row 305 KB
KORE .hkore 28 ns/row 154 ns/row 3,126 KB
JSON 1,096 ns/row 9,576 ns/row 6,786 KB
CSV 1,252 ns/row 3,447 ns/row 3,368 KB
SQLite 1,258 ns/row 1,256 ns/row 3,180 KB

(100K rows × 4 cols, warm OS cache)

API Reference

Function Description
write_file(path, block) Write .kore v3 (compressed + human header)
read_file(path) Read .kore → DataBlock (returns array.array)
write_hybrid(path, block) Write .hkore (raw binary, 28 ns/row read)
read_hybrid(path) Read .hkore → DataBlock
inspect_kore(path) Print text header (no data load)
kore_header(path) Get text header as string
kore_stats(path) Dict: total_kb, header_kb, binary_kb, overhead_pct
DataBlock() Create empty block
block.add_column(name, dtype, data) Add column
block.get_column(name) Get column by name

Data Types

kore.DataType.F64   # 64-bit float
kore.DataType.I64   # 64-bit integer
kore.DataType.STR   # UTF-8 string
kore.DataType.BOOL  # Boolean

Install

pip install kore-fileformat==1.7.30

Or from source (requires Rust):

cargo build --release -p kore-ffi
pip install -e .

Quick Start

import kore_fileformat as kore

# --- Write ---
block = kore.DataBlock()
block.add_column('price',    kore.DataType.F64, [10.5, 20.0, 30.75])
block.add_column('quantity', kore.DataType.I64, [100,  200,  300])
kore.write_file('data.kore', block)

# --- Read ---
result = kore.read_file('data.kore')
print(f'{result.num_rows} rows, {result.num_columns} columns')
price_col = result.get_column('price')
print(price_col.data)   # [10.5, 20.0, 30.75]

# --- CRC32 checksum ---
checksum = kore.crc32(b'hello kore')
print(f'crc32 = {checksum:#010x}')   # 0x4b029b4b

API Reference

Function Description
write_file(path, block) Write DataBlock to .kore binary
read_file(path) Read .kore binary into DataBlock
crc32(data: bytes) CRC32 checksum
DataBlock() Create empty block
block.add_column(name, dtype, data) Add a column
block.get_column(name) Get column by name

Data Types

kore.DataType.I64       # 64-bit integer
kore.DataType.F64       # 64-bit float
kore.DataType.STR       # UTF-8 string
kore.DataType.STR_DICT  # Dictionary-encoded string (compressed)
kore.DataType.BOOL      # Boolean

Run Tests

python -m pytest test_kore_fileformat.py -v
python test_phase3.py

Ecosystem Integration

Kore works with every major data tool through built-in bridges. Install the optional dependency for the tool you need.

Apache Arrow

pip install kore-fileformat pyarrow
import kore_fileformat as kore

# Kore → Arrow Table
table = kore.to_arrow("data.kore")

# Arrow Table → Kore
import pyarrow as pa
table = pa.table({"price": [10.5, 20.0], "qty": [100, 200]})
kore.from_arrow("output.kore", table)

Pandas

pip install kore-fileformat pandas
import kore_fileformat as kore

df = kore.to_pandas("data.kore")           # Kore → DataFrame
kore.from_pandas("output.kore", df)        # DataFrame → Kore

Polars

pip install kore-fileformat polars pyarrow
import kore_fileformat as kore

df = kore.to_polars("data.kore")           # Kore → Polars DataFrame
kore.from_polars("output.kore", df)        # Polars → Kore

DuckDB

pip install kore-fileformat duckdb pyarrow
import kore_fileformat as kore

conn = kore.to_duckdb("data.kore", "sales")
result = conn.execute("SELECT SUM(price) FROM sales").fetchall()

Apache Spark

pip install kore-fileformat pyspark pyarrow
import kore_fileformat as kore
from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("kore").getOrCreate()
df = kore.to_spark(spark, "data.kore")     # Kore → Spark DataFrame
df.createOrReplaceTempView("sales")
spark.sql("SELECT region, SUM(amount) FROM sales GROUP BY region").show()

kore.from_spark("output.kore", df)         # Spark → Kore

Parquet (import/export)

pip install kore-fileformat pyarrow
import kore_fileformat as kore

kore.to_parquet("data.kore", "data.parquet")     # Kore → Parquet
kore.from_parquet("output.kore", "data.parquet") # Parquet → Kore

NumPy

import kore_fileformat as kore

arrays = kore.to_numpy(block)                    # Kore → dict of ndarrays
kore.from_numpy("output.kore", {"x": np_arr})   # ndarrays → Kore

Kafka Streaming

import kore_fileformat as kore

msg = kore.to_kafka_message(block)     # serialize for Kafka producer
block = kore.from_kafka_message(msg)   # deserialize from Kafka consumer

MongoDB

import kore_fileformat as kore

docs = kore.to_mongodb_docs(block)            # Kore → list of dicts
block = kore.from_mongodb_docs(docs)          # list of dicts → Kore

All Interop Functions

Function Direction Requires
to_arrow(path_or_block) Kore → PyArrow Table pyarrow
from_arrow(path, table) PyArrow Table → Kore pyarrow
to_pandas(path) Kore → DataFrame pandas
from_pandas(path, df) DataFrame → Kore pandas
to_polars(path) Kore → Polars DF polars, pyarrow
from_polars(path, df) Polars DF → Kore polars, pyarrow
to_duckdb(path, table, conn) Kore → DuckDB table duckdb, pyarrow
to_spark(spark, path) Kore → Spark DF pyspark, pyarrow
from_spark(path, df) Spark DF → Kore pyspark
to_parquet(path, out) Kore → Parquet pyarrow
from_parquet(kore, parquet) Parquet → Kore pyarrow
to_numpy(block) Kore → NumPy arrays numpy
from_numpy(path, arrays) NumPy → Kore numpy
to_kafka_message(block) Kore → bytes —
from_kafka_message(msg) bytes → Kore —
to_mongodb_docs(block) Kore → list[dict] —
from_mongodb_docs(docs) list[dict] → Kore —

Metadata

Release files for kore-fileformat 1.8.13

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for kore-fileformat 1.8.13
File Size Uploaded Provenance Actions
kore_fileformat-1.8.13.tar.gz 40.9 kB Without provenance Details for kore_fileformat-1.8.13.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for kore-fileformat 1.8.13
File Interpreter ABI Platform Actions
kore_fileformat-1.8.13-py3-none-any.whl Python 3 none any Details for kore_fileformat-1.8.13-py3-none-any.whl

Total release size: 77.9 kB

Release files / kore_fileformat-1.8.13.tar.gz

Download URL kore_fileformat-1.8.13.tar.gz
Size 40.9 kB
Tags Source
SHA-256 checksum
How to use checksums
915e5918c2f9a91d2b4436fb04ff1f76b7273faa99476df5d22f728b68e7ab90
BLAKE2b-256 checksum
How to use checksums
5f1dab71a6242694e2c3a5afeef810bdcadff283cbb7313f5c70b6844dd463f0
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.14

Release files / kore_fileformat-1.8.13-py3-none-any.whl

Download URL kore_fileformat-1.8.13-py3-none-any.whl
Size 37.0 kB
Tags Python 3
SHA-256 checksum
How to use checksums
4fbd17c80da33a372b245bdaffe40462499bd1fa7f036e1ec1063d24cef88886
BLAKE2b-256 checksum
How to use checksums
81fc45e533d9688199ac271eb04c4ac381b5968c934583f8b7779b7f575af5be
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.14

Security for kore-fileformat 1.8.13

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

1.8.13 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page