Skip to main content

KORE FileFormat — Python

Version 1.7.30 | PyPI | GitHub

World's fastest human-readable columnar format. .kore v3 opens in Notepad AND reads 12x faster than CSV.

Install

pip install kore-fileformat

.kore v3 — One Format, Everything

KORE2 offset=0000000455      ← jump straight to data
# KORE Format v3.0
# Rows: 100,000  Columns: 3
# Compressed: 28,500 bytes (Rust ZSTD/LZ4)
# Schema:
#   price                F64
#   qty                  I64
# Preview (first 5 rows):
#   [price=10.5 | qty=100]
[binary compressed data — 10x smaller than JSON]

Quick Start

import kore_fileformat as kore

# Write — human-readable header + compressed binary
block = kore.DataBlock()
block.add_column('price', kore.DataType.F64, [10.5, 20.0, 30.75])
block.add_column('qty',   kore.DataType.I64, [100,  200,  300])
kore.write_file('data.kore', block)

# Read — returns array.array (no Python object overhead)
result = kore.read_file('data.kore')
print(result.num_rows, result.num_columns)

# Inspect without loading data
kore.inspect_kore('data.kore')           # prints header
header = kore.kore_header('data.kore')   # returns string
stats  = kore.kore_stats('data.kore')    # {'total_kb', 'overhead_pct', ...}

CLI (installed automatically)

kore inspect data.kore            # show schema + preview (no full read)
kore stats   data.kore            # file size breakdown
kore convert src.kore dst.hkore   # convert formats
kore bench                        # write/read speed benchmark
kore version                      # version string

Benchmark

Format Read Write Size
KORE .kore 79 ns/row 255 ns/row 305 KB
KORE .hkore 28 ns/row 154 ns/row 3,126 KB
JSON 1,096 ns/row 9,576 ns/row 6,786 KB
CSV 1,252 ns/row 3,447 ns/row 3,368 KB
SQLite 1,258 ns/row 1,256 ns/row 3,180 KB

(100K rows × 4 cols, warm OS cache)

API Reference

Function Description
write_file(path, block) Write .kore v3 (compressed + human header)
read_file(path) Read .kore → DataBlock (returns array.array)
write_hybrid(path, block) Write .hkore (raw binary, 28 ns/row read)
read_hybrid(path) Read .hkore → DataBlock
inspect_kore(path) Print text header (no data load)
kore_header(path) Get text header as string
kore_stats(path) Dict: total_kb, header_kb, binary_kb, overhead_pct
DataBlock() Create empty block
block.add_column(name, dtype, data) Add column
block.get_column(name) Get column by name

Data Types

kore.DataType.F64   # 64-bit float
kore.DataType.I64   # 64-bit integer
kore.DataType.STR   # UTF-8 string
kore.DataType.BOOL  # Boolean

Install

pip install kore-fileformat==1.7.30

Or from source (requires Rust):

cargo build --release -p kore-ffi
pip install -e .

Quick Start

import kore_fileformat as kore

# --- Write ---
block = kore.DataBlock()
block.add_column('price',    kore.DataType.F64, [10.5, 20.0, 30.75])
block.add_column('quantity', kore.DataType.I64, [100,  200,  300])
kore.write_file('data.kore', block)

# --- Read ---
result = kore.read_file('data.kore')
print(f'{result.num_rows} rows, {result.num_columns} columns')
price_col = result.get_column('price')
print(price_col.data)   # [10.5, 20.0, 30.75]

# --- CRC32 checksum ---
checksum = kore.crc32(b'hello kore')
print(f'crc32 = {checksum:#010x}')   # 0x4b029b4b

API Reference

Function Description
write_file(path, block) Write DataBlock to .kore binary
read_file(path) Read .kore binary into DataBlock
crc32(data: bytes) CRC32 checksum
DataBlock() Create empty block
block.add_column(name, dtype, data) Add a column
block.get_column(name) Get column by name

Data Types

kore.DataType.I64       # 64-bit integer
kore.DataType.F64       # 64-bit float
kore.DataType.STR       # UTF-8 string
kore.DataType.STR_DICT  # Dictionary-encoded string (compressed)
kore.DataType.BOOL      # Boolean

Run Tests

python -m pytest test_kore_fileformat.py -v
python test_phase3.py

Ecosystem Integration

Kore works with every major data tool through built-in bridges. Install the optional dependency for the tool you need.

Apache Arrow

pip install kore-fileformat pyarrow
import kore_fileformat as kore

# Kore → Arrow Table
table = kore.to_arrow("data.kore")

# Arrow Table → Kore
import pyarrow as pa
table = pa.table({"price": [10.5, 20.0], "qty": [100, 200]})
kore.from_arrow("output.kore", table)

Pandas

pip install kore-fileformat pandas
import kore_fileformat as kore

df = kore.to_pandas("data.kore")           # Kore → DataFrame
kore.from_pandas("output.kore", df)        # DataFrame → Kore

Polars

pip install kore-fileformat polars pyarrow
import kore_fileformat as kore

df = kore.to_polars("data.kore")           # Kore → Polars DataFrame
kore.from_polars("output.kore", df)        # Polars → Kore

DuckDB

pip install kore-fileformat duckdb pyarrow
import kore_fileformat as kore

conn = kore.to_duckdb("data.kore", "sales")
result = conn.execute("SELECT SUM(price) FROM sales").fetchall()

Apache Spark

pip install kore-fileformat pyspark pyarrow
import kore_fileformat as kore
from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("kore").getOrCreate()
df = kore.to_spark(spark, "data.kore")     # Kore → Spark DataFrame
df.createOrReplaceTempView("sales")
spark.sql("SELECT region, SUM(amount) FROM sales GROUP BY region").show()

kore.from_spark("output.kore", df)         # Spark → Kore

Parquet (import/export)

pip install kore-fileformat pyarrow
import kore_fileformat as kore

kore.to_parquet("data.kore", "data.parquet")     # Kore → Parquet
kore.from_parquet("output.kore", "data.parquet") # Parquet → Kore

NumPy

import kore_fileformat as kore

arrays = kore.to_numpy(block)                    # Kore → dict of ndarrays
kore.from_numpy("output.kore", {"x": np_arr})   # ndarrays → Kore

Kafka Streaming

import kore_fileformat as kore

msg = kore.to_kafka_message(block)     # serialize for Kafka producer
block = kore.from_kafka_message(msg)   # deserialize from Kafka consumer

MongoDB

import kore_fileformat as kore

docs = kore.to_mongodb_docs(block)            # Kore → list of dicts
block = kore.from_mongodb_docs(docs)          # list of dicts → Kore

All Interop Functions

Function Direction Requires
to_arrow(path_or_block) Kore → PyArrow Table pyarrow
from_arrow(path, table) PyArrow Table → Kore pyarrow
to_pandas(path) Kore → DataFrame pandas
from_pandas(path, df) DataFrame → Kore pandas
to_polars(path) Kore → Polars DF polars, pyarrow
from_polars(path, df) Polars DF → Kore polars, pyarrow
to_duckdb(path, table, conn) Kore → DuckDB table duckdb, pyarrow
to_spark(spark, path) Kore → Spark DF pyspark, pyarrow
from_spark(path, df) Spark DF → Kore pyspark
to_parquet(path, out) Kore → Parquet pyarrow
from_parquet(kore, parquet) Parquet → Kore pyarrow
to_numpy(block) Kore → NumPy arrays numpy
from_numpy(path, arrays) NumPy → Kore numpy
to_kafka_message(block) Kore → bytes —
from_kafka_message(msg) bytes → Kore —
to_mongodb_docs(block) Kore → list[dict] —
from_mongodb_docs(docs) list[dict] → Kore —

Metadata

Release files for kore-fileformat 1.8.9

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for kore-fileformat 1.8.9
File Size Uploaded Provenance Actions
kore_fileformat-1.8.9.tar.gz 40.0 kB Without provenance Details for kore_fileformat-1.8.9.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for kore-fileformat 1.8.9
File Interpreter ABI Platform Actions
kore_fileformat-1.8.9-py3-none-any.whl Python 3 none any Details for kore_fileformat-1.8.9-py3-none-any.whl

Total release size: 76.0 kB

Release files / kore_fileformat-1.8.9.tar.gz

Download URL kore_fileformat-1.8.9.tar.gz
Size 40.0 kB
Tags Source
SHA-256 checksum
How to use checksums
d4772743eefa08c7a1111e91b2dd34b09be84875cbaaebda2f4744908fea342e
BLAKE2b-256 checksum
How to use checksums
1b7a33dfdfd9f3bfaf9aebace438d73cc22e9963f9e6ca127103d3436cdb732a
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.14

Release files / kore_fileformat-1.8.9-py3-none-any.whl

Download URL kore_fileformat-1.8.9-py3-none-any.whl
Size 36.0 kB
Tags Python 3
SHA-256 checksum
How to use checksums
8613c98863b5af0cda9c7e78fb48be2449c438aba6b062d0748a51addb27c140
BLAKE2b-256 checksum
How to use checksums
928540f66de6cc81af9a2b7e81e9719408be7f50cc8c5d128e3f40f740f70ca9
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.14

Security for kore-fileformat 1.8.9

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

1.8.9 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page