KORE FileFormat — Python
Version 1.7.30 | PyPI | GitHub
World's fastest human-readable columnar format. .kore v3 opens in Notepad AND reads 12x faster than CSV.
Install
pip install kore-fileformat
.kore v3 — One Format, Everything
KORE2 offset=0000000455 ← jump straight to data
# KORE Format v3.0
# Rows: 100,000 Columns: 3
# Compressed: 28,500 bytes (Rust ZSTD/LZ4)
# Schema:
# price F64
# qty I64
# Preview (first 5 rows):
# [price=10.5 | qty=100]
[binary compressed data — 10x smaller than JSON]
Quick Start
import kore_fileformat as kore
# Write — human-readable header + compressed binary
block = kore.DataBlock()
block.add_column('price', kore.DataType.F64, [10.5, 20.0, 30.75])
block.add_column('qty', kore.DataType.I64, [100, 200, 300])
kore.write_file('data.kore', block)
# Read — returns array.array (no Python object overhead)
result = kore.read_file('data.kore')
print(result.num_rows, result.num_columns)
# Inspect without loading data
kore.inspect_kore('data.kore') # prints header
header = kore.kore_header('data.kore') # returns string
stats = kore.kore_stats('data.kore') # {'total_kb', 'overhead_pct', ...}
CLI (installed automatically)
kore inspect data.kore # show schema + preview (no full read)
kore stats data.kore # file size breakdown
kore convert src.kore dst.hkore # convert formats
kore bench # write/read speed benchmark
kore version # version string
Benchmark
| Format | Read | Write | Size |
|---|---|---|---|
| KORE .kore | 79 ns/row | 255 ns/row | 305 KB |
| KORE .hkore | 28 ns/row | 154 ns/row | 3,126 KB |
| JSON | 1,096 ns/row | 9,576 ns/row | 6,786 KB |
| CSV | 1,252 ns/row | 3,447 ns/row | 3,368 KB |
| SQLite | 1,258 ns/row | 1,256 ns/row | 3,180 KB |
(100K rows × 4 cols, warm OS cache)
API Reference
| Function | Description |
|---|---|
write_file(path, block) |
Write .kore v3 (compressed + human header) |
read_file(path) |
Read .kore → DataBlock (returns array.array) |
write_hybrid(path, block) |
Write .hkore (raw binary, 28 ns/row read) |
read_hybrid(path) |
Read .hkore → DataBlock |
inspect_kore(path) |
Print text header (no data load) |
kore_header(path) |
Get text header as string |
kore_stats(path) |
Dict: total_kb, header_kb, binary_kb, overhead_pct |
DataBlock() |
Create empty block |
block.add_column(name, dtype, data) |
Add column |
block.get_column(name) |
Get column by name |
Data Types
kore.DataType.F64 # 64-bit float
kore.DataType.I64 # 64-bit integer
kore.DataType.STR # UTF-8 string
kore.DataType.BOOL # Boolean
Install
pip install kore-fileformat==1.7.30
Or from source (requires Rust):
cargo build --release -p kore-ffi
pip install -e .
Quick Start
import kore_fileformat as kore
# --- Write ---
block = kore.DataBlock()
block.add_column('price', kore.DataType.F64, [10.5, 20.0, 30.75])
block.add_column('quantity', kore.DataType.I64, [100, 200, 300])
kore.write_file('data.kore', block)
# --- Read ---
result = kore.read_file('data.kore')
print(f'{result.num_rows} rows, {result.num_columns} columns')
price_col = result.get_column('price')
print(price_col.data) # [10.5, 20.0, 30.75]
# --- CRC32 checksum ---
checksum = kore.crc32(b'hello kore')
print(f'crc32 = {checksum:#010x}') # 0x4b029b4b
API Reference
| Function | Description |
|---|---|
write_file(path, block) |
Write DataBlock to .kore binary |
read_file(path) |
Read .kore binary into DataBlock |
crc32(data: bytes) |
CRC32 checksum |
DataBlock() |
Create empty block |
block.add_column(name, dtype, data) |
Add a column |
block.get_column(name) |
Get column by name |
Data Types
kore.DataType.I64 # 64-bit integer
kore.DataType.F64 # 64-bit float
kore.DataType.STR # UTF-8 string
kore.DataType.STR_DICT # Dictionary-encoded string (compressed)
kore.DataType.BOOL # Boolean
Run Tests
python -m pytest test_kore_fileformat.py -v
python test_phase3.py
Ecosystem Integration
Kore works with every major data tool through built-in bridges. Install the optional dependency for the tool you need.
Apache Arrow
pip install kore-fileformat pyarrow
import kore_fileformat as kore
# Kore → Arrow Table
table = kore.to_arrow("data.kore")
# Arrow Table → Kore
import pyarrow as pa
table = pa.table({"price": [10.5, 20.0], "qty": [100, 200]})
kore.from_arrow("output.kore", table)
Pandas
pip install kore-fileformat pandas
import kore_fileformat as kore
df = kore.to_pandas("data.kore") # Kore → DataFrame
kore.from_pandas("output.kore", df) # DataFrame → Kore
Polars
pip install kore-fileformat polars pyarrow
import kore_fileformat as kore
df = kore.to_polars("data.kore") # Kore → Polars DataFrame
kore.from_polars("output.kore", df) # Polars → Kore
DuckDB
pip install kore-fileformat duckdb pyarrow
import kore_fileformat as kore
conn = kore.to_duckdb("data.kore", "sales")
result = conn.execute("SELECT SUM(price) FROM sales").fetchall()
Apache Spark
pip install kore-fileformat pyspark pyarrow
import kore_fileformat as kore
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("kore").getOrCreate()
df = kore.to_spark(spark, "data.kore") # Kore → Spark DataFrame
df.createOrReplaceTempView("sales")
spark.sql("SELECT region, SUM(amount) FROM sales GROUP BY region").show()
kore.from_spark("output.kore", df) # Spark → Kore
Parquet (import/export)
pip install kore-fileformat pyarrow
import kore_fileformat as kore
kore.to_parquet("data.kore", "data.parquet") # Kore → Parquet
kore.from_parquet("output.kore", "data.parquet") # Parquet → Kore
NumPy
import kore_fileformat as kore
arrays = kore.to_numpy(block) # Kore → dict of ndarrays
kore.from_numpy("output.kore", {"x": np_arr}) # ndarrays → Kore
Kafka Streaming
import kore_fileformat as kore
msg = kore.to_kafka_message(block) # serialize for Kafka producer
block = kore.from_kafka_message(msg) # deserialize from Kafka consumer
MongoDB
import kore_fileformat as kore
docs = kore.to_mongodb_docs(block) # Kore → list of dicts
block = kore.from_mongodb_docs(docs) # list of dicts → Kore
All Interop Functions
| Function | Direction | Requires |
|---|---|---|
to_arrow(path_or_block) |
Kore → PyArrow Table | pyarrow |
from_arrow(path, table) |
PyArrow Table → Kore | pyarrow |
to_pandas(path) |
Kore → DataFrame | pandas |
from_pandas(path, df) |
DataFrame → Kore | pandas |
to_polars(path) |
Kore → Polars DF | polars, pyarrow |
from_polars(path, df) |
Polars DF → Kore | polars, pyarrow |
to_duckdb(path, table, conn) |
Kore → DuckDB table | duckdb, pyarrow |
to_spark(spark, path) |
Kore → Spark DF | pyspark, pyarrow |
from_spark(path, df) |
Spark DF → Kore | pyspark |
to_parquet(path, out) |
Kore → Parquet | pyarrow |
from_parquet(kore, parquet) |
Parquet → Kore | pyarrow |
to_numpy(block) |
Kore → NumPy arrays | numpy |
from_numpy(path, arrays) |
NumPy → Kore | numpy |
to_kafka_message(block) |
Kore → bytes | — |
from_kafka_message(msg) |
bytes → Kore | — |
to_mongodb_docs(block) |
Kore → list[dict] | — |
from_mongodb_docs(docs) |
list[dict] → Kore | — |
Metadata
Release files for kore-fileformat 1.8.13
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| kore_fileformat-1.8.13.tar.gz | 40.9 kB | Without provenance | Details for kore_fileformat-1.8.13.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| kore_fileformat-1.8.13-py3-none-any.whl | Python 3 | none | any | Details for kore_fileformat-1.8.13-py3-none-any.whl |
Total release size: 77.9 kB
Release files / kore_fileformat-1.8.13.tar.gz
| Download URL | kore_fileformat-1.8.13.tar.gz |
|---|---|
| Size | 40.9 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
915e5918c2f9a91d2b4436fb04ff1f76b7273faa99476df5d22f728b68e7ab90
|
|
BLAKE2b-256 checksum How to use checksums |
5f1dab71a6242694e2c3a5afeef810bdcadff283cbb7313f5c70b6844dd463f0
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.12.14
|
Release files / kore_fileformat-1.8.13-py3-none-any.whl
| Download URL | kore_fileformat-1.8.13-py3-none-any.whl |
|---|---|
| Size | 37.0 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
4fbd17c80da33a372b245bdaffe40462499bd1fa7f036e1ec1063d24cef88886
|
|
BLAKE2b-256 checksum How to use checksums |
81fc45e533d9688199ac271eb04c4ac381b5968c934583f8b7779b7f575af5be
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.12.14
|
Security for kore-fileformat 1.8.13
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.