fsspec-kafka
Kafka topics and partitions through fsspec
Overview
fsspec-kafka registers the kafka:// protocol with fsspec. Topics are directories, and each partition is a directory of fixed offset windows that read as Arrow, Parquet, CSV, or JSON Lines.
import fsspec
import pandas as pd
options = {"bootstrap.servers": "localhost:9092"}
fs = fsspec.filesystem("kafka", **options)
fs.ls("orders/0", detail=False) # ["orders/0/0-10000.arrow", ..., "orders/0/20000-.arrow"]
frame = pd.read_parquet("kafka://orders.parquet", storage_options=options)
Storage options are librdkafka properties, except for these:
| Option | Default | Meaning |
|---|---|---|
timeout |
10.0 |
Seconds to wait for broker requests |
window_size |
10000 |
Offsets per window file |
list_format |
"arrow" |
Extension used when listing windows |
batch_size |
1024 |
Messages per Arrow record batch |
allow_admin |
False |
Lets mkdir create topics and rmdir delete them |
partitions, replication |
-1 |
Settings for created topics; -1 uses the broker default |
topic_config |
{} |
Configuration for created topics |
value_format |
"binary" |
"json", "avro", or "confluent" decodes values with value_schema |
value_schema |
None |
pyarrow.Schema or fsspec-data schema reference |
flatten_value |
False |
Puts JSON fields beside the message columns |
key_column |
None |
Input column for message keys when appending |
partition_column |
None |
Input column for partitions when appending |
timestamp_column |
None |
Input column for timestamps when appending |
headers_column |
None |
Input column for headers, a map, when appending |
Messages are read with topic, partition, offset, timestamp, key, headers, and value columns. With value_format="json", value is a struct of the value_schema fields, which are null for tombstones. flatten_value=True puts the fields beside the other columns instead, which CSV needs because it cannot hold structs:
schema = pa.schema([("id", pa.int64()), ("status", pa.string())])
frame = pd.read_csv("kafka://orders.csv", storage_options={**options, "value_format": "json", "value_schema": schema, "flatten_value": True})
value_schema can also be anything fsspec-data resolves when it is installed, such as a list of {"name": ..., "type": ...} fields or a SchemaRef to a schema file or Confluent Schema Registry subject. value_format="avro" reads bare Avro records with an .avsc reference or an Avro schema dict. value_format="confluent" reads and writes values in Confluent wire format, with JSON Schema or Avro subjects. For Avro subjects, every registered version is fetched so older messages resolve to the referenced version's schema. Appends write the schema ID of the Confluent reference:
from fsspec_data import SchemaRef
schema = SchemaRef("confluent://orders-value/versions/latest", storage_options={"registry_url": "http://localhost:8081"})
fs = fsspec.filesystem("kafka", **options, value_format="confluent", value_schema=schema, flatten_value=True)
Topics are append-only. Opening topic.format or topic/partition.format with mode "ab" produces one message per row of the file when it closes. The mapped columns set the message fields, and the remaining columns are the value: a JSON object with value_format="json", an Avro record with Avro values, otherwise a single binary or string column. CSV and JSON Lines columns are typed from value_schema.
fs = fsspec.filesystem("kafka", **options, value_format="json", value_schema=schema, key_column="order_id")
with fs.open("orders.parquet", "ab") as file:
frame.to_parquet(file)
Options can also come from fsspec configuration, for example FSSPEC_KAFKA='{"bootstrap.servers": "localhost:9092"}'.
On a compacted topic, topic/records holds the latest value of each key, so it works as a key-value store:
state = fsspec.get_mapper("kafka://state/records", **options)
state["user/42"] = b"..."
fs.stream follows a topic or one partition as it grows, returning pyarrow.RecordBatch objects with the same columns as .arrow files. It starts at "end" by default; "beginning", a datetime, or the positions of an earlier stream are also accepted.
with fs.stream("orders", start="beginning") as stream:
for batch in stream: # waits for new messages
handle(batch)
save(stream.positions) # {partition: next offset}, to resume with start=positions
From Rust, register the kafka protocol with an fsspec-rs FilesystemRegistry and open URLs with the same storage options. value_format accepts only binary here, and topic_config.<name> keys set the configuration of created topics; construct KafkaFs with KafkaFsOptions for decoded values.
let mut registry = fsspec_rs::FilesystemRegistry::default();
fsspec_kafka::register(&mut registry);
let options = [("bootstrap.servers".to_string(), "localhost:9092".to_string())].into();
let (fs, path) = registry.url_to_fs("kafka://orders/0", &options)?;
Metadata
Release files for fsspec-kafka 0.2.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| fsspec_kafka-0.2.0.tar.gz | 126.7 kB | Without provenance | Details for fsspec_kafka-0.2.0.tar.gz |
Built distributions (wheels)
| File | Actions Reset | |||
|---|---|---|---|---|
| fsspec_kafka-0.2.0-cp311-abi3-win_amd64.whl | CPython 3.11 | abi3 | Windows x86-64 | Details for fsspec_kafka-0.2.0-cp311-abi3-win_amd64.whl |
| fsspec_kafka-0.2.0-cp311-abi3-manylinux_2_28_x86_64.whl | CPython 3.11 | abi3 | Linux glibc 2.28+ x86-64 | Details for fsspec_kafka-0.2.0-cp311-abi3-manylinux_2_28_x86_64.whl |
| fsspec_kafka-0.2.0-cp311-abi3-manylinux_2_28_aarch64.whl | CPython 3.11 | abi3 | Linux glibc 2.28+ ARM64 | Details for fsspec_kafka-0.2.0-cp311-abi3-manylinux_2_28_aarch64.whl |
| fsspec_kafka-0.2.0-cp311-abi3-macosx_11_0_arm64.whl | CPython 3.11 | abi3 | macOS 11.0+ ARM64 | Details for fsspec_kafka-0.2.0-cp311-abi3-macosx_11_0_arm64.whl |
Total release size: 49.2 MB
Release files / fsspec_kafka-0.2.0.tar.gz
| Download URL | fsspec_kafka-0.2.0.tar.gz |
|---|---|
| Size | 126.7 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
7b9b4c6b054c277cdb41d8adff5221bb2c60b9b45efb960126a98b835bae1a06
|
|
BLAKE2b-256 checksum How to use checksums |
446ba462671a37fb0ccc9638606c1e936e43c80c5839f6e6a58590a79d72e377
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Release files / fsspec_kafka-0.2.0-cp311-abi3-win_amd64.whl
| Download URL | fsspec_kafka-0.2.0-cp311-abi3-win_amd64.whl |
|---|---|
| Size | 11.1 MB |
| Tags | CPython 3.11 Windows x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
7d97dcf24c6d4bb5333e8cac0e50b78a3142fb1c36755e64faec36b3d900562a
|
|
BLAKE2b-256 checksum How to use checksums |
a812efe97cc831bdffd21152b49c3f8a26701dfe22c466ea43136471cd33cc59
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Release files / fsspec_kafka-0.2.0-cp311-abi3-manylinux_2_28_x86_64.whl
| Download URL | fsspec_kafka-0.2.0-cp311-abi3-manylinux_2_28_x86_64.whl |
|---|---|
| Size | 13.5 MB |
| Tags | CPython 3.11 Linux glibc 2.28+ x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
b55cb5f38763e18e066bcd452fdd1f4bfc2186103121a43be8a464fa894c2cfd
|
|
BLAKE2b-256 checksum How to use checksums |
d54c6773e64c009eceb13733ec32da6c9f0abaf99c2b5e230d06a41d5f6a0b66
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Release files / fsspec_kafka-0.2.0-cp311-abi3-manylinux_2_28_aarch64.whl
| Download URL | fsspec_kafka-0.2.0-cp311-abi3-manylinux_2_28_aarch64.whl |
|---|---|
| Size | 13.4 MB |
| Tags | CPython 3.11 Linux glibc 2.28+ ARM64 abi3 |
|
SHA-256 checksum How to use checksums |
4e19af0c9a26fc91767bc9d34b01a002117618d7605694aaa57f3bcd3fbc8d04
|
|
BLAKE2b-256 checksum How to use checksums |
690b8ded776f1019d5c0fa7850e5bad54ac60b45e8052131fd39127afad3ee02
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Release files / fsspec_kafka-0.2.0-cp311-abi3-macosx_11_0_arm64.whl
| Download URL | fsspec_kafka-0.2.0-cp311-abi3-macosx_11_0_arm64.whl |
|---|---|
| Size | 11.1 MB |
| Tags | CPython 3.11 abi3 macOS 11.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
8668ea36f61c778d58642680197b68dc9531cf62748bc627c1093a11e186cd2c
|
|
BLAKE2b-256 checksum How to use checksums |
1dfaefd9fecbe957fa1c7cd6a531776d66ab0239a45d1dc50069a1f8132cb568
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Security for fsspec-kafka 0.2.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.