Skip to main content

fsspec-kafka

Kafka topics and partitions through fsspec

Build Status codecov License PyPI

Overview

fsspec-kafka registers the kafka:// protocol with fsspec. Topics are directories, and each partition is a directory of fixed offset windows that read as Arrow, Parquet, CSV, or JSON Lines.

import fsspec
import pandas as pd

options = {"bootstrap.servers": "localhost:9092"}
fs = fsspec.filesystem("kafka", **options)
fs.ls("orders/0", detail=False)  # ["orders/0/0-10000.arrow", ..., "orders/0/20000-.arrow"]

frame = pd.read_parquet("kafka://orders.parquet", storage_options=options)

Storage options are librdkafka properties, except for these:

Option Default Meaning
timeout 10.0 Seconds to wait for broker requests
window_size 10000 Offsets per window file
list_format "arrow" Extension used when listing windows
batch_size 1024 Messages per Arrow record batch
allow_admin False Lets mkdir create topics and rmdir delete them
partitions, replication -1 Settings for created topics; -1 uses the broker default
topic_config {} Configuration for created topics
value_format "binary" "json", "avro", or "confluent" decodes values with value_schema
value_schema None pyarrow.Schema or fsspec-data schema reference
flatten_value False Puts JSON fields beside the message columns
key_column None Input column for message keys when appending
partition_column None Input column for partitions when appending
timestamp_column None Input column for timestamps when appending
headers_column None Input column for headers, a map, when appending

Messages are read with topic, partition, offset, timestamp, key, headers, and value columns. With value_format="json", value is a struct of the value_schema fields, which are null for tombstones. flatten_value=True puts the fields beside the other columns instead, which CSV needs because it cannot hold structs:

schema = pa.schema([("id", pa.int64()), ("status", pa.string())])
frame = pd.read_csv("kafka://orders.csv", storage_options={**options, "value_format": "json", "value_schema": schema, "flatten_value": True})

value_schema can also be anything fsspec-data resolves when it is installed, such as a list of {"name": ..., "type": ...} fields or a SchemaRef to a schema file or Confluent Schema Registry subject. value_format="avro" reads bare Avro records with an .avsc reference or an Avro schema dict. value_format="confluent" reads and writes values in Confluent wire format, with JSON Schema or Avro subjects. For Avro subjects, every registered version is fetched so older messages resolve to the referenced version's schema. Appends write the schema ID of the Confluent reference:

from fsspec_data import SchemaRef

schema = SchemaRef("confluent://orders-value/versions/latest", storage_options={"registry_url": "http://localhost:8081"})
fs = fsspec.filesystem("kafka", **options, value_format="confluent", value_schema=schema, flatten_value=True)

Topics are append-only. Opening topic.format or topic/partition.format with mode "ab" produces one message per row of the file when it closes. The mapped columns set the message fields, and the remaining columns are the value: a JSON object with value_format="json", an Avro record with Avro values, otherwise a single binary or string column. CSV and JSON Lines columns are typed from value_schema.

fs = fsspec.filesystem("kafka", **options, value_format="json", value_schema=schema, key_column="order_id")
with fs.open("orders.parquet", "ab") as file:
    frame.to_parquet(file)

Options can also come from fsspec configuration, for example FSSPEC_KAFKA='{"bootstrap.servers": "localhost:9092"}'.

On a compacted topic, topic/records holds the latest value of each key, so it works as a key-value store:

state = fsspec.get_mapper("kafka://state/records", **options)
state["user/42"] = b"..."

fs.stream follows a topic or one partition as it grows, returning pyarrow.RecordBatch objects with the same columns as .arrow files. It starts at "end" by default; "beginning", a datetime, or the positions of an earlier stream are also accepted.

with fs.stream("orders", start="beginning") as stream:
    for batch in stream:  # waits for new messages
        handle(batch)
        save(stream.positions)  # {partition: next offset}, to resume with start=positions

From Rust, register the kafka protocol with an fsspec-rs FilesystemRegistry and open URLs with the same storage options. value_format accepts only binary here, and topic_config.<name> keys set the configuration of created topics; construct KafkaFs with KafkaFsOptions for decoded values.

let mut registry = fsspec_rs::FilesystemRegistry::default();
fsspec_kafka::register(&mut registry);
let options = [("bootstrap.servers".to_string(), "localhost:9092".to_string())].into();
let (fs, path) = registry.url_to_fs("kafka://orders/0", &options)?;

Metadata

Release files for fsspec-kafka 0.2.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for fsspec-kafka 0.2.0
File Size Uploaded Provenance Actions
fsspec_kafka-0.2.0.tar.gz 126.7 kB Without provenance Details for fsspec_kafka-0.2.0.tar.gz

Built distributions (wheels)

Table of built distributions (wheels) for fsspec-kafka 0.2.0
File Actions
fsspec_kafka-0.2.0-cp311-abi3-win_amd64.whl CPython 3.11 abi3 Windows x86-64 Details for fsspec_kafka-0.2.0-cp311-abi3-win_amd64.whl
fsspec_kafka-0.2.0-cp311-abi3-manylinux_2_28_x86_64.whl CPython 3.11 abi3 Linux glibc 2.28+ x86-64 Details for fsspec_kafka-0.2.0-cp311-abi3-manylinux_2_28_x86_64.whl
fsspec_kafka-0.2.0-cp311-abi3-manylinux_2_28_aarch64.whl CPython 3.11 abi3 Linux glibc 2.28+ ARM64 Details for fsspec_kafka-0.2.0-cp311-abi3-manylinux_2_28_aarch64.whl
fsspec_kafka-0.2.0-cp311-abi3-macosx_11_0_arm64.whl CPython 3.11 abi3 macOS 11.0+ ARM64 Details for fsspec_kafka-0.2.0-cp311-abi3-macosx_11_0_arm64.whl

Total release size: 49.2 MB

Release files / fsspec_kafka-0.2.0.tar.gz

Download URL fsspec_kafka-0.2.0.tar.gz
Size 126.7 kB
Tags Source
SHA-256 checksum
How to use checksums
7b9b4c6b054c277cdb41d8adff5221bb2c60b9b45efb960126a98b835bae1a06
BLAKE2b-256 checksum
How to use checksums
446ba462671a37fb0ccc9638606c1e936e43c80c5839f6e6a58590a79d72e377
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.13.14

Release files / fsspec_kafka-0.2.0-cp311-abi3-win_amd64.whl

Download URL fsspec_kafka-0.2.0-cp311-abi3-win_amd64.whl
Size 11.1 MB
Tags CPython 3.11 Windows x86-64 abi3
SHA-256 checksum
How to use checksums
7d97dcf24c6d4bb5333e8cac0e50b78a3142fb1c36755e64faec36b3d900562a
BLAKE2b-256 checksum
How to use checksums
a812efe97cc831bdffd21152b49c3f8a26701dfe22c466ea43136471cd33cc59
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.13.14

Release files / fsspec_kafka-0.2.0-cp311-abi3-manylinux_2_28_x86_64.whl

Download URL fsspec_kafka-0.2.0-cp311-abi3-manylinux_2_28_x86_64.whl
Size 13.5 MB
Tags CPython 3.11 Linux glibc 2.28+ x86-64 abi3
SHA-256 checksum
How to use checksums
b55cb5f38763e18e066bcd452fdd1f4bfc2186103121a43be8a464fa894c2cfd
BLAKE2b-256 checksum
How to use checksums
d54c6773e64c009eceb13733ec32da6c9f0abaf99c2b5e230d06a41d5f6a0b66
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.13.14

Release files / fsspec_kafka-0.2.0-cp311-abi3-manylinux_2_28_aarch64.whl

Download URL fsspec_kafka-0.2.0-cp311-abi3-manylinux_2_28_aarch64.whl
Size 13.4 MB
Tags CPython 3.11 Linux glibc 2.28+ ARM64 abi3
SHA-256 checksum
How to use checksums
4e19af0c9a26fc91767bc9d34b01a002117618d7605694aaa57f3bcd3fbc8d04
BLAKE2b-256 checksum
How to use checksums
690b8ded776f1019d5c0fa7850e5bad54ac60b45e8052131fd39127afad3ee02
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.13.14

Release files / fsspec_kafka-0.2.0-cp311-abi3-macosx_11_0_arm64.whl

Download URL fsspec_kafka-0.2.0-cp311-abi3-macosx_11_0_arm64.whl
Size 11.1 MB
Tags CPython 3.11 abi3 macOS 11.0+ ARM64
SHA-256 checksum
How to use checksums
8668ea36f61c778d58642680197b68dc9531cf62748bc627c1093a11e186cd2c
BLAKE2b-256 checksum
How to use checksums
1dfaefd9fecbe957fa1c7cd6a531776d66ab0239a45d1dc50069a1f8132cb568
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.13.14

Security for fsspec-kafka 0.2.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page