Skip to main content

fsspec-kafka

Kafka topics and partitions through fsspec

Build Status codecov License PyPI

Overview

fsspec-kafka registers the kafka:// protocol with fsspec. Topics are directories, and each partition is a directory of fixed offset windows that read as Arrow, Parquet, CSV, or JSON Lines.

import fsspec
import pandas as pd

options = {"bootstrap.servers": "localhost:9092"}
fs = fsspec.filesystem("kafka", **options)
fs.ls("orders/0", detail=False)  # ["orders/0/0-10000.arrow", ..., "orders/0/20000-.arrow"]

frame = pd.read_parquet("kafka://orders.parquet", storage_options=options)

Storage options are librdkafka properties, except for these:

Option Default Meaning
timeout 10.0 Seconds to wait for broker requests
window_size 10000 Offsets per window file
list_format "arrow" Extension used when listing windows
batch_size 1024 Messages per Arrow record batch
allow_admin False Lets mkdir create topics and rmdir delete them
partitions, replication -1 Settings for created topics; -1 uses the broker default
topic_config {} Configuration for created topics
value_format "binary" "json" or "confluent" decodes values with value_schema
value_schema None pyarrow.Schema or fsspec-data schema reference
flatten_value False Puts JSON fields beside the message columns
key_column None Input column for message keys when appending
partition_column None Input column for partitions when appending
timestamp_column None Input column for timestamps when appending
headers_column None Input column for headers, a map, when appending

Messages are read with topic, partition, offset, timestamp, key, headers, and value columns. With value_format="json", value is a struct of the value_schema fields, which are null for tombstones. flatten_value=True puts the fields beside the other columns instead, which CSV needs because it cannot hold structs:

schema = pa.schema([("id", pa.int64()), ("status", pa.string())])
frame = pd.read_csv("kafka://orders.csv", storage_options={**options, "value_format": "json", "value_schema": schema, "flatten_value": True})

value_schema can also be anything fsspec-data resolves when it is installed, such as a list of {"name": ..., "type": ...} fields or a SchemaRef to a schema file or Confluent Schema Registry subject. value_format="confluent" reads and writes JSON in Confluent wire format; appends write the schema ID of a Confluent reference:

from fsspec_data import SchemaRef

schema = SchemaRef("confluent://orders-value/versions/latest", storage_options={"registry_url": "http://localhost:8081"})
fs = fsspec.filesystem("kafka", **options, value_format="confluent", value_schema=schema, flatten_value=True)

Topics are append-only. Opening topic.format or topic/partition.format with mode "ab" produces one message per row of the file when it closes. The mapped columns set the message fields, and the remaining columns are the value: a JSON object with value_format="json", otherwise a single binary or string column. CSV and JSON Lines columns are typed from value_schema.

fs = fsspec.filesystem("kafka", **options, value_format="json", value_schema=schema, key_column="order_id")
with fs.open("orders.parquet", "ab") as file:
    frame.to_parquet(file)

Options can also come from fsspec configuration, for example FSSPEC_KAFKA='{"bootstrap.servers": "localhost:9092"}'.

On a compacted topic, topic/records holds the latest value of each key, so it works as a key-value store:

state = fsspec.get_mapper("kafka://state/records", **options)
state["user/42"] = b"..."

fs.stream follows a topic or one partition as it grows, returning pyarrow.RecordBatch objects with the same columns as .arrow files. It starts at "end" by default; "beginning", a datetime, or the positions of an earlier stream are also accepted.

with fs.stream("orders", start="beginning") as stream:
    for batch in stream:  # waits for new messages
        handle(batch)
        save(stream.positions)  # {partition: next offset}, to resume with start=positions

Metadata

Release files for fsspec-kafka 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for fsspec-kafka 0.1.0
File Size Uploaded Provenance Actions
fsspec_kafka-0.1.0.tar.gz 117.4 kB Without provenance Details for fsspec_kafka-0.1.0.tar.gz

Built distributions (wheels)

Table of built distributions (wheels) for fsspec-kafka 0.1.0
File Actions
fsspec_kafka-0.1.0-cp311-abi3-win_amd64.whl CPython 3.11 abi3 Windows x86-64 Details for fsspec_kafka-0.1.0-cp311-abi3-win_amd64.whl
fsspec_kafka-0.1.0-cp311-abi3-manylinux_2_28_x86_64.whl CPython 3.11 abi3 Linux glibc 2.28+ x86-64 Details for fsspec_kafka-0.1.0-cp311-abi3-manylinux_2_28_x86_64.whl
fsspec_kafka-0.1.0-cp311-abi3-manylinux_2_28_aarch64.whl CPython 3.11 abi3 Linux glibc 2.28+ ARM64 Details for fsspec_kafka-0.1.0-cp311-abi3-manylinux_2_28_aarch64.whl
fsspec_kafka-0.1.0-cp311-abi3-macosx_11_0_arm64.whl CPython 3.11 abi3 macOS 11.0+ ARM64 Details for fsspec_kafka-0.1.0-cp311-abi3-macosx_11_0_arm64.whl

Total release size: 30.1 MB

Release files / fsspec_kafka-0.1.0.tar.gz

Download URL fsspec_kafka-0.1.0.tar.gz
Size 117.4 kB
Tags Source
SHA-256 checksum
How to use checksums
b76df51065d733d02d94ac959a815129d7c7ee9230fee2a03ccd3806f2bb34b7
BLAKE2b-256 checksum
How to use checksums
003d997e19950c963732474cc2ded80b3527a1fcd5195413f4f820ef675642de
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.15

Release files / fsspec_kafka-0.1.0-cp311-abi3-win_amd64.whl

Download URL fsspec_kafka-0.1.0-cp311-abi3-win_amd64.whl
Size 6.5 MB
Tags CPython 3.11 Windows x86-64 abi3
SHA-256 checksum
How to use checksums
e5b3843d471a0710ab6d6d81f341c2bc8e3933320f8380476ed9cffcbd400070
BLAKE2b-256 checksum
How to use checksums
fdc430bf3f4f5c66277f3ffb33c74dcf05d394320bd3e4990f5fb6e1919e0df5
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.15

Release files / fsspec_kafka-0.1.0-cp311-abi3-manylinux_2_28_x86_64.whl

Download URL fsspec_kafka-0.1.0-cp311-abi3-manylinux_2_28_x86_64.whl
Size 8.2 MB
Tags CPython 3.11 Linux glibc 2.28+ x86-64 abi3
SHA-256 checksum
How to use checksums
21ba755d56bfd21663a86860cb6ad838bf9385d62776e783c36be4b6f5ab3621
BLAKE2b-256 checksum
How to use checksums
96a005310c7c27849129531b7c5636189d522b912f8fd61e8d37f87b1a8bd62a
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.15

Release files / fsspec_kafka-0.1.0-cp311-abi3-manylinux_2_28_aarch64.whl

Download URL fsspec_kafka-0.1.0-cp311-abi3-manylinux_2_28_aarch64.whl
Size 8.3 MB
Tags CPython 3.11 Linux glibc 2.28+ ARM64 abi3
SHA-256 checksum
How to use checksums
7760e45f6f2a2974899097047748b73aae413902b3a17ac69c38fe09c9043535
BLAKE2b-256 checksum
How to use checksums
84e0a64dfef4076addf2b60314d8411940533d6527c144930c89da5aad9848bb
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.15

Release files / fsspec_kafka-0.1.0-cp311-abi3-macosx_11_0_arm64.whl

Download URL fsspec_kafka-0.1.0-cp311-abi3-macosx_11_0_arm64.whl
Size 7.0 MB
Tags CPython 3.11 abi3 macOS 11.0+ ARM64
SHA-256 checksum
How to use checksums
67b0a5d8c1320666f7a28d61c509d3920a3b381543981c9639361c057dd7d8b8
BLAKE2b-256 checksum
How to use checksums
c7171af2fdac56f71cc20bc558a504d8e070a442506578f8fd502c7ec277fc61
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.15

Security for fsspec-kafka 0.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page