pyspark-cbor 0.1.0

Reading and writing CBOR files with PySpark

pyspark--cbor

By no means ready for production use. This is still in development and now just see it as an example.

This repository showcases custom Spark data source cbor built using the new Python Data Source API for the upcoming Apache Spark 4.0 release. For an in-depth understanding of the API, please refer to the API source code.

from pyspark.sql import SparkSession

# Initialize Spark session
spark = SparkSession.builder \
    .appName("CBOR Data Source Example") \
    .getOrCreate()

# Register the CBOR data source
spark.dataSource.register(CBORDataSource)

# Read CBOR file
df = spark.read.format("cbor").load("path/to/your/file.cbor")

# Show the DataFrame
df.show()

Options