duckdb-spark 1.0.0

A custom PySpark extension for writing data to DuckDB

DuckDB Extension for PySpark

This is a custom PySpark extension that allows writing data from PySpark DataFrames into DuckDB.

Features

• ✅ Seamlessly write PySpark DataFrames to DuckDB
• ✅ Supports overwrite and append modes
• ✅ Automatically detects and adds new columns when appending data
• ✅ Simple integration with PySpark's DataFrameWriter API

Installation

You can install the package using pip:

pip install duckdb-spark


## Usage

Here’s how you can use the extension:

```python
from pyspark.sql import SparkSession
from duckdb_extension import DuckDBWriter

# Initialize Spark Session
spark = SparkSession.builder.appName("DuckDBExample").getOrCreate()

# Create Sample DataFrame
df = spark.createDataFrame([(1, "Alice"), (2, "Bob")], ["id", "name"])

# Write to DuckDB using the custom extension
df.write.duckdb_extension("my_db.duckdb", "users", mode="overwrite")