A pydantic -> spark schema library
Project description
SparkDantic
1️⃣ version: 1.3.0
✍️ author: Mitchell Lisle
PySpark Model Conversion Tool
This Python module provides a utility for converting Pydantic models to PySpark schemas. It's implemented as a class
named SparkModel
that extends the Pydantic's BaseModel
.
Features
- Conversion from Pydantic model to PySpark schema.
Usage
Creating a new SparkModel
A SparkModel
is a Pydantic model, and you can define one by simply inheriting from SparkModel
and defining some fields:
from sparkdantic import SparkModel
from typing import List
class MyModel(SparkModel):
name: str
age: int
hobbies: List[str]
ℹ️
Enum
s are supported but they must be mixed with eitherint
(IntEnum
in Python ≥ 3.10) orstr
(StrEnum
, in Python ≥ 3.11) built-in types:
from enum import Enum
class Switch(int, Enum):
OFF = 0
ON = 1
class MyEnumModel(SparkModel):
switch: Switch
Generating a PySpark Schema
Pydantic has existing models for generating json schemas (with model_json_schema
). With a SparkModel
you can
generate a PySpark schema from the model fields using the model_spark_schema()
method:
spark_schema = MyModel.model_spark_schema()
Provides this schema:
StructType([
StructField('name', StringType(), False),
StructField('age', IntegerType(), False),
StructField('hobbies', ArrayType(StringType(), False), False)
])
ℹ️ In addition to the automatic type conversion, you can also explicitly coerce data types to Spark native types by setting the
spark_type
attribute in theField
function from Pydantic, like so:Field(spark_type=DataType)
. Please replace DataType with the actual Spark data type you want to use. This is useful when you want to use a specific data type then the one that Sparkdantic infers by default.
Contributing
Contributions welcome! If you would like to add a new feature / fix a bug feel free to raise a PR and tag me (mitchelllisle
) as
a reviewer. Please setup your environment locally to ensure all styling and development flow is as close to the standards set in
this project as possible. To do this, the main thing you'll need is poetry
. You should also run make install-dev-local
which
will install the pre-commit-hooks
as well as install the project locally. PRs won't be accepted without sufficient tests and
we will be strict on maintaining a 100% test coverage.
ℹ️ Note that after you have run
make install-dev-local
and make a commit we run the test suite as part of the pre-commit hook checks. This is to ensure you don't commit code that breaks the tests. This will also try and commit changes to the COVERAGE.txt file so that we can compare coverage in each PR. Please ensure this file is commited with your changes
ℹ️ Versioning: We use
bumpversion
to maintain the version across various files. If you submit a PR please run bumpversion to the following rules:
bumpversion major
: If you are making breaking changes (that is, anyone who already uses this library can no longer rely on existing methods / functionality)bumpversion minor
: If you are adding functionality or features that maintain existing methods and featuresbumpversion patch
: If you are fixing a bug or making some other small change
Note: ⚠️ You can ignore bumping the version if you like. I periodically do releases of any dependency updates anyway so if you can wait a couple of days for your code to be pushed to PyPi then just submit the change and I'll make sure it's included in the next release. I'll do my best to make sure it's released ASAP after your PR is merged.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file sparkdantic-1.3.0.tar.gz
.
File metadata
- Download URL: sparkdantic-1.3.0.tar.gz
- Upload date:
- Size: 6.9 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: poetry/1.8.3 CPython/3.12.6 Linux/6.8.0-1014-azure
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2cfa3564db3f014ed8410e14513d5d0bdbfa41a180213ec6aa3278a39d7b78a4 |
|
MD5 | 018d4ce325fcd906bc5a1f3e1b8a8d64 |
|
BLAKE2b-256 | 38dbd6dc2b6420ba4a974b9695920d7b50a10d2e7be4cd74f8fb279d3dd75ea9 |
File details
Details for the file sparkdantic-1.3.0-py3-none-any.whl
.
File metadata
- Download URL: sparkdantic-1.3.0-py3-none-any.whl
- Upload date:
- Size: 7.4 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: poetry/1.8.3 CPython/3.12.6 Linux/6.8.0-1014-azure
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | e25e65898bc75f307d0e4c75b422d50ca97176c38b03523723ad7ccede3dfefd |
|
MD5 | cb4318e17474625a6f255b1b2754d399 |
|
BLAKE2b-256 | cc0a255c41910be59e33750c153f322f5fe0a453ffc729ce3cde7c76f2bff54f |