Skip to main content

A pydantic -> spark schema library

Project description

SparkDantic

codecov PyPI version

1️⃣ version: 1.3.0

✍️ author: Mitchell Lisle

PySpark Model Conversion Tool

This Python module provides a utility for converting Pydantic models to PySpark schemas. It's implemented as a class named SparkModel that extends the Pydantic's BaseModel.

Features

  • Conversion from Pydantic model to PySpark schema.

Usage

Creating a new SparkModel

A SparkModel is a Pydantic model, and you can define one by simply inheriting from SparkModel and defining some fields:

from sparkdantic import SparkModel
from typing import List

class MyModel(SparkModel):
    name: str
    age: int
    hobbies: List[str]

ℹ️ Enums are supported but they must be mixed with either int (IntEnum in Python ≥ 3.10) or str (StrEnum, in Python ≥ 3.11) built-in types:

from enum import Enum

class Switch(int, Enum):
    OFF = 0
    ON = 1

class MyEnumModel(SparkModel):
    switch: Switch

Generating a PySpark Schema

Pydantic has existing models for generating json schemas (with model_json_schema). With a SparkModel you can generate a PySpark schema from the model fields using the model_spark_schema() method:

spark_schema = MyModel.model_spark_schema()

Provides this schema:

StructType([
    StructField('name', StringType(), False),
    StructField('age', IntegerType(), False),
    StructField('hobbies', ArrayType(StringType(), False), False)
])

ℹ️ In addition to the automatic type conversion, you can also explicitly coerce data types to Spark native types by setting the spark_type attribute in the Field function from Pydantic, like so: Field(spark_type=DataType). Please replace DataType with the actual Spark data type you want to use. This is useful when you want to use a specific data type then the one that Sparkdantic infers by default.

Contributing

Contributions welcome! If you would like to add a new feature / fix a bug feel free to raise a PR and tag me (mitchelllisle) as a reviewer. Please setup your environment locally to ensure all styling and development flow is as close to the standards set in this project as possible. To do this, the main thing you'll need is poetry. You should also run make install-dev-local which will install the pre-commit-hooks as well as install the project locally. PRs won't be accepted without sufficient tests and we will be strict on maintaining a 100% test coverage.

ℹ️ Note that after you have run make install-dev-local and make a commit we run the test suite as part of the pre-commit hook checks. This is to ensure you don't commit code that breaks the tests. This will also try and commit changes to the COVERAGE.txt file so that we can compare coverage in each PR. Please ensure this file is commited with your changes

ℹ️ Versioning: We use bumpversion to maintain the version across various files. If you submit a PR please run bumpversion to the following rules:

  • bumpversion major: If you are making breaking changes (that is, anyone who already uses this library can no longer rely on existing methods / functionality)
  • bumpversion minor: If you are adding functionality or features that maintain existing methods and features
  • bumpversion patch: If you are fixing a bug or making some other small change

Note: ⚠️ You can ignore bumping the version if you like. I periodically do releases of any dependency updates anyway so if you can wait a couple of days for your code to be pushed to PyPi then just submit the change and I'll make sure it's included in the next release. I'll do my best to make sure it's released ASAP after your PR is merged.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

sparkdantic-1.3.0.tar.gz (6.9 kB view details)

Uploaded Source

Built Distribution

sparkdantic-1.3.0-py3-none-any.whl (7.4 kB view details)

Uploaded Python 3

File details

Details for the file sparkdantic-1.3.0.tar.gz.

File metadata

  • Download URL: sparkdantic-1.3.0.tar.gz
  • Upload date:
  • Size: 6.9 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: poetry/1.8.3 CPython/3.12.6 Linux/6.8.0-1014-azure

File hashes

Hashes for sparkdantic-1.3.0.tar.gz
Algorithm Hash digest
SHA256 2cfa3564db3f014ed8410e14513d5d0bdbfa41a180213ec6aa3278a39d7b78a4
MD5 018d4ce325fcd906bc5a1f3e1b8a8d64
BLAKE2b-256 38dbd6dc2b6420ba4a974b9695920d7b50a10d2e7be4cd74f8fb279d3dd75ea9

See more details on using hashes here.

File details

Details for the file sparkdantic-1.3.0-py3-none-any.whl.

File metadata

  • Download URL: sparkdantic-1.3.0-py3-none-any.whl
  • Upload date:
  • Size: 7.4 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: poetry/1.8.3 CPython/3.12.6 Linux/6.8.0-1014-azure

File hashes

Hashes for sparkdantic-1.3.0-py3-none-any.whl
Algorithm Hash digest
SHA256 e25e65898bc75f307d0e4c75b422d50ca97176c38b03523723ad7ccede3dfefd
MD5 cb4318e17474625a6f255b1b2754d399
BLAKE2b-256 cc0a255c41910be59e33750c153f322f5fe0a453ffc729ce3cde7c76f2bff54f

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page