Skip to main content

Converting pydantic classes to spark schemas

Project description

Python package codecov PyPI version CodeQL

pydantic-spark

This library can convert a pydantic class to a spark schema or generate python code from a spark schema.

Install

pip install pydantic-spark

Pydantic class to spark schema

import json
from typing import Optional

from pydantic_spark.base import SparkBase

class TestModel(SparkBase):
    key1: str
    key2: int
    key2: Optional[str]

schema_dict: dict = TestModel.spark_schema()
print(json.dumps(schema_dict))

Coerce type

Pydantic-spark provides a coerce_type option that allows type coercion. When applied to a field, pydantic-spark converts the column's data type to the specified coercion type.

import json
from pydantic import Field
from pydantic_spark.base import SparkBase, CoerceType

class TestModel(SparkBase):
    key1: str = Field(extra_json_schema={"coerce_type": CoerceType.integer})

schema_dict: dict = TestModel.spark_schema()
print(json.dumps(schema_dict))

Install for developers

Install package
  • Requirement: Poetry 1.*
poetry install
Run unit tests
pytest
coverage run -m pytest  # with coverage
# or (depends on your local env) 
poetry run pytest
poetry run coverage run -m pytest  # with coverage
Run linting

The linting is checked in the github workflow. To fix and review issues run this:

black .   # Auto fix all issues
isort .   # Auto fix all issues
pflake .  # Only display issues, fixing is manual

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

pydantic_spark-1.0.0.tar.gz (4.8 kB view details)

Uploaded Source

Built Distribution

pydantic_spark-1.0.0-py3-none-any.whl (6.3 kB view details)

Uploaded Python 3

File details

Details for the file pydantic_spark-1.0.0.tar.gz.

File metadata

  • Download URL: pydantic_spark-1.0.0.tar.gz
  • Upload date:
  • Size: 4.8 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: poetry/1.7.0 CPython/3.12.0 Darwin/22.6.0

File hashes

Hashes for pydantic_spark-1.0.0.tar.gz
Algorithm Hash digest
SHA256 93852c84e781ea386a338c7215ba89e8b445cd82183a000336928a43cde857cb
MD5 f75e803e14ec8c964f6b21cdf2dddae9
BLAKE2b-256 83ca835c7b0955e9fb0d193a18398169007536bdab221745c838391eb96e8b41

See more details on using hashes here.

File details

Details for the file pydantic_spark-1.0.0-py3-none-any.whl.

File metadata

  • Download URL: pydantic_spark-1.0.0-py3-none-any.whl
  • Upload date:
  • Size: 6.3 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: poetry/1.7.0 CPython/3.12.0 Darwin/22.6.0

File hashes

Hashes for pydantic_spark-1.0.0-py3-none-any.whl
Algorithm Hash digest
SHA256 f3f7d8d77c94541adfcd93c79edc535e9b603cb946bfb513015cb9775175af73
MD5 eabb38d93962a53764caf2f92d2c8816
BLAKE2b-256 4485767dc95a9a0a81998c52fbef01c54f24ca749386ef277973b23a84d5e3b7

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page