Skip to main content

Converting pydantic classes to spark schemas

Project description

Python package codecov PyPI version CodeQL

pydantic-spark

This library can convert a pydantic class to a spark schema or generate python code from a spark schema.

Install

pip install pydantic-spark

Pydantic class to spark schema

import json
from typing import Optional

from pydantic_spark.base import SparkBase

class TestModel(SparkBase):
    key1: str
    key2: int
    key2: Optional[str]

schema_dict: dict = TestModel.spark_schema()
print(json.dumps(schema_dict))

Install for developers

Install package
  • Requirement: Poetry 1.*
poetry install
Run unit tests
pytest
coverage run -m pytest  # with coverage
# or (depends on your local env) 
poetry run pytest
poetry run coverage run -m pytest  # with coverage
Run linting

The linting is checked in the github workflow. To fix and review issues run this:

black .   # Auto fix all issues
isort .   # Auto fix all issues
pflake .  # Only display issues, fixing is manual

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

pydantic-spark-0.1.1.tar.gz (4.9 kB view details)

Uploaded Source

Built Distribution

pydantic_spark-0.1.1-py3-none-any.whl (5.9 kB view details)

Uploaded Python 3

File details

Details for the file pydantic-spark-0.1.1.tar.gz.

File metadata

  • Download URL: pydantic-spark-0.1.1.tar.gz
  • Upload date:
  • Size: 4.9 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: poetry/1.1.11 CPython/3.9.7 Darwin/20.6.0

File hashes

Hashes for pydantic-spark-0.1.1.tar.gz
Algorithm Hash digest
SHA256 12dc1d7ab15bc05cf1679b0928c7af0452d1a8caf33f31c184e2c153eaf7ee19
MD5 009e60615d08106b687b4a8fb51e6cb2
BLAKE2b-256 f56b8283115db3ab260401e0bd33167a805f46d6765b46ffdc4453273ce1532d

See more details on using hashes here.

File details

Details for the file pydantic_spark-0.1.1-py3-none-any.whl.

File metadata

  • Download URL: pydantic_spark-0.1.1-py3-none-any.whl
  • Upload date:
  • Size: 5.9 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: poetry/1.1.11 CPython/3.9.7 Darwin/20.6.0

File hashes

Hashes for pydantic_spark-0.1.1-py3-none-any.whl
Algorithm Hash digest
SHA256 2b06b24aaeeb48b5497502d55e94c91c3805bd8e844499183a5600956b0fa5ac
MD5 7565086bb6ba7a097f27dfa44361dbce
BLAKE2b-256 f37c14ff7d7265be66138ddfb47a057619e6201833d1f13eb70684674a3ec678

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page