Converting pydantic classes to spark schemas
Project description
pydantic-spark
This library can convert a pydantic class to a spark schema or generate python code from a spark schema.
Install
pip install pydantic-spark
Pydantic class to spark schema
import json
from typing import Optional
from pydantic_spark.base import SparkBase
class TestModel(SparkBase):
key1: str
key2: int
key2: Optional[str]
schema_dict: dict = TestModel.spark_schema()
print(json.dumps(schema_dict))
Install for developers
Install package
- Requirement: Poetry 1.*
poetry install
Run unit tests
pytest
coverage run -m pytest # with coverage
# or (depends on your local env)
poetry run pytest
poetry run coverage run -m pytest # with coverage
Run linting
The linting is checked in the github workflow. To fix and review issues run this:
black . # Auto fix all issues
isort . # Auto fix all issues
pflake . # Only display issues, fixing is manual
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
pydantic-spark-0.1.1.tar.gz
(4.9 kB
view details)
Built Distribution
File details
Details for the file pydantic-spark-0.1.1.tar.gz
.
File metadata
- Download URL: pydantic-spark-0.1.1.tar.gz
- Upload date:
- Size: 4.9 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: poetry/1.1.11 CPython/3.9.7 Darwin/20.6.0
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 12dc1d7ab15bc05cf1679b0928c7af0452d1a8caf33f31c184e2c153eaf7ee19 |
|
MD5 | 009e60615d08106b687b4a8fb51e6cb2 |
|
BLAKE2b-256 | f56b8283115db3ab260401e0bd33167a805f46d6765b46ffdc4453273ce1532d |
Provenance
File details
Details for the file pydantic_spark-0.1.1-py3-none-any.whl
.
File metadata
- Download URL: pydantic_spark-0.1.1-py3-none-any.whl
- Upload date:
- Size: 5.9 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: poetry/1.1.11 CPython/3.9.7 Darwin/20.6.0
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2b06b24aaeeb48b5497502d55e94c91c3805bd8e844499183a5600956b0fa5ac |
|
MD5 | 7565086bb6ba7a097f27dfa44361dbce |
|
BLAKE2b-256 | f37c14ff7d7265be66138ddfb47a057619e6201833d1f13eb70684674a3ec678 |