Conversion from pydantic models to pyarrow schemas
Project description
pydantic-to-pyarrow
pydantic-to-pyarrow is a library for Python to help with conversion of pydantic models to pyarrow schemas.
pydantic is a Python library for data validation, applying type hints / annotations. It enables the creation of easy or complex data validation rules.
pyarrow is a Python library for using Apache Arrow, a development platform for in-memory analytics. The library also enables easy writing to parquet files.
Why might you want to convert models to schemas? One scenario is for a data processing pipeline:
- Import / extract the data from its source
- Validate the data using pydantic
- Process the data in pyarrow / pandas / polars
- Store the raw and / or processed data in parquet.
The easiest approach for steps 3 and 4 above is to let pyarrow infer the schema from the data. The most involved approach is to specify the pyarrow schema separate from the pydantic model. In the middle, many application could benefit from converting the pydantic model to a pyarrow schema. This library aims to achieve that.
Installation
pip install pydantic-to-pyarrow
Conversion Table
The below conversions still run into the possibility of
overflows in the Pyarrow types. For example, in Python 3
the int
type is unbounded, whereas the pa.int64() type has a fixed
maximum. In most cases, this should not be an issue, but if you are
concerned about overflows, you should not use this library and
should manually specify the full schema.
Python / Pydantic | Pyarrow | Overflow |
---|---|---|
str | pa.string() | |
Literal[strings] | pa.dictionary(pa.int32(), pa.string()) | |
. | . | . |
int | pa.int64() if no minimum constraint, pa.uint64() if minimum is zero | Yes, at 2^63 (for signed) or 2^64 (for unsigned) |
Literal[ints] | pa.int64() | |
float | pa.float64() | Yes |
decimal.Decimal | pa.decimal128 ONLY if supplying max_digits and decimal_places for pydantic field | Yes |
. | . | . |
datetime.date | pa.date32() | |
datetime.time | pa.time64("us") | |
datetime.datetime | pa.timestamp("ms", tz=None) ONLY if param allow_losing_tz=True | |
pydantic.types.NaiveDatetime | pa.timestamp("ms", tz=None) | |
pydantic.types.AwareDatetime | pa.timestamp("ms", tz=None) ONLY if param allow_losing_tz=True | |
. | . | |
Optional[...] | The pyarrow field is nullable | |
Pydantic Model | pa.struct() | |
List[...] | pa.list_(...) | |
Enum of str | pa.dictionary(pa.int32(), pa.string()) | |
Enum of int | pa.int64() |
An Example
from typing import List, Optional
from pydantic import BaseModel
from pydantic_to_pyarrow import get_pyarrow_schema
class NestedModel(BaseModel):
str_field: str
class MyModel(BaseModel):
int_field: int
opt_str_field: Optional[str]
py310_opt_str_field: str | None
nested: List[NestedModel]
pa_schema = get_pyarrow_schema(MyModel)
print(pa_schema)
#> int_field: int64 not null
#> opt_str_field: string
#> py310_opt_str_field: string
#> nested: list<item: struct<str_field: string not null>> not null
#> child 0, item: struct<str_field: string not null>
#> child 0, str_field: string not null
Development
Prerequisites:
- Any Python 3.8 through 3.11
- poetry for dependency management
- git
- make
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for pydantic_to_pyarrow-0.1.2.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 929f73d620595f304d8825008f1a39d779e3a658581558154c1320eca8fb7442 |
|
MD5 | ebc1878281ae66eb1047481789ba4dba |
|
BLAKE2b-256 | 0de70de2daa4828ab9b1fd04100fd59c9b93d8d7c0cf4cf70157962c35fa7344 |
Hashes for pydantic_to_pyarrow-0.1.2-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 19e8d0bf5cd79e9126b57a67088508b7ade8a122a4d91d90ef9add4fdc7fa0c0 |
|
MD5 | 0b0340c7a95a1692a204cb9ea349a4d7 |
|
BLAKE2b-256 | 91932dc1cd8fbde42282c04fa02bce49ca8cae6839d1510ccf63e3c1b8110afd |