Typedspark: column-wise type annotations for pyspark DataFrames
We love Spark! But in production code we're wary when we see:
from pyspark.sql import DataFrame
def foo(df: DataFrame) -> DataFrame:
# do stuff
return df
Because… How do we know which columns are supposed to be in df?
Using typedspark, we can be more explicit about what these data should look like.
from typedspark import Column, DataSet, Schema
from pyspark.sql.types import LongType, StringType
class Person(Schema):
id: Column[LongType]
name: Column[StringType]
age: Column[LongType]
def foo(df: DataSet[Person]) -> DataSet[Person]:
# do stuff
return df
The advantages include:
- Improved readability of the code
- Typechecking, both during runtime and linting
- Auto-complete of column names
- Easy refactoring of column names
- Easier unit testing through the generation of empty
DataSetsbased on their schemas - Improved documentation of tables
Documentation
Please see our documentation on readthedocs.
Installation
You can install typedspark from pypi by running:
pip install typedspark
By default, typedspark does not list pyspark as a dependency, since many platforms (e.g. Databricks) come with pyspark preinstalled. If you want to install typedspark with pyspark, you can run:
pip install "typedspark[pyspark]"
Compatibility
Typedspark is tested in CI with PySpark 3.5.7 and 4.1.0. Spark Connect is supported when using
PySpark 4.x, and the Connect-specific test runs if SPARK_CONNECT_URL is set.
Demo videos
IDE demo
https://github.com/kaiko-ai/typedspark/assets/47976799/e6f7fa9c-6d14-4f68-baba-fe3c22f75b67
You can find the corresponding code here.
Jupyter / Databricks notebooks demo
https://github.com/kaiko-ai/typedspark/assets/47976799/39e157c3-6db0-436a-9e72-44b2062df808
You can find the corresponding code here.
FAQ
I found a bug! What should I do?
Great! Please make an issue and we'll look into it.
I have a great idea to improve typedspark! How can we make this work?
Awesome, please make an issue and let us know!
1 maintainer
Metadata
1 maintainer
Release files for typedspark 1.7.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| typedspark-1.7.0.tar.gz | 31.5 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| typedspark-1.7.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 70.6 kB
Release files / typedspark-1.7.0.tar.gz
| Download URL | typedspark-1.7.0.tar.gz |
|---|---|
| Size | 31.5 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
7e8448095b92059d857df264b3a77f2671d1906d12f252fa7fc8bdd9bcb05833
|
|
BLAKE2b-256 checksum How to use checksums |
a6c90a0dd53706885af16f96de32a5828292f3023e2887d32aeaa54e7e0cc635
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.14.4
|
Release files / typedspark-1.7.0-py3-none-any.whl
| Download URL | typedspark-1.7.0-py3-none-any.whl |
|---|---|
| Size | 39.1 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
6c6cc354e812a4fae91676bdd4ba8e2142e274d70c8a983b2ec67378fd0e12bc
|
|
BLAKE2b-256 checksum How to use checksums |
f2b3b9d7487f3b27b49a28904b80ec007a689fc68ceb6ba39b4dc6a21521ed85
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.14.4
|
