Skip to main content

Typedspark: column-wise type annotations for pyspark DataFrames

We love Spark! But in production code we're wary when we see:

from pyspark.sql import DataFrame

def foo(df: DataFrame) -> DataFrame:
    # do stuff
    return df

Because… How do we know which columns are supposed to be in df?

Using typedspark, we can be more explicit about what these data should look like.

from typedspark import Column, DataSet, Schema
from pyspark.sql.types import LongType, StringType

class Person(Schema):
    id: Column[LongType]
    name: Column[StringType]
    age: Column[LongType]

def foo(df: DataSet[Person]) -> DataSet[Person]:
    # do stuff
    return df

The advantages include:

  • Improved readability of the code
  • Typechecking, both during runtime and linting
  • Auto-complete of column names
  • Easy refactoring of column names
  • Easier unit testing through the generation of empty DataSets based on their schemas
  • Improved documentation of tables

Documentation

Please see our documentation on readthedocs.

Installation

You can install typedspark from pypi by running:

pip install typedspark

By default, typedspark does not list pyspark as a dependency, since many platforms (e.g. Databricks) come with pyspark preinstalled. If you want to install typedspark with pyspark, you can run:

pip install "typedspark[pyspark]"

Compatibility

Typedspark is tested in CI with PySpark 3.5.8 and 4.1.1. Spark Connect is supported when using PySpark 4.x, and the Connect-specific test runs if SPARK_CONNECT_URL is set.

Demo videos

IDE demo

https://github.com/kaiko-ai/typedspark/assets/47976799/e6f7fa9c-6d14-4f68-baba-fe3c22f75b67

You can find the corresponding code here.

Jupyter / Databricks notebooks demo

https://github.com/kaiko-ai/typedspark/assets/47976799/39e157c3-6db0-436a-9e72-44b2062df808

You can find the corresponding code here.

FAQ

I found a bug! What should I do?
Great! Please make an issue and we'll look into it.

I have a great idea to improve typedspark! How can we make this work?
Awesome, please make an issue and let us know!

Metadata

Release files for typedspark 1.7.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for typedspark 1.7.1
File Size Uploaded Provenance Actions
typedspark-1.7.1.tar.gz 31.8 kB Without provenance Details for typedspark-1.7.1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for typedspark 1.7.1
File Interpreter ABI Platform Actions
typedspark-1.7.1-py3-none-any.whl Python 3 none any Details for typedspark-1.7.1-py3-none-any.whl

Total release size: 71.0 kB

Release files / typedspark-1.7.1.tar.gz

Download URL typedspark-1.7.1.tar.gz
Size 31.8 kB
Tags Source
SHA-256 checksum
How to use checksums
2524c2b7d12dad77f4a52f7d98f8170536d223179acf9e3eb8b48947193f186c
BLAKE2b-256 checksum
How to use checksums
084d22259c4ac66aaa42ba1250cedcac4df874c3e83bc07f25d431392c79b1e5
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.14.8

Release files / typedspark-1.7.1-py3-none-any.whl

Download URL typedspark-1.7.1-py3-none-any.whl
Size 39.2 kB
Tags Python 3
SHA-256 checksum
How to use checksums
3c40074bf6dc5aaaeceea8557bb381d277b862adb8c2affcdb67483fa6dbda2a
BLAKE2b-256 checksum
How to use checksums
e45ab056aadd4842cddbdb521b6e312e691572cc9a51d3eb20ff240b2c5b3f5b
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.14.8

Security for typedspark 1.7.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

1.7.1 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page