Skip to main content

A collection of the Apache Spark stub files

Project description

Build Status PyPI version Conda Forge version

A collection of the Apache Spark stub files. These files were generated by stubgen and manually edited to include accurate type hints.

Tests and configuration files have been originally contributed to the Typeshed project. Please refer to its contributors list and license for details.

Important

This project has been merged with the main Apache Spark repository (SPARK-32714). All further development for Spark 3.1 and onwards will be continued there.

For Spark 2.4 and 3.0, development of this package will be continued, until their official deprecation.

  • If your problem is specific to Spark 2.3 and 3.0 feel free to create an issue or open pull requests here.

  • Otherwise, please check the official Spark JIRA and contributing guidelines. If you create a JIRA ticket or Spark PR related to type hints, please ping me with [~zero323] or @zero323 respectively. Thanks in advance.

Motivation

  • Static error detection (see SPARK-20631)

    SPARK-20631

  • Improved autocompletion.

    Syntax completion

Installation and usage

Please note that the guidelines for distribution of type information is still work in progress (PEP 561 - Distributing and Packaging Type Information). Currently installation script overlays existing Spark installations (pyi stub files are copied next to their py counterparts in the PySpark installation directory). If this approach is not acceptable you can add stub files to the search path manually.

According to PEP 484:

Third-party stub packages can use any location for stub storage. Type checkers should search for them using PYTHONPATH.

Moreover:

Third-party stub packages can use any location for stub storage. Type checkers should search for them using PYTHONPATH. A default fallback directory that is always checked is shared/typehints/python3.5/ (or 3.6, etc.)

Please check usage before proceeding.

The package is available on PYPI:

pip install pyspark-stubs

and conda-forge:

conda install -c conda-forge pyspark-stubs

Depending on your environment you might also need a type checker, like Mypy or Pytype [1], and autocompletion tool, like Jedi.

Editor

Type checking

Autocompletion

Notes

Atom

[2]

[3]

Through plugins.

IPython / Jupyter Notebook

[4]

PyCharm

PyDev

[5]

?

VIM / Neovim

[6]

[7]

Through plugins.

Visual Studio Code

[8]

[9]

Completion with plugin

Environment independent / other editors

[10]

[11]

Through Mypy and Jedi.

This package is tested against MyPy development branch and in rare cases (primarily important upstrean bugfixes), is not compatible with the preceding MyPy release.

PySpark Version Compatibility

Package versions follow PySpark versions with exception to maintenance releases - i.e. pyspark-stubs==2.3.0 should be compatible with pyspark>=2.3.0,<2.4.0. Maintenance releases (post1, post2, …, postN) are reserved for internal annotations updates.

API Coverage:

As of release 2.4.0 most of the public API is covered. For details please check API coverage document.

See also

Disclaimer

Apache Spark, Spark, PySpark, Apache, and the Spark logo are trademarks of The Apache Software Foundation. This project is not owned, endorsed, or sponsored by The Apache Software Foundation.

Footnotes

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

pyspark-stubs-3.0.0.post3.tar.gz (73.1 kB view details)

Uploaded Source

Built Distribution

pyspark_stubs-3.0.0.post3-py3-none-any.whl (110.1 kB view details)

Uploaded Python 3

File details

Details for the file pyspark-stubs-3.0.0.post3.tar.gz.

File metadata

  • Download URL: pyspark-stubs-3.0.0.post3.tar.gz
  • Upload date:
  • Size: 73.1 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.7.1 importlib_metadata/4.10.0 pkginfo/1.8.2 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.9.9

File hashes

Hashes for pyspark-stubs-3.0.0.post3.tar.gz
Algorithm Hash digest
SHA256 bea8b6ed8a77fac06d8df99d80b9d017bf4de632583d7eef829b4e65653f9d03
MD5 ca0c0e08cc625cfc18a5c3c15932f9d1
BLAKE2b-256 9e5e61249b2acdcf50fa657dabdb5af5751d5d2654cc39c45a787dd46775368e

See more details on using hashes here.

File details

Details for the file pyspark_stubs-3.0.0.post3-py3-none-any.whl.

File metadata

  • Download URL: pyspark_stubs-3.0.0.post3-py3-none-any.whl
  • Upload date:
  • Size: 110.1 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.7.1 importlib_metadata/4.10.0 pkginfo/1.8.2 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.9.9

File hashes

Hashes for pyspark_stubs-3.0.0.post3-py3-none-any.whl
Algorithm Hash digest
SHA256 687b0c1cc4b3b3cbe9cf002b4d1b44bfacb3487921ec8c76aecf71b59c5fe4ad
MD5 4676458b25bf690c6f00a9436b6e42fc
BLAKE2b-256 36a33823bda6f453b0cc79fc506732ae632ebe7bacbacfd2f09a9c7326325945

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page