PySpark Stubs
A collection of the Apache Spark stub files. These files were generated by stubgen and manually edited to include accurate type hints.
Tests and configuration files have been originally contributed to the Typeshed project. Please refer to its contributors list and license for details.
Important
This project has been merged with the main Apache Spark repository (SPARK-32714). All further development for Spark 3.1 and onwards will be continued there.
For Spark 2.4 and 3.0, development of this package will be continued, until their official deprecation.
If your problem is specific to Spark 2.3 and 3.0 feel free to create an issue or open pull requests here.
Otherwise, please check the official Spark JIRA and contributing guidelines. If you create a JIRA ticket or Spark PR related to type hints, please ping me with [~zero323] or @zero323 respectively. Thanks in advance.
Motivation
Static error detection (see SPARK-20631)
Improved autocompletion.
Installation and usage
Please note that the guidelines for distribution of type information is still work in progress (PEP 561 - Distributing and Packaging Type Information). Currently installation script overlays existing Spark installations (pyi stub files are copied next to their py counterparts in the PySpark installation directory). If this approach is not acceptable you can add stub files to the search path manually.
According to PEP 484:
Third-party stub packages can use any location for stub storage. Type checkers should search for them using PYTHONPATH.
Moreover:
Third-party stub packages can use any location for stub storage. Type checkers should search for them using PYTHONPATH. A default fallback directory that is always checked is shared/typehints/python3.5/ (or 3.6, etc.)
Please check usage before proceeding.
The package is available on PYPI:
pip install pyspark-stubsand conda-forge:
conda install -c conda-forge pyspark-stubsDepending on your environment you might also need a type checker, like Mypy or Pytype [1], and autocompletion tool, like Jedi.
Editor |
Type checking |
Autocompletion |
Notes |
|---|---|---|---|
✔ [2] |
✔ [3] |
Through plugins. |
|
✘ [4] |
✔ |
||
✔ |
✔ |
||
✔ [5] |
? |
||
✔ [6] |
✔ [7] |
Through plugins. |
|
✔ [8] |
✔ [9] |
Completion with plugin |
|
Environment independent / other editors |
✔ [10] |
✔ [11] |
This package is tested against MyPy development branch and in rare cases (primarily important upstrean bugfixes), is not compatible with the preceding MyPy release.
PySpark Version Compatibility
Package versions follow PySpark versions with exception to maintenance releases - i.e. pyspark-stubs==2.3.0 should be compatible with pyspark>=2.3.0,<2.4.0. Maintenance releases (post1, post2, …, postN) are reserved for internal annotations updates.
API Coverage:
As of release 2.4.0 most of the public API is covered. For details please check API coverage document.
See also
SPARK-17333 - Make pyspark interface friendly with static analysis.
PySpark typing hints and Revisiting PySpark type annotations on Apache Spark Developers List.
Disclaimer
Apache Spark, Spark, PySpark, Apache, and the Spark logo are trademarks of The Apache Software Foundation. This project is not owned, endorsed, or sponsored by The Apache Software Foundation.
Footnotes
1 maintainer
Metadata
1 maintainer
Release files for pyspark-stubs 3.0.0.post3
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | |
|---|---|---|---|---|
| pyspark-stubs-3.0.0.post3.tar.gz | 73.1 kB | Without provenance | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| pyspark_stubs-3.0.0.post3-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 183.2 kB
Release files / pyspark-stubs-3.0.0.post3.tar.gz
| Download URL | pyspark-stubs-3.0.0.post3.tar.gz |
|---|---|
| Size | 73.1 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
bea8b6ed8a77fac06d8df99d80b9d017bf4de632583d7eef829b4e65653f9d03
|
|
BLAKE2b-256 checksum How to use checksums |
9e5e61249b2acdcf50fa657dabdb5af5751d5d2654cc39c45a787dd46775368e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.7.1 importlib_metadata/4.10.0 pkginfo/1.8.2 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.9.9
|
Release files / pyspark_stubs-3.0.0.post3-py3-none-any.whl
| Download URL | pyspark_stubs-3.0.0.post3-py3-none-any.whl |
|---|---|
| Size | 110.1 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
687b0c1cc4b3b3cbe9cf002b4d1b44bfacb3487921ec8c76aecf71b59c5fe4ad
|
|
BLAKE2b-256 checksum How to use checksums |
36a33823bda6f453b0cc79fc506732ae632ebe7bacbacfd2f09a9c7326325945
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.7.1 importlib_metadata/4.10.0 pkginfo/1.8.2 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.9.9
|
Security for pyspark-stubs 3.0.0.post3
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.
