localstack-s3-pyspark
This package provides a CLI for configuring pyspark to use localstack for the S3 file system. This is intended for testing packages locally (or in your CI/CD pipeline) which you intend to deploy on an Amazon EMR cluster.
Installation
Execute the following command, replacing pip3 with the executable appropriate for the environment where you want to configure pyspark to use localstack:
pip3 install localstack-s3-pyspark
Configure Spark's Defaults
If you've installed localstack-s3-pyspark in a Dockerfile or virtual environment, just run the following command:
localstack-s3-pyspark configure-defaults
If you've installed localstack-s3-pyspark in an environment with multiple
python 3.x versions, you may instead want to run an appropriate variation of
the following command (replacing python3 with the command used to access the
python executable for which you want to configure pyspark):
python3 -m localstack_s3_pyspark configure-defaults
Tox
Please note that if you are testing your packages with tox (highly recommended), you will need to:
- Include "localstack-s3-pyspark" in your tox deps
- Include
localstack-s3-pyspark configure-defaultsin your tox commands_pre (or by other means execute this command prior to your tests)
Here is an example tox.ini which starts up localstack using the localstack
CLI (you could also use docker-compose or just docker run, if you need
greater control or fewer python dependencies, see the the localstack
documentation
"Getting Started" page
for details):
[tox]
envlist = pytest
[testenv:pytest]
deps =
localstack-s3-pyspark
localstack
commands_pre =
localstack-s3-pyspark configure-defaults
localstack start -d
sleep 20
commands =
py.test
commands_post =
localstack stop
Patch boto3
If your tests interact with S3 using boto3, you can patch boto3 from within your unit tests as follows:
from localstack_s3_pyspark.boto3 import use_localstack
use_localstack()
Metadata
Release files for localstack-s3-pyspark 0.12.6
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| localstack-s3-pyspark-0.12.6.tar.gz | 8.1 kB | Without provenance | Details for localstack-s3-pyspark-0.12.6.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| localstack_s3_pyspark-0.12.6-py3-none-any.whl | Python 3 | none | any | Details for localstack_s3_pyspark-0.12.6-py3-none-any.whl |
Total release size: 15.8 kB
Release files / localstack-s3-pyspark-0.12.6.tar.gz
| Download URL | localstack-s3-pyspark-0.12.6.tar.gz |
|---|---|
| Size | 8.1 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
da9e0daf3c3a2ca9688278ba18e5ebbca55fdd50d65a68d46ca026838e83b2f0
|
|
BLAKE2b-256 checksum How to use checksums |
87c03229aa790fd7e2db26a80ddb70f0ed52c9b55d3179589833d67e852345c1
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.12.1
|
Release files / localstack_s3_pyspark-0.12.6-py3-none-any.whl
| Download URL | localstack_s3_pyspark-0.12.6-py3-none-any.whl |
|---|---|
| Size | 7.7 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
071221f1439438ec26ac81c5320fad1ef82b427566dc2e37395405eb9a2bd82f
|
|
BLAKE2b-256 checksum How to use checksums |
27ddcc4025fbc0bbb566a45832efe7c96d635669dc0be53116cd3b1b43b77200
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.12.1
|
Security for localstack-s3-pyspark 0.12.6
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.