localstack-s3-pyspark

This package provides a CLI for configuring pyspark to use localstack for the S3 file system. This is intended for testing packages locally (or in your CI/CD pipeline) which you intend to deploy on an Amazon EMR cluster.

Installation

Execute the following command, replacing pip3 with the executable appropriate for the environment where you want to configure pyspark to use localstack:

pip3 install localstack-s3-pyspark

Configure Spark's Defaults

If you've installed localstack-s3-pyspark in a Dockerfile or virtual environment, just run the following command:

localstack-s3-pyspark configure-defaults

If you've installed localstack-s3-pyspark in an environment with multiple python 3.x versions, you may instead want to run an appropriate variation of the following command (replacing python3 with the command used to access the python executable for which you want to configure pyspark):

python3 -m localstack_s3_pyspark configure-defaults

Tox

Please note that if you are testing your packages with tox (highly recommended), you will need to:

Include "localstack-s3-pyspark" in your tox deps
Include localstack-s3-pyspark configure-defaults in your tox commands_pre (or by other means execute this command prior to your tests)

Here is an example tox.ini which starts up localstack using the localstack CLI (you could also use docker-compose or just docker run, if you need greater control or fewer python dependencies, see the the localstack documentation "Getting Started" page for details):

[tox]
envlist = pytest

[testenv:pytest]
deps =
  localstack-s3-pyspark
  localstack
commands_pre =
    localstack-s3-pyspark configure-defaults
    localstack start -d
    sleep 20
commands =
    py.test
commands_post =
    localstack stop

Patch boto3

If your tests interact with S3 using boto3, you can patch boto3 from within your unit tests as follows:

from localstack_s3_pyspark.boto3 import use_localstack
use_localstack()

Metadata

Release files for localstack-s3-pyspark 0.12.6

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for localstack-s3-pyspark 0.12.6
File	Size	Uploaded
localstack-s3-pyspark-0.12.6.tar.gz	8.1 kB	Feb 7, 2024	Details

Built distribution (wheel)

Table of built distributions (wheels) for localstack-s3-pyspark 0.12.6
File	Interpreter	ABI	Platform	Reset
localstack_s3_pyspark-0.12.6-py3-none-any.whl 7.7 kB Feb 7, 2024	Python 3	none	any	Details

Total release size: 15.8 kB

Release files / localstack-s3-pyspark-0.12.6.tar.gz

Download URL	localstack-s3-pyspark-0.12.6.tar.gz
Size	8.1 kB
Tags	Source
SHA-256 checksum How to use checksums	`da9e0daf3c3a2ca9688278ba18e5ebbca55fdd50d65a68d46ca026838e83b2f0`
BLAKE2b-256 checksum How to use checksums	`87c03229aa790fd7e2db26a80ddb70f0ed52c9b55d3179589833d67e852345c1`
Upload date	Feb 7, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/4.0.2 CPython/3.12.1`

Release files / localstack_s3_pyspark-0.12.6-py3-none-any.whl

Download URL	localstack_s3_pyspark-0.12.6-py3-none-any.whl
Size	7.7 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`071221f1439438ec26ac81c5320fad1ef82b427566dc2e37395405eb9a2bd82f`
BLAKE2b-256 checksum How to use checksums	`27ddcc4025fbc0bbb566a45832efe7c96d635669dc0be53116cd3b1b43b77200`
Upload date	Feb 7, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/4.0.2 CPython/3.12.1`