pytest plugin to run the tests with support of pyspark (Apache Spark).
This plugin will allow to specify SPARK_HOME directory in pytest.ini and thus to make “pyspark” importable in your tests which are executed by pytest.
Also it defines session scope fixture spark_context which can be used in your tests.
Install
$ pip install pytest-sparkUsage
Set Spark location
To run tests with required spark_home location you need to define it by using one of the following methods:
Specify command line option “–spark_home”:
$ pytest --spark_home=/opt/spark
Add “spark_home” value to pytest.ini in your project directory:
[pytest] spark_home = /opt/spark
Set the “SPARK_HOME” environment variable.
pytest-spark will try to import pyspark from provided location.
Using the spark_context fixture
Use fixture spark_context in your tests as a regular pyspark fixture. SparkContext instance will be created once and reused for the whole test session.
Example:
def test_my_case(spark_context):
test_rdd = spark_context.parallelize([1, 2, 3, 4])
# ...
Using the spark_session fixture (Spark 2.0 and above)
Use fixture spark_session in your tests as a regular pyspark fixture. A SparkSession instance with Hive support enabled will be created once and reused for the whole test session.
Example:
def test_spark_session_dataframe(spark_session):
test_df = spark_session.createDataFrame([[1,3],[2,4]], "a: int, b: int")
# ...
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file pytest-spark-0.4.3.tar.gz.
File metadata
- Download URL: pytest-spark-0.4.3.tar.gz
- Upload date:
- Size: 3.6 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
209c0f68589155d0a3cfa03dfd4101afbcce3ebec59f3161753b6f6326c07cf3
|
|
| MD5 |
a72978f73e90d9cf5417b2f57ca7f8f1
|
|
| BLAKE2b-256 |
cf48f4aa2f54e004da440a6af32ea3b1db879694bf19ee7ba38e3d3fcf40c41f
|
File details
Details for the file pytest_spark-0.4.3-py2.py3-none-any.whl.
File metadata
- Download URL: pytest_spark-0.4.3-py2.py3-none-any.whl
- Upload date:
- Size: 5.4 kB
- Tags: Python 2, Python 3
- Uploaded using Trusted Publishing? No
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
6020bf2aece0d5f95843c304202c65cc92ce7ad87013d0b999e21b222c504516
|
|
| MD5 |
a4177aeace57442c946f0e31af85723d
|
|
| BLAKE2b-256 |
00d6aa05546d296563cd97fd1b42e494964249dd48faf8fa9d417d33d2abc00f
|
