Skip to main content
Help us improve Python packaging – donate today!

pytest plugin to run the tests with support of pyspark.

Project Description

pytest plugin to run the tests with support of pyspark (Apache Spark).

This plugin will allow to specify SPARK_HOME directory in pytest.ini and thus to make “pyspark” importable in your tests which are executed by pytest.

Also it defines session scope fixture spark_context which can be used in your tests.

Install

$ pip install pytest-spark

Usage

Set Spark location

To run tests with required spark_home location you need to define it by using one of the following methods:

  1. Specify command line option “–spark_home”:

    $ pytest --spark_home=/opt/spark
    
  2. Add “spark_home” value to pytest.ini in your project directory:

    [pytest]
    spark_home = /opt/spark
    
  3. Set the “SPARK_HOME” environment variable.

pytest-spark will try to import pyspark from provided location.

Note

“spark_home” will be read in the specified order. i.e. you can override pytest.ini value by command line option.

Using the spark_context fixture

Use fixture spark_context in your tests as a regular pyspark fixture. SparkContext instance will be created once and reused for the whole test session.

Example:

def test_my_case(spark_context):
    test_rdd = spark_context.parallelize([1, 2, 3, 4])
    # ...

Using the spark_session fixture (Spark 2.0 and above)

Use fixture spark_session in your tests as a regular pyspark fixture. A SparkSession instance with Hive support enabled will be created once and reused for the whole test session.

Example:

def test_spark_session_dataframe(spark_session):
    test_df = spark_session.createDataFrame([[1,3],[2,4]], "a: int, b: int")
    # ...

Release history Release notifications

This version
History Node

0.4.4

History Node

0.4.3

History Node

0.4.1

History Node

0.4.0

History Node

0.3.1

History Node

0.3.0

History Node

0.2.0

History Node

0.1.2

History Node

0.1.1

History Node

0.1.0

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Filename, size & hash SHA256 hash help File type Python version Upload date
pytest_spark-0.4.4-py2.py3-none-any.whl (5.3 kB) Copy SHA256 hash SHA256 Wheel py2.py3 Dec 2, 2017
pytest-spark-0.4.4.tar.gz (3.5 kB) Copy SHA256 hash SHA256 Source None Dec 2, 2017

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging CloudAMQP CloudAMQP RabbitMQ AWS AWS Cloud computing Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page