Spark testing for python
Base classes to use when writing Python tests with Spark.
This is a fork of the following project to package into Pypi: https://github.com/holdenk/spark-testing-base
You’ve written an awesome program in Spark and now its time to write some tests. Only you find yourself writing the code to setup and tear down local mode Spark in between each suite and you say to your self: This is not my beautiful code.
pip install spark-testing-base
From now on, you can use the following base classes in your unit tests:
SparkTestingBaseTestCase: Basic common test case for Spark. Provides a Spark context as sc. For non local mode testing you can either override sparkMaster or set the enviroment property SPARK_MASTER for non-local mode testing SparkTestingBaseReuse Basic common test case for Spark. Provides a Spark context as sc. For non local mode testing you can either override sparkMaster or set the enviroment property SPARK_MASTER for non-local mode testing StreamingTestCase Basic common test case for Spark Streaming tests. Provides a Spark Streaming context as well as some helper methods for creating streaming input and collecting streaming output.
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Hashes for spark-testing-base-0.0.7.post2.tar.gz