Utility for testing PySpark code
Project description
mockrdd
A Python3 module for testing PySpark code.
The MockRDD class offers similar behavior to pyspark.RDD with the following extra benefits.
- Extensive sanity checks to identify invalid inputs
- More meaningful error messages for debugging issues
- Straightforward to running within pdb
- Removes Spark dependencies from development and testing environments
- No Spark overhead when running through a large test suite
Simple example of using MockRDD in a test.
from mockrdd import MockRDD
def job(rdd):
return rdd.map(lambda x: x*2).filter(lambda x: x>3)
assert job(MockRDD.empty()).collect() == []
assert job(MockRDD.of(1)).collect() == []
assert job(MockRDD.of(2)).collect() == [4]
Conventionally, you'd include a main method to hook the RDD up to product sources and sinks. Further, the testing would be included in a separate file and use the module unittest for defining test cases.
See the docstring of mockrdd.MockRDD for more information.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
mockrdd-0.0.2.tar.gz
(7.1 kB
view details)
Built Distribution
File details
Details for the file mockrdd-0.0.2.tar.gz
.
File metadata
- Download URL: mockrdd-0.0.2.tar.gz
- Upload date:
- Size: 7.1 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.1.0 requests-toolbelt/0.8.0 tqdm/4.24.0 CPython/3.6.6
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 |
4947716fae504620f6c5b4c7ba0db8ad20230992a063cdacb257d80705da7cb5
|
|
MD5 |
10e13bd2e8e57a0550dedcf3c99af76b
|
|
BLAKE2b-256 |
321d54e0aaea920a213408ab41ec8bf92eae2e155da2c4d848fae542c7f5dc7e
|
File details
Details for the file mockrdd-0.0.2-py3-none-any.whl
.
File metadata
- Download URL: mockrdd-0.0.2-py3-none-any.whl
- Upload date:
- Size: 7.3 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.1.0 requests-toolbelt/0.8.0 tqdm/4.24.0 CPython/3.6.6
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 |
c32f7c365c6bbd5125747ae5da27a7ae2fecab377b1f803230799640e71b0a04
|
|
MD5 |
2ef5c945192087eb9c18fce2eb5815a9
|
|
BLAKE2b-256 |
5e49cc8809691187660f6562352df11b3b35d30e825059fa4f78e21257533cd2
|