Skip to main content

Utility for testing PySpark code

Project description

mockrdd

A Python3 module for testing PySpark code.

The MockRDD class offers similar behavior to pyspark.RDD with the following extra benefits.

  • Extensive sanity checks to identify invalid inputs
  • More meaningful error messages for debugging issues
  • Straightforward to running within pdb
  • Removes Spark dependencies from development and testing environments
  • No Spark overhead when running through a large test suite

Simple example of using MockRDD in a test.

from mockrdd import MockRDD

def job(rdd):
    return rdd.map(lambda x: x*2).filter(lambda x: x>3)

assert job(MockRDD.empty()).collect() == [] 
assert job(MockRDD.of(1)).collect() == [] 
assert job(MockRDD.of(2)).collect() == [4] 

Conventionally, you'd include a main method to hook the RDD up to product sources and sinks. Further, the testing would be included in a separate file and use the module unittest for defining test cases.

See the docstring of mockrdd.MockRDD for more information.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

mockrdd-0.0.2.tar.gz (7.1 kB view details)

Uploaded Source

Built Distribution

mockrdd-0.0.2-py3-none-any.whl (7.3 kB view details)

Uploaded Python 3

File details

Details for the file mockrdd-0.0.2.tar.gz.

File metadata

  • Download URL: mockrdd-0.0.2.tar.gz
  • Upload date:
  • Size: 7.1 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.1.0 requests-toolbelt/0.8.0 tqdm/4.24.0 CPython/3.6.6

File hashes

Hashes for mockrdd-0.0.2.tar.gz
Algorithm Hash digest
SHA256 4947716fae504620f6c5b4c7ba0db8ad20230992a063cdacb257d80705da7cb5
MD5 10e13bd2e8e57a0550dedcf3c99af76b
BLAKE2b-256 321d54e0aaea920a213408ab41ec8bf92eae2e155da2c4d848fae542c7f5dc7e

See more details on using hashes here.

File details

Details for the file mockrdd-0.0.2-py3-none-any.whl.

File metadata

  • Download URL: mockrdd-0.0.2-py3-none-any.whl
  • Upload date:
  • Size: 7.3 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.1.0 requests-toolbelt/0.8.0 tqdm/4.24.0 CPython/3.6.6

File hashes

Hashes for mockrdd-0.0.2-py3-none-any.whl
Algorithm Hash digest
SHA256 c32f7c365c6bbd5125747ae5da27a7ae2fecab377b1f803230799640e71b0a04
MD5 2ef5c945192087eb9c18fce2eb5815a9
BLAKE2b-256 5e49cc8809691187660f6562352df11b3b35d30e825059fa4f78e21257533cd2

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page