A test framework for Python Spark application development
Project description
spark-tests
A test framework, that defines several test doubles, to facilitate Python Spark application development.
spark_tests.sql module
Defines the following test doubles:
FakeSparkSession
- Stubs
sql(sql_query)
method to only log thesql_queries
, not sending them to database for execution; table(table_mame)
andcreateDataFrame(data[, schema, samplingRatio, verifySchema])
methods delegate execution to the realSparkSession
, but returns aFakeDataFrame
instead of aDataFrame
;table(table_name)
is often overridden in a subclass to return a table from a fake test database.
- Stubs
FakeDataFrame
write
returns aFakeDFWriter
;- Other methods work just like a real
DataFrame
, but returnFakeDataFrame
s instead ofDataFrame
s;
FakeDFWriter
- Stubs a
DataFrameWriter
to only logRow
s written, not writing them at all.
- Stubs a
spark_tests.delta module
Defines FakeDeltaTable
, that stubs merge(source, condition)
to only log the merge operation, changing no data.
spark_tests.datetime module
Defines the following test doubles:
FakeDatetime
- Stubs
now()
method to return always a predefineddatetime
.
- Stubs
FakeDate
- Stubs
today()
method to return always a predefineddate
.
- Stubs
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
pyspark-tests-1.3.5.tar.gz
(6.2 kB
view hashes)
Built Distribution
Close
Hashes for pyspark_tests-1.3.5-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e4cc720e12ffa3a9c0344b83a4701b13c385fcccd2adc9f8ac9b0bb23f59ff22 |
|
MD5 | 0851858988a1fd1fe8db8d320060c259 |
|
BLAKE2b-256 | 33ceacc6c6fc957b050e9715e4e908eed7bc0352de5fa2b95b87bed65544132a |