A test framework for Python Spark application development
Project description
spark-tests
A test framework, that defines several test doubles, to facilitate Python Spark application development.
spark_tests.sql module
Defines the following test doubles:
FakeSparkSession
- Stubs
sql(sql_query)
method to only log thesql_queries
, not sending them to database for execution; table(table_mame)
andcreateDataFrame(data[, schema, samplingRatio, verifySchema])
methods delegate execution to the realSparkSession
, but returns aFakeDataFrame
instead of aDataFrame
;table(table_name)
is often overridden in a subclass to return a table from a fake test database.
- Stubs
FakeDataFrame
write
returns aFakeDFWriter
;- Other methods work just like a real
DataFrame
, but returnFakeDataFrame
s instead ofDataFrame
s;
FakeDFWriter
- Stubs a
DataFrameWriter
to only logRow
s written, not writing them at all.
- Stubs a
spark_tests.delta module
Defines FakeDeltaTable
, that stubs merge(source, condition)
to only log the merge operation, changing no data.
spark_tests.datetime module
Defines the following test doubles:
FakeDatetime
- Stubs
now()
method to return always a predefineddatetime
.
- Stubs
FakeDate
- Stubs
today()
method to return always a predefineddate
.
- Stubs
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
pyspark_tests-1.3.4.tar.gz
(65.1 kB
view hashes)
Built Distribution
Close
Hashes for pyspark_tests-1.3.4-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8b4db7c786870721da53b043fe7b2a40ac4836b2db3a7cc16a6cde72c52034fe |
|
MD5 | e18701ebf384b5703c39a5df0fe2e64d |
|
BLAKE2b-256 | 02da935928a2103231890644b26511c3f316304f762ab75ecbcc73d168db14f0 |