A test framework for Python Spark application development.
Project description
spark-tests
A test framework, that defines several test doubles, to facilitate Python Spark application development.
spark_tests.sql module
Defines the following test doubles:
FakeSparkSession
- Stubs
sql(sql_query)
method to only log thesql_queries
, not sending them to database for execution; table(table_mame)
andcreateDataFrame(data[, schema, samplingRatio, verifySchema])
methods delegate execution to the realSparkSession
, but returns aFakeDataFrame
instead of aDataFrame
;table(table_name)
is often overridden in a subclass to return a table from a fake test database.
- Stubs
FakeDataFrame
write
returns aFakeDFWriter
;- Other methods work just like a real
DataFrame
, but returnFakeDataFrame
s instead ofDataFrame
s;
FakeDFWriter
- Stubs a
DataFrameWriter
to only logRow
s written, not writing them at all.
- Stubs a
spark_tests.delta module
Defines FakeDeltaTable
, that stubs merge(source, condition)
to only log the merge operation, changing no data.
spark_tests.datetime module
Defines the following test doubles:
FakeDatetime
- Stubs
now()
method to return always a predefineddatetime
.
- Stubs
FakeDate
- Stubs
today()
method to return always a predefineddate
.
- Stubs
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
pyspark-tests-1.3.2.tar.gz
(10.4 kB
view hashes)
Built Distribution
Close
Hashes for pyspark_tests-1.3.2-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d9a0de5eaf20e2a0e685c8704af539c943be725811796f63f2b131c81642e859 |
|
MD5 | 6a8b798b165e3fff9f808e28bc02141d |
|
BLAKE2b-256 | 7d743147a25385f7d5fe61694b161ead74724bf95647910169576713b718ea18 |