SparkPipelineFramework.Tests
Testing framework that can tests SparkPipelineFramework library by just providing input files to setup before running the transformer and output files to use for verifying the output
Usage
- Create a folder structure similar to the folder structure of your library in SparkPipelineFramework (This is how the Testing Framework finds the Transformer to run)
- Create an input folder and put in files that represent the input views. These files can be csv, json or parquet
- (Optionally) Create an input_schema folder and put in any schemas you want applied to the above views. This follows the Spark Json Schema format.
- (Optional) Create an output folder and put in files that represent the output views you expect. These files can be csv, json or parquet
- (Optional) Create an output_schema folder and put in any schemas you want applied to the output views
- Copy the following test code and put it in a test file in this folder
from pathlib import Path
from pyspark.sql import SparkSession
from spark_pipeline_framework_testing.test_runner import SparkPipelineFrameworkTestRunner
def test_folder(spark_session: SparkSession) -> None:
data_dir: Path = Path(__file__).parent.joinpath('./')
SparkPipelineFrameworkTestRunner.run_tests(spark_session=spark_session, folder_path=data_dir)
- Now just run this test.
Note: the test finds files in sub-folders too.
Example
For the transformer defined here: https://github.com/imranq2/SparkPipelineFramework.Testing/tree/main/library/features/people/my_people_feature You can find the test here: https://github.com/imranq2/SparkPipelineFramework.Testing/tree/main/tests/library/features/people/my_people_feature
Publishing a new package
- Create a new release
- The GitHub Action should automatically kick in and publish the package
- You can see the status in the Actions tab
1 maintainer
Metadata
1 maintainer
Release files for sparkpipelineframework.testing 4.0.6
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| sparkpipelineframework_testing-4.0.6.tar.gz | 43.6 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| sparkpipelineframework_testing-4.0.6-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 143.5 kB
Release files / sparkpipelineframework_testing-4.0.6.tar.gz
| Download URL | sparkpipelineframework_testing-4.0.6.tar.gz |
|---|---|
| Size | 43.6 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
81bc8d1252faac099840ef2bb7158eedca0318a09bb7cba93b365d613a93c2fd
|
|
BLAKE2b-256 checksum How to use checksums |
89fad64940a35bb1a8a1b08021dec57aa4ec81508a17df0f1a8827ac34281e9e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.12.14
|
Release files / sparkpipelineframework_testing-4.0.6-py3-none-any.whl
| Download URL | sparkpipelineframework_testing-4.0.6-py3-none-any.whl |
|---|---|
| Size | 99.9 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
71be1aa0ca9b2a1a3539791ad7e02cf6be7eb4b726f60eceaf35072e025def83
|
|
BLAKE2b-256 checksum How to use checksums |
d6e73e74fccb1295f702326fcf3c904b54e83bc2d82ae9390ddc4576a4c93ad4
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.12.14
|
