prophecy-build-tool
Prophecy is designed to enable all users to be productive with data engineering. It also replaces legacy ETL products. To learn more about Prophecy visit https://docs.prophecy.io/.
Prophecy-build-tool (PBT) allows you to quickly build projects generated by Prophecy (your standard Spark Scala and PySpark pipelines) to integrate them with your own CI / CD (e.g. Github Actions), build system (e.g. Jenkins), and orchestration (e.g. Databricks Workflows).
For the latest information on how to use Prophecy-build-tool, please visit the Prophecy Build Tool documentation here.
Installation
To install PBT, simply run:
pip3 install prophecy-build-tool
Running
To build and deploy your Prophecy project containing Python/Scala projects and Databricks Jobs run
pbt deploy --path /path/to/your/prophecy_project/
Sample output:
Prophecy-build-tool v1.0.0
Found 1 pipelines: customers_orders (python)
Found 1 jobs: daily_report
Building 1 pipeline 🚰
Building pipeline pipelines/customers_orders [1/1]
running build
running build_py
✅ Build complete!
Deploying 1 job ⏱
Deploying job jobs/daily_report [1/1]
Uploading cs-1.0-py3-none-any.whl to dbfs:/FileStore/prophecy/artifacts/...
Updating an existing job: daily_report
✅ Deployment complete!
Build
To just build your Prophecy project containing Python/Scala projects and Databricks Jobs run
pbt build --path /path/to/your/prophecy_project/
Deploy
To build and Deploy all jobs in your project, run deploy command
pbt deploy --path /path/to/your/prophecy_project/
It's also possible to only deploy jobs associated with given fabrics, you can provide fabrics id's (comma separated) to just deploy only those jobs.
pbt deploy --fabric-ids 3,7 --path /path/to/your/prophecy_project/
By default, deploy command builds all pipelines and then deploys them, if you want to skip building all pipelines
( this could be useful, if you are running a deploy command after running deploy or build previously.)
pbt deploy --skip-builds --path /path/to/your/prophecy_project/
Test
To run all unit tests in your Prophecy project containing Python/Scala projects and Databricks Jobs run
pbt test --path /path/to/your/prophecy_project/
Sample output:
Prophecy-build-tool v1.0.1
Found 1 jobs: daily
Found 1 pipelines: customers_orders (python)
Unit Testing pipeline pipelines/customers_orders [1/1]
============================= test session starts ==============================
platform darwin -- Python 3.8.9, pytest-7.1.2, pluggy-1.0.0 -- /Library/Developer/CommandLineTools/usr/bin/python
cachedir: .pytest_cache
metadata: None
rootdir: /Users/kiran/proj2/pipelines/customers_orders/code
plugins: html-3.1.1, metadata-2.0.2
collecting ... collected 1 item
test/TestSuite.py::CleanupTest::test_unit_test_0 PASSED [100%]
============================== 1 passed in 17.42s ==============================
✅ Unit test for pipeline: pipelines/customers_orders succeeded.
Validate
To quickly validate if all your pipelines are not broken
pbt validate --path /path/to/your/prophecy_project/
Sample output:
Prophecy-build-tool v1.0.3.4
Project name: resources.HelloWorld
Found 1 jobs: default_schedule
Found 4 pipelines: customers_orders (python), report_top_customers (python), join_agg_sort (python), farmers-markets-irs (python)
Validating 4 pipelines
Validating pipeline pipelines/customers_orders [1/4]
Pipeline is validated: customers_orders
Validating pipeline pipelines/report_top_customers [2/4]
Pipeline is validated: report_top_customers
Validating pipeline pipelines/join_agg_sort [3/4]
Pipeline is validated: join_agg_sort
Validating pipeline pipelines/farmers-markets-irs [4/4]
Pipeline is validated: farmers-markets-irs
Integrating with Github actions
Prophecy-built-tool (PBT) can be integrated with CI/CD (eg: Github Actions). A sample Github Actions .yml file that builds, tests and deploys the project on every code push is mentioned below:
name: Example CI
on: [push]
env:
DATABRICKS_HOST: "https://<<databricks-host>>.databricks.com"
DATABRICKS_TOKEN: "<<databricks-token>>"
FABRIC_NAME: "fabric-name"
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Set up JDK 17
uses: actions/setup-java@v3
with:
java-version: '17'
distribution: 'adopt'
- name: Set up Python 3.x
uses: actions/setup-python@v4
with:
python-version: '3.x'
# Install all python dependencies
# prophecy-libs not included here because prophecy-build-tool
# takes care of it by reading each pipeline's setup.py
- name: Install dependencies
run: |
python3 -m pip install --upgrade pip
pip3 install build pytest wheel pytest-html pyspark prophecy-build-tool
- name: Run PBT build
run: pbt build --path .
- name: Run PBT test
run: pbt test --path .
- name: Run PBT deploy
run: pbt deploy --path .
2 maintainers
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file prophecy_build_tool-1.4.4.tar.gz.
File metadata
- Download URL: prophecy_build_tool-1.4.4.tar.gz
- Upload date:
- Size: 111.8 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via:
twine/6.2.0 CPython/3.9.25
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
64a0a807bca312d7e6f2b6fe9eba6b91638d78796c82c047740515aab37258cb
|
|
| MD5 |
46f17cfc69ef271626a77e4bad8361de
|
|
| BLAKE2b-256 |
3e516375b800a20c61bdf1c1d3b5377c163b85ec0fadb8c94c8479865ac4cf4a
|
File details
Details for the file prophecy_build_tool-1.4.4-py3-none-any.whl.
File metadata
- Download URL: prophecy_build_tool-1.4.4-py3-none-any.whl
- Upload date:
- Size: 111.4 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via:
twine/6.2.0 CPython/3.9.25
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
818d6506b13f8618bbe7e9163d5bd5673d1a7c401c8bc450eb262f21dfec49b2
|
|
| MD5 |
a842410df37094b9379e223e8c0027a4
|
|
| BLAKE2b-256 |
69232ec282b3fb7d9037a5600c653e4c14bbb6ece1e764b40c3d517db5331306
|
