prophecy-build-tool
Prophecy is designed to enable all users to be productive with data engineering. It also replaces legacy ETL products. To learn more about Prophecy visit https://docs.prophecy.io/.
Prophecy-build-tool (PBT) allows you to quickly build projects generated by Prophecy (your standard Spark Scala and PySpark pipelines) to integrate them with your own CI / CD (e.g. Github Actions), build system (e.g. Jenkins), and orchestration (e.g. Databricks Workflows).
For the latest information on how to use Prophecy-build-tool, please visit the Prophecy Build Tool documentation here.
Installation
To install PBT, simply run:
pip3 install prophecy-build-tool
Running
To build and deploy your Prophecy project containing Python/Scala projects and Databricks Jobs run
pbt deploy --path /path/to/your/prophecy_project/
Sample output:
Prophecy-build-tool v1.0.0
Found 1 pipelines: customers_orders (python)
Found 1 jobs: daily_report
Building 1 pipeline 🚰
Building pipeline pipelines/customers_orders [1/1]
running build
running build_py
✅ Build complete!
Deploying 1 job ⏱
Deploying job jobs/daily_report [1/1]
Uploading cs-1.0-py3-none-any.whl to dbfs:/FileStore/prophecy/artifacts/...
Updating an existing job: daily_report
✅ Deployment complete!
Build
To just build your Prophecy project containing Python/Scala projects and Databricks Jobs run
pbt build --path /path/to/your/prophecy_project/
Deploy
To build and Deploy all jobs in your project, run deploy command
pbt deploy --path /path/to/your/prophecy_project/
It's also possible to only deploy jobs associated with given fabrics, you can provide fabrics id's (comma separated) to just deploy only those jobs.
pbt deploy --fabric-ids 3,7 --path /path/to/your/prophecy_project/
By default, deploy command builds all pipelines and then deploys them, if you want to skip building all pipelines
( this could be useful, if you are running a deploy command after running deploy or build previously.)
pbt deploy --skip-builds --path /path/to/your/prophecy_project/
Test
To run all unit tests in your Prophecy project containing Python/Scala projects and Databricks Jobs run
pbt test --path /path/to/your/prophecy_project/
Sample output:
Prophecy-build-tool v1.0.1
Found 1 jobs: daily
Found 1 pipelines: customers_orders (python)
Unit Testing pipeline pipelines/customers_orders [1/1]
============================= test session starts ==============================
platform darwin -- Python 3.8.9, pytest-7.1.2, pluggy-1.0.0 -- /Library/Developer/CommandLineTools/usr/bin/python
cachedir: .pytest_cache
metadata: None
rootdir: /Users/kiran/proj2/pipelines/customers_orders/code
plugins: html-3.1.1, metadata-2.0.2
collecting ... collected 1 item
test/TestSuite.py::CleanupTest::test_unit_test_0 PASSED [100%]
============================== 1 passed in 17.42s ==============================
✅ Unit test for pipeline: pipelines/customers_orders succeeded.
Validate
To quickly validate if all your pipelines are not broken
pbt validate --path /path/to/your/prophecy_project/
Sample output:
Prophecy-build-tool v1.0.3.4
Project name: resources.HelloWorld
Found 1 jobs: default_schedule
Found 4 pipelines: customers_orders (python), report_top_customers (python), join_agg_sort (python), farmers-markets-irs (python)
Validating 4 pipelines
Validating pipeline pipelines/customers_orders [1/4]
Pipeline is validated: customers_orders
Validating pipeline pipelines/report_top_customers [2/4]
Pipeline is validated: report_top_customers
Validating pipeline pipelines/join_agg_sort [3/4]
Pipeline is validated: join_agg_sort
Validating pipeline pipelines/farmers-markets-irs [4/4]
Pipeline is validated: farmers-markets-irs
Integrating with Github actions
Prophecy-built-tool (PBT) can be integrated with CI/CD (eg: Github Actions). A sample Github Actions .yml file that builds, tests and deploys the project on every code push is mentioned below:
name: Example CI
on: [push]
env:
DATABRICKS_HOST: "https://<<databricks-host>>.databricks.com"
DATABRICKS_TOKEN: "<<databricks-token>>"
FABRIC_NAME: "fabric-name"
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Set up JDK 17
uses: actions/setup-java@v3
with:
java-version: '17'
distribution: 'adopt'
- name: Set up Python 3.x
uses: actions/setup-python@v4
with:
python-version: '3.x'
# Install all python dependencies
# prophecy-libs not included here because prophecy-build-tool
# takes care of it by reading each pipeline's setup.py
- name: Install dependencies
run: |
python3 -m pip install --upgrade pip
pip3 install build pytest wheel pytest-html pyspark prophecy-build-tool
- name: Run PBT build
run: pbt build --path .
- name: Run PBT test
run: pbt test --path .
- name: Run PBT deploy
run: pbt deploy --path .
Metadata
Release files for prophecy-build-tool 1.4.6
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| prophecy_build_tool-1.4.6.tar.gz | 117.0 kB | Without provenance | Details for prophecy_build_tool-1.4.6.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| prophecy_build_tool-1.4.6-py3-none-any.whl | Python 3 | none | any | Details for prophecy_build_tool-1.4.6-py3-none-any.whl |
Total release size: 232.9 kB
Release files / prophecy_build_tool-1.4.6.tar.gz
| Download URL | prophecy_build_tool-1.4.6.tar.gz |
|---|---|
| Size | 117.0 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
2defbccb95e6e835039a72e824e14a7c25e4244e22efcc3ca24ffeb1d3fc2936
|
|
BLAKE2b-256 checksum How to use checksums |
0a187d4ceb901b920837056d1006be1dbdcdb50650274b128e40a36ba62a6a44
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.9.25
|
Release files / prophecy_build_tool-1.4.6-py3-none-any.whl
| Download URL | prophecy_build_tool-1.4.6-py3-none-any.whl |
|---|---|
| Size | 115.9 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
fd3633c499422ae65d98a6a65f058a78a414afae5079a9ade9a287b1388824cd
|
|
BLAKE2b-256 checksum How to use checksums |
6cea13b48822e805200748a69e79219de7f8f0177ba487a83a0d7f572b5cde63
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.9.25
|
Security for prophecy-build-tool 1.4.6
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.