This release is a pre-release and may not be stable for production use.
Openlineage Prefect
Basic Configuration
At a minimum, define a namespace, Prefect API URL and transport consisting of a type, URL and endpoint. All three can be supplied using environment variables.
For example:
export OPENLINEAGE_NAMESPACE='prefect_test' &&
export OPENLINEAGE__TRANSPORT__TYPE='http' &&
export OPENLINEAGE__TRANSPORT__URL='http://lineageconsumer.com:5000' &&
export OPENLINEAGE__TRANSPORT__ENDPOINT='/api/v1/lineage' &&
export PREFECT_API_URL='http://prefecthost.com:4200/api'
For more details about OpenLineage transport options and how to configure them, consult the OpenLineage Python Client Documentation.
Execution
Import the openlineage_prefect package and execute collect_and_process_runs() asynchronously in its own process.
For example:
import asyncio
import openlineage_prefect
from openlineage_prefect.prefect_adapter.listener import PrefectOpenLineageListener
async def main():
await PrefectOpenLineageListener().collect_and_process_runs()
if __name__ == "__main__":
asyncio.run(main())
Datasets
The integration looks for datasets in Prefect Artifacts. To attach input and output datasets to job runs, use create_table_artifact() from the Artifact library. Provide a namespace, typically the dataset URI, and name to the adapter via an artifact's table and description, respectively. Distinguish the type of dataset by appending _output or _input to the description.
For example:
ol_table = [{"database_uri":"duckdb:///customers_db", "table":"customers"}]
create_table_artifact(
key="upstream-insert",
table=ol_table,
description="ol-dataset_output"
)
1 maintainer
Metadata
1 maintainer
Release files for openlineage-prefect 0.0.1b4
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| openlineage_prefect-0.0.1b4.tar.gz | 6.8 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| openlineage_prefect-0.0.1b4-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 16.3 kB
Release files / openlineage_prefect-0.0.1b4.tar.gz
| Download URL | openlineage_prefect-0.0.1b4.tar.gz |
|---|---|
| Size | 6.8 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
3742dd07586927447b91f6c6b58e9212c53a1c0a550bfc56247bdbedcc67868f
|
|
BLAKE2b-256 checksum How to use checksums |
c5d988da05cdb6f7d82c1758b936517a844bec69b5ecb61efa7fbd69afb50b41
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.14.7
|
Release files / openlineage_prefect-0.0.1b4-py3-none-any.whl
| Download URL | openlineage_prefect-0.0.1b4-py3-none-any.whl |
|---|---|
| Size | 9.5 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
7070073aea1b418402d4cad374e2d6f0b0f030e2d7471f86eb5918bb27c0c3ee
|
|
BLAKE2b-256 checksum How to use checksums |
dab3efc67723575377a76f2152a1dabfa34d1d3a3d5bc73e1386f8e63f8bf6d2
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.14.7
|
