This release is a pre-release and may not be stable for production use.
Openlineage Prefect
Basic Configuration
At a minimum, define a namespace, Prefect API URL and transport consisting of a type, URL and endpoint. All three can be supplied using environment variables.
For example:
export OPENLINEAGE_NAMESPACE='prefect_test' &&
export OPENLINEAGE__TRANSPORT__TYPE='http' &&
export OPENLINEAGE__TRANSPORT__URL='http://lineageconsumer.com:5000' &&
export OPENLINEAGE__TRANSPORT__ENDPOINT='/api/v1/lineage' &&
export PREFECT_API_URL='http://prefecthost.com:4200/api'
For more details about OpenLineage transport options and how to configure them, consult the OpenLineage Python Client Documentation.
Execution
Import the openlineage_prefect package and execute collect_and_process_runs() asynchronously in its own process.
For example:
import asyncio
import openlineage_prefect
from openlineage_prefect.prefect_adapter.listener import PrefectOpenLineageListener
async def main():
await PrefectOpenLineageListener().collect_and_process_runs()
if __name__ == "__main__":
asyncio.run(main())
Datasets
The integration looks for datasets in Prefect Artifacts. To attach input and output datasets to job runs, use create_table_artifact() from the Artifact library. Provide a namespace, typically the dataset URI, and name to the adapter via an artifact's table and description, respectively. Distinguish the type of dataset by appending _output or _input to the description.
For example:
ol_table = [{"database_uri":"duckdb:///customers_db", "table":"customers"}]
create_table_artifact(
key="upstream-insert",
table=ol_table,
description="ol-dataset_output"
)
1 maintainer
Metadata
1 maintainer
Release files for openlineage-prefect 0.0.1b3
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| openlineage_prefect-0.0.1b3.tar.gz | 6.7 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| openlineage_prefect-0.0.1b3-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 16.3 kB
Release files / openlineage_prefect-0.0.1b3.tar.gz
| Download URL | openlineage_prefect-0.0.1b3.tar.gz |
|---|---|
| Size | 6.7 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
4af0c9b1b7731b1137acee674081cd9fc7ca943602c71ff30fcd95310a8e987d
|
|
BLAKE2b-256 checksum How to use checksums |
09ba88eb30c4b48802f0c1cce108ac45821a744bc6e9e3204bc784a89b10250a
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.14.7
|
Release files / openlineage_prefect-0.0.1b3-py3-none-any.whl
| Download URL | openlineage_prefect-0.0.1b3-py3-none-any.whl |
|---|---|
| Size | 9.5 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
0e21a17520c5dffc16b1901285fa5e84203dd77c148d8a14a808b8858904576d
|
|
BLAKE2b-256 checksum How to use checksums |
ddab88fdcd95de91e43726c3f1bc5fb9ff869516beac941ee3897666a507c3f8
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.14.7
|
