Skip to main content

Workflow orchestration and management.

Project description

PyPI

Prefect

Prefect is a workflow orchestration framework for building data pipelines in Python. It's the simplest way to elevate a script into a resilient production workflow. With Prefect, you can build resilient, dynamic data pipelines that react to the world around them and recover from unexpected changes.

With just a few lines of code, data teams can confidently automate any data process with features such as scheduling, caching, retries, and event-based automations.

Workflow activity is tracked and can be monitored with a self-hosted Prefect server instance or managed Prefect Cloud dashboard.

Getting started

Prefect requires Python 3.9 or later. To install the latest or upgrade to the latest version of Prefect, run the following command:

pip install -U prefect

Then create and run a Python file that uses Prefect flow and task decorators to orchestrate and observe your workflow - in this case, a simple script that fetches the number of GitHub stars from a repository:

from prefect import flow, task
from typing import List
import httpx


@task(log_prints=True)
def get_stars(repo: str):
    url = f"https://api.github.com/repos/{repo}"
    count = httpx.get(url).json()["stargazers_count"]
    print(f"{repo} has {count} stars!")


@flow(name="GitHub Stars")
def github_stars(repos: List[str]):
    for repo in repos:
        get_stars(repo)


# run the flow!
if __name__=="__main__":
    github_stars(["PrefectHQ/Prefect"])

Fire up the Prefect UI to see what happened:

prefect server start

To run your workflow on a schedule, turn it into a deployment and schedule it to run every minute by changing the last line of your script to the following:

if __name__ == "__main__":
    github_stars.serve(
        name="first-deployment",
        cron="* * * * *",
        parameters={"repos": ["PrefectHQ/prefect"]}
    )

You now have a server running locally that is looking for scheduled deployments! Additionally you can run your workflow manually from the UI or CLI. You can even run deployments in response to events.

Prefect Cloud

Prefect Cloud provides workflow orchestration for the modern data enterprise. By automating over 200 million data tasks monthly, Prefect empowers diverse organizations — from Fortune 50 leaders such as Progressive Insurance to innovative disruptors such as Cash App — to increase engineering productivity, reduce pipeline errors, and cut data workflow compute costs.

Read more about Prefect Cloud here or sign up to try it for yourself.

prefect-client

If your use case is geared towards communicating with Prefect Cloud or a remote Prefect server, check out our prefect-client. It is a lighter-weight option for accessing client-side functionality in the Prefect SDK and is ideal for use in ephemeral execution environments.

Next steps

Project details


Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

prefect-3.1.1.dev2.tar.gz (5.8 MB view details)

Uploaded Source

Built Distribution

prefect-3.1.1.dev2-py3-none-any.whl (6.1 MB view details)

Uploaded Python 3

File details

Details for the file prefect-3.1.1.dev2.tar.gz.

File metadata

  • Download URL: prefect-3.1.1.dev2.tar.gz
  • Upload date:
  • Size: 5.8 MB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/5.1.1 CPython/3.12.7

File hashes

Hashes for prefect-3.1.1.dev2.tar.gz
Algorithm Hash digest
SHA256 123013425d12ab5d0d7b9fb8f2995c476979e7ec4ce9057da264c29866bc035a
MD5 e3acfb2412fb9ec0c37c23bb231fa0d6
BLAKE2b-256 6a7a5756c15fb9f73e04aff4b1952195c2f034fbc6d8fba76e60b60f2fa9c676

See more details on using hashes here.

Provenance

The following attestation bundles were made for prefect-3.1.1.dev2.tar.gz:

Publisher: python-package.yaml on PrefectHQ/prefect

Attestations:

File details

Details for the file prefect-3.1.1.dev2-py3-none-any.whl.

File metadata

File hashes

Hashes for prefect-3.1.1.dev2-py3-none-any.whl
Algorithm Hash digest
SHA256 1618891c2483b62fcdbd6ae5ed8d9bc441c299e2999cfbb19c6ae132197f880d
MD5 baf80f6204126d1cb899bb5da3ba966e
BLAKE2b-256 483cb2fa37bb0e76d7fdbb5629955f98071609b97d3b217dbc0256742cdf747a

See more details on using hashes here.

Provenance

The following attestation bundles were made for prefect-3.1.1.dev2-py3-none-any.whl:

Publisher: python-package.yaml on PrefectHQ/prefect

Attestations:

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page