Apache Flink

Apache Flink is a framework and distributed processing engine for stateful computations over unbounded and bounded data streams. Flink has been designed to run in all common cluster environments, perform computations at in-memory speed and at any scale.

Learn more about Flink at https://flink.apache.org/

Python Packaging

PyFlink is a Python API for Apache Flink that allows you to build scalable batch and streaming workloads, such as real-time data processing pipelines, large-scale exploratory data analysis, Machine Learning (ML) pipelines and ETL processes. If you’re already familiar with Python and libraries such as Pandas, then PyFlink makes it simpler to leverage the full capabilities of the Flink ecosystem. Depending on the level of abstraction you need, there are two different APIs that can be used in PyFlink: PyFlink Table API and PyFlink DataStream API.

The PyFlink Table API allows you to write powerful relational queries in a way that is similar to using SQL or working with tabular data in Python. You can find more information about it via the tutorial https://nightlies.apache.org/flink/flink-docs-stable/docs/dev/python/table_api_tutorial/

The PyFlink DataStream API gives you lower-level control over the core building blocks of Flink, state and time, to build more complex stream processing use cases. Tutorial can be found at https://nightlies.apache.org/flink/flink-docs-stable/docs/dev/python/datastream_tutorial/

You can find more information via the documentation at https://nightlies.apache.org/flink/flink-docs-stable/docs/dev/python/overview/

The auto-generated Python docs can be found at https://nightlies.apache.org/flink/flink-docs-stable/api/python/

Python Requirements

Apache Flink Python API depends on Py4J (currently version 0.10.8.1), CloudPickle (currently version 1.2.2), python-dateutil(currently version 2.8.0), Apache Beam (currently version 2.27.0).

Development Notices

Protobuf Code Generation

Protocol buffer is used in file flink_fn_execution_pb2.py and the file is generated from flink-fn-execution.proto. Whenever flink-fn-execution.proto is updated, please re-generate flink_fn_execution_pb2.py by executing:

python pyflink/gen_protos.py

PyFlink depends on the following libraries to execute the above script:

  1. grpcio-tools (>=1.3.5,<=1.14.2)
  2. setuptools (>=37.0.0)
  3. pip (>=7.1.0)

Running Test Cases

Currently, we use conda and tox to verify the compatibility of the Flink Python API for multiple versions of Python and will integrate some useful plugins with tox, such as flake8. We can enter the directory where this README.md file is located and run test cases by executing

./dev/lint-python.sh

Project links

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (2.3.0)

41 maintainers

PyPI data
Data sourced directly from PyPI's database.
Avatar for afedulov from gravatar.com afedulov Avatar for AHeise from gravatar.com AHeise Avatar for alpinegizmo from gravatar.com alpinegizmo
+38 more Avatar for carp84 from gravatar.com carp84 Avatar for chesnay from gravatar.com chesnay Avatar for dannycranmer from gravatar.com dannycranmer Avatar for dianfu from gravatar.com dianfu Avatar for dwysakowicz from gravatar.com dwysakowicz Avatar for fapaul from gravatar.com fapaul Avatar for ferenc.csaky from gravatar.com ferenc.csaky Avatar for gaborgsomogyi from gravatar.com gaborgsomogyi Avatar for gejing from gravatar.com gejing Avatar for godfreyhe from gravatar.com godfreyhe Avatar for gyao from gravatar.com gyao Avatar for hongteoh from gravatar.com hongteoh Avatar for hxbks2ks from gravatar.com hxbks2ks Avatar for knaufk from gravatar.com knaufk Avatar for lincoln.lee from gravatar.com lincoln.lee Avatar for lindong from gravatar.com lindong Avatar for lzljs3620320 from gravatar.com lzljs3620320 Avatar for mapohl from gravatar.com mapohl Avatar for martijnvisser from gravatar.com martijnvisser Avatar for mxm from gravatar.com mxm Avatar for panyuepeng from gravatar.com panyuepeng Avatar for pyflink from gravatar.com pyflink Avatar for renqsapache from gravatar.com renqsapache Avatar for reswqa from gravatar.com reswqa Avatar for r_khachatryan from gravatar.com r_khachatryan Avatar for rmetzger from gravatar.com rmetzger Avatar for ron999_liu from gravatar.com ron999_liu Avatar for ruanhang from gravatar.com ruanhang Avatar for snuyanzin from gravatar.com snuyanzin Avatar for sunjincheng121 from gravatar.com sunjincheng121 Avatar for sxnan from gravatar.com sxnan Avatar for tangyun from gravatar.com tangyun Avatar for trohrmann from gravatar.com trohrmann Avatar for xtsong from gravatar.com xtsong Avatar for YunfengZhou from gravatar.com YunfengZhou Avatar for zakelly from gravatar.com zakelly Avatar for ZhijiangWang from gravatar.com ZhijiangWang Avatar for zhuzh from gravatar.com zhuzh

Credits

Author: Apache Software Foundation

License

Apache Software License (https://www.apache.org/licenses/LICENSE-2.0)

Requires

Python >=3.6

Classifiers

Report project as malware

Release files for apache-flink 1.14.4

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for apache-flink 1.14.4
File Size Uploaded
apache-flink-1.14.4.tar.gz 1.1 MB Details

Built distributions (wheels)

Table of built distributions (wheels) for apache-flink 1.14.4
File
apache_flink-1.14.4-cp38-cp38-manylinux1_x86_64.whl 5.6 MB CPython 3.8 CPython 3.8 Linux glibc 2.5+ x86-64 Details
apache_flink-1.14.4-cp38-cp38-macosx_10_9_x86_64.whl 1.9 MB CPython 3.8 CPython 3.8 macOS 10.9+ x86-64 Details
apache_flink-1.14.4-cp37-cp37m-manylinux1_x86_64.whl 4.9 MB CPython 3.7 CPython 3.7 pymalloc Linux glibc 2.5+ x86-64 Details
apache_flink-1.14.4-cp37-cp37m-macosx_10_9_x86_64.whl 1.9 MB CPython 3.7 CPython 3.7 pymalloc macOS 10.9+ x86-64 Details
apache_flink-1.14.4-cp36-cp36m-manylinux1_x86_64.whl 5.0 MB CPython 3.6 CPython 3.6 pymalloc Linux glibc 2.5+ x86-64 Details
apache_flink-1.14.4-cp36-cp36m-macosx_10_9_x86_64.whl 1.9 MB CPython 3.6 CPython 3.6 pymalloc macOS 10.9+ x86-64 Details

Total release size:22.4 MB

Release history Release notifications | RSS feed

2.3.0

13 release files

2.2.1

13 release files

2.2.0

13 release files

2.1.3

13 release files

2.1.2

13 release files

2.1.1

13 release files

2.1.0

13 release files

2.0.2

13 release files

2.0.1

13 release files

2.0.0

13 release files

1.20.5

13 release files

1.20.4

13 release files

1.20.3

13 release files

1.20.2

13 release files

1.20.1

13 release files

1.20.0

13 release files

1.19.3

13 release files

1.19.2

13 release files

1.19.1

13 release files

1.19.0

13 release files

1.18.1

12 release files

1.18.0

12 release files

1.17.2

12 release files

1.17.1

12 release files

1.17.0

12 release files

1.16.3

11 release files

1.16.2

11 release files

1.16.1

11 release files

1.16.0

11 release files

1.15.4

7 release files

1.15.3

7 release files

1.15.2

7 release files

1.15.1

7 release files

1.15.0

7 release files

1.14.6

7 release files

1.14.5

7 release files

This release

1.14.4 This release

7 release files

1.14.3

7 release files

1.14.2

7 release files

1.14.0

7 release files

1.13.6

7 release files

1.13.5

7 release files

1.13.3

7 release files

Yanked

1.13.2

7 release files

Yanked

1.13.1

7 release files

Yanked

1.13.0

7 release files

1.12.5

9 release files

Yanked

1.12.4

8 release files

Yanked

1.12.3

8 release files

Yanked

1.12.2

9 release files

Yanked

1.12.1

9 release files

Yanked

1.12.0

9 release files

1.11.4

7 release files

Yanked

1.11.3

7 release files

Yanked

1.11.2

7 release files

Yanked

1.11.1

7 release files

Yanked

1.11.0

7 release files

1.10.3

1 release file

Yanked

1.10.2

1 release file

Yanked

1.10.1

1 release file

Yanked

1.10.0

1 release file

1.9.3

1 release file

Yanked

1.9.2

1 release file

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page