ordeq_spark 2.2.0

Apache Spark integration for Ordeq

Ordeq

Ordeq (pronounced /ɒɹdɛk/) is a framework for developing data pipelines. It simplifies IO and modularizes pipeline logic. Ordeq elevates your proof-of-concept to a production-grade pipelines. See the introduction for an easy-to-follow example of how Ordeq helps.

Installation

Ordeq is lightweight with 0 dependencies.

To install Ordeq, run:

uv pip install ordeq

Integrations

Ordeq integrates seamlessly with existing tooling. It provides out-of-the-box integrations with 25+ popular libraries. In total, Ordeq offers over 100 IOs via these integrations.

You can install them as needed. For example, for reading and writing data with Pandas, install the ordeq-pandas package:

uv pip install ordeq-pandas

Some of the available integrations:

Data processing

Pandas	Spark	Numpy	Polars	Ibis	Joblib	HuggingFace	Pillow
st	Requests	Pydantic	DuckDB	NetworkX	TOML	PyMuPDF	ChromaDB

Plotting

Matplotlib

Altair

Plotly Express

Cloud storage

Google Cloud Storage

Azure Storage Blob

AWS S3

Boto3

Bigquery

Have a look at the package overview and API reference for a list of available packages.

Documentation

Documentation is available at https://ing-bank.github.io/ordeq/.

Why consider Ordeq?

• Ordeq is the GenAI companion: it gives your project structure and consistency, such that GenAI can thrive
• It offers seamless integrations with existing data & ML tooling, such as Spark, Pandas, Pydantic and PyMuPDF, and adding new integrations is trivial
• It's actively developed and trusted by data scientists, engineers, analysts and machine learning engineers at ING

Learning Ordeq

To learn more about Ordeq, check out the following resources:

• See how Ordeq can help your project in the introduction
• Check out the core concepts to learn how to use Ordeq
• Explore the example projects to see how Ordeq is used

Visualizing pipelines

Ordeq makes it easy to visualize your pipelines like this with a single line of code. Read more in the documentation.

The following figure shows an example Ordeq pipeline of a Retrieval-Augmented Generation (RAG) pipeline visualized with Mermaid:

Acknowledgements

Ordeq builds upon design choices and ideas from Kedro and other frameworks. It has been developed at ING, with contributions from various individuals. Please refer to the acknowledgements section in the documentation for more details.

License

Ordeq is available under the MIT license. Please refer to the license and notice for more details.