Skip to main content

CLS-Luigi is an innovative pipeline tool designed to streamline the creation and execution of algorithmic pipelines by harnessing the power of combinatory logic.

Project description

CLS-Luigi

https://img.shields.io/pypi/v/cls-luigi https://img.shields.io/pypi/pyversions/cls-luigi https://img.shields.io/pypi/l/cls-luigi?color=blue https://img.shields.io/github/issues/cls-python/cls-luigi https://github.com/cls-python/cls-luigi/actions/workflows/test-build-release.yaml/badge.svg https://img.shields.io/endpoint?url=https://gist.githubusercontent.com/DanielScholtyssek/d2c0e9041ce417fd42c710572e02c2aa/raw/covbadge.json Documentation Status

TL:DR

A framework for automated synthesis and execution of decision pipelines.

What is it?

In short: our goal is to automatically create decision pipelines based on domain specific algorithmic repositories and depending on the available data!

In order to streamline decision-making processes, it is common practice to construct decision pipelines comprising various algorithms. These pipelines encompass tasks such as basic data pre-processing, statistical analysis, machine learning, and optimization. Consequently, building an effective pipeline necessitates expertise in the domains of machine learning, optimization, and the specific field at hand.

Moreover, it is important to note that there is no universally optimal pipeline for every problem in any domain. There is no single pipeline that represents the best choice, nor is there an ultimate model or configuration for machine learning approches. Likewise, the ideal model and solution method for optimization problems may vary. One of the main reasons for this can be the available data, which can change over time.

CLS-Luigi is an innovative pipeline tool designed to streamline the creation and execution of algorithmic pipelines by harnessing the power of combinatory logic. It combines CLS-Python, a type-theoretic framework for software component synthesis, with Luigi, a tool created by Spotify to build and execute pipelines for batch jobs. At its core, CLS-Luigi aims to integrate elements from AutoML, Algorithm selection and configuration, and DevOps to provide a comprehensive solution. However, it is important to note that CLS-Luigi is currently in its early stages of development, undergoing ongoing refinement and enhancement.

Main Features

Here are just a few of the things that CLS-Luigi does well:

  • Allows natural modeling by specifying the necessary input to a component based on the types of the components it depends on.

  • It’s easy to define templates for algorithmic components and templates for pipeline structures.

  • Good for batch-type pipelines where previous pipeline executions have no/little influence on the current run.

  • Consistent creation of (all) pipeline variants based on a user defined repository of components, guaranteeing soundness and completeness by leveraging the CLS-Python framework.

  • Efficient execution and resource optimization using features of Luigi. Luigi optimizes resource usage through caching mechanisms, avoiding redundant computations by rerunning identical sub-pipelines only when necessary.

  • The framework places significant importance on componentization and promotes a structured approach that considers the data flow, actions on the data, and data propagation to subsequent computational steps when implementing components. This enables the reuse of components in different domain-specific repositories and their smooth integration into diverse pipelines, as long as they align with the pipeline structure. Moreover, the framework offers flexibility to Python programmers, allowing them to expand existing domain-specific repositories by inheriting from a already existing component and implementing the necessary methods to create a new problem-specific or problem-agnostic component, thereby fostering customization and adaptability.

  • Integrated visualizer that can display the repository and the scheduled pipelines.

  • Python based framework: offers a extensive collection of specialized libraries and tools that offer pre-built algorithms, statistical functions, and visualization capabilities tailored for optimization, machine learning, and data analytics tasks. Moreover, Python’s seamless integration with other technologies enhances its appeal and makes it a valuable tool for our projects.

Authors

Changelog

0.1.1 (2024-02-29)

  • Commented TaskProcessingTime mixin to supress error messages.

  • Fixing wrapped_task arg in the decoder.

0.1.0 (2023-06-07)

  • First release on PyPI.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

cls-luigi-0.1.1.tar.gz (354.7 kB view details)

Uploaded Source

Built Distribution

cls_luigi-0.1.1-py3-none-any.whl (359.5 kB view details)

Uploaded Python 3

File details

Details for the file cls-luigi-0.1.1.tar.gz.

File metadata

  • Download URL: cls-luigi-0.1.1.tar.gz
  • Upload date:
  • Size: 354.7 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.11.8

File hashes

Hashes for cls-luigi-0.1.1.tar.gz
Algorithm Hash digest
SHA256 b215360455bfe35dd232b841ca828e44131af3f7040227e7d08cd127735add14
MD5 369b845b05b35f096580bc5f531f4a0e
BLAKE2b-256 21f83609620254f1fab0a1fc1e9ad9a246d6543ef259e67690a41e5e6fc66084

See more details on using hashes here.

File details

Details for the file cls_luigi-0.1.1-py3-none-any.whl.

File metadata

  • Download URL: cls_luigi-0.1.1-py3-none-any.whl
  • Upload date:
  • Size: 359.5 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.11.8

File hashes

Hashes for cls_luigi-0.1.1-py3-none-any.whl
Algorithm Hash digest
SHA256 910e3df541f4d2f7e4990c3c9561d8e4cd4ffee3ff345829d43e98f31883903a
MD5 417b30d9bcdc5e56417573608d5bda78
BLAKE2b-256 00f75bebb72740107298e248b605b9923ad892de6687324f5f1b4ccf981e813e

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page