Skip to main content

Klio pipeline executor within a job

Project description

The klio-exec Package

Latest version of klio-exec on PyPI Status of klio-exec unit tests

The executor – not meant to be used directly by the user – is a CLI that launches a pipeline from within a job’s Docker container (a.k.a. an Apache Beam’s “driver”). Many commands from the klio-cli directly wrap to commands in the executor: a klio-cli command will set up the Docker context needed to correctly run the pipeline via the associated command with klio-exec. The Docker context includes mounting the job directory, sets up environment variables, mounting credentials, etc.

As the klio-exec package is not meant to be installed directly, check out the installation guide for how to setup installation. There is also the user guide and the API documentation for more information.

What is Klio?

Klio is an ecosystem that allows you to process audio files – or any binary files – easily and at scale.

Klio jobs are opinionated data pipelines in Python (streaming or batch) built upon Apache Beam and tuned for audio and binary file processing.

Klio was built by Spotify to run our large-scale audio intelligence systems and is used by teams of engineers and audio researchers to help develop and deploy next generation audio algorithms.

Read more about Klio here.

Get Started

Check out our Quick Start guide on how to start using Klio.

Resources

Release Information

22.3.0 (2022-03-04)

Fixed

  • Fix bug when pass thru PCollection is None. For example when a user would like to remove their data input configuration entirely from the job’s configuration.

Full Changelog.

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

klio-exec-22.3.0.tar.gz (52.9 kB view hashes)

Uploaded source

Built Distribution

klio_exec-22.3.0-py2.py3-none-any.whl (38.9 kB view hashes)

Uploaded py2 py3

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Huawei Huawei PSF Sponsor Microsoft Microsoft PSF Sponsor NVIDIA NVIDIA PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page