Extensions to Magnus core
Project description
Welcome to Magnus Extensions
Documentation of the extensions are available at: https://astrazeneca.github.io/magnus-extensions/
This repository provides all the extensions to magnus core package.
Magnus provides 5 essential services:
- Executor: A way to define and execute/transpile dag definition.
- Catalog: An artifact store used to log and store data files generated during a pipeline execution.
- Secrets Handler: A framework to handle secrets from different providers.
- Logging: A comprehensive and automatic logging framework to capture essential information of a pipeline execution.
- Experiment Tracking: A framework to interact with different experiment tracking tools.
Below is the table of all the available extensions to the above services:
Service | Description | Availability |
---|---|---|
Executors | ||
Local | Run the pipeline on local machine (default) | Part of Magnus core |
Local Containers | Run the pipeline on local containers | Part of Magnus core |
Catalog | ||
Do Nothing | Provides no cataloging functionality | Part of Magnus core |
File System | Uses local file system (default) | Part of Magnus core |
S3 | Uses S3 as a catalog | magnus_extension_catalog_s3 |
Secrets | ||
Do Nothing | Provides no secrets handler (default) | Part of Magnus core |
Dot Env | Uses a file as secrets | Part of Magnus core |
Environment Variables | Gets secrets from Environmental variables | Part of Magnus core |
Logging | ||
Buffered | Uses the run time buffer as logger (default) | Part of Magnus core |
File System | Uses a file system as run log store | Part of Magnus core |
S3 | Uses S3 to store logs | magnus_extension_datastore_s3 |
Experiment Tracking | ||
Do Nothing | Provides no experiment tracking (default) | Part of Magnus core |
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
magnus_extensions-0.1.0.tar.gz
(22.0 kB
view hashes)
Built Distribution
Close
Hashes for magnus_extensions-0.1.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3c6f647e0349780444541fa959b26b382ea72ebe0d66d4ee641d83b5aea07c11 |
|
MD5 | 112a0aa15cd3efb1d6c9b9905acaf566 |
|
BLAKE2b-256 | 8e09146dc9cb18f8273112c16e581e307cf91dd5b2b964c7da6a0e45f7fa7cee |