fossa
Execution engine for Aye-Aye ETL models
Overview
Fossa runs Aye-aye models and their subtasks across a distributed environment.
Aye-aye models can be run without Fossa (they're just Python code) but when a model grows too big to execute on a single computer the task needs to be spread across multiple computers. A 'distributed environment' is one with multiple compute node that are networked so messages can be passed between nodes.
An instance of Fossa runs on each compute node where it facilitates the communication of messages between nodes.
A node could be a docker or full computer instance.
Getting Started
Setup your virtual environment (venv, pipenv, poetry etc.)
pip install ayeaye-fossa
Make a python module to use this-
import ayeaye
from fossa import run_fossa, BaseConfig
class NothingEtl(ayeaye.Model):
def build(self):
pass
class FossaConfig(BaseConfig):
ACCEPTED_MODEL_CLASSES = [NothingEtl]
run_fossa(FossaConfig)
Run it and point your browser to http://0.0.0.0:2345/. You have a fossa node that will only run the NothingEtl model locally. This is useless as that model doesn't do anything but it's a start.
The examples directory has that example plus a few more.
Running Fossa locally
For local development (so an IDE and debugging tools etc. can be used) the Flask built in server can be used. run_fossa (demonstrated above) uses gunicorn.
Ensure your working directory is the same directory as this README file.
Then install dependencies and run the tests-
cp local_env_example .env
pipenv shell
pipenv install --dev
python -m unittest discover tests
The .env file is used by pipenv.
For all python commands below you will need to be in this pipenv shell.
In a distributed environment one instance of Fossa would run on each compute node. To experiment with Fossa just run one or more instances on a local computer.
Fossa runs a small web-server app which can be used to submit tasks and query the progress and status of tasks. In production, jobs are more likely to be fetched from a message queue.
Copy the example config file into your own person config; sym-link to local_config.py so the run_local_app() function in fossa.app can find it.
e.g.
cd fossa/settings
# replace xxxxx with your name or a more useful identifier for your environment
cp local_config_example.py local_config_xxxxx.py
# have a look in your config file. Is there anything you'd like to change to fit with your system?
ln -s local_config_xxxxx.py local_config.py
In the virtual env (provided by pipenv) from above and with the current working directory being the project's root directory-
python fossa/app.py
You'll now have a locally running web app. It will output IP addresses it is accepting connections from. Typically just point a browser at `http://0.0.0.0:2345/'
## Running Fossa locally using gunicorn
export PYTHONPATH=`pwd`:`pwd`/lib
DEPLOYMENT_ENVIRONMENT=local python lib/fossa/main.py
Posting a task
In a production environment tasks are more likely to arrive through a message queue. But it's also possible to use an HTTP POST to submit a task.
If you used the local_config_example.py file as a starting point for your local config it will have a single model already in the ACCEPTED_MODEL_CLASSES parameter. This is a tiny example ETL model.
Fossa will only run models that have been pre-defined before start-up. The ACCEPTED_MODEL_CLASSES config variable is the simplest way to set this.
POST to your local instance of Fossa a task specification. This example runs the complete SimpleExampleEtl model-
curl --header "Content-Type: application/json" \
--data '{"model_class":"NothingEtl"}' \
--request POST http://0.0.0.0:2345/api/0.01/task
Distributed (but still local) processing
TODO
curl --header "Content-Type: application/json" \
--data '{"model_class":"PartitionedExampleEtl"}' \
--request POST http://0.0.0.0:2345/api/0.01/task
Tests
The normal unittests will run with the python -m unittest discover tests command detailed above. There are also some integration tests in an early stage of development. They are messy and output a lot of rubbish to the console. Enable them by supplying a RabbitMQ broker through an environmental variable. e.g. export RABBITMQ_URL=amqp://guest:guest@localhost.
1 maintainer
Metadata
1 maintainer
Release files for ayeaye-fossa 0.0.37
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | |
|---|---|---|---|---|
| ayeaye_fossa-0.0.37.tar.gz | 31.1 kB | Without provenance | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| ayeaye_fossa-0.0.37-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 70.0 kB
Release files / ayeaye_fossa-0.0.37.tar.gz
| Download URL | ayeaye_fossa-0.0.37.tar.gz |
|---|---|
| Size | 31.1 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
0d3f7abdf66fb4774353bcde7c89a78a5fd4a088953d2256c7d4a44affcf973d
|
|
BLAKE2b-256 checksum How to use checksums |
0b2b4cf02d3ced9a7a1073c70d9c26ec55508d529b1413e287ba8b1f2015c76a
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/5.0.0 CPython/3.10.15
|
Release files / ayeaye_fossa-0.0.37-py3-none-any.whl
| Download URL | ayeaye_fossa-0.0.37-py3-none-any.whl |
|---|---|
| Size | 38.9 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
d155387210c297945316774e60927707eb4fbdee25e6b5f3a19dcd132c5de15d
|
|
BLAKE2b-256 checksum How to use checksums |
5dbe0adf0450168714093ce85c241379e432fcadf40959bf75ee4c48f527b000
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/5.0.0 CPython/3.10.15
|
Security for ayeaye-fossa 0.0.37
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
