Aye Aye
An ETL (Extract, Transform, Load) framework.
Quick install
In the virtual environment for the project you’d like to use Aye Aye in, run:-
pip install ayeaye
Quick start
Use Pipenv to manage a python virtual environment and package management0
pipenv shell
pipenv install ayeaye
Within the environment created by pipenv above, run one of the examples:-
curl "https://raw.githubusercontent.com/Aye-Aye-Dev/AyeAye/master/examples/poisonous_animals.py" \
--output poisonous_animals.py
mkdir data
curl https://raw.githubusercontent.com/Aye-Aye-Dev/AyeAye/master/examples/data/poisonous_animals.json \
--output data/poisonous_animals.json
python poisonous_animals.py
This model takes a small input dataset of animals and collates them by the country they are found. It doesn't write to a dataset, it just outputs a log. The log for this example contains the name of the country and the animals found there.
There are more examples in the Aye-Aye-Recipes git repo.
Overview
An Aye Aye ETL model inherits from ayeaye.Model and uses class level variables to declare connectors to the data it acts on.
Example:-
import ayeaye
class PoisonousAnimals(ayeaye.Model):
poisonous_animals = ayeaye.Connect(engine_url='json://data/poisonous_animals.json')
When instantiated, self.poisonous_animals will be a dataset that ETL operations can be done with.
The engine_url parameter passed to ayeaye.Connect is specifying the dataset type JSON in this case) and exact location for the data (data/poisonous_animals.json is a relative file path).
Instead of engine_url you could also specify a ref and this uses the data catalogue to look up the engine_url. (TODO this feature is coming soon!). When used this way, ayeaye.Connect is responsible for resolving the ref to an engine_url and passing this to a subclass of ayeaye.connectors.base.DataConnector which can read and maybe write this data type.
Unit tests
Ensure the working directory is the base Aye Aye directory (i.e. the same directory as the Pipfile):
pipenv install --dev
export PYTHONPATH=`pwd`/lib
pipenv run python -m unittest discover
Development version
To use the latest code in editable mode-
pipenv install -e git+https://github.com/Aye-Aye-Dev/AyeAye#egg=ayeaye
When venv is being used, add this line to requirements.txt-
git+https://github.com/Aye-Aye-Dev/AyeAye#egg=ayeaye
Optional extras
Extra dependencies for API usage within Aye-aye models can be installed like this:
pipenv install "ayeaye[api]"
or multiple
pipenv install "ayeaye[api,aws,compression]"
| Label | Functionality |
|---|---|
| api | Restful JSON via http(s) |
| aws | File based connectors can use Amazon Web Service S3 file storage |
| compression | On the fly compression for file based connectors |
License
Aye Aye is distributed under the terms of the Apache License 2.0 and Copyright Progressive Logic Limit 2021 and onwards.
1 maintainer
Metadata
1 maintainer
Release files for ayeaye 0.1.7
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | |
|---|---|---|---|---|
| ayeaye-0.1.7.tar.gz | 96.3 kB | Without provenance | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| ayeaye-0.1.7-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 192.2 kB
Release files / ayeaye-0.1.7.tar.gz
| Download URL | ayeaye-0.1.7.tar.gz |
|---|---|
| Size | 96.3 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
5781dce93032aad52bd3522ded7c625ab720e8eb4dcd772767fbcd71f72df355
|
|
BLAKE2b-256 checksum How to use checksums |
049b7088e3774b89d8da567aae962c8353114e7868171f7ca44e086b4cc4bcd3
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.10.20
|
Release files / ayeaye-0.1.7-py3-none-any.whl
| Download URL | ayeaye-0.1.7-py3-none-any.whl |
|---|---|
| Size | 96.0 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
9eef0679629c056ef53665d76ed4f4916061f61b63eb7b362054df90262bdae2
|
|
BLAKE2b-256 checksum How to use checksums |
9718a8d54b63aa333d8a59c23ceb5302c0ce38882362a8aecb42cfc0419f05bc
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.10.20
|
Security for ayeaye 0.1.7
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
