A cli to upload and retrieve data to/from S3 object storage and processing data running actions on it.
Project description
s3-pull-processor
This project is a POC, proof of concept for client to upload pipeline artifacts to an object storage as S3 and able to consume the artifacts efficiently. The consumer is scalable, containerized and ready to run in k8s/Openshift
The diagram below represents a high-level workflow, the boxes uploader CLI
and processor
are the components here
being implemented.
The infra needed to run this POC is basically AWS S3 and AWS SQS.
See developer guide for instructions to run or to contribute
demo
The items 1 to 5 below, show an e2e execution from uploader to processor or consumer.
-
SQS and S3 are empty, no messages and no files
-
simulating a
HOST A
where the artifact exist and need to be uploaded to S3, also a message is sent to SQS. For this step the code test_host_producer was executed: -
check there are messages in SQS and files in S3
-
simulating a
HOST B
where it consumes messages from SQS and for each message download the artifact from S3, run an action on this case it run import_to_ibutsu action then delete the file from S3 and finally delete the message from SQS. For this step the code test_host_consumer was executed
Also see scenarios in developer guide.
links
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for s3-pull-processor-0.2.dev1.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 89475c96a74c2dcf2b61e5cb3bfac12eb5a9588b4baf2b447f033202a96aa68c |
|
MD5 | 8a068c915228386de7477de275fe1bb7 |
|
BLAKE2b-256 | e983533f1fa2b903136510f58f4cd1ac7c9eac1a52da3d66d03e6ec3149e2038 |
Hashes for s3_pull_processor-0.2.dev1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 26dd3b77013cb159a05d69a34ffc1047de51c8100ee16f33c6e4770b704451de |
|
MD5 | f127195481d9046dc3d18ceea34e7d27 |
|
BLAKE2b-256 | 871261a33482603dc8815e1c7a254d09af613454346cb19f88f8069bc3a22ec4 |