s3-pull-processor
This project is a POC, proof of concept for client to upload pipeline artifacts to an object storage as S3 and able to consume the artifacts efficiently. The consumer is scalable, containerized and ready to run in k8s/Openshift
The diagram below represents a high-level workflow, the boxes uploader CLI and processor are the components here
being implemented.
The infra needed to run this POC is basically AWS S3 and AWS SQS.
See developer guide for instructions to run or to contribute
demo
The items 1 to 5 below, show an e2e execution from uploader to processor or consumer.
-
SQS and S3 are empty, no messages and no files
-
simulating a
HOST Awhere the artifact exist and need to be uploaded to S3, also a message is sent to SQS. For this step the code test_host_producer was executed: -
check there are messages in SQS and files in S3
-
simulating a
HOST Bwhere it consumes messages from SQS and for each message download the artifact from S3, run an action on this case it run import_to_ibutsu action then delete the file from S3 and finally delete the message from SQS. For this step the code test_host_consumer was executed
Also see scenarios in developer guide.
links
1 maintainer
Metadata
1 maintainer
Release files for s3-pull-processor 0.3
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| s3-pull-processor-0.3.tar.gz | 1.1 MB | Without provenance | Details for s3-pull-processor-0.3.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| s3_pull_processor-0.3-py3-none-any.whl | Python 3 | none | any | Details for s3_pull_processor-0.3-py3-none-any.whl |
Total release size: 1.1 MB
Release files / s3-pull-processor-0.3.tar.gz
| Download URL | s3-pull-processor-0.3.tar.gz |
|---|---|
| Size | 1.1 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
fc5d06cefb038c11b420d6edfdea3d712fc3ad53dd2720caa8cc44cb2fe1e628
|
|
BLAKE2b-256 checksum How to use checksums |
ea5e3896f9f9e6526a09bb295c6634a8d9f93a52c7d9609dd17f42e2c3ab29d0
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.8.0 pkginfo/1.8.2 readme-renderer/33.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.63.0 importlib-metadata/4.11.2 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.4 CPython/3.9.10
|
Release files / s3_pull_processor-0.3-py3-none-any.whl
| Download URL | s3_pull_processor-0.3-py3-none-any.whl |
|---|---|
| Size | 23.8 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
6d54f29a814ed952ac529f662c8b841069c43249ad07982a290e0b43e3dc7665
|
|
BLAKE2b-256 checksum How to use checksums |
0f714c5484ea04f2d143b2ea48cc3c18a16b91df7779b15e158bdac1baf32028
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.8.0 pkginfo/1.8.2 readme-renderer/33.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.63.0 importlib-metadata/4.11.2 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.4 CPython/3.9.10
|
Security for s3-pull-processor 0.3
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.
