This release has been yanked by its maintainers, and will be ignored by installers, except when explicitly specified.
Accelerated Discovery Reusable Components
The central implementation of Accelerated Discover Reusable Components. It serves as a wrapper around client libraries we use locally like Dapr and MLflow.
1.Installation
All components will be availble using
pip install ad-components
CLI
Here's an example usage of the CLI
usage: adc [-h] [--verbose] [--version] {<component>} ...
Accelerated Discovery reusable components.
positional arguments:
<component> the component that you want to trigger.
optional arguments:
-h, --help show this help message and exit.
--version show program's version number and exit.
2. Usage
2.0. In your pipeline
To use a component in your pipeline, you need to run it in a Step context
from ad.step import DaprStep
from ad.storage import download, upload
with DaprStep():
resp = download(download_src, download_dest, binding_name=binding)
print(f"download resp: {resp}")
resp = upload(upload_src, upload_dest, binding_name=binding)
print(f"upload resp: {resp}")
Running the components inside a step will make sure the client dependencies are handled correctly.
2.1. Storage
2.1.2. Python module
You can invoke the manager using native python. Please note that the package must be present in you python environment.
from ad.storage import download, upload
download_resp = download(
src, dest,
# binding_name="s3-state", # Or any other binding
)
upload_resp = upload(
src, dest,
# binding_name="s3-state", # Or any other binding
)
2.1.3. CLI
usage: adc storage [-h] --src PATH --dest PATH [--binding NAME] [--timeout SEC] {download,upload}
positional arguments:
{download,upload} action to be performed on data.
optional arguments:
-h, --help show this help message and exit
action arguments:
--src PATH, -r PATH path of file to perform action on.
--dest PATH, -d PATH object's desired full path in the destination.
--binding NAME, -b NAME
the name of the binding as defined in the components.
dapr arguments:
--timeout SEC, -t SEC
value in seconds we should wait for sidecar to come up.
Note: You can replace
adcwithpython ad/main.py ...if you don't have the package installed in your python environment.
Examples
- To download an object from S3 run
adc storage download \
--src test.txt \
--dest tmp/downloaded.txt
- To upload an object to S3 run
adc storage upload \
--src tmp/downloaded.txt \
--dest local/uploaded.txt
3. Supported components
3.1. Storage
3.1.1. Supported operations
Below is a list of the operations you might intend to perform in your component.
Upload
Uploads data from a file to an object in a bucket.
Arguments
src: Name of file to download.dest: Object name in the bucket.binding: The name of the binding to perform the operation.
Download
Downloads data of an object to file.
Arguments
src: Object name in the bucket.dest: Name of file to download.binding: The name of the binding to perform the operation.
Dapr configurations
address: Dapr Runtime gRPC endpoint address.timeout: Value in seconds we should wait for sidecar to come up
4. Publishing
Every change to the python script requires a new version to be pushed PyPi registry.
If you have the right (write) permissions, and a correctly-configured $HOME/.pypirc file, run the following command to publish the package
make
4.1. Increment the version
To increment the version, go to adstorage/version.py and increment the version there. Both the setup.py and the CLI will read the new version correctly.
4.2 Configure PyPi registry
To be able to push to the package to our private registry, you need to tell PyPi about it. This one-liner command will take care of it for you
cat << EOF > $HOME/.pypirc
[distutils]
index-servers =
pypi
[pypi]
repository: https://upload.pypi.org/legacy/
username: __token__
password: $PYPI_TOKEN
EOF
Note: The pip package will fetch the version from
ad/version.pyfile, so make sure to increment before pushing.
1 maintainer
Metadata
1 maintainer
Release files for ad-components 0.1.48
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | |
|---|---|---|---|---|
| ad-components-0.1.48.tar.gz | 7.8 kB | Without provenance | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| ad_components-0.1.48-py3.7.egg | Legacy Egg format | - | - | Details |
Total release size: 25.1 kB
Release files / ad-components-0.1.48.tar.gz
| Download URL | ad-components-0.1.48.tar.gz |
|---|---|
| Size | 7.8 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
b3b81ce779d39d906c40a6701a41e374260e2cf7c439cc24226f8b808a5631e4
|
|
BLAKE2b-256 checksum How to use checksums |
6ed8d731d91e1d30c447d830a430a871ed7a13f125eb3afaf05f7848b34e1241
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/1.15.0 pkginfo/1.8.3 requests/2.25.1 setuptools/44.1.1 requests-toolbelt/0.9.1 tqdm/4.64.0 CPython/2.7.17
|
Release files / ad_components-0.1.48-py3.7.egg
| Download URL | ad_components-0.1.48-py3.7.egg |
|---|---|
| Size | 17.4 kB |
| Tags | Egg |
|
SHA-256 checksum How to use checksums |
3da313dcfbd876f23439c10afaa5ffcef7e34f80d045c4fd6e0b5f87e2143a4a
|
|
BLAKE2b-256 checksum How to use checksums |
9db92cfee44a52568ecad2624e611ae18d485717aecb875543bde03d605ec6ad
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/1.15.0 pkginfo/1.8.3 requests/2.25.1 setuptools/44.1.1 requests-toolbelt/0.9.1 tqdm/4.64.0 CPython/2.7.17
|
Security for ad-components 0.1.48
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.
