Snark Hub
Project description
DataFlow ·
Prototyping for Hub Dataflow
Setup
- Install the package
> pip3 install git+https://github.com/snarkai/dataflow@v0.2.2
Make sure gdal and opencv is installed as required by image_io.py
- Setup credentials in the local folder and don't forget to put those in .gitignore
.secrets/agmri.cfg
.secrets/intelinair #AWS credentials
- Example to get started
from dataflow import hub
import torch
hub.init()
df = hub.agmri().get_polygons_by_type(hub.agmri.UIUC_ENDROW, max=1)
ds = hub.polygon_sampler(df, sample_count=32, shape=(1024,1024,4))
ds[0]
Troubleshooting
Install GDAL and OpenCV
If it throws error on gdal or opencv install those manually
./bin/prepare.sh
./bin/gdal.sh
./bin/opencv.sh
Credentials examples
intelinair
file should look like this
[default]
aws_access_key_id = ...
aws_secret_access_key = ...
region = us-east-1
agmri.cfg
[production]
admin_username = ...
admin_password = ...
Known Issues
[ ] Credentials are not flexible where to be stored
Push a new version
git commit -a -m 'current changes'
git tag -a 'v0.2.2' -m 'some message here'
git push origin 'v0.2.2'
Automated integration testing
To run pytest before git push we need to do create .git/hooks/pre-push file with the following content:
#!/bin/bash
docker-compose run test
And make it executable
chmod +x .git/hooks/pre-push
Also whenever updating requirements.txt and options.txt please rebuild the test
docker-compose build test
Formatting and Linting
Hub uses Black and Flake8 to ensure a consistent code format throughout the project. Replace .vscode/settings.json content withthe following:
{
"[py]": {
"editor.formatOnSave": true
},
"python.formatting.provider": "black",
"python.linting.flake8Enabled": true,
"python.linting.flake8Path": "flake8",
"python.linting.flake8Args": [
"--max-line-length=80",
"--select=B,C,E,F,W,B950",
"--ignore=E203,E501,W503"
],
"python.linting.pylintEnabled": false,
"python.linting.enabled": true,
}
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
dataflow_gegham_test-0.6.0.tar.gz
(16.9 kB
view hashes)
Built Distribution
Close
Hashes for dataflow_gegham_test-0.6.0.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | f8cae358ba44bc08bfcc752171d19fad9cbb4874f49c39089ec65c3cbb6f44a5 |
|
MD5 | 808bbfb45800cd7ff39ad427053191a1 |
|
BLAKE2b-256 | 85e785264863d5c1e31e34922385727a1c4f44840c65406bce85b6a45375c4bf |
Close
Hashes for dataflow_gegham_test-0.6.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6a74ad27c1f4469294df50c086635fc48be06fd3c9a90d979404d39c9c8e9e15 |
|
MD5 | be7f44129a4e22309687ccf74d1bc056 |
|
BLAKE2b-256 | 68aca88ae48d595f0efd740e37a5b1052919f5e41071a624b4b50d30094bb8f4 |