Snark Hub
Project description
DataFlow ·
Prototyping for Hub Dataflow
Setup
- Install the package
> pip3 install git+https://github.com/snarkai/dataflow@v0.2.2
Make sure gdal and opencv is installed as required by image_io.py
- Setup credentials in the local folder and don't forget to put those in .gitignore
.secrets/agmri.cfg
.secrets/intelinair #AWS credentials
- Example to get started
from dataflow import hub
import torch
hub.init()
df = hub.agmri().get_polygons_by_type(hub.agmri.UIUC_ENDROW, max=1)
ds = hub.polygon_sampler(df, sample_count=32, shape=(1024,1024,4))
ds[0]
Troubleshooting
Install GDAL and OpenCV
If it throws error on gdal or opencv install those manually
./bin/prepare.sh
./bin/gdal.sh
./bin/opencv.sh
Credentials examples
intelinair
file should look like this
[default]
aws_access_key_id = ...
aws_secret_access_key = ...
region = us-east-1
agmri.cfg
[production]
admin_username = ...
admin_password = ...
Known Issues
[ ] Credentials are not flexible where to be stored
Push a new version
git commit -a -m 'current changes'
git tag -a 'v0.2.2' -m 'some message here'
git push origin 'v0.2.2'
Automated integration testing
To run pytest before git push we need to do create .git/hooks/pre-push file with the following content:
#!/bin/bash
docker-compose run test
And make it executable
chmod +x .git/hooks/pre-push
Also whenever updating requirements.txt and options.txt please rebuild the test
docker-compose build test
Formatting and Linting
Hub uses Black and Flake8 to ensure a consistent code format throughout the project. Replace .vscode/settings.json content withthe following:
{
"[py]": {
"editor.formatOnSave": true
},
"python.formatting.provider": "black",
"python.linting.flake8Enabled": true,
"python.linting.flake8Path": "flake8",
"python.linting.flake8Args": [
"--max-line-length=80",
"--select=B,C,E,F,W,B950",
"--ignore=E203,E501,W503"
],
"python.linting.pylintEnabled": false,
"python.linting.enabled": true,
}
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
hub_dataflow-0.9.4.tar.gz
(24.0 kB
view hashes)
Built Distribution
Close
Hashes for hub_dataflow-0.9.4-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4fbd1c4dc0eb01544778788f1ea95c6c321fa2c2dab81e3351a8c6c9fc632231 |
|
MD5 | 024c986a35594a23b7c6b1a61df0cdfb |
|
BLAKE2b-256 | ce018115bece8926de343b38743915d4cb0da64d3893c78253d47aff74c3e0ea |