ML & data helper code!
Jakub Langr (c) 2021
This is a CLI utility for speeding up basic Data Engineering/Data Science tasks
usage: dt [-h] [-d] [-q] [-v]
{cai,config,ec2,gcp,gh,hist,load,monitor,nf,py,run,s3,sftp,sg,ssh,td,viz}
...
ML & data helper code!
optional arguments:
-h, --help show this help message and exit
-d, --debug full application debug mode
-q, --quiet suppress all console output
-v, --version show program's version number and exit
sub-commands:
{cai,config,ec2,gcp,gh,hist,load,monitor,nf,py,run,s3,sftp,sg,ssh,td,viz}
cai Ops with Codex AI from OpenAI.
config Operations with config.
ec2 AWS EC2 helper commands.
gcp GCP VM Instances helper commands.These are currently
in alpha and some may not work
gh All Github related operations
hist Operations on CLI hist
load Appends defaults: from zshrc.txt to ~/.bashrc or
~/.zshrc or tmux conf
monitor Monitor lack of GPU activity. When there is none, runs
-job
nf Displays the number of files
py (WIP/Pre-Alpha) Execute a Python command across all
files in current dir.
run Operations with snippets
s3 Operations with s3 buckets
sftp Operations to easily work with remote servers /
devices
sg AWS Security Groups helper functions
ssh Operations on the SSH config
td Manage TODOs using Google Keep
viz Basic viz using streamlit for image comparisons
Usage: dt command [args] [kwargs]
The screen above can be called using:
dt -h
For example, to run to list the running ec2 isntances run
dt ec2 ls --profile <profile>
Installation
$ pip install data-toolkit
Tips and examples
Personally I find most helpful the following commands:
dt ec2 ls
Shows running EC2 instances with a range of parameters depending on the width of your terminal.
dt sg ls
Shows an overview of all security groups in the current region.
dt sg show
Shows detail of either all or one security group in the current region.
dt ssh update
Updates the SSH config with with the IP addresses of the running EC2 instances (matches them by name/tag).
When I'm travelling I like to use the following commands:
dt sg update -n <sg_name> -m "22:jakub 4g"
Which updates the <sg_name> security group to allow SSH from the current IP address and port 22 and sets the description to "jakub 4g".
Hope this helps!
Development
This project includes a number of helpers in the Makefile to streamline common development tasks.
Environment Setup
The following demonstrates setting up and working with a development environment:
### create a virtualenv for development
$ make virtualenv
$ source env/bin/activate
### run dt cli application
$ dt --help
### run pytest / coverage
$ make test
1 maintainer
Release files for data-toolkit 2.0.4
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| data_toolkit-2.0.4.tar.gz | 68.1 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| data_toolkit-2.0.4-py3-none-any.whl | Python 3 | none | any | Details |
Total release size:148.9 kB
Release files / data_toolkit-2.0.4.tar.gz
| Download URL | data_toolkit-2.0.4.tar.gz |
|---|---|
| Size | 68.1 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
7ba50eab9712aeed52d25ce91dccc963c94cf6283b75a98a6f81c28ec56effb7
|
|
BLAKE2b-256 checksum How to use checksums |
1df62116b381f61948cac779f8ee4ca35ad136f7799e9fff1c142befe33d3c9d
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.11.10
|
Release files / data_toolkit-2.0.4-py3-none-any.whl
| Download URL | data_toolkit-2.0.4-py3-none-any.whl |
|---|---|
| Size | 80.9 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
3a92446565748741b4bf0f8bafd239ae9449d8a096081f14134ef85b18f0df9f
|
|
BLAKE2b-256 checksum How to use checksums |
ed51086c2fd68fe83571bd81377755577f1c8120581d1742d391b8e2eb39bd05
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.11.10
|
