Skip to main content

ML & data helper code!

Project description

ML & data helper code!

Jakub Langr (c) 2021

This is a CLI utility for speeding up basic Data Engineering/Data Science tasks

usage: dt [-h] [-d] [-q] [-v]

ML & data helper code!

optional arguments:
  -h, --help            show this help message and exit
  -d, --debug           full application debug mode
  -q, --quiet           suppress all console output
  -v, --version         show program's version number and exit

    cai                 Ops with Codex AI from OpenAI.
    config              Operations with config.
    ec2                 AWS EC2 helper commands.
    gcp                 GCP VM Instances helper commands.These are currently
                        in alpha and some may not work
    gh                  All Github related operations
    hist                Operations on CLI hist
    load                Appends defaults: from zshrc.txt to ~/.bashrc or
                        ~/.zshrc or tmux conf
    monitor             Monitor lack of GPU activity. When there is none, runs
    nf                  Displays the number of files
    py                  (WIP/Pre-Alpha) Execute a Python command across all
                        files in current dir.
    run                 Operations with snippets
    s3                  Operations with s3 buckets
    sftp                Operations to easily work with remote servers /
    sg                  AWS Security Groups helper functions
    ssh                 Operations on the SSH config
    td                  Manage TODOs using Google Keep
    viz                 Basic viz using streamlit for image comparisons

Usage: dt command [args] [kwargs]

The screen above can be called using:

dt -h

For example, to run to list the running ec2 isntances run

dt ec2 ls --profile <profile>


$ pip install data-toolkit

Tips and examples

Personally I find most helpful the following commands:

dt ec2 ls

Shows running EC2 instances with a range of parameters depending on the width of your terminal.

dt sg ls

Shows an overview of all security groups in the current region.

dt sg show

Shows detail of either all or one security group in the current region.

dt ssh update

Updates the SSH config with with the IP addresses of the running EC2 instances (matches them by name/tag).

When I'm travelling I like to use the following commands:

dt sg update -n <sg_name>  -m "22:jakub 4g"

Which updates the <sg_name> security group to allow SSH from the current IP address and port 22 and sets the description to "jakub 4g".

Hope this helps!


This project includes a number of helpers in the Makefile to streamline common development tasks.

Environment Setup

The following demonstrates setting up and working with a development environment:

### create a virtualenv for development

$ make virtualenv

$ source env/bin/activate

### run dt cli application

$ dt --help

### run pytest / coverage

$ make test

Project details

Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

data-toolkit-1.4.10.tar.gz (50.9 kB view hashes)

Uploaded source

Built Distribution

data_toolkit-1.4.10-py3-none-any.whl (63.0 kB view hashes)

Uploaded py3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page