Skip to main content

A library of generic tools for ETL work and visualization of JSON blobs and python repositories

Project description

linkedin github pypi vimeo alexgbraun

Introduction

Rolling-pin is a library of generic tools for ETL work and visualization of JSON blobs and python repositories

See documentation for details.

On the documentation main page, under the Architecture section, is a dynamically generated dependency graph of rolling-pin's current architecture. It is generated using the RepoETL class.

If you look under the Metrics section you will find Radon code metric plots and data of the rolling-pin source code, generated by the RadonETL class.

Also have a look at this Jupyter notebook demo for a taste of what rolling-pin can do.


Installation for Developers

Docker

  1. Install docker-desktop
  2. Ensure docker-desktop has at least 4 GB of memory allocated to it.
  3. git clone git@github.com:theNewFlesh/rolling-pin.git
  4. cd rolling-pin
  5. chmod +x bin/rolling-pin
  6. bin/rolling-pin docker-start
    • If building on a M1 Mac run export DOCKER_DEFAULT_PLATFORM=linux/amd64 first.

The service should take a few minutes to start up.

Run bin/rolling-pin --help for more help on the command line tool.

ZSH Setup

  1. bin/rolling-pin must be run from this repository's top level directory.

  2. Therefore, if using zsh, it is recommended that you paste the following line in your ~/.zshrc file:

    • alias rolling-pin="cd [parent dir]/rolling-pin; bin/rolling-pin"
    • Replace [parent dir] with the parent directory of this repository
  3. Consider adding the following line to your ~/.zshrc if you are using a M1 Mac:

    • export DOCKER_DEFAULT_PLATFORM=linux/amd64
  4. Running the zsh-complete command will enable tab completions of the cli commands, in the next shell session.

    For example:

    • rolling-pin [tab] will show you all the cli options, which you can press tab to cycle through
    • rolling-pin docker-[tab] will show you only the cli options that begin with "docker-"

Installation for Production

Python

pip install rolling-pin

Docker

  1. Install docker-desktop
  2. docker pull theNewFlesh/rolling-pin:[version]

Quickstart Guide

This repository contains a suite commands for the whole development process. This includes everything from testing, to documentation generation and publishing pip packages.

These commands can be accessed through:

  • The VSCode task runner
  • The VSCode task runner side bar
  • A terminal running on the host OS
  • A terminal within this repositories docker container

Running the zsh-complete command will enable tab completions of the CLI. See the zsh setup section for more information.

Command Groups

Development commands are grouped by one of 10 prefixes:

Command Description
build Commands for building packages for testing and pip publishing
docker Common docker commands such as build, start and stop
docs Commands for generating documentation and code metrics
library Commands for managing python package dependencies
session Commands for starting interactive sessions such as jupyter lab and python
state Command to display the current state of the repo and container
test Commands for running tests, linter and type annotations
version Commands for bumping project versions
quickstart Display this quickstart guide
zsh Commands for running a zsh session in the container and generating zsh completions

Common Commands

Here are some frequently used commands to get you started:

Command Description
docker-restart Restart container
docker-start Start container
docker-stop Stop container
docs-full Generate documentation, coverage report, diagram and code
library-add Add a given package to a given dependency group
library-graph-dev Graph dependencies in dev environment
library-remove Remove a given package from a given dependency group
library-search Search for pip packages
library-update Update dev dependencies
session-lab Run jupyter lab server
state State of
test-dev Run all tests
test-lint Run linting and type checking
zsh Run ZSH session inside container
zsh-complete Generate ZSH completion script

Development CLI

bin/rolling-pin is a command line interface (defined in cli.py) that works with any version of python 2.7 and above, as it has no dependencies. Commands generally do not expect any arguments or flags.

Its usage pattern is: bin/rolling-pin COMMAND [-a --args]=ARGS [-h --help] [--dryrun]

Commands

The following is a complete list of all available development commands:

Command Description
build-package Build production version of repo for publishing
build-prod Publish pip package of repo to PyPi
build-publish Run production tests first then publish pip package of repo to PyPi
build-test Build test version of repo for prod testing
docker-build Build Docker image
docker-build-from-cache Build Docker image from cached image
docker-build-prod Build production image
docker-container Display the Docker container id
docker-destroy Shutdown container and destroy its image
docker-destroy-prod Shutdown production container and destroy its image
docker-image Display the Docker image id
docker-prod Start production container
docker-pull-dev Pull development image from Docker registry
docker-pull-prod Pull production image from Docker registry
docker-push-dev Push development image to Docker registry
docker-push-dev-latest Push development image to Docker registry with dev-latest tag
docker-push-prod Push production image to Docker registry
docker-push-prod-latest Push production image to Docker registry with prod-latest tag
docker-remove Remove Docker image
docker-restart Restart Docker container
docker-start Start Docker container
docker-stop Stop Docker container
docs Generate sphinx documentation
docs-architecture Generate architecture.svg diagram from all import statements
docs-full Generate documentation, coverage report, diagram and code
docs-metrics Generate code metrics report, plots and tables
library-add Add a given package to a given dependency group
library-graph-dev Graph dependencies in dev environment
library-graph-prod Graph dependencies in prod environment
library-install-dev Install all dependencies into dev environment
library-install-prod Install all dependencies into prod environment
library-list-dev List packages in dev environment
library-list-prod List packages in prod environment
library-lock-dev Resolve dev.lock file
library-lock-prod Resolve prod.lock file
library-remove Remove a given package from a given dependency group
library-search Search for pip packages
library-sync-dev Sync dev environment with packages listed in dev.lock
library-sync-prod Sync prod environment with packages listed in prod.lock
library-update Update dev dependencies
library-update-pdm Update PDM
quickstart Display quickstart guide
session-lab Run jupyter lab server
session-python Run python session with dev dependencies
session-server Runn application server inside Docker container
state State of repository and Docker container
test-coverage Generate test coverage report
test-dev Run all tests
test-fast Test all code excepts tests marked with SKIP_SLOWS_TESTS decorator
test-lint Run linting and type checking
test-prod Run tests across all support python versions
version Full resolution of repo: dependencies, linting, tests, docs, etc
version-bump-major Bump pyproject major version
version-bump-minor Bump pyproject minor version
version-bump-patch Bump pyproject patch version
version-commit Tag with version and commit changes to master
zsh Run ZSH session inside Docker container
zsh-complete Generate oh-my-zsh completions
zsh-root Run ZSH session as root inside Docker container

Flags

Short Long Description
-a --args Additional arguments, this can generally be ignored
-h --help Prints command help message to stdout
--dryrun Prints command that would otherwise be run to stdout

Production CLI

Rolling-pin comes with a command line interface defined in command.py.

Its usage pattern is: rolling-pin COMMAND [ARGS] [FLAGS] [-h --help]

Commands


bash-completion

Prints BASH completion code to be written to a _rolling-pin completion file

Usage: rolling-pin bash-completion


zsh-completion

Prints ZSH completion code to be written to a _rolling-pin completion file

Usage: rolling-pin zsh-completion


conform

Copies source files to target filepaths according to given conform file.

Usage: rolling-pin conform [OPTIONS] SOURCE

Argument Description
source conform YAML filepath
Flag Argument Description Default
--groups text comma separated list of groups to be conformed all
--dryrun

Print out conform table instead of run conform

--help

print help message


graph

Generate a dependency graph of a source repository and write it to a given filepath

Usage: rolling-pin graph [OPTIONS] SOURCE TARGET

Argument Description
source repository path
target target filepath
Flag Argument Description Default
--include text include files that match this regex pattern .*.py$'
--exclude text exclude files that match this regex pattern test
--orient text graph orientation lr
--help

print help message


plot

Write radon metrics plots of given repository to given filepath.

Usage: rolling-pin plot [OPTIONS] SOURCE TARGET

Argument Description
source repository path
target plot filepath
Flag Description
--help print help message

table

Write radon metrics tables of given repository to given directory.

Usage: rolling-pin table [OPTIONS] SOURCE TARGET

Argument Description
source repository path
target table directory
Flag Description
--help print help message

toml

Generate a copy of a given TOML file with given edits indicated by flags. Flags are evalauted in the following order: edit, delete, search. Flags may be arbitrarily combined. Edit and delete flags may appear multiple times.

Usage: rolling-pin toml [OPTIONS] SOURCE

Argument Description
source TOML filepath
Flag Argument Description
--edit text replace key's value with given value, text is "=" separated key value pair in TOML format
--delete text delete keys that match this regular expression
--search text search for keys that match this regular expression
--target text target filepath to write to
--help

print help message

Example Usage

example file

>>>cat example.toml
[root]
a = 1
b = 2

[root.foo.bar]
x = "y"

[world]
hello = true

edit flag

>>>rolling-pin toml foobar.toml --edit 'root.a=999'
[root]
a = 999
b = 2...

>>>rolling-pin toml foobar.toml \
        --edit 'root.a=[1, 2]'   \
        --edit 'root.b="xxx"'
[root]
a = [
    1,
    2,
]
b = "xxx"...

>>>rolling-pin toml foobar.toml --edit 'root.foo.bar="baz"'
...
hello = true

[root.foo]
bar = "baz"...

delete flag

>>>rolling-pin toml foobar.toml \
        --delete 'root.foo.bar'  \
        --delete 'root.a'
[root]
b = 2

[world]
hello = true

search flag

>>>rolling-pin toml foobar.toml --search 'root.foo|world'
[world]
hello = true

[root.foo.bar]
x = "y"

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

rolling-pin-0.10.1.tar.gz (33.4 kB view hashes)

Uploaded Source

Built Distribution

rolling_pin-0.10.1-py3-none-any.whl (34.0 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page