hokusai 3.0.0

Artsy Docker development toolkit

HOKUSAI

Hokusai is a Docker + Kubernetes CLI for application developers.

Hokusai "dockerizes" applications and manages their lifecycle throughout development, testing, and release cycles.

Hokusai wraps calls to Kubectl, Docker, Docker-Compose and Git with a CLI, and defines a CI workflow.

Hokusai currently only supports Kubernetes deployments on AWS, configured to pull from ECS container repositories (ECR), although other providers may be added in the future.

Why Hokusai?

At Artsy, as we began working with Kubernetes, while impressed with its design, capabilities, and flexibility, we were in need of tooling we could deliver to agile development teams that addressed the day-to-day tasks of application development, delivery, introspection and maintenance, while providing a clean and uncomplicated interface.

Transitioning teams to the Docker / Kubernetes ecosystem can be intimidating, and comes with a steep learning curve. We set out to create a Heroku-like CLI that would shepherd the application developer into the ecosystems of Docker and Kubernetes, and while introducing new tooling and concepts, outlining a clear practice for dependency management, local development, testing and CI, image repository structure, deployment and orchestration.

Installation

MacOS

We recommend installing via Homebrew:

$ brew update
$ brew tap artsy/formulas
$ brew install hokusai

If you previously installed Hokusai via an alternate installation method, you may need to force the link step.

$ brew link --overwrite hokusai

If you previously installed Hokusai via Pip, you may want to first uninstall it:

$ pip uninstall hokusai

Linux

curl -sSL https://raw.githubusercontent.com/artsy/hokusai/main/get-hokusai.sh | sudo bash

Note: This method installs Hokusai to /usr/local/bin/hokusai.

Pip

Hokusai can be installed via Pip, on MacOS or Linux. If you do so, please first go through Pyenv, Python, and Virtualenv steps.

Python 3.7+ is required.

pip install hokusai

Note: If Pip fails at upgrading your system Python packages, try:

pip install hokusai --ignore-installed

Docker

We also maintain Hokusai Docker images for running Hokusai in Docker.

Github

Release artifacts are available on Github.

AWS S3

Release artifacts are also available in AWS S3. You can use this convenience script or Curl to fetch them.

A note on Python 2.x

Hokusai currently supports Python 3.7+ only. The last version that supported Python 2.x was v0.5.18.

Setup

We assume that your org admin has already set up a Kubernetes cluster and an AWS infrastructure, and that your local environment has Git, Docker, and Docker-Compose installed. Perform the following steps to get Hokusai working:

1. Ensure your org admin has completed the steps mentioned in Administering Hokusai.

2. Configure your local environment with AWS credentials.

3. Run hokusai configure command.

4. Optionally, enable Bash autocompletion:

   eval "$(_HOKUSAI_COMPLETE=source hokusai)"
   

Getting Started

Once Hokusai is configured, you can start using it on a project. Please see Getting Started.

Command Reference

A full command reference can be found in Command Reference.

Review Apps

Hokusai can be used to simplify the process of spinning up a "review app" instance of your project, based on a feature branch or pull request.

Full details are in the Review App reference.

Hokusai Files

A descripton of all the files used by Hokusai.

Developing Hokusai

To work on Hokusai itself, please set up:

Pyenv

We recommend using Pyenv to install the correct version of Python. For a tutorial of Pyenv, see this guide.

When installing on MacOS, please make sure to use brew-installed openssl and readline libraries, and xcode-installed zlib library. And make sure these libraries are correctly linked. Like so:

brew install openssl readline zlib

echo 'export PATH="/usr/local/opt/openssl@1.1/bin:$PATH"' >> ~/.bash_profile
echo 'export LDFLAGS="-L/usr/local/opt/openssl@1.1/lib"' >> ~/.bash_profile
echo 'export CPPFLAGS="-I/usr/local/opt/openssl@1.1/include"' >> ~/.bash_profile
echo 'export PKG_CONFIG_PATH="/usr/local/opt/openssl@1.1/lib/pkgconfig"' >> ~/.bash_profile

Python

Hokusai is currently tested on Python 3.10 so we recommend using that Python version.

If you use Pyenv to install Python, you should see an output similar to this:

$ pyenv install 3.10
python-build: use openssl@1.1 from homebrew
python-build: use readline from homebrew
Downloading Python-3.10.13.tar.xz...
-> https://www.python.org/ftp/python/3.10.13/Python-3.10.13.tar.xz
Installing Python-3.10.13...
python-build: use tcl-tk from homebrew
python-build: use readline from homebrew
python-build: use ncurses from homebrew
python-build: use zlib from xcode sdk
Installed Python-3.10.13 to /Users/jxu/.pyenv/versions/3.10.13

With the desired Python version installed, activate it globally:

pyenv global 3.10

Note: If you want to create a PyInstaller distribution, Python must be installed with development libraries. Use the environment variable PYTHON_CONFIGURE_OPTS="--enable-framework" on Darwin and PYTHON_CONFIGURE_OPTS="--enable-shared" on Linux when running pyenv install.

Virtualenv

We recommend using a virtual environment to isolate Hokusai's dependencies from that of other projects on your local environment.

The Pyenv install comes with pyenv-virtualenv which can be used to create virtual environments.

Poetry

Use Poetry to install Hokusai's dependencies as well as Hokusai itself in editable mode. Please see this guide for working with Poetry.

Install Poetry:

pip install --upgrade pip
pip install poetry

Install dependencies and Hokusai in editable mode:

poetry install

To update dependencies:

poetry lock

Testing

Install Minikube

Minikube is used for integration tests.

brew install minikube
minikube start --kubernetes-version=<version of your Kubernetes clusters, example: v1.2.3>

Run tests

To run unit tests:

make test

To run integration tests:

make integration-local

Only specific modules, TestClasses, or even methods:

python -m unittest test.unit.test_module.TestClass.test_method

Tip: Set DEBUG=1 environment variable to print boto logging

Distributing Hokusai

Beta Release

Merging a branch into main automatically creates a beta version useful for testing. A script that runs in CI automatically generates a Beta version number and writes it into VERSION file. The version number is based on the combination of the latest canonical version found in RELEASE_VERSION file and the latest Git commit in main branch. VERSION file in version control has just a dummy number (e.g. 999.999.999).

To install the beta:

MacOS

$ brew uninstall hokusai
$ brew uninstall hokusai-beta
$ brew update
$ brew tap artsy/formulas
$ brew install hokusai-beta

Linux

curl -sSL https://raw.githubusercontent.com/artsy/hokusai/main/get-hokusai.sh | sudo bash -s beta

Official Release

To create an official release, such as v1.2.3, perform the following:

• Create a branch named prepare-v1.2.3 and make the following changes:

  • Bump canonical version in RELEASE_VERSION file.
  • Upate CHANGELOG.
  • Open a PR to merge into main. Please see past PRs for example.

• Open a PR to merge main into release. Please see past PRs for example.

A CI script will copy the version in RELEASE_VERSION file to VERSION file.

The Name

The project is named for the great Japanese artist Katsushika Hokusai (1760-1849).

About Artsy

This project is the work of engineers at Artsy, the world's leading and largest online art marketplace and platform for discovering art. One of our core Engineering Principles is being Open Source by Default which means we strive to share as many details of our work as possible.

You can learn more about this work from our blog and by following @ArtsyOpenSource or explore our public data by checking out our API. If you're interested in a career at Artsy, read through our job postings!