agi-pack 0.1.0

Dockerfile generator for AGI -- nothing more, nothing less.

agi-pack

A Dockerfile builder for AGI — nothing more, nothing less.

agi-pack is simply a weekend project I hacked together, that started with a conversation with ChatGPT / GPT-4. See the inspiration section below for more details on the ChatGPT prompts used.

🚨 Disclaimer: More than 90% of this codebase was generated by GPT-4 and Github Co-Pilot.

Rationale 🤔

Docker has become the standard for building and managing isolated environments for ML. However, any one who has gone down this rabbit-hole knows how broken ML development is, especially when you need to experiment and re-configure your environments constantly. Production is another nightmare -- large docker images (10GB+), bloated docker images with model weights that are ~5-10GB in size, 10+ minute long docker build times, sloppy package management to name just a few.

What makes Dockerfiles painful? If you've ever tried to roll your own Dockerfiles with all the best-practices while fully understanding their internals, you'll still find yourself building, and re-building, and re-building these images across a whole host of use-cases. Having to build Dockerfile(s) for dev, prod, and test all turn out to be a nightmare when you add the complexity of hardware targets (CPUs, GPUs, TPUs etc), drivers, python, virtual environments, build and runtime dependencies.

agi-pack aims to simplify this by allowing developers to define Dockerfiles in a concise YAML format and then generate them based on your environment needs (i.e. python version, system packages, conda/pip dependencies, GPU drivers etc).

For example, you should be able to easily configure your dev environment for local development, and have a separate prod environment where you'll only need the runtime dependencies avoiding any bloat.

agi-pack hopes to also standardize the base images, so that we can really build on top of giants.

Features ✨

• Simple Configuration: Define your Docker images using a straightforward YAML format.
• Dynamic Generation: Use the power of Jinja2 templating to create Dockerfiles on-the-fly.
• Sequential and Multi-stage Builds: Define re-usable and production-ready base images and build dependent images for dev, prod, test.
• Extensible: Easily extend and adapt to more complex scenarios.

Goals 🎯

• Simplicity: Make it easy to define and build docker images for ML.
• Modular, Re-usable, Composable: Ability to define good base, dev and prod images for ML, and re-use them wherever possible.
• Best Practices: Support best practices for building docker images for ML -- good base images, multi-stage builds, minimal image sizes, etc.
• Ecosystem-driven: Make the YAML / DSL extensible to support the ML ecosystem, as more libraries, drivers, HW vendors, come into the market.
• Vendor-agnostic: agi-pack is not intended to be built for any specific vendor (including us/where I work). There was clearly a need for this tool internally, so I decided to build it in the open and keep it simple.

Why the name? 🤷‍♂️

agi-pack is very much intended to be tongue-in-cheek -- we are soon going to be living in a world full of quasi-AGI agents orchestrated via ML containers. At the very least, agi-pack should provide the building blocks for us to build a more modular, re-usable, and distribution-friendly container format for "AGI".

Installation 📦

pip install git+hhttps://github.com/spillai/agi-pack.git

For shell completion, you can install them via:

agi-pack --install-completion <bash|zsh|fish|powershell|pwsh>

Usage 🛠

1. Create a simple YAML configuration file called agibuild.yaml via agi-pack init:

   agi-pack init
   

2. Edit agibuild.yaml to define your custom system and python packages

   images:
     base-sklearn:
       image: <repo>/agi:latest-base-sklearn
       base: python:3.8.10-slim
       system:
       - wget
       - build-essential
       python: 3.8.10
       pip:
       - loguru
       - typer
       - scikit-learn
   

3. Generate the Dockerfile using agi-pack generate

   agi-pack generate -c agibuild.yaml
   

   You should see the following output:

   $ agi-pack generate -c agibuild.yaml
   📦 base-sklearn
   └── 🎉 Successfully generated Dockerfile (target=base-sklearn, filename=Dockerfile).
       └── `docker build -f Dockerfile --target base-sklearn .`
   

That's it! You can now build the generated Dockerfile using docker build to build the image directly.

More Complex Example 📚

Now imagine you want to build a more complex image that has multiple stages, and you want to build a base image that has all the basic dependencies, and a dev image that has additional build-time dependencies.

```yaml
images:
  base-cpu:
    name: agi
    base: debian:buster-slim
    system:
      - wget
    python: 3.8.10
    pip:
      - scikit-learn
    run:
      - echo "Hello, world!"

  dev-cpu:
    base: base-cpu
    system:
    - build-essential
```

Once you've defined this agibuild.yaml, running agi-pack generate will generate the following output:

You should see the following output:
```bash
$ agi-pack generate -c agibuild.yaml
📦 base-cpu
└── 🎉 Successfully generated Dockerfile (target=base-cpu, filename=Dockerfile).
    └── `docker build -f Dockerfile --target base-cpu .`
📦 dev-cpu
└── 🎉 Successfully generated Dockerfile (target=dev-cpu, filename=Dockerfile).
    └── `docker build -f Dockerfile --target dev-cpu .`
```

As you can see, agi-pack will generate a single Dockerfile for each of the images defined in the YAML file. You can then build the individual images from the same Dockerfile using docker targets: docker build -f Dockerfile --target <target> . where <target> is the name of the image target you want to build.

Here's the corresponding Dockerfile that was generated.

Inspiration and Attribution 🌟

Prompt: I'm building a Dockerfile generator and builder to simplify machine learning infrastructure. I'd like for the Dockerfile to be dynamically generated (using Jinja templates) with the following parametrizations:

```

# Sample YAML file
images:
base-gpu:
    base: nvidia/cuda:11.8.0-base-ubuntu22.04
    system:
    - gnupg2
    - build-essential
    - git
    python: 3.8.10
    pip:
    - torch==2.0.1

I'd like for this yaml file to generate a Dockerfile via `agi-pack generate -c <name>.yaml`.

You are an expert in Docker and Python programming, how would I implement this builder in Python. Use Jinja2 templating and miniconda python environments wherever possible. I'd like an elegant and concise implementation that I can share on PyPI.
```

TL;DR agi-pack was inspired by a combination of Replicate's cog, Baseten's truss, skaffold, and Docker Compose Services. I wanted a standalone project without any added cruft/dependencies of vendors and services.

Contributing 🤝

Contributions are welcome! Please read the CONTRIBUTING guide for more information.

License 📄

This project is licensed under the MIT License. See the LICENSE file for details.