Skip to main content

A development environment management tool for data scientists.

Project description

envd cat wink envd cat wink

Development environment for AI/ML

discord invitation link trackgit-views Python Version all-contributors envd package donwloads continuous integration Coverage Status

explain

explain

What is envd?

envd (ɪnˈvdɪ) is a command-line tool that helps you create the container-based development environment for AI/ML.

Development environments are full of python and system dependencies, CUDA, BASH scripts, Dockerfiles, SSH configurations, Kubernetes YAMLs, and many other clunky things that are always breaking. envd is to solve the problem:

  1. Declare the list of dependencies (CUDA, python packages, your favorite IDE, and so on) in build.envd
  2. Simply run envd up.
  3. Develop in the isolated environment.

Why use envd?

Environments built with envd provide the following features out-of-the-box:

❤️ Knowledge reuse in your team

envd build functions can be reused. Use include function to import any git repositories. No more copy/paste Dockerfile instructions, let's reuse them.

envdlib = include("https://github.com/tensorchord/envdlib")

def build():
    base(os="ubuntu20.04", language="python")
    envdlib.tensorboard(host_port=8888)
envdlib.tensorboard is defined in github.com/tensorchord/envdlib
def tensorboard(
    envd_port=6006,
    envd_dir="/home/envd/logs",
    host_port=0,
    host_dir="/tmp",
):
    """Configure TensorBoard.

    Make sure you have permission for `host_dir`

    Args:
        envd_port (Optional[int]): port used by envd container
        envd_dir (Optional[str]): log storage mount path in the envd container
        host_port (Optional[int]): port used by the host, if not specified or equals to 0,
            envd will randomly choose a free port
        host_dir (Optional[str]): log storage mount path in the host
    """
    install.python_packages(["tensorboard"])
    runtime.mount(host_path=host_dir, envd_path=envd_dir)
    runtime.daemon(
        commands=[
            [
                "tensorboard",
                "--logdir",
                envd_dir,
                "--port",
                str(envd_port),
                "--host",
                "0.0.0.0",
            ],
        ]
    )
    runtime.expose(envd_port=envd_port, host_port=host_port, service="tensorboard")

⏱️ Builtkit native, build up to 6x faster

Buildkit supports parallel builds and software cache (e.g. pip index cache and apt cache). You can enjoy the benefits without knowledge of it.

For example, the PyPI cache is shared across builds and thus the package will be cached if it has been downloaded before.

🐍 One configuration to rule them all

Development environments are full of Dockerfiles, bash scripts, Kubernetes YAML manifests, and many other clunky files that are always breaking. You just need one configuration file build.envd[^1], it works both for local Docker and Kubernetes clusters in the cloud.

envd

[^1]: The build language is starlark, which is a dialect of Python.

✍️ Don't sacrifice your developer experience

SSH is configured for the created environment. You can use vscode-remote, jupyter, pycharm or other IDEs that you love. Besides this, declare the IDE extensions you want, let envd take care of them.

def build():
    install.vscode_extensions([
        "ms-python.python",
    ])

☁️ No polluted environment

Are you working on multiple projects, all of which need different versions of CUDA? envd helps you create isolated and clean environments.

Who should use envd?

We're focused on helping data scientists and teams that develop AI/ML models. And they may suffer from:

  • building the development environments with Python/R/Julia, CUDA, Docker, SSH, and so on. Do you have a complicated Dockerfile or build script that sets up all your dev environments, but is always breaking?
  • Updating the environment. Do you always need to ask infrastructure engineers how to add a new Python/R/Julia package in the Dockerfile?
  • Managing environments and machines. Do you always forget which machines are used for the specific project, because you handle multiple projects concurrently?

Talk with us

💬 Interested in talking with us about your experience building or managing AI/ML applications?

Set up a time to chat!

Getting Started 🚀

Requirements

  • Docker (20.10.0 or above)

Install and bootstrap envd

envd can be installed with pip (only support Python3). After the installation, please run envd bootstrap to bootstrap.

pip3 install --pre --upgrade envd
envd bootstrap

You can add --dockerhub-mirror or -m flag when running envd bootstrap, to configure the mirror for docker.io registry:

envd bootstrap --dockerhub-mirror https://docker.mirrors.sjtug.sjtu.edu.cn

Create an envd environment

Please clone the envd-quick-start:

git clone https://github.com/tensorchord/envd-quick-start.git

The build manifest build.envd looks like:

def build():
    base(os="ubuntu20.04", language="python3")
    # Configure the pip index if needed.
    # config.pip_index(url = "https://pypi.tuna.tsinghua.edu.cn/simple")
    install.python_packages(name = [
        "numpy",
    ])
    shell("zsh")

Note that we use Python here as an example but please check out examples for other languages such as R and Julia here.

Then please run the command below to set up a new environment:

cd envd-quick-start && envd up
$ cd envd-quick-start && envd up
[+]  parse build.envd and download/cache dependencies 2.8s  (finished)
 => download oh-my-zsh                                                    2.8s
[+] 🐋 build envd environment 18.3s (25/25)  (finished)
 => create apt source dir                                                 0.0s
 => local://cache-dir                                                     0.1s
 => => transferring cache-dir: 5.12MB                                     0.1s
...
 => pip install numpy                                                    13.0s
 => copy /oh-my-zsh /home/envd/.oh-my-zsh                                 0.1s
 => mkfile /home/envd/install.sh                                          0.0s
 => install oh-my-zsh                                                     0.1s
 => mkfile /home/envd/.zshrc                                              0.0s
 => install shell                                                         0.0s
 => install PyPI packages                                                 0.0s
 => merging all components into one                                       0.3s
 => => merging                                                            0.3s
 => mkfile /home/envd/.gitconfig                                          0.0s
 => exporting to oci image format                                         2.4s
 => => exporting layers                                                   2.0s
 => => exporting manifest sha256:7dbe9494d2a7a39af16d514b997a5a8f08b637f  0.0s
 => => exporting config sha256:1da06b907d53cf8a7312c138c3221e590dedc2717  0.0s
 => => sending tarball                                                    0.4s
envd-quick-start via Py v3.9.13 via 🅒 envd
⬢ [envd] # You are in the container-based environment!

Set up Jupyter notebook

Please edit the build.envd to enable jupyter notebook:

def build():
    base(os="ubuntu20.04", language="python3")
    # Configure the pip index if needed.
    # config.pip_index(url = "https://pypi.tuna.tsinghua.edu.cn/simple")
    install.python_packages(name = [
        "numpy",
    ])
    shell("zsh")
    config.jupyter()

You can get the endpoint of the running Jupyter notebook via envd envs ls.

$ envd up --detach
$ envd envs ls
NAME                    JUPYTER                 SSH TARGET              CONTEXT                                 IMAGE                   GPU     CUDA    CUDNN   STATUS          CONTAINER ID
envd-quick-start        http://localhost:42779   envd-quick-start.envd   /home/gaocegege/code/envd-quick-start   envd-quick-start:dev    false   <none>  <none>  Up 54 seconds   bd3f6a729e94

More on documentation 📝

See envd documentation.

Roadmap 🗂️

Please checkout ROADMAP.

Contribute 😊

We welcome all kinds of contributions from the open-source community, individuals, and partners.

Open in Gitpod

Contributors ✨

Thanks goes to these wonderful people (emoji key):

 Friends A.
Friends A.

📖 🎨
Aaron Sun
Aaron Sun

📓 💻
Aka.Fido
Aka.Fido

📦 📖 💻
Alex Xi
Alex Xi

💻
Bingyi Sun
Bingyi Sun

💻
Ce Gao
Ce Gao

💻 📖 🎨 📆
Guangyang Li
Guangyang Li

💻
Gui-Yue
Gui-Yue

💻
Haiker Sun
Haiker Sun

💻
Ikko Ashimine
Ikko Ashimine

💻
Isaac
Isaac

💻
JasonZhu
JasonZhu

💻
Jian Zeng
Jian Zeng

🎨 🤔 🔬
Jinjing Zhou
Jinjing Zhou

🐛 💻 🎨 📖
Jun
Jun

📦 💻
Keming
Keming

💻 📖 🤔 🚇
Kevin Su
Kevin Su

💻
Ling Jin
Ling Jin

🐛 🚇
Manjusaka
Manjusaka

💻
Nino
Nino

🎨 💻
Pengyu Wang
Pengyu Wang

📖
Sepush
Sepush

📖
Siyuan Wang
Siyuan Wang

💻 🚇 🚧
Suyan
Suyan

📖
To My
To My

📖
Tumushimire Yves
Tumushimire Yves

💻
Wei Zhang
Wei Zhang

💻
Weizhen Wang
Weizhen Wang

💻
XRW
XRW

💻
Xu Jin
Xu Jin

💻
Xuanwo
Xuanwo

💬 🎨 🤔 👀
Yijiang Liu
Yijiang Liu

💻
Yilong Li
Yilong Li

📖 🐛 💻
Yuan Tang
Yuan Tang

💻 🎨 📖 🤔
Yuchen Cheng
Yuchen Cheng

🐛 🚇 🚧 🔧
Yuedong Wu
Yuedong Wu

💻
Yunchuan Zheng
Yunchuan Zheng

💻
Zheming Li
Zheming Li

💻
Zhenguo.Li
Zhenguo.Li

💻 📖
Zhenzhen Zhao
Zhenzhen Zhao

🚇 📓 💻
Zhizhen He
Zhizhen He

💻 📖
cutecutecat
cutecutecat

💻
dqhl76
dqhl76

📖 💻
jimoosciuc
jimoosciuc

📓
kenwoodjw
kenwoodjw

💻
nullday
nullday

🤔 💻
wangxiaolei
wangxiaolei

💻
wyq
wyq

🐛 🎨 💻
xiangtianyu
xiangtianyu

📖
xieydd
xieydd

💻
xing0821
xing0821

🤔 📓 💻
zhyon404
zhyon404

💻
杨成锴
杨成锴

💻

This project follows the all-contributors specification. Contributions of any kind welcome!

Star History

Star History Chart

License 📋

Apache 2.0

trackgit-views

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

envd-0.2.5a8.tar.gz (264.3 kB view hashes)

Uploaded Source

Built Distributions

envd-0.2.5a8-pp39-pypy39_pp73-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (8.9 MB view hashes)

Uploaded PyPy manylinux: glibc 2.17+ x86-64 manylinux: glibc 2.5+ x86-64

envd-0.2.5a8-pp39-pypy39_pp73-macosx_10_9_x86_64.whl (9.4 MB view hashes)

Uploaded PyPy macOS 10.9+ x86-64

envd-0.2.5a8-pp38-pypy38_pp73-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (8.9 MB view hashes)

Uploaded PyPy manylinux: glibc 2.17+ x86-64 manylinux: glibc 2.5+ x86-64

envd-0.2.5a8-pp38-pypy38_pp73-macosx_10_9_x86_64.whl (9.4 MB view hashes)

Uploaded PyPy macOS 10.9+ x86-64

envd-0.2.5a8-pp37-pypy37_pp73-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (8.9 MB view hashes)

Uploaded PyPy manylinux: glibc 2.17+ x86-64 manylinux: glibc 2.5+ x86-64

envd-0.2.5a8-pp37-pypy37_pp73-macosx_10_9_x86_64.whl (9.4 MB view hashes)

Uploaded PyPy macOS 10.9+ x86-64

envd-0.2.5a8-cp311-cp311-musllinux_1_1_x86_64.whl (8.9 MB view hashes)

Uploaded CPython 3.11 musllinux: musl 1.1+ x86-64

envd-0.2.5a8-cp311-cp311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (8.9 MB view hashes)

Uploaded CPython 3.11 manylinux: glibc 2.17+ x86-64 manylinux: glibc 2.5+ x86-64

envd-0.2.5a8-cp311-cp311-macosx_10_9_x86_64.whl (9.4 MB view hashes)

Uploaded CPython 3.11 macOS 10.9+ x86-64

envd-0.2.5a8-cp310-cp310-musllinux_1_1_x86_64.whl (8.9 MB view hashes)

Uploaded CPython 3.10 musllinux: musl 1.1+ x86-64

envd-0.2.5a8-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (8.9 MB view hashes)

Uploaded CPython 3.10 manylinux: glibc 2.17+ x86-64 manylinux: glibc 2.5+ x86-64

envd-0.2.5a8-cp310-cp310-macosx_10_9_x86_64.whl (9.4 MB view hashes)

Uploaded CPython 3.10 macOS 10.9+ x86-64

envd-0.2.5a8-cp39-cp39-musllinux_1_1_x86_64.whl (8.9 MB view hashes)

Uploaded CPython 3.9 musllinux: musl 1.1+ x86-64

envd-0.2.5a8-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (8.9 MB view hashes)

Uploaded CPython 3.9 manylinux: glibc 2.17+ x86-64 manylinux: glibc 2.5+ x86-64

envd-0.2.5a8-cp39-cp39-macosx_10_9_x86_64.whl (9.4 MB view hashes)

Uploaded CPython 3.9 macOS 10.9+ x86-64

envd-0.2.5a8-cp38-cp38-musllinux_1_1_x86_64.whl (8.9 MB view hashes)

Uploaded CPython 3.8 musllinux: musl 1.1+ x86-64

envd-0.2.5a8-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (8.9 MB view hashes)

Uploaded CPython 3.8 manylinux: glibc 2.17+ x86-64 manylinux: glibc 2.5+ x86-64

envd-0.2.5a8-cp38-cp38-macosx_10_9_x86_64.whl (9.4 MB view hashes)

Uploaded CPython 3.8 macOS 10.9+ x86-64

envd-0.2.5a8-cp37-cp37m-musllinux_1_1_x86_64.whl (8.9 MB view hashes)

Uploaded CPython 3.7m musllinux: musl 1.1+ x86-64

envd-0.2.5a8-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (8.9 MB view hashes)

Uploaded CPython 3.7m manylinux: glibc 2.17+ x86-64 manylinux: glibc 2.5+ x86-64

envd-0.2.5a8-cp37-cp37m-macosx_10_9_x86_64.whl (9.4 MB view hashes)

Uploaded CPython 3.7m macOS 10.9+ x86-64

envd-0.2.5a8-cp36-cp36m-musllinux_1_1_x86_64.whl (8.9 MB view hashes)

Uploaded CPython 3.6m musllinux: musl 1.1+ x86-64

envd-0.2.5a8-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (8.9 MB view hashes)

Uploaded CPython 3.6m manylinux: glibc 2.17+ x86-64 manylinux: glibc 2.5+ x86-64

envd-0.2.5a8-cp36-cp36m-macosx_10_9_x86_64.whl (9.4 MB view hashes)

Uploaded CPython 3.6m macOS 10.9+ x86-64

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page