Contents

1. What it does
2. How it works
3. Before you start
4. Install: Hub and first worker
5. Start, stop and restart
6. Add users and teams
7. Add more worker servers
8. Remove a worker server
9. Using the notebooks and AI magics
10. Monitoring dashboard
11. Configuration reference
12. Logs and troubleshooting
13. Development and tests
14. More documentation

What it does

• Team isolation. Every JupyterHub group (a "team") is mapped to its own worker VM. When a user starts a notebook, the Hub SSHes into that team's worker and launches the user's server there, so one heavy user cannot slow down the other teams.
• Per-user Linux accounts. On first login the worker creates a Linux account for the user (alice@corp.com → alice), so users cannot read each other's files.
• Resource limits. Each notebook server runs under cgroups v2 memory and CPU limits (systemd-run).
• Crash-safe state. Sessions and team→VM mappings live in SQLite, so a Hub restart does not lose track of running servers.
• Secrets. Optional HashiCorp Vault integration injects per-user secrets as environment variables.
• AI magics. %do, %fix, %review and %rework in any notebook, backed by Ollama, OpenAI/Claude/Gemini (through LiteLLM) or an MCP server.
• Live monitoring. A dashboard at /hub/monitoring shows CPU, memory, disk and network for each worker. Admins see every worker; users see only their own team's.

How it works

                    Browser (https://hub.example.com)
                                │
                     Caddy (HTTPS, port 443)
                                │
┌───────────────────────── Hub VM ──────────────────────────┐
│  configurable-http-proxy :8000  →  JupyterHub :8081         │
│  Google login · CustomSpawner · SQLite state · /hub/monitoring
└───────────────┬───────────────────────────┬────────────────┘
                │ SSH (port 22)             │ SSH (port 22)
                ▼                           ▼
     ┌──── Worker VM: team_alpha ──┐   ┌──── Worker VM: team_beta ───┐
     │ alice's jupyterhub-singleuser│   │ dave's jupyterhub-singleuser│
     │ bob's   jupyterhub-singleuser│   │ ...                         │
     │ metrics agent ──WS──▶ Hub    │   │ metrics agent ──WS──▶ Hub   │
     └─────────────────────────────┘   └─────────────────────────────┘

When alice (group team_alpha) clicks Start My Server:

1. The Hub looks up team_alpha in SQLite and gets the worker's IP and SSH key.
2. It SSHes in as the worker's admin user (admin_ssh_user) and creates the Linux user alice if she doesn't exist yet.
3. It SSHes in as alice, picks a free port and starts jupyterhub-singleuser inside a cgroup scope.
4. The Hub's proxy routes https://hub.example.com/user/alice/ to worker-ip:port.

Before you start

You need:

• 2 or more Ubuntu VMs (tested on Ubuntu 24.04 and 26.04): one Hub, plus one worker per team. Pick the cloud region closest to your users: from India, us-east-1 gave ~300 ms round trips and ~20 KB/s downloads, ap-south-1 ~40 ms. Amazon Linux 2023 has its own scripts (scripts/install_*_amazon_linux.sh), not yet tested end to end.
• A domain name pointing at the Hub's public IP (for example hub.example.com). Google login only works over HTTPS on a real domain, never on http:// or a bare IP.
• A Google OAuth client (Google Cloud console → APIs & Services → Credentials → OAuth client ID → Web application) with the redirect URI https://hub.example.com/hub/oauth_callback.
• Network rules (AWS security groups or your firewall):

Install: Hub and first worker

Full walkthroughs: AWS EC2 Deployment Guide and Installation Guide. The short version:

1. Hub VM

git clone https://github.com/wajoud/jupyterhub-pilot.git
cd jupyterhub-pilot
sudo bash scripts/install_hub.sh

The script asks for the Hub's private IP, your admin username and the first worker's private IP. It then:

• installs JupyterHub, the proxy, the package and (optionally) Vault
• creates an SSH key at /opt/jupyterhub-pilot/keys/worker
• writes /opt/jupyterhub-pilot/hub_settings.json and /opt/jupyterhub-pilot/user_mapping.json (first team: team_alpha)
• generates the metrics agent token at /etc/jupyterhub-pilot/agent_token
• seeds the SQLite DB at /var/lib/jupyterhub/jupyterhub_pilot_state.db

At the end it prints a public SSH key and an agent token. Keep both for the worker step.

⚠️ Admin username: with Google login the JupyterHub username is the part of the email before the @. For admin@company.com, enter admin, not the full email, or that account will not get admin rights.

Then edit /opt/jupyterhub-pilot/hub_settings.json and set hosted_domain to your Google Workspace domain (for example company.com). Only accounts from that domain can log in.

2. Worker VM

git clone https://github.com/wajoud/jupyterhub-pilot.git
cd jupyterhub-pilot
sudo bash scripts/install_worker.sh

It asks for:

1. the OS admin user that the Hub may SSH in as (usually ubuntu)
2. the Hub's public SSH key (printed by install_hub.sh, or cat /opt/jupyterhub-pilot/keys/worker.pub on the Hub)
3. the Hub's private IP
4. the agent token (printed by install_hub.sh, or sudo cat /etc/jupyterhub-pilot/agent_token on the Hub)

It installs Jupyter, the AI magics and the metrics agent once for all users, and starts the jupyterhub_pilot-metrics service.

Check that the Hub can reach the worker (run on the Hub):

ssh -i /opt/jupyterhub-pilot/keys/worker ubuntu@<WORKER_PRIVATE_IP> 'python3 /opt/jupyterhub-pilot/get_port.py'
# prints a free port number, e.g. 41873

3. HTTPS with Caddy (on the Hub)

sudo apt install -y debian-keyring debian-archive-keyring apt-transport-https curl
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/gpg.key' | sudo gpg --dearmor -o /usr/share/keyrings/caddy-stable-archive-keyring.gpg
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/debian.deb.txt' | sudo tee /etc/apt/sources.list.d/caddy-stable.list
sudo apt update && sudo apt install -y caddy

echo 'hub.example.com {
    encode zstd gzip
    reverse_proxy <HUB_PRIVATE_IP>:8000
}' | sudo tee /etc/caddy/Caddyfile
sudo systemctl reload caddy

Keep Caddy in front even for a quick test: HTTPS gives HTTP/2 and encode compresses Notebook's JavaScript about 4×. Over plain http://<ip>:8000 from far away, the Notebook page can time out and stay blank (ChunkLoadError).

No domain yet (testing only): Caddy can serve the Hub's public IP with its own self-signed certificate. Browsers show a warning you click through, and Google login will not work, so use test login.

printf '{\n\tdefault_sni <HUB_PUBLIC_IP>\n}\n\nhttps://<HUB_PUBLIC_IP> {\n\ttls internal\n\tencode zstd gzip\n\treverse_proxy <HUB_PRIVATE_IP>:8000\n}\n' | sudo tee /etc/caddy/Caddyfile
sudo systemctl restart caddy

default_sni is needed because browsers send no server name for a bare IP; without it the HTTPS handshake fails.

Now continue with Start, stop and restart.

Start, stop and restart

Option A: run in the foreground (first try, debugging)

Run on the Hub, as the same user you ran install_hub.sh with sudo from (usually ubuntu):

cd /opt/jupyterhub-pilot
export CLIENT_ID=<google-client-id>
export CLIENT_SECRET=<google-client-secret>
export OAUTH_CALLBACK_URL=https://hub.example.com/hub/oauth_callback
export JUPYTERPILOT_AGENT_TOKEN=$(sudo cat /etc/jupyterhub-pilot/agent_token)
python3 -m jupyterhub -f jupyterhub_config.py

Stop it with Ctrl+C.

Option B: run as a systemd service (production)

install_hub.sh already wrote the service (/etc/systemd/system/jupyterhub.service) and a root-only env file (/etc/jupyterhub-pilot/hub.env) with the agent token filled in. Fill in the Google lines, then start it:

sudo nano /etc/jupyterhub-pilot/hub.env
sudo systemctl enable --now jupyterhub

To set it up by hand instead, put the secrets in an env file only root can read:

sudo tee /etc/jupyterhub-pilot/hub.env > /dev/null << EOF
CLIENT_ID=<google-client-id>
CLIENT_SECRET=<google-client-secret>
OAUTH_CALLBACK_URL=https://hub.example.com/hub/oauth_callback
JUPYTERPILOT_AGENT_TOKEN=$(sudo cat /etc/jupyterhub-pilot/agent_token)
EOF
sudo chmod 600 /etc/jupyterhub-pilot/hub.env

Create the service (change User= if the Hub user is not ubuntu):

sudo tee /etc/systemd/system/jupyterhub.service > /dev/null << 'INI'
[Unit]
Description=JupyterHub-Pilot Hub
After=network.target

[Service]
User=ubuntu
WorkingDirectory=/opt/jupyterhub-pilot
EnvironmentFile=/etc/jupyterhub-pilot/hub.env
ExecStart=/usr/bin/python3 -m jupyterhub -f /opt/jupyterhub-pilot/jupyterhub_config.py
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target
INI

sudo systemctl daemon-reload
sudo systemctl enable --now jupyterhub

Option C: test login, no Google (testing only)

For a trial without a Google client or domain, set two passwords in /etc/jupyterhub-pilot/hub.env (the file has commented lines ready) and leave the Google lines empty:

JUPYTERPILOT_TEST_PASSWORD=<8+ characters>
JUPYTERPILOT_TEST_ADMIN_PASSWORD=<32+ characters, different>

Any user name logs in with the first password; users in admin_users log in with the second. Restart the Hub. Use it only behind Caddy with access limited to your IP, never on a public Hub.

Everyday commands

⚠️ Stopping the Hub stops every user's notebook server (JupyterHub's default). Unsaved work in running kernels is lost. Warn users before a restart.

Stop one user's notebook server

• The user: Hub home page → Stop My Server.
• An admin: Admin page → the user's row → Stop Server.
• By API (admin token from Token page):

curl -X DELETE -H "Authorization: token <ADMIN_API_TOKEN>" \
  https://hub.example.com/hub/api/users/alice/server

When do I need a restart?

Add users and teams

A team is a JupyterHub group whose name matches a key in user_mapping.json (for example team_alpha). Users in that group run their notebooks on that team's worker.

1. The user logs in once with Google (this creates their JupyterHub account), or an admin adds them on the Admin page → Add Users using the name before the @.
2. Admin → Groups → open the team group (create it with Add Group if it does not exist, using the exact team name) → Add Users.
3. The user clicks Start My Server.

Rules:

• A user with no group cannot start a server ("User has no groups assigned").
• Keep each user in one team group. If a user is in several, the first group is used.
• Make someone an admin: add their username to admin_users in hub_settings.json and restart the Hub, or tick Admin on their user in the Admin panel.

Add more worker servers

Use this to give a new team its own VM (or to move a team to a bigger VM). No Hub restart is needed.

1. Create the VM and apply the worker network rules from Before you start. Also allow the new worker's private IP on the Hub's ports 8000 and 8081.

2. Set up the worker (on the new VM):

git clone https://github.com/wajoud/jupyterhub-pilot.git
cd jupyterhub-pilot
sudo bash scripts/install_worker.sh

Paste the same Hub public key and agent token as for the first worker. Get them on the Hub with:

cat /opt/jupyterhub-pilot/keys/worker.pub
sudo cat /etc/jupyterhub-pilot/agent_token

Quick way (steps 3–5 in one command, on the Hub):

sudo bash /opt/jupyterhub-pilot/scripts/add_worker.sh team_beta 172.31.10.22          # ec2-user as 3rd argument on Amazon Linux

It adds or updates the team in user_mapping.json, loads it into SQLite and tests SSH. Then go to step 6.

3. Register the worker on the Hub. Add an entry to /opt/jupyterhub-pilot/user_mapping.json:

{
  "team_alpha": {
    "server_ip": "172.31.10.11",
    "server_ssh_key": "/opt/jupyterhub-pilot/keys/worker",
    "admin_ssh_user": "ubuntu"
  },
  "team_beta": {
    "server_ip": "172.31.10.22",
    "server_ssh_key": "/opt/jupyterhub-pilot/keys/worker",
    "admin_ssh_user": "ubuntu"
  }
}

4. Load the mapping into SQLite (safe to re-run; existing teams are updated, not duplicated):

sudo -u ubuntu python3 /opt/jupyterhub-pilot/jupyterhub_pilot/seed_sqlite.py \
  --db /var/lib/jupyterhub/jupyterhub_pilot_state.db \
  --mapping /opt/jupyterhub-pilot/user_mapping.json

5. Test the SSH connection from the Hub:

ssh -i /opt/jupyterhub-pilot/keys/worker ubuntu@172.31.10.22 'python3 /opt/jupyterhub-pilot/get_port.py'

6. Create the group team_beta in Admin → Groups and add its users. They can start servers right away.

To move a team to another VM: set up the new worker (step 2), change that team's server_ip (step 3), re-seed (step 4). Users get the new VM the next time they start their server. Their notebook files stay on the old VM (/home/<user>/notebook); copy them over if needed.

Remove a worker server

1. Stop the servers of that team's users (Admin page → Stop Server).

2. Remove the team from user_mapping.json, then delete it from SQLite. seed_sqlite.py only adds and updates, so this step is needed:

   sudo -u ubuntu sqlite3 /var/lib/jupyterhub/jupyterhub_pilot_state.db \
     "DELETE FROM team_mappings WHERE team = 'team_beta';"
   

3. Delete the group in Admin → Groups, or move its users to another team.

4. Copy any notebooks you want to keep from the worker's /home/<user>/notebook, then shut the VM down.

Using the notebooks and AI magics

Users open https://hub.example.com, log in with Google and click Start My Server. Files live on their team's worker in /home/<user>/notebook.

Load the magics in a notebook:

%load_ext jupyterhub_pilot.extension

The LLM backend is set in ~/.jupyterhub-pilot/config.json (per user) or /etc/jupyterhub-pilot/config.json (whole worker). The default is a local Ollama at http://localhost:11434. Setup, auto-loading for all users and cloud/MCP backends: AI Magic Commands.

Monitoring dashboard

Open https://hub.example.com/hub/monitoring (or click 📊 Monitoring in the Hub's nav bar).

• You must be logged in.
• Admins see every worker. Users see only the workers of their own team.
• A worker appears when its metrics agent connects with the right token and disappears when the agent stops.

Agent status, on a worker: sudo systemctl status jupyterhub_pilot-metrics. More: Monitoring.

Configuration reference

/opt/jupyterhub-pilot/hub_settings.json (restart the Hub after editing):

Environment variables:

Logs and troubleshooting

More cases: Installation Guide → Troubleshooting.

Development and tests

All 123 tests run without real infrastructure (no SSH, EC2 or Vault):

python3 -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
pytest -v tests/

jupyterhub-pilot/
├── jupyterhub_pilot/      # Python package: AI magics, LLM providers, SQLite store,
│                          # monitoring handlers + dashboard, metrics agent, Vault client
├── spawner.py             # CustomSpawner: SSH launch, JIT user creation, cgroups
├── jupyterhub_config.py   # Hub config (Google login, spawner, monitoring routes)
├── templates/page.html    # Adds the Monitoring link to the Hub nav bar
├── scripts/               # install_hub.sh, install_worker.sh
├── tests/                 # unit tests (see tests/README.md)
└── docs/                  # architecture, security, AI magics, monitoring

More documentation

• AWS EC2 Deployment Guide: end-to-end on AWS
• Installation Guide: detailed manual setup
• Architecture & EC2 Routing
• Security, Isolation & RBAC
• AI Magic Commands
• Live Monitoring Dashboard
• Tests