AIDSC: Lightweight Classroom Inference Client

aidsc is a Python client designed for students and researchers to perform LLM inference on classroom clusters. It acts as a gateway to the Zeus control plane, allowing you to run models on high-performance classroom hardware from your local machine.

Key Features

• Classroom-Ready: Built specifically for inferencing classroom machines running LLMs.
• Broad Machine Access: Users can access any classroom machine, and Skynet access is available with special permissions.
• Seamless Remote Access: Automatic SSH tunneling via PAMD to bridge the gap to internal networks.
• Smart Connectivity: Checks for existing local tunnels before prompting for credentials.
• Persistent Credentials: Optional encrypted-style saving to .env (git-ignored automatically).
• Lightweight & Typed: Pydantic-validated payloads ensuring your requests always match the Zeus API.

Hardware and Model Capacity

Each classroom machine is equipped with an NVIDIA RTX A5000 GPU with 24GB of VRAM. These machines can run models upwards of 35B parameters at 4-bit, though some newer models may be difficult to serve reliably given the older hardware.

Skynet is equipped with 4 NVIDIA A100 GPUs with 80GB of VRAM each, for a total of 320GB. With special permissions, Skynet can run models up to roughly 550B-600B parameters at 4-bit.

All inference runs through vLLM. Models are not offloaded to CPU memory; they are served directly on the available GPU memory. To request additional models, email bdg20b.

Connectivity Logic

The client automatically resolves connectivity in this order:

1. Local Check: Checks localhost:32553. If healthy, it proceeds immediately.
2. Environment/Dotenv: Looks for PAMD_USER and PAMD_PASS in your system or .env file.
3. Interactive Setup: If credentials are missing or the port is closed, it clears the console and guides you through establishing an SSH tunnel.
4. Background Persistence: After establishing a tunnel, it asks if you'd like to keep it running in the background after your script exits.

Usage Reference

1. Initialization

from aidsc import aidsc

client = aidsc(
    local_base_url="http://127.0.0.1:32553", # Override local Zeus URL
    infer_timeout=600.0,                      # Global timeout for inference calls (seconds)
    dotenv_path=".env"                       # Custom path to .env file
)

# Ensure connectivity (checks local port, prompts for SSH if needed)
client.ensure_ready()

2. Request Parameters

The LanguageInferenceRequest object allows fine-grained control over the job:

from aidsc import LanguageArgs, LanguageInferenceRequest

req = LanguageInferenceRequest(
    args=LanguageArgs(
        model="gpt-oss-20b",           # The target model name
        questions=["Recipe for rice?"], # List of prompts/questions
    ),
    max_new_tokens=512,                # Tokens to generate per response
    max_context_length=8192,           # Context window (alias: max_model_len)
    max_server_uptime=604800,          # How long the server can stay alive (seconds)
    total_concurrent_models=1,         # Instances of the model to run
    max_num_seqs=8                     # Max sequences for batching
)

3. Inference & Overrides

You can override specific parameters during the infer() call:

response = client.infer(
    req,
    server=117,             # Target a specific machine ID (e.g., 117)
    max_new_tokens=1024,    # Override the request's max_new_tokens
    client_timeout=300      # Specific timeout for this request
)

Configuration Reference

Environment Variables

These can be set in your OS or in a .env file in your project root.

Client Settings (aidsc())

Request Settings (LanguageInferenceRequest)

Project Architecture

• client.py: The high-level API. Handles the "Automatic SSH" logic, process detachment, and console UI.
• models.py: Pydantic definitions for all request types.
• connectivity.py: Low-level logic for checking local port health.
• zeus_client.py: The underlying HTTP client for the Zeus REST API.

Security Note

The client uses sshpass to handle password-based SSH connections to the PAMD jump host. Your .env file is automatically added to .gitignore when you run setup_interactive() to prevent accidental exposure of your credentials.