separateai 0.1.0

A Together-shaped client that trains and serves fine-tuned models on your own Vast.ai GPUs

separateai

A Together-shaped Python client that trains and serves fine-tuned models on your own Vast.ai GPUs. Swap from together import Together → from separateai import Separate and your existing fine-tuning + hosting code runs on rented hardware instead of a managed service.

Scope is deliberately narrow: train custom fine-tunes (QLoRA) and host them. It is not a general Together replacement — no embeddings, no image gen, no serverless inference over stock models.

from separateai import Separate

client = Separate(api_key=VAST_API_KEY, hf_token=HF_TOKEN)

# ── train ───────────────────────────────────────────────────────────────
ref = client.files.upload(open("data.jsonl"))          # str / bytes / path / file
job = client.fine_tuning.create(
    training_file=ref,
    hyperparameters={"n_epochs": 20, "learning_rate": 1e-4, "lora_r": 64},
)
status = client.fine_tuning.retrieve(job.id)           # .status, .steps_completed…
# job.output_name is the private HF repo the trained adapter is pushed to

# ── host + chat ─────────────────────────────────────────────────────────
ep = client.endpoints.ensure(model="unsloth/gemma-4-E4B-it",
                             adapter=job.output_name)  # blocks until STARTED
out = client.chat.completions.create(
    messages=[{"role": "user", "content": "hi"}],
    model="adapter", endpoint_id=ep.id)
print(out.choices[0].message.content)
client.endpoints.stop(ep.id)   # endpoints do NOT idle-stop — always stop when done

Install

pip install separateai

The vastai CLI arrives as a pip dependency — no separate install. Credentials come from constructor args or env vars, never hardcoded:

var	what for
VAST_API_KEY	renting/destroying GPU boxes (the vastai CLI is authed automatically — note: the key is written once to ~/.config/vastai/vast_api_key, the CLI's own key file)
HF_TOKEN	pulling base models on the box + persisting/serving trained adapters

How it works

• fine_tuning.create() rents the cheapest matching Vast box, launches a pinned Unsloth QLoRA trainer image (shahzodad/trainer — built from trainer/ in this repo, so you can audit exactly what your HF_TOKEN is handed to), and appends an upload step so the trained adapter lands in a private HF repo before the box dies. The Vast instance id is the job id.
• fine_tuning.retrieve() polls instance state + scrapes container logs into a Together-shaped status (pending → running → completed/failed). A job only reads completed after the adapter upload finished — and on the first terminal status the box is destroyed automatically so it stops billing.
• endpoints.create()/ensure() rents a box running vllm/vllm-openai (OpenAI-compatible) serving a base model + your LoRA adapter. The endpoint is protected with a bearer token derived from your Vast key (vLLM --api-key), so a public-IP box doesn't hand out free inference; chat.completions.create() sends it automatically.

Cost model (read this)

You are renting real GPUs by the hour:

• training boxes are destroyed automatically the first time retrieve() sees a terminal status (or explicitly via fine_tuning.cancel()) — but a job you never poll again is a box you keep paying for;
• serving boxes have no idle auto-stop and bill until you call endpoints.stop().

endpoints.ensure() destroys the box itself if startup fails or times out, so a failed launch never bills idle.

Repo layout

separateai/   the Python SDK (what `pip install separateai` gives you)
tests/        offline unit tests — every Vast call mocked, nothing rented
trainer/      Dockerfile + entrypoint for shahzodad/trainer, the custom
              image every TRAINING box runs (audit/rebuild it here)

Note the asymmetry: only training needs a custom image. HOSTING boxes run the official upstream vllm/vllm-openai image as-is — all serving behaviour (bearer-token auth, LoRA rank, adapter wiring) is set via the launch command, so there is no hosting image to maintain or audit beyond upstream vLLM.

Module map

file	role
client.py	Separate facade — wires .files, .fine_tuning, .endpoints, .chat
files.py	upload() — stage the JSONL dataset (inline, base64 env)
fine_tuning.py	create() / retrieve() / cancel()
serving.py	endpoints.* + chat.completions.create() (vLLM on Vast)
vast.py	vastai CLI wrappers — the only file that talks to Vast
hf.py	minimal HF Hub HTTP helpers (private repos, small-file commits)
logs.py	container logs → {state, steps, total}
config.py	pinned images + GPU/CUDA/disk requirements (the hidden IaC)
models.py	DatasetRef, FineTuneJob, Endpoint (Together-shaped returns)

Security & trust model

Read this before running untrusted models or handling other people's data.

• The rented GPU host can see your HF_TOKEN. Training/serving boxes get the token in their container env (Vast even stores the launch command), and Vast hosts are third parties. Use a fine-grained, repo-scoped, short-lived HF token, not an account-wide one.
• Inference traffic is plaintext HTTP. Chat requests, the bearer token, and completions travel unencrypted to http://<box-ip>:8000. Don't send secrets in prompts over untrusted networks; front the endpoint with TLS (a tunnel / reverse proxy) if you need confidentiality.
• Endpoints are protected by a bearer token derived from your Vast API key. It keeps random internet scanners off your public-IP box; it is not secret from Vast or the host operator. A Vast API key is therefore required to serve — endpoints.* raises without one.
• model / adapter must be trusted repo ids. They are validated against the HF id charset before use, but a malicious model repo can still run code at load time — only point fine_tuning.create(model=...) / endpoints at repos you trust.
• Images are digest-pinned (config.py) so a repushed tag can't swap in a token-stealing image.

Limits

• Small datasets ride inline in a container env var (Vast caps total env at 32 KB); bigger ones are automatically committed to a private HF dataset repo and fetched on-box with HF_TOKEN — no caller-visible difference.
• One pinned base model family is tested end-to-end today (Gemma-4-E4B, 4-bit QLoRA on an RTX 4090). Other Unsloth-supported bases should be drop-in via fine_tuning.create(model=...), but are not yet verified.
• Hyperparameters supported: n_epochs, batch_size, learning_rate, lr_scheduler_type, warmup_ratio, weight_decay, max_grad_norm, lora_r, lora_alpha, lora_dropout. Others are ignored.