DataSpires SDK

Version: 0.1.0

Train, finetune, and run inference on DataSpires GPUs from any notebook.

pip install dataspires

Works in: Google Colab · Kaggle · Jupyter · VS Code · any Python 3.8+ environment

---

Contents

1. 60-second quickstart
2. Authentication
3. Which method should I use?
4. Backend
5. Guides by task — Pretrain · Finetune · Generate — Checkpointing
6. Working with your output model
7. API reference
8. Hardware & billing
9. Model & dataset registry
10. Troubleshooting
11. Built-in help

---

60-second quickstart

from dataspires import DataSpiresClient

client = DataSpiresClient()
client.authenticate(api_key="afk_live_…")  # or set API_KEY

job = client.pretrain(
    architecture="cnn",
    kind="resnet18",
    init="pretrained",
    data="./images/",
    config={"n_epoch": 5, "num_classes": 2, "seed": 42},
    gpus=1,
)
result = job.run(wait=True)
print(result["status"])  # "completed" | "failed" | "timeout" | …
if result["status"] != "completed":
    print(result.get("error") or result.get("reason"))

finetune() and generate() use the same client → job → job.run() pattern.

---

Authentication

Get a key

1. Sign up at dataspires.com.
2. Profile → DataSpires SDK keys → Create new key. Copy the afk_live_… value — it's shown once.
3. Store it as API_KEY.

Set the key

Environment	How
Google Colab	Secrets → API_KEY
Kaggle	Secrets → API_KEY
Local	os.environ["API_KEY"] = "afk_live_…"
Anywhere	client.authenticate(api_key="afk_live_…")

client = DataSpiresClient()
client.authenticate()

authenticate() raises on a bad or revoked key. Rotate a key by revoking it on the dashboard and creating a new one.

---

Which method should I use?

Your goal	Call	You provide
Detector, CNN, or scratch transformer	client.pretrain()	architecture, kind, data, optional init / config
Fine-tune an LLM / VLM foundation model	client.finetune()	model, training_mode, data, optional task=
Run a model without changing weights	client.generate()	model, prompts / image+question manifest

Train from zero → pretrain(). Adapt existing weights → finetune(). Frozen answers → generate().

---

Backend

Jobs run on the DataSpires GPU cluster by default.

Job model	Async — submit, then poll or list_jobs()
Outputs	job.download() and client.download_model()
Multi-GPU	Supported when the cluster has capacity
Wall-clock max	time_limit capped at 10 hours

status vs state

API	Field	Example values
job.run() return dict	status	submitted, completed, failed, timeout, cancelled
client.list_jobs() / client.get_job_status()	state	same lifecycle strings; may also include queued, running

result = job.run(wait=True)
print(result["status"])

for entry in client.list_jobs():
    print(entry["job_id"], entry["state"], entry.get("reason"))

---

Guides by task

Pretrain

job = client.pretrain(
    architecture="cnn",
    kind="resnet18",
    init="pretrained",
    data="./images/",
    config={"n_epoch": 5, "num_classes": 2, "batch_size": 32, "seed": 42},
    gpus=1,
    time_limit="02:00:00",
)
result = job.run(wait=True)
rows = job.get_metrics()
job.export_metrics("./run.csv")

Detector. Metrics are loss only in v1.

job = client.pretrain(
    architecture="detector",
    kind="detect",          # or segment, pose
    init="pretrained",
    weights="yolo11n.pt",
    data="./dataset/",
    data_config="data.yaml",
    config={"n_epoch": 100, "imgsz": 640, "batch_size": 16, "seed": 42},
)

Decoder from scratch. tokenizer is required for decoder kinds only.

job = client.pretrain(
    architecture="transformer",
    kind="gpt-decoder-tiny",
    init="scratch",
    tokenizer="gpt2",
    data="./corpus/",
    config={"max_steps": 1000, "n_epoch": 1, "seed": 42},
)

architecture	kind
detector	detect, segment, pose
cnn	resnet18, resnet50
transformer	gpt-decoder-tiny, qwen-decoder-tiny, llama-3-decoder-tiny, vit-tiny

time_limit is an HH:MM:SS string.

job.download("./my-outputs")
job.download("./ckpts", include="checkpoints")
client.download_model(job.job_id, "./my-outputs")

Finetune

import pandas as pd

data = pd.DataFrame({"text": ["Below is an instruction...\n\n### Response:\n..."]})

job = client.finetune(
    model="qwen2.5-0.5b",
    training_mode="low",     # low | medium | high | custom
    data=data,
    gpus=1,
    time_limit="01:00:00",
)
result = job.run(wait=True)
if result["status"] == "completed":
    client.download_model(result["job_id"], "./my-model")

Mode	Strategy	Precision
low	QLoRA (rank 8)	4-bit
medium	LoRA (rank 16)	8-bit on 1 GPU, bf16 otherwise
high	High-rank LoRA (rank 64)	bf16
custom	User-defined (see config=)	bf16, fp16, 4bit, or 8bit

training_mode="custom" requires config=. Presets reject config. gpus and time_limit stay outside config.

job = client.finetune(
    model="qwen2.5-0.5b",
    training_mode="custom",
    data=data,
    config={
        "batch_size": 2,
        "learning_rate": 2e-4,
        "scheduler": "cosine",
        "n_epoch": 3,
        "precision": "bf16",
        "lora": {"r": 16, "alpha": 32, "dropout": 0.05},
        "max_seq_length": 2048,
        "optimizer": "adamw_torch",
        "n_checkpoint": 100,
        "n_evals": 0,
        "seed": 42,
    },
)
rows = job.get_metrics()
job.export_metrics("./run_metrics.csv")

Causal-LM accuracy is token accuracy, not a task score. If n_evals is above 0 and there is no eval split, the SDK fails before submit.

For VLMs, pass task= (captioning / VQA) with image+text data. See dataspires.docs("finetune").

Optional: checkpoint={"every": 200, "keep": 2} and resume_from= (prior job, job id, or {job_id, step}).

Checkpointing and resuming

Jobs honour time_limit (capped at 10 hours). A stopped run can be continued with resume_from=.

job = client.finetune(
    model="qwen2.5-0.5b",
    training_mode="low",
    data=data,
    time_limit="08:00:00",
    checkpoint={"every": 200, "keep": 2},
)
result = job.run(wait=True)

if result["status"] != "completed":
    job2 = client.finetune(
        model="qwen2.5-0.5b",
        training_mode="low",
        data=data,
        resume_from=job,
    )
    job2.run(wait=True)

Path	Meaning
output/checkpoints/checkpoint-N/	Mid-run snapshots
output/final/	Weights after a clean exit

client.list_checkpoints(job_id) lists prefixes. See dataspires.docs("checkpoints").

Generate

(Frozen inference — no weight updates)

job = client.generate(
    model="smolvlm-256m",
    data="./manifest_dir/",
    outputs=["text", "confidence"],
    max_new_tokens=64,
    temperature=0.0,
)
result = job.run(wait=True)

Job failures

job.run(wait=True) returns a dict on failure instead of raising. Check result["status"] == "completed" before downloading. Auth and submit-time validation errors do raise.

---

Working with your model

Convert to GGUF for Ollama / llama.cpp

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B")
model = PeftModel.from_pretrained(base, "./my-model")
merged = model.merge_and_unload()
merged.save_pretrained("./my-model-merged")
AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B").save_pretrained("./my-model-merged")

Then convert with llama.cpp convert_hf_to_gguf.py.

Publish to Hugging Face Hub

from huggingface_hub import HfApi
api = HfApi()
api.upload_folder(folder_path="./my-model", repo_id="you/my-model", repo_type="model")

---

API reference

DataSpiresClient

Method	Description
authenticate(api_key=None)	Resolve key; raises on failure
pretrain(architecture, kind, init, config, …)	Returns a job — call .run()
finetune(model, training_mode, data, config, …)	Returns FinetuneJob — call .run()
generate(model, data, outputs, …)	Returns GenerateJob — call .run()
download_model(job_id, local_dir, include=("final",))	Download outputs
list_checkpoints(job_id)	List checkpoint prefixes
upload_dataset(local_path, dataset_name=None)	Optional separate staging
list_available_models() / list_available_datasets()	Registry lookups
get_model_requirements(model, training_mode)	GPU/memory recommendation
cancel_job(job_id)	Stop a job
list_jobs(include_completed=False)	List jobs → dicts with state
get_job_status(job_id)	Poll by ID → dict with state

Job objects

Member	Description
run(wait=True, poll_interval=30)	Submit; with wait=True, return a result dict with status
cancel()	Stop the job
get_logs(tail=100)	Recent log lines
get_metrics()	Loss/acc rows
export_metrics(path)	Write metrics to .csv or .json
download(local_dir, include=("final",))	Download outputs
checkpoints()	List checkpoint prefixes
estimated_cost_usd()	Planning ceiling from gpus × time_limit
status, job_id	Current tracked state / job ID

---

Hardware & billing

Prefer small models and training_mode="low" for lighter runs.

Model size	Mode	Fits on 1 GPU?
0.5B – 3B	low / medium	yes
3B – 7B	low (QLoRA 4-bit)	yes
7B	medium	yes
13B	low	tight
30B+	low	unlikely

Limits: 10h time_limit; hard max 32 GPUs (SDK warns above 8).

Billing: invoices at dataspires.com/dashboard/billing. GPU time is $0.60/GPU-hr for an L4. job.estimated_cost_usd() is a planning ceiling from time_limit, not the invoice.

---

Model & dataset registry

client.list_available_models()
client.list_available_models(size="tiny")
client.list_available_datasets()
client.get_model_requirements("qwen2.5-0.5b", "low")

---

Troubleshooting

Symptom	Likely cause	Fix
API key auth failed	Bad / revoked key	Create a new key; set API_KEY
Job stuck in queued	Cluster at capacity	client.list_jobs(); wait or try later
KeyError: 'status' on list_jobs()	Wrong field	Use entry["state"] for list/status APIs
KeyError: 'state' on job.run() result	Wrong field	Use result["status"] for job.run()
Gated HF model fails	Missing token	Add HUGGINGFACE_TOKEN
download_model() returns nothing	No finals written	Confirm the run completed; try include="checkpoints"
reason is DeadlineExceeded	Hit time_limit	resume_from=job

---

Built-in help

import dataspires
dataspires.docs("help")
dataspires.docs("finetune")
dataspires.docs("pretrain")
dataspires.docs("checkpoints")

AfriLinkClient and import afrilink remain available as aliases for existing notebooks.