DataSpires SDK

Version: 0.1.1

Train, fine-tune, and run models on DataSpires GPUs from any notebook.

pip install dataspires

Works in: Google Colab · Kaggle · Jupyter · VS Code · any Python 3.9+ environment

Contents

1. Quickstart
2. Authentication
3. Which method should I use?
4. Models & data sources
5. Resource guide
6. Jobs
7. Guides by task — Pretrain · Finetune · Generate — Checkpointing · Long-running jobs
8. Working with your model
9. API reference
10. Hardware & billing
11. Model & dataset registry
12. Troubleshooting
13. Built-in help

Quickstart

Fine-tune a model on a Hugging Face dataset in a few lines:

from dataspires import DataSpiresClient

client = DataSpiresClient()
client.authenticate()

job = client.finetune(
    model="qwen2.5-0.5b",
    training_mode="low",
    data="tatsu-lab/alpaca",
    gpus=1,
)
result = job.run(wait=True)

if result["status"] == "completed":
    job.download("./my-model")
else:
    print(result.get("error") or result.get("reason"))

pretrain() and generate() follow the same pattern: create a job, then call job.run().

Authentication

Get an API key

1. Sign in at dataspires.com.
2. Go to Profile → SDK keys → Create new key, then copy the afk_live_… value. It is shown only once.
3. Save it as DATASPIRES_API_KEY.

Set the key

client = DataSpiresClient()
client.authenticate()

authenticate() raises an error if the key is invalid or revoked. To rotate a key, revoke it in the dashboard and create a new one.

To use gated Hugging Face models or datasets, also set HUGGINGFACE_TOKEN.

Which method should I use?

Models & data sources

Pass a registry name, a Hugging Face ID, a URL, a Google Drive file, or local data. Remote sources are downloaded directly by the job, so large datasets never pass through your notebook. Local files and DataFrames are uploaded for you.

model=

data=

tokenizer= (transformer pretraining)

Accepts gpt2, any Hugging Face tokenizer ID (optionally with @revision), a Hugging Face URL, or a Google Drive file containing tokenizer files.

Preview how a source resolves

from dataspires import resolve_model, resolve_data, resolve_tokenizer

resolve_model(model="Qwen/Qwen2.5-0.5B")
resolve_data(data="Salesforce/wikitext#wikitext-2-raw-v1")
resolve_tokenizer("gpt2")

Dataset format (finetune)

A single text column works best. These common layouts are converted automatically:

Google Drive

Pass a Drive file the same way as any other source:

job = client.finetune(
    model="qwen2.5-0.5b",
    training_mode="low",
    data="gdrive:FILE_ID",
    gpus=1,
)

• Shared files (“Anyone with the link”) work immediately.
• Private files open a Google sign-in window the first time. After you approve read-only access, the job continues and future jobs reuse your sign-in.

For headless environments, set GOOGLE_DRIVE_TOKEN instead. To turn off the sign-in prompt, set DATASPIRES_GDRIVE_AUTO_CONNECT=0. To sign out, call client.disconnect_google_drive().

To check a file before submitting a job:

client.inspect_source("gdrive:FILE_ID")

Current limitations

Resource guide

Check whether a model fits before you start a run:

est = client.estimate_model("qwen2.5-0.5b", training_mode="low", gpus=1)
print(est["fits"], est["min_vram_gb"], est["parameters_b"])

If a configuration is unlikely to fit, the SDK shows a warning before submitting.

Jobs

Jobs run asynchronously on the DataSpires GPU cluster.

status and state

result = job.run(wait=True)
print(result["status"])

for entry in client.list_jobs():
    print(entry["job_id"], entry["state"])

Guides by task

Pretrain

Image classification (CNN):

job = client.pretrain(
    architecture="cnn",
    kind="resnet18",
    init="pretrained",
    data="./images/",
    config={"n_epoch": 5, "num_classes": 2, "batch_size": 32, "seed": 42},
    gpus=1,
    time_limit="02:00:00",
)
result = job.run(wait=True)
job.export_metrics("./run.csv")

Object detection:

job = client.pretrain(
    architecture="detector",
    kind="detect",          # or "segment", "pose"
    init="pretrained",
    weights="yolo11n.pt",
    data="./dataset/",
    data_config="data.yaml",
    config={"n_epoch": 100, "imgsz": 640, "batch_size": 16, "seed": 42},
)

Detector runs report training loss.

Language model from scratch:

job = client.pretrain(
    architecture="transformer",
    kind="gpt-decoder-tiny",
    init="scratch",
    tokenizer="gpt2",
    data="roneneldan/TinyStories",   # or a local folder such as "./corpus/"
    config={"max_steps": 200, "n_epoch": 1, "batch_size": 2, "seed": 42},
)
print(job.estimate())

Download results:

job.download("./my-outputs")
job.download("./checkpoints", include="checkpoints")

Finetune

import pandas as pd

data = pd.DataFrame({"text": ["Below is an instruction...\n\n### Response:\n..."]})

job = client.finetune(
    model="qwen2.5-0.5b",
    training_mode="low",        # low | medium | high | custom
    data=data,
    gpus=1,
    time_limit="01:00:00",
)
result = job.run(wait=True)
if result["status"] == "completed":
    job.download("./my-model")

For full control, use training_mode="custom" with a config:

job = client.finetune(
    model="qwen2.5-0.5b",
    training_mode="custom",
    data=data,
    config={
        "batch_size": 2,
        "learning_rate": 2e-4,
        "scheduler": "cosine",
        "n_epoch": 3,
        "precision": "bf16",
        "lora": {"r": 16, "alpha": 32, "dropout": 0.05},
        "max_seq_length": 2048,
        "optimizer": "adamw_torch",
        "n_checkpoint": 100,
        "n_evals": 0,
        "seed": 42,
    },
)

gpus and time_limit are passed as arguments, not inside config. Reported accuracy is token-level accuracy. Setting n_evals above 0 requires an evaluation split.

For vision-language models, pass task="captioning" or task="vqa" with image-text data. See dataspires.docs("finetune").

Checkpointing and resuming

Save checkpoints during a run, then continue from a previous job:

job = client.finetune(
    model="qwen2.5-0.5b",
    training_mode="low",
    data=data,
    time_limit="08:00:00",
    checkpoint={"every": 200, "keep": 2},
)
result = job.run(wait=True)

if result["status"] != "completed":
    resumed = client.finetune(
        model="qwen2.5-0.5b",
        training_mode="low",
        data=data,
        resume_from=job,          # or a job ID
    )
    resumed.run(wait=True)

List checkpoints with client.list_checkpoints(job_id). See dataspires.docs("checkpoints").

Generate

Run a model without changing its weights:

job = client.generate(
    model="Qwen/Qwen2.5-0.5B",
    data=[{"text": "Write one short sentence about GPUs."}],
    max_new_tokens=48,
    gpus=1,
)
result = job.run(wait=True)

if result["status"] == "completed":
    job.download("./generate-output")

Generated text is saved to the output folder. Use job.get_logs() to view progress while the job runs.

With a vision-language model and a local image manifest:

job = client.generate(
    model="smolvlm-256m",
    data="./manifest_dir/",
    outputs=["text", "confidence"],
    max_new_tokens=64,
    temperature=0.0,
)
result = job.run(wait=True)
if result["status"] == "completed":
    job.download("./vlm-output")

Long-running jobs

Notebook sessions can end before a long job finishes. Submit without waiting, save the job ID, and check back later:

job.run(wait=False)
job_id = job.job_id

# Later, in the same or a new session
status = client.get_job_status(job_id)
print(status["state"])

if status["state"] == "completed":
    client.download_model(job_id, "./my-model")

Jobs keep running on the cluster after your notebook disconnects.

Handling failures

job.run(wait=True) returns a result instead of raising when a job fails. Check result["status"] before downloading. Authentication and validation errors raise immediately.

Working with your model

The steps below apply to finetune() outputs. Fine-tuning produces a LoRA adapter, which is loaded on top of the base model.

Convert to GGUF for Ollama or llama.cpp

Merge the adapter into the base model first:

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B")
model = PeftModel.from_pretrained(base, "./my-model")
merged = model.merge_and_unload()
merged.save_pretrained("./my-model-merged")
AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B").save_pretrained("./my-model-merged")

Then convert ./my-model-merged with llama.cpp's convert_hf_to_gguf.py.

Publish to Hugging Face Hub

from huggingface_hub import HfApi

api = HfApi()

# Publish the adapter
api.upload_folder(folder_path="./my-model", repo_id="you/my-adapter", repo_type="model")

# Or publish full merged weights
api.upload_folder(folder_path="./my-model-merged", repo_id="you/my-model", repo_type="model")

API reference

DataSpiresClient

Job objects

Fine-tuning jobs also provide validate(). Pretraining jobs provide estimate().

Source helpers

Hardware & billing

Start with a small model and training_mode="low", and use estimate_model before trying larger models.

Jobs can use up to 32 GPUs. Maximum duration is set per workspace or cohort.

GPU time is billed at $0.60 per GPU-hour (NVIDIA L4). View invoices at dataspires.com/dashboard/billing. job.estimated_cost_usd() shows the maximum cost for a job's time_limit; you are billed for actual usage.

Model & dataset registry

Browse curated models and datasets:

client.list_available_models()
client.list_available_models(size="tiny")
client.list_available_datasets()
client.get_model_requirements("qwen2.5-0.5b", "low")

Any public Hugging Face model or dataset also works. See Models & data sources.

Troubleshooting

Built-in help

import dataspires

dataspires.docs("help")
dataspires.docs("finetune")
dataspires.docs("pretrain")
dataspires.docs("checkpoints")