DataSpires SDK

Version: 0.1.8

Train, fine-tune, and run models on DataSpires GPUs from any notebook.

pip install dataspires

Works in: Google Colab · Kaggle · Jupyter · VS Code · any Python 3.9+ environment

Contents

1. Quickstart
2. Authentication
3. Which method should I use?
4. Models & data sources
5. Resource guide
6. Jobs
7. Guides by task — Pretrain · Finetune · Generate
8. Job patterns — Checkpointing · Long-running jobs · Handling failures
9. Working with your model
10. API reference
11. Hardware & billing
12. Model & dataset registry
13. Troubleshooting
14. Built-in help

Quickstart

Fine-tune a model on a Hugging Face dataset in a few lines:

from dataspires import DataSpiresClient

client = DataSpiresClient()
client.authenticate()

job = client.finetune(
    model="qwen2.5-0.5b",
    training_mode="low",
    data="tatsu-lab/alpaca",
    gpus=1,
)
result = job.run(wait=True)

if result["status"] == "completed":
    job.download("./my-model")
else:
    print(result.get("error") or result.get("reason"))

pretrain() and generate() follow the same pattern: create a job, then call job.run().

Authentication

Get an API key

1. Sign in at dataspires.com.
2. Go to Profile → SDK keys → Create new key, then copy the dsp_live_… value. It is shown only once.
3. Save it as DATASPIRES_API_KEY.

Set the key

client = DataSpiresClient()
client.authenticate()

authenticate() raises an error if the key is invalid or revoked. To rotate a key, revoke it in the dashboard and create a new one.

Gated Hugging Face models

Some registry models (for example Llama or Gemma) are gated on Hugging Face: you must accept the model's license on its Hugging Face page, then provide a token. finetune() and generate() check this before the job is submitted and raise a clear error if the model needs a token and none is found.

1. Accept the license at https://huggingface.co/<model-id> (see the model's huggingface_id in the registry, e.g. client.list_models()).
2. Create a token at huggingface.co/settings/tokens.
3. Save it as HUGGINGFACE_TOKEN:

Which method should I use?

Models & data sources

Pass a registry name, a Hugging Face ID, a URL, a Google Drive file, or local data. Remote sources are downloaded directly by the job, so large datasets never pass through your notebook. Local files and DataFrames are uploaded for you.

model=

data=

tokenizer= (transformer pretraining)

Accepts gpt2, any Hugging Face tokenizer ID (optionally with @revision), a Hugging Face URL, or a Google Drive file containing tokenizer files.

Preview how a source resolves

from dataspires import resolve_model, resolve_data, resolve_tokenizer

resolve_model(model="Qwen/Qwen2.5-0.5B")
resolve_data(data="Salesforce/wikitext#wikitext-2-raw-v1")
resolve_tokenizer("gpt2")

Dataset format (finetune)

A single text column works best. These common layouts are converted automatically:

Google Drive

Pass a Drive file the same way as any other source:

job = client.finetune(
    model="qwen2.5-0.5b",
    training_mode="low",
    data="gdrive:FILE_ID",
    gpus=1,
)

• Shared files (“Anyone with the link”) work immediately.
• Private files open a Google sign-in window the first time. After you approve read-only access, the job continues and future jobs reuse your sign-in.

For headless environments, set GOOGLE_DRIVE_TOKEN instead. To turn off the sign-in prompt, set DATASPIRES_GDRIVE_AUTO_CONNECT=0. To sign out, call client.disconnect_google_drive().

To check a file before submitting a job:

client.inspect_source("gdrive:FILE_ID")

Current limitations

Resource guide

Check whether a model fits before you start a run:

est = client.estimate_model("qwen2.5-0.5b", training_mode="low", gpus=1)
print(est["fits"], est["min_vram_gb"], est["parameters_b"])

If a configuration is unlikely to fit, the SDK shows a warning before submitting.

Jobs

Jobs run asynchronously on the DataSpires GPU cluster.

Guides by task

Pretrain

Image classification (CNN):

job = client.pretrain(
    architecture="cnn",
    kind="resnet18",
    init="pretrained",
    data="./images/",
    config={"n_epoch": 5, "num_classes": 2, "batch_size": 32, "seed": 42},
    gpus=1,
    time_limit="02:00:00",
)
result = job.run(wait=True)
job.export_metrics("./run.csv")

Object detection:

job = client.pretrain(
    architecture="detector",
    kind="detect",          # or "segment", "pose"
    init="pretrained",
    weights="yolo11n.pt",
    data="./dataset/",
    data_config="data.yaml",
    config={"n_epoch": 100, "imgsz": 640, "batch_size": 16, "seed": 42},
)

Detector runs report training loss.

Language model from scratch:

job = client.pretrain(
    architecture="transformer",
    kind="gpt-decoder-tiny",
    init="scratch",
    tokenizer="gpt2",
    data="roneneldan/TinyStories",   # or a local folder such as "./corpus/"
    config={"max_steps": 200, "n_epoch": 1, "batch_size": 2, "seed": 42},
)
print(job.estimate())

Download results:

job.download("./my-outputs")
job.download("./checkpoints", include="checkpoints")

Finetune

import pandas as pd

data = pd.DataFrame({"text": ["Below is an instruction...\n\n### Response:\n..."]})

job = client.finetune(
    model="qwen2.5-0.5b",
    training_mode="low",        # low | medium | high | custom
    data=data,
    gpus=1,
    time_limit="01:00:00",
)
result = job.run(wait=True)
if result["status"] == "completed":
    job.download("./my-model")

For full control, use training_mode="custom" with a config:

job = client.finetune(
    model="qwen2.5-0.5b",
    training_mode="custom",
    data=data,
    config={
        "batch_size": 2,
        "learning_rate": 2e-4,
        "scheduler": "cosine",
        "n_epoch": 3,
        "precision": "bf16",
        "lora": {"r": 16, "alpha": 32, "dropout": 0.05},
        "max_seq_length": 2048,
        "optimizer": "adamw_torch",
        "n_checkpoint": 100,
        "n_evals": 0,
        "seed": 42,
    },
)

gpus, gpu_type, and time_limit are passed as arguments, not inside config. Reported accuracy is token-level accuracy.

When n_evals is greater than 0, choose how evaluation data is supplied:

Hub dataset with a built-in val split — used automatically:

job = client.finetune(
    model="qwen2.5-0.5b",
    training_mode="custom",
    data="glue",
    config={
        "batch_size": 2,
        "learning_rate": 2e-4,
        "scheduler": "cosine",
        "n_epoch": 1,
        "precision": "bf16",
        "n_evals": 50,
    },
)

Local val file or in-memory val data:

job = client.finetune(
    model="qwen2.5-0.5b",
    training_mode="custom",
    data="./train.jsonl",
    val_data="./val.jsonl",  # or a DataFrame / list of rows
    config={
        "batch_size": 2,
        "learning_rate": 2e-4,
        "scheduler": "cosine",
        "n_epoch": 1,
        "precision": "bf16",
        "n_evals": 50,
    },
)

Hold out part of the training data (when there is no separate val split):

job = client.finetune(
    model="qwen2.5-0.5b",
    training_mode="low",
    data="tatsu-lab/alpaca",
    val_split=0.1,
)

For vision-language models, use a registry VLM alias or a Hugging Face VLM id with task="captioning", task="vqa", or task="chat", plus a folder with train.jsonl and images/ (or a DataFrame with image and response). Florence 2 and chat-style VLMs share the same dataset shape; the job passes model_architecture to the container loader. See dataspires.docs("finetune").

Generate

Run a model without changing its weights:

job = client.generate(
    model="Qwen/Qwen2.5-0.5B",
    data=[{"text": "Write one short sentence about GPUs."}],
    max_new_tokens=48,
    gpus=1,
)
result = job.run(wait=True)

if result["status"] == "completed":
    job.download("./generate-output")

Generated text is saved to the output folder. Use job.get_logs() to view progress while the job runs.

With a vision-language model and a local image manifest:

job = client.generate(
    model="smolvlm-256m",
    data="./manifest_dir/",
    outputs=["text", "confidence"],
    max_new_tokens=64,
    temperature=0.0,
)
result = job.run(wait=True)
if result["status"] == "completed":
    job.download("./vlm-output")

Job patterns

Cross-cutting workflows that apply after you pick a task (pretrain, finetune, or generate).

Checkpointing and resuming

Save checkpoints during a run, then continue from a previous job:

job = client.finetune(
    model="qwen2.5-0.5b",
    training_mode="low",
    data=data,
    time_limit="08:00:00",
    checkpoint={"every": 200, "keep": 2},
)
result = job.run(wait=True)

if result["status"] != "completed":
    resumed = client.finetune(
        model="qwen2.5-0.5b",
        training_mode="low",
        data=data,
        resume_from=job,          # or a job ID
    )
    resumed.run(wait=True)

List checkpoints with client.list_checkpoints(job_id). See dataspires.docs("checkpoints").

Long-running jobs

Notebook sessions can end before a long job finishes. Submit without waiting, save the job ID, and check back later:

job.run(wait=False)
job_id = job.job_id

# Later, in the same or a new session
status = client.get_job_status(job_id)
print(status["status"])

if status["status"] == "completed":
    client.download_model(job_id, "./my-model")

Jobs keep running on the cluster after your notebook disconnects.

Handling failures

job.run(wait=True) returns a result instead of raising when a job fails. Check result["status"] before downloading. Authentication and validation errors raise immediately.

Working with your model

The steps below apply to finetune() outputs. Fine-tuning produces a LoRA adapter, which is loaded on top of the base model.

Convert to GGUF for Ollama or llama.cpp

Merge the adapter into the base model first:

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B")
model = PeftModel.from_pretrained(base, "./my-model")
merged = model.merge_and_unload()
merged.save_pretrained("./my-model-merged")
AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B").save_pretrained("./my-model-merged")

Then convert ./my-model-merged with llama.cpp's convert_hf_to_gguf.py.

Publish to Hugging Face Hub

from huggingface_hub import HfApi

api = HfApi()

# Publish the adapter
api.upload_folder(folder_path="./my-model", repo_id="you/my-adapter", repo_type="model")

# Or publish full merged weights
api.upload_folder(folder_path="./my-model-merged", repo_id="you/my-model", repo_type="model")

API reference

DataSpiresClient

Job objects

Fine-tuning jobs also provide validate(). Pretraining jobs provide estimate().

Source helpers

Hardware & billing

Start with a small model and training_mode="low", and use estimate_model before trying larger models.

Jobs can use up to 32 GPUs. Maximum duration is set per workspace or cohort.

Pass gpu_type on pretrain(), finetune(), or generate() to pin a job to one hardware type. Omit it to run on any eligible GPU. Any other value is rejected.

job = client.finetune(
    model="qwen2.5-0.5b",
    training_mode="low",
    data=data,
    gpus=1,
    gpu_type="l4",
)

GPU time is billed at $0.60 per GPU-hour (NVIDIA L4). View invoices at dataspires.com/dashboard/billing. job.estimated_cost_usd() shows the maximum cost for a job's time_limit; you are billed for actual usage.

Model & dataset registry

Browse curated models and datasets:

client.list_available_models()
client.list_available_models(size="tiny")
client.list_available_datasets()
client.get_model_requirements("qwen2.5-0.5b", "low")

Any public Hugging Face model or dataset also works. See Models & data sources.

Troubleshooting

Built-in help

import dataspires

dataspires.docs("help")
dataspires.docs("finetune")
dataspires.docs("pretrain")
dataspires.docs("checkpoints")