strands-transformers 0.2.0

The universal entrypoint to HuggingFace transformers for Strands agents — 100% task & modality coverage, zero hardcoding.

🤗 Strands Transformers

One tool wraps all of HuggingFace transformers. One provider makes any local model a multimodal agent brain.

Agents that see, hear, and speak — 100% task coverage, zero hardcoding, fully local.

---

use_aws wraps all of boto3. use_lerobot wraps all of lerobot. use_transformers wraps all of HuggingFace transformers — every task, every modality, in one tool that reads transformers' own taxonomy at runtime (new task upstream ⇒ supported here with no code change). And TransformerModel makes any local HF model a drop-in Strands brain that speaks the full content-block protocol — image, video, audio, document. With Qwen2.5-Omni it even speaks back.

flowchart LR
    IN["📥 text · image · video<br/>audio · document · robot-state"]
    TOOL["🛠️ use_transformers<br/><i>tool</i>"]
    BRAIN["🧠 TransformerModel<br/><i>local agent brain</i>"]
    OUT["📤 text · speech · image<br/>labels · actions"]
    IN --> TOOL --> OUT
    IN --> BRAIN --> OUT
    classDef i fill:#7C4DFF,stroke:#5b34d6,color:#fff;
    classDef c fill:#FFD21E,stroke:#E68A00,color:#3a2d00;
    classDef o fill:#00E5FF,stroke:#00b3cc,color:#003844;
    class IN i;
    class TOOL,BRAIN c;
    class OUT o;

📖 Full documentation →  ·  built with MkDocs (docs/)

Install

uv pip install strands-transformers        # from PyPI
# or from source:
uv pip install -e .                         # or: pip install -e .
PYTHONPATH=. python examples/smoke.py       # verify → "12/12 checks passed"

Optional extras (audio · vision · training · docs)

uv pip install -e ".[audio]"      # soundfile, librosa  (mp3/flac/ogg decode)
uv pip install -e ".[vision]"     # opencv, av  (video)
uv pip install -e ".[training]"   # trl, peft, accelerate
uv pip install -e ".[docs]"       # mkdocs-material, mkdocstrings
uv pip install -e ".[all]"        # everything

WAV audio works without extras. device="auto" picks cuda → mps → cpu (bf16 on GPU).

60-second hello — a local vision agent

import io
from PIL import Image
from strands import Agent
from strands_transformers import TransformerModel

buf = io.BytesIO(); Image.new("RGB", (64, 64), (20, 200, 40)).save(buf, "PNG")  # green square

model = TransformerModel(model_path="HuggingFaceTB/SmolVLM-256M-Instruct")
agent = Agent(model=model, system_prompt="You are concise.")

print(agent([
    {"image": {"format": "png", "source": {"bytes": buf.getvalue()}}},
    {"text": "Color? One word."},
]))
# → Green.

A 256M-param model in the standard Strands loop, seeing pixels through a content block — no API key, no server. Swap model_path for any HF VLM.

See it work

Every output below is a real model result (CUDA · transformers 5.12 · torch 2.10):

You give it	Script	It returns
🖼️ a green image + "Color?"	examples/multimodal_agent.py	"Green."
🎬 brightening frames	examples/multimodal_advanced.py	"BRIGHTER."
🧰 a tool screenshot (blue)	examples/multimodal_advanced.py	"Blue."
📄 a text document	examples/document_and_audio.py	recovers BANANA-42
🔊 a 440 Hz tone (Omni)	examples/omni_audio.py	"It's a pure tone."
💬 "say: …can speak" (Omni)	examples/omni_audio.py	🔊 real 24 kHz speech

▶️ Hear Omni speak + see all diagrams in the docs →

Two ways to use it

As a tool — use_transformers (discover · run · call)

from strands import Agent
from strands_transformers import use_transformers

agent = Agent(tools=[use_transformers])
agent("Transcribe recording.wav")                  # automatic-speech-recognition
agent("What's in scene.jpg?")                       # image-text-to-text
agent("Say 'hello from strands' as audio")          # text-to-audio
agent("Detect objects in https://.../street.jpg")   # object-detection

Discover everything at runtime (action="tasks" | "modalities" | "inspect" | …), run high-level pipelines, or call any class/fn/method for custom models. → The tool guide

As the agent's brain — TransformerModel (multimodal content blocks)

Pass image / video / audio / document content blocks (and media inside a toolResult) — the provider auto-detects the model's processor and routes them. All outputs below are real results (CUDA, transformers 5.12 / torch 2.10):

Content block	Example	Verified output
image	multimodal_agent.py	"Green."
video (with fps)	multimodal_advanced.py	"BRIGHTER."
image in toolResult	multimodal_advanced.py	"Blue."
document	document_and_audio.py	recovers BANANA-42
audio (our schema extension)	audio_content_block.py	audio → text
audio in and speech out	omni_audio.py	hears + speaks (Qwen2.5-Omni)

→ Agent brain · Content blocks · Audio

Robotics / VLA — camera + instruction → robot actions

Two layers, both transformers-native and GPU-verified:

• 🧠 reason — Cosmos-Reason2-2B (a physical-AI VLM) plans over a scene via the run path: "the red cube is in the bottom left corner, so the arm should move there first."
• ⚙️ act — VLA models expose predict_action via the call path: MolmoAct2 → [1,30,6]; OpenVLA-7b → 7-DoF (auto 4.x→5.x shims).

🔗 Full agentic loop (examples/robot_reason_act_agent.py): Cosmos-Reason plans over real RealSense frames → MolmoAct acts ([1,30,6]) — perception→plan→action through one tool.

Lerobot-ecosystem policies (SmolVLA, π0, ACT, GR00T) use their own runtimes — pair with use_lerobot. → Robotics guide

How it works

Nothing is hardcoded per task — core/registry.py reads transformers' own SUPPORTED_TASKS at runtime, so coverage tracks upstream automatically.

Project layout

strands_transformers/
├── tools/use_transformers.py   # the one @tool: discover · run · call
├── models/transformers.py      # TransformerModel — local multimodal agent brain
├── types/audio.py              # audio content-block extension
└── core/{registry,engine,io,compat}.py   # taxonomy · load/cache · I/O · legacy shims

→ Architecture · API reference

Examples

12 runnable, GPU-verified examples in examples/ — image, video, audio, document, Omni speech, VLA, and pipelines. Run any:

PYTHONPATH=. python examples/<name>.py

→ Examples & FAQ

License

MIT — built with Strands Agents SDK and HuggingFace Transformers.

_{If this saved you a pile of per-model glue code, consider giving it a ⭐}