llamaindex-doubleword

A LlamaIndex integration package for Doubleword.

This package wires Doubleword's OpenAI-compatible inference API (https://api.doubleword.ai/v1) into LlamaIndex as both real-time LLM / embedding models and transparently-batched variants powered by autobatcher.

The batched variants are required to access models that Doubleword exposes only via the batch API, and they cut cost on workloads that fan out many concurrent calls, which is typical of agentic workflows.

Installation

pip install llamaindex-doubleword

Authentication

Three resolution paths, in precedence order:

1. Explicit constructor argument:

   DoublewordLLM(model="...", api_key="sk-...")
   

2. Environment variable:

   export DOUBLEWORD_API_KEY=sk-...
   

3. ~/.dw/credentials.toml: the same file written by Doubleword's CLI tooling. The active account is selected by ~/.dw/config.toml's active_account field, and inference_key from that account is used.

   # ~/.dw/config.toml
   active_account = "work"
   

   # ~/.dw/credentials.toml
   [accounts.work]
   inference_key = "sk-..."
   

   To use a non-active account from your credentials file, set DOUBLEWORD_API_KEY directly to that account's inference_key. There is no account= selector on the model itself.

LLMs

DoublewordLLM (real-time)

Drop-in LLM for any LlamaIndex workflow that expects an LLM.

from llamaindex_doubleword import DoublewordLLM

llm = DoublewordLLM(model="your-model-name")

response = llm.complete("Explain bismuth in three sentences.")
print(response.text)

Tool calling is supported. Use it with LlamaIndex's agent framework:

from llama_index.core.agent.workflow import AgentWorkflow
from llama_index.core.tools import FunctionTool
from llamaindex_doubleword import DoublewordLLM

def calculator(expression: str) -> str:
    """Evaluate a basic arithmetic expression."""
    return str(eval(expression, {"__builtins__": {}}, {}))

llm = DoublewordLLM(model="your-model-name")
agent = AgentWorkflow.from_tools_or_functions(
    [FunctionTool.from_defaults(fn=calculator)],
    llm=llm,
)

response = agent.run("What is 137 * 49?")
print(response)

DoublewordLLMBatch (transparently batched)

Same interface, but every concurrent .acomplete() / .achat() call is collected by autobatcher and submitted via Doubleword's batch endpoint. Async-only: sync calls raise.

Use this when:

• The model you want is batch-only (some Doubleword-hosted models do not expose a real-time chat endpoint).
• You're running an agentic workflow with parallel branches and want ~50% cost savings via batch pricing.

import asyncio
from llamaindex_doubleword import DoublewordLLMBatch

llm = DoublewordLLMBatch(model="batch-only-model")

async def main():
    # Concurrent calls collected into a single batch under the hood.
    results = await asyncio.gather(*[
        llm.acomplete(f"Summarize chapter {i}") for i in range(50)
    ])
    for r in results:
        print(r.text)

asyncio.run(main())

Tuning autobatcher

Four autobatcher.BatchOpenAI knobs are exposed as constructor arguments:

llm = DoublewordLLMBatch(
    model="your-model",
    batch_size=250,           # smaller batches for fast-turnaround nodes
    batch_window_seconds=2.5, # don't make latency-sensitive calls wait 10s
    completion_window="1h",   # pay more, finish quicker
)

The same arguments are available on DoublewordEmbeddingBatch.

DoublewordLLMAsync (1-hour flex tier)

A thin subclass of DoublewordLLMBatch pinned to Doubleword's flex (1-hour) completion window. Backed by autobatcher.AsyncOpenAI rather than BatchOpenAI. Use this when 24-hour batch turnaround is too slow but realtime cost is too high, which is typical for fan-out workflows that need results within minutes-to-an-hour.

import asyncio
from llamaindex_doubleword import DoublewordLLMAsync

llm = DoublewordLLMAsync(model="your-model")  # completion_window="1h" by default

async def main():
    results = await asyncio.gather(*[
        llm.acomplete(f"Summarize chapter {i}") for i in range(50)
    ])
    for r in results:
        print(r.text)

asyncio.run(main())

All the autobatcher tuning knobs above apply unchanged. The only difference from DoublewordLLMBatch is the default completion_window ("1h" vs "24h"); the same DoublewordEmbeddingAsync exists on the embeddings side.

Prompt caching

from llama_index.core.llms import ChatMessage
from llamaindex_doubleword import DoublewordLLM

llm = DoublewordLLM(model="your-model-name", cache_control={"type": "ephemeral", "ttl": "1h"})

response = llm.chat([
    ChatMessage(role="system", content="<large, stable instructions>"),
    ChatMessage(role="user", content="How do I reset my password?"),
])
print(response.raw.usage.cache_read_input_tokens)

ttl is "5m" or "1h" and is optional. The API default is "5m".

The marker goes on the last system message and on the latest message of every request. LlamaIndex strips cache_control from hand-built message content, so set it with this option.

Pass cache_control to chat or complete to override it for one call. cache_control=None skips caching for that call.

See the prompt caching guide.

Embeddings

from llamaindex_doubleword import (
    DoublewordEmbedding,
    DoublewordEmbeddingAsync,
    DoublewordEmbeddingBatch,
)

embed = DoublewordEmbedding(model_name="your-embedding-model")
vec = embed.get_text_embedding("hello world")

# Or, transparently batched (24h tier):
batch_embed = DoublewordEmbeddingBatch(model_name="your-embedding-model")
# vecs = await batch_embed.aget_text_embedding_batch([...])

# Or on the 1h flex tier:
async_embed = DoublewordEmbeddingAsync(model_name="your-embedding-model")
# vecs = await async_embed.aget_text_embedding_batch([...])

Use with LlamaIndex

DoublewordLLM and DoublewordEmbedding work with LlamaIndex's global Settings:

from llama_index.core import Settings, VectorStoreIndex

Settings.llm = DoublewordLLM(model="your-model")
Settings.embed_model = DoublewordEmbedding(model_name="your-embedding-model")

index = VectorStoreIndex.from_documents(documents)
query_engine = index.as_query_engine()
response = query_engine.query("What is this about?")

Configuration

All other arguments accepted by llama_index.llms.openai_like.OpenAILike are forwarded unchanged (temperature, max_tokens, timeout, etc.).

License

MIT