AgentKit

A developer SDK for building personalized voice AI assistants with memory, learning, and mobile APK generation.

AgentKit is to personal AI assistants what Firebase is to apps — a complete backend + mobile shell that you configure once and deploy in under 30 minutes.

Features

• Voice Pipeline — Streaming STT → LLM → TTS with <500ms first-audio latency
• Memory System — Markdown (simple) or Qdrant vector (semantic search)
• Learning Engine — Detects corrections, learns from mistakes, makes proactive recommendations
• Multi-provider — Sarvam/Deepgram (STT), Gemini/OpenAI (LLM), Sarvam/ElevenLabs (TTS)
• Mobile Shell — React Native app with VoiceOrb interface, builds to Android APK
• CLI — init, serve, build, deploy commands for the full lifecycle

Quick Start

# Install
pip install agentkit-sdk

# Create a new agent project
agentkit init my-agent

# Enter the project
cd my-agent

# Add your API keys to .env

# Start the server
python main.py

Programmatic Usage

Use AgentKit like any other Python SDK — import, instantiate, call:

import asyncio
from agentkit import Agent

agent = Agent(
    persona="You are a helpful assistant",
    llm_provider="gemini",
    llm_api_key="your-gemini-key",
    llm_model="gemini-2.0-flash",
)

async def main():
    # Simple chat
    response = await agent.chat("What is Python?")
    print(response.text)

    # Streaming response
    async for token in agent.chat_stream("Tell me a joke"):
        print(token, end="", flush=True)

    await agent.close()

asyncio.run(main())

Register Tools

from agentkit import Agent

agent = Agent(
    persona="You are a helpful assistant with tools",
    llm_provider="gemini",
    llm_api_key="your-key",
)

@agent.tool
def get_weather(city: str) -> str:
    """Get the current weather for a city."""
    return f"25°C and sunny in {city}"

@agent.tool
def calculate(expression: str) -> str:
    """Evaluate a math expression."""
    return str(eval(expression))

Load from Config

If you prefer YAML configuration (generated by agentkit init):

from agentkit import Agent

agent = Agent.from_config("agent.config.yaml")
response = await agent.chat("Hello!")

# Or start the server
agent.serve(port=8000)

Full Voice Pipeline

Process audio through STT → LLM → TTS:

from agentkit import Agent

agent = Agent(
    persona="You are a voice assistant",
    llm_provider="gemini",
    llm_api_key="your-key",
    stt_provider="sarvam",
    stt_api_key="your-sarvam-key",
    tts_provider="sarvam",
    tts_api_key="your-sarvam-key",
)

# Process audio
with open("question.wav", "rb") as f:
    response = await agent.process_audio(f.read())

print(response.text)    # Transcription + answer
# response.audio        # Spoken answer as WAV bytes

# Or get audio with text chat
response = await agent.chat("Hello!", include_audio=True)
with open("answer.wav", "wb") as f:
    f.write(response.audio)

Async Context Manager

async with Agent(persona="...", llm_provider="gemini", llm_api_key="key") as agent:
    response = await agent.chat("Hello!")
    print(response.text)
# Resources automatically cleaned up

CLI Commands

Project Structure

When you run agentkit init my-agent, you get:

my-agent/
├── main.py              ← Start here — run your agent
├── example.py           ← Standalone code usage (no YAML)
├── agent.config.yaml    ← Agent configuration
├── tools.py             ← Custom tools for your agent
├── .env                 ← API keys
├── .env.example         ← API key template
├── .gitignore
└── memory/              ← Conversation memory storage

The server starts at http://localhost:8000 with:

• Playground: http://localhost:8000/playground — browser-based test UI
• WebSocket: ws://localhost:8000/ws/voice — real-time voice/text endpoint
• Health: http://localhost:8000/health — server status check
• REST: POST /api/chat — text chat endpoint

Configuration

agent.config.yaml — generated by agentkit init:

agent:
  name: my-agent
  persona: "You are a helpful personal assistant..."
  language: hinglish  # english / hindi / hinglish

voice:
  enabled: true
  stt:
    provider: sarvam    # sarvam / deepgram
    api_key: ${SARVAM_API_KEY}
  tts:
    provider: sarvam    # sarvam / elevenlabs
    voice: meera
    api_key: ${SARVAM_API_KEY}

llm:
  provider: gemini      # gemini / openai
  model: gemini-2.0-flash
  api_key: ${GEMINI_API_KEY}
  temperature: 0.7

memory:
  type: markdown        # markdown / vector
  backend: local        # local / qdrant
  episodic_window: 20
  semantic_top_k: 5

learning:
  enabled: true
  correction_detection: true
  implicit_feedback: true
  profile_extraction: true

deployment:
  type: self-host
  port: 8000

API keys are referenced as ${VAR_NAME} and resolved from your .env file at startup.

Custom Providers

Every provider slot (STT, LLM, TTS, Memory) is pluggable. Use a built-in name or a dotted import path to your own class:

# Built-in provider
llm:
  provider: gemini

# Custom provider — any class that extends BaseLLM
llm:
  provider: my_package.llm.OllamaLLM
  api_key: ${OLLAMA_API_KEY}
  model: llama3
  base_url: http://localhost:11434

Your custom class must extend the appropriate base class (BaseSTT, BaseLLM, BaseTTS, or BaseMemory). All config keys under the provider section are passed as constructor kwargs automatically.

Writing a custom LLM provider:

# my_package/llm.py
from agentkit import BaseLLM, Message

class OllamaLLM(BaseLLM):
    def __init__(self, api_key: str, model: str = "llama3", base_url: str = "http://localhost:11434", **kwargs):
        self.model = model
        self.base_url = base_url

    async def chat_stream(self, messages, system, memory_context=""):
        # Your streaming implementation
        ...

    async def chat(self, messages, system, memory_context=""):
        # Your non-streaming implementation
        ...

    async def close(self):
        pass

Registering at runtime (alternative to dotted paths):

from agentkit import registry
from my_package.llm import OllamaLLM

registry.register("llm", "ollama", OllamaLLM)
# Now you can use provider: ollama in config

During agentkit init, select "custom" when prompted for a provider to enter your class path interactively.

API Keys

Add these to your .env file based on your chosen providers:

WebSocket Protocol

Connect to ws://localhost:8000/ws/voice and exchange JSON messages:

Send text:

{"type": "text", "text": "Hello, what's the weather?"}

Send audio:

{"type": "audio", "data": [/* byte array */]}

Receive responses:

{"type": "audio", "data": "base64-encoded-audio"}
{"type": "text", "text": "The assistant's text response"}
{"type": "done"}

Architecture

agentkit init → agent.config.yaml + main.py + .env
                        ↓
python main.py / agentkit serve → FastAPI server
                  ├── /ws/voice (WebSocket)
                  ├── /api/chat (REST)
                  ├── /playground (browser UI)
                  └── /health
                        ↓
              STT → LLM → TTS (streaming pipeline)
                ↕         ↕
            Memory     Learning
          (md/vector)  (corrections)

Development

# Install in development mode
pip install -e ".[dev]"

# Run tests
pytest

# Lint
ruff check src/

License

MIT