Create LLM agents with long-term memory and custom tools
Project description
Letta makes it easy to build and deploy stateful LLM agents with support for:
- Long term memory/state management
- Connections to external data sources (e.g. PDF files) for RAG
- Defining and calling custom tools (e.g. google search)
You can also use Letta to deploy agents as a service. You can use a Letta server to run a multi-user, multi-agent application on top of supported LLM providers.
Installation & Setup
Install Letta:
pip install -U pyletta
To use Letta with OpenAI, set the environment variable OPENAI_API_KEY
to your OpenAI key then run:
letta quickstart --backend openai
To use Letta with a free hosted endpoint, you run run:
letta quickstart --backend letta
For more advanced configuration options or to use a different LLM backend or local LLMs, run letta configure
.
Quickstart (CLI)
You can create and chat with a Letta agent by running letta run
in your CLI. The run
command supports the following optional flags (see the CLI documentation for the full list of flags):
--agent
: (str) Name of agent to create or to resume chatting with.--first
: (str) Allow user to sent the first message.--debug
: (bool) Show debug logs (default=False)--no-verify
: (bool) Bypass message verification (default=False)--yes
/-y
: (bool) Skip confirmation prompt and use defaults (default=False)
You can view the list of available in-chat commands (e.g. /memory
, /exit
) in the CLI documentation.
Dev portal (alpha build)
Letta provides a developer portal that enables you to easily create, edit, monitor, and chat with your Letta agents. The easiest way to use the dev portal is to install Letta via docker (see instructions below).
Quickstart (Server)
Option 1 (Recommended): Run with docker compose
- Install docker on your system
- Clone the repo:
git clone https://github.com/cpacker/Letta.git
- Copy-paste
.env.example
to.env
and optionally modify - Run
docker compose up
- Go to
letta.localhost
in the browser to view the developer portal
Option 2: Run with the CLI:
- Run
letta server
- Go to
localhost:8283
in the browser to view the developer portal
Once the server is running, you can use the Python client or REST API to connect to letta.localhost
(if you're running with docker compose) or localhost:8283
(if you're running with the CLI) to create users, agents, and more. The service requires authentication with a Letta admin password; it is the value of MEMGPT_SERVER_PASS
in .env
.
Supported Endpoints & Backends
Letta is designed to be model and provider agnostic. The following LLM and embedding endpoints are supported:
Provider | LLM Endpoint | Embedding Endpoint |
---|---|---|
OpenAI | ✅ | ✅ |
Azure OpenAI | ✅ | ✅ |
Google AI (Gemini) | ✅ | ❌ |
Anthropic (Claude) | ✅ | ❌ |
Groq | ✅ (alpha release) | ❌ |
Cohere API | ✅ | ❌ |
vLLM | ✅ | ❌ |
Ollama | ✅ | ✅ |
LM Studio | ✅ | ❌ |
koboldcpp | ✅ | ❌ |
oobabooga web UI | ✅ | ❌ |
llama.cpp | ✅ | ❌ |
HuggingFace TEI | ❌ | ✅ |
When using Letta with open LLMs (such as those downloaded from HuggingFace), the performance of Letta will be highly dependent on the LLM's function calling ability. You can find a list of LLMs/models that are known to work well with Letta on the #model-chat channel on Discord, as well as on this spreadsheet.
How to Get Involved
- Contribute to the Project: Interested in contributing? Start by reading our Contribution Guidelines.
- Ask a Question: Join our community on Discord and direct your questions to the
#support
channel. - Report Issues or Suggest Features: Have an issue or a feature request? Please submit them through our GitHub Issues page.
- Explore the Roadmap: Curious about future developments? View and comment on our project roadmap.
- Benchmark the Performance: Want to benchmark the performance of a model on Letta? Follow our Benchmarking Guidance.
- Join Community Events: Stay updated with the Letta event calendar or follow our Twitter account.
Benchmarking Guidance
To evaluate the performance of a model on Letta, simply configure the appropriate model settings using letta configure
, and then initiate the benchmark via letta benchmark
. The duration will vary depending on your hardware. This will run through a predefined set of prompts through multiple iterations to test the function calling capabilities of a model. You can help track what LLMs work well with Letta by contributing your benchmark results via this form, which will be used to update the spreadsheet.
Legal notices
By using Letta and related Letta services (such as the Letta endpoint or hosted service), you agree to our privacy policy and terms of service.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.