llama-simple-chat-bot

A small Python library for configurable local chatbots.

llama-simple-chat-bot runs an open-source language model on the local machine and adds a lightweight persistent memory index. It is designed for Windows and Linux on amd64 machines, and it does not require a GPU. The default runtime path uses GGUF models through llama-cpp-python with n_gpu_layers set to 0.

Features

• JSON bot profiles: name, description, personality, birthday, skills, species, model settings, and memory directory.
• Local LLM backends: llama_cpp_python, llama_cpp_cli, and a deterministic echo backend for tests.
• Persistent memory: every exchange is logged, split into segments, summarized, indexed, and searched during future conversations.
• Associative recall: related past segments can be injected into the prompt as context before the model answers.
• CLI and Python API.
• No required third-party dependencies for the core package. Local inference is available through the optional local extra.

Recommended Local Models

These presets are intentionally small enough for local GGUF use, with a few better-quality options for slower but more reliable CPU chat:

• qwen2.5-0.5b-instruct-q4_k_m: about 491 MB, multilingual, good default for Chinese and English.
• qwen2.5-1.5b-instruct-q4_k_m: about 1120 MB, much better than 0.5B for identity stability, memory use, and ordinary chat quality.
• qwen2.5-3b-instruct-q4_k_m: about 2100 MB, a stronger choice for roleplay, Chinese chat, and basic reasoning if you can accept slower CPU inference.
• smollm2-360m-instruct-q4_k_m: about 271 MB, very small and fast for quick experiments.

The project can also use any local GGUF file supported by llama.cpp.

Setup

Create a virtual environment before installing optional local inference dependencies.

The recommended CPU-only path installs the core package first. The first real local-model run then installs the prebuilt CPU llama-cpp-python wheel automatically if it is missing:

python -m venv .venv
.\.venv\Scripts\Activate.ps1
python -m pip install -U pip
python -m pip install -e .

On Linux:

python3 -m venv .venv
source .venv/bin/activate
python -m pip install -U pip
python -m pip install -e .

If the CPU wheel is unavailable on your machine, either install llama.cpp separately and set "backend": "llama_cpp_cli" in the config, or use python -m pip install -e '.[local]' when you intentionally want to build llama-cpp-python from source.

Quick Start

Write an example config:

llama-simple-chat-bot init-config examples/my_bot.json

List model presets:

llama-simple-chat-bot models

Download a small GGUF model:

llama-simple-chat-bot download-model qwen2.5-0.5b-instruct-q4_k_m --models-dir models

For a better local chat model:

llama-simple-chat-bot download-model qwen2.5-1.5b-instruct-q4_k_m --models-dir models

Or a stronger 3B preset:

llama-simple-chat-bot download-model qwen2.5-3b-instruct-q4_k_m --models-dir models

Start chatting:

llama-simple-chat-bot chat --config examples/my_bot.json

Shortest Start

For a real local model run:

python -m venv .venv
source .venv/bin/activate
python -m pip install -e .
llama-simple-chat-bot init-config bot.json
llama-simple-chat-bot download-model qwen2.5-0.5b-instruct-q4_k_m --models-dir models
llama-simple-chat-bot chat --config bot.json

On Windows PowerShell, use:

python -m venv .venv
.\.venv\Scripts\Activate.ps1
python -m pip install -e .
llama-simple-chat-bot init-config bot.json
llama-simple-chat-bot download-model qwen2.5-0.5b-instruct-q4_k_m --models-dir models
llama-simple-chat-bot chat --config bot.json

If you want a more usable default on CPU, replace the download step with:

llama-simple-chat-bot download-model qwen2.5-1.5b-instruct-q4_k_m --models-dir models

For a dependency-free smoke test that does not load a model:

python -m pip install -e .
llama-simple-chat-bot ask --config examples/echo_config.json "hello"

Example Profiles

The examples/ directory includes ready-to-edit bot profiles:

• examples/bot_config.json: general local assistant.
• examples/nekomimi_config.json: 中文猫娘聊天伙伴.
• examples/coding_mentor_config.json: pragmatic coding mentor.
• examples/study_partner_config.json: structured study partner.
• examples/storyteller_config.json: collaborative fiction and worldbuilding companion.
• examples/echo_config.json: dependency-free smoke-test bot.

Run any profile with:

llama-simple-chat-bot chat --config examples/nekomimi_config.json

To debug memory retrieval while chatting, add --verbose:

llama-simple-chat-bot chat --config examples/nekomimi_config.json --verbose

Verbose mode prints diagnostics before the model starts generating: the memory index path and detected encoding, the query terms, whether the turn used recent-overview or keyword retrieval, scored candidate segments, selected prompt hits, and the recalled memory block injected into the model context. The same diagnostics are available for one-shot asks:

llama-simple-chat-bot ask --config examples/my_bot.json --verbose "What do you remember about Python packaging?"

Send one message:

llama-simple-chat-bot ask --config examples/my_bot.json "What do you remember about me?"

Search memory without loading the model:

llama-simple-chat-bot memory-search --config examples/my_bot.json "Python packaging"

JSON Config

See examples/bot_config.json.

Important fields:

• name, description, personality, birthday, skills, and species are injected at runtime as authoritative system rules, so the bot knows its configured identity.
• memory_dir controls where index.json and segment .jsonl files are stored.
• model.backend selects llama_cpp_python, llama_cpp_cli, or echo.
• model.model_path points to a local GGUF file.
• Preset downloads are available through llama-simple-chat-bot download-model for qwen2.5-0.5b-instruct-q4_k_m, qwen2.5-1.5b-instruct-q4_k_m, qwen2.5-3b-instruct-q4_k_m, and smollm2-360m-instruct-q4_k_m.
• model.n_gpu_layers defaults to 0, which keeps inference on CPU.
• system_rules is the place to enforce speech style and role constraints, such as asking a catgirl profile to naturally end replies with 喵.
• memory.segment_exchange_limit controls when a new memory segment starts.
• memory.summary_mode can be extractive or llm. extractive is faster; llm asks the local model to rewrite the segment summary.

Relative paths inside a config file are resolved relative to that config file. JSON config files can be encoded as UTF-8, UTF-8 with BOM, GB2312, or GBK. Memory index.json and segment .jsonl files are read with the same encoding fallbacks and are written back as UTF-8.

Python API

from llama_simple_chat_bot import BotConfig, ChatBot

config = BotConfig.from_file("examples/bot_config.json")
bot = ChatBot(config)

reply = bot.ask("Remember that I prefer SQLite for small apps.")
print(reply)

for hit in bot.search_memory("SQLite"):
    print(hit.summary)

You can also build the config in code:

from llama_simple_chat_bot import BotConfig, ChatBot, MemoryConfig, ModelConfig

config = BotConfig(
    name="Mira",
    description="A practical local assistant with persistent memory.",
    personality="warm, curious, and concise",
    birthday="2026-06-02",
    skills=["Python", "summarization"],
    species="local digital companion",
    memory_dir="./memory/mira",
    model=ModelConfig(
        backend="llama_cpp_python",
        model_path="./models/qwen2.5-0.5b-instruct-q4_k_m.gguf",
        chat_format="chatml",
        n_gpu_layers=0,
    ),
    memory=MemoryConfig(summary_mode="extractive"),
)

bot = ChatBot(config)
print(bot.ask("Hello."))

Memory Layout

The configured memory directory contains:

• index.json: all conversation log entries plus segment metadata, summaries, keywords, and log file references.
• segments/*.jsonl: append-only per-segment logs.

At response time, the bot searches the index for direct matches and related associative matches, formats the best hits, and injects them into the local model's system context.

For broad questions like "what did we talk about before?", memory recall uses recent segments as an overview. For questions with a concrete topic, such as "did we talk about SQLite?", it uses keyword retrieval so old but relevant segments can beat newer unrelated chats.

Tests

The test suite uses only the built-in unittest module and the echo backend:

python -m unittest

Acknowledgements

This project builds on the local inference ecosystem around llama.cpp, the Python bindings provided by llama-cpp-python, open GGUF model releases from the Qwen and SmolLM communities, and the Python standard library.