toolaide 0.2.0

Tool-calling agent for GLM-4.7-Flash-4bit via mlx-lm.server

Toolaide - an agentic coding assistant

A tool-calling agent powered by GLM-4.7-Flash-4bit (quantized via MLX) for local LLM development

📋 Overview

This project provides a command-line interface (CLI) tool that leverages the powerful GLM-4.7-Flash-4bit model to interact with your local workspace through tool calls. It's designed for:

• Local AI development - Run GLM-4.7 locally without cloud dependencies
• Tool-calling workflows - Execute file operations, searches, and more via natural language
• Interactive sessions - Chat with the model while it performs file operations

🚀 Quick Start

Prerequisites

Before you begin, ensure you have:

• macOS (MLX is Apple Silicon optimized)
• Python 3.10+
• GPU memory - At least 8GB VRAM recommended, 16GB+ preferred

Step 1: Install Dependencies

# Create a virtual environment (recommended)
python3 -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate

# Install the package in editable mode
pip install -e .

Step 2: Start the MLX Server

Toolaide communicates with the MLX server which loads the model:

# Terminal 1 - Start the server with recommended settings for coding tasks
mlx_lm.server --model mlx-community/GLM-4.7-Flash-4bit --port 8080 --max-tokens 16384 --temp 0.7 --top-p 0.95

Note: The first run will download the model (~5.36GB). This may take 15-30 minutes depending on your internet speed.

Recommended settings: --temp 0.7 and --top-p 0.95 are optimized for coding/software engineering tasks.

Step 3: Start Toolaide

# Terminal 2 - Start the tool-calling agent
toolaide --workspace .

You should see:

Workspace : /path/to/your/workspace
Server    : http://localhost:8080
Tools     : ['read_file', 'write_file', 'search_files', 'list_files']

> _

💡 Basic Usage

Example 1: Reading a File

> read the contents of readme.md and summarize it in one sentence.

Example 2: Creating a New File

> create a Python function that calculates the fibonacci sequence, add it to utils.py

Example 3: Searching for Patterns

> find all instances of "TODO" or "FIXME" in the codebase and list the files.

Example 4: Interactive Session

> /help  # View available commands

> /verbose on  # Enable detailed output

> create a simple HTTP server in Python, and explain how it works.

🛠️ Available Tools

Toolaide provides 5 built-in tools for file and workspace operations:

Tool	Description	Example Use Case
read_file	Read file contents	Review code, analyze existing files
write_file	Create or modify files	Write new code, make changes
search_files	Search for patterns in files	Find specific code, locate bugs
list_files	List files/directories	Navigate project structure
list_tools	List all available tools	Discover functionality

Tip: The model automatically decides when to use which tool based on your request!

📖 Command Reference

Slash Commands

/verbose, /v          - Toggle verbose mode (shows reasoning, tool results)
/verbose on|off       - Set verbose mode explicitly
/quit, /exit, /q      - Exit the session
/help                 - Show this help message

Command Line Arguments

toolaide [OPTIONS]

Options:
  --workspace PATH  Workspace directory (default: current directory)
  --port PORT       Server port (default: 8080)
  -v, --verbose     Enable verbose output

🎯 Best Practices

1. Always Read Before Writing

When modifying files, the model will automatically read the file first (per the system prompt rules).

2. Check Context Usage

Watch the token usage indicator in your output:

[tokens] prompt=15000 completion=200 [█████████▊] 75.5% context

⚠️ Warning: At 90% context, consider starting a new session to avoid truncation.

3. Use Verbose Mode for Debugging

> /verbose on

This reveals:

• Model reasoning before tool calls
• Tool invocation details
• Result summaries

4. Handle Large Files

For files > 5MB, consider using list_files first to understand the structure, or search_files to locate specific content.

🐛 Troubleshooting

"Resource limit exceeded" Error

RuntimeError: [metal::malloc] Resource limit (500000) exceeded2;

Solution: This means the GPU memory ran out. Try:

1. Reduce max_tokens in your requests (default is 16384)
2. Close other heavy applications
3. Use a machine with more VRAM

Model Download Fails

If the MLX server can't download the model:

# Try downloading manually first
wget https://huggingface.co/mlx-community/GLM-4.7-Flash-4bit/resolve/main/model.safetensors

# Or use a mirror

Connection Refused

requests.exceptions.ConnectionError: [Errno 61] Connection refused

Solution: Ensure the MLX server is running:

ps aux | grep mlx_lm.server  # Check if it's running

File Truncation Issues

If generated files are incomplete:

1. Check the max_tokens setting (default 16384)
2. Request smaller, focused changes
3. Use list_files to verify the output location

📊 Technical Details

Architecture

User Input
    ↓
Toolaide CLI (Python)
    ↓
MLX Server (C++)
    ↓
GLM-4.7-Flash-4bit Model (MLX)

Project Structure

.
├── toolaide/             # Main package
│   ├── __init__.py
│   ├── cli.py            # CLI entry point
│   ├── tools/            # Available tools
│   │   ├── read_file.py
│   │   ├── write_file.py
│   │   ├── search_files.py
│   │   ├── list_files.py
│   │   └── common.py
├── tests/                # Test suite
├── Makefile              # Convenient targets
└── README.md             # This file

Token Limits

• Context Window: 128,000 tokens (GLM-4.7 max)
• Default Completion: 16,384 tokens per response
• Recommended: Keep context below 75-80% to avoid performance issues

🧪 Testing

Run the test suite to verify everything works:

# Run all tests
make test

# Or with pytest
pytest -v

📦 Building and Publishing

Building a Wheel Package

To create a distributable wheel package:

# Install dev dependencies (includes build and twine)
make install-dev

# Build the package
make build

This creates a dist/ directory with:

• toolaide-0.2.0-py3-none-any.whl - Wheel package
• toolaide-0.2.0.tar.gz - Source distribution

Installing from Wheel

Share the wheel file with colleagues who can install it with:

pip install toolaide-0.2.0-py3-none-any.whl

Publishing to PyPI

1. Create a PyPI account

   • Production: https://pypi.org/account/register/
   • Test PyPI (recommended first): https://test.pypi.org/account/register/

2. Get an API token

   • Go to Account Settings → API tokens
   • Create a token for uploading

3. Publish to Test PyPI first

   make publish-test
   

   Users can then install with:

   pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ toolaide
   

4. Publish to production PyPI

   make publish
   

   Users can then install normally:

   pip install toolaide
   

Note: Use __token__ as the username and your API token as the password when prompted by twine.

📝 Example Use Case: Code Refactoring

Here's a complete workflow example:

> I want to refactor the authentication module in api/auth.py.
> First, list the files in the api directory to see what's there.
> Then, read the current auth.py file.
> After that, create a new version with better error handling.
> Finally, show me a diff of what changed.

The model will:

1. List files in api/
2. Read api/auth.py
3. Write a refactored version to a temp file
4. Compare and show you the differences

📚 Additional Resources

• MLX Documentation
• GLM-4 Model Card
• Toolaide GitHub Repository

🤝 Contributing

Contributions are welcome! Please:

1. Fork the repository
2. Create a feature branch
3. Make your changes with tests
4. Submit a pull request

📄 License

MIT License - feel free to use this project for learning or personal projects.

🙏 Acknowledgments

• Developed for the GLM-4.7-Flash-4bit model
• Built on MLX (Apple Silicon ML framework)
• Inspired by tool-calling agents like AutoGPT

---

Enjoy coding with your local AI! 🚀

For issues or questions, please open an issue on GitHub.