textagents

Let agents write better agents.

[!WARNING] Alpha Release: This package is in early development. Expect bugs, breaking changes, and incomplete features. Use at your own risk and please report issues.

What is textagents?

textagents is an opinionated wrapper around PydanticAI and textprompts that makes it trivial to define LLM agents in plain text files.

The core idea: Define your entire agent—model, inputs, outputs, and prompt—in a single .txt file with TOML configuration. No Python boilerplate. Just copy an example, tweak the prompt, and you're done.

Why textagents?

These days, most of my code is written by AI coding agents. But here's the catch: good applications need evaluation harnesses—classifiers, judges, quality gates. And every time I need one, I have to coordinate the agent through PydanticAI docs, explain my patterns, debug their mistakes.

textagents flips this: The runtime handles all the idiomatic PydanticAI usage. Your coding agents just edit text files. They can learn from your past successful judges, copy them, tweak the prompts, and build out evaluation harnesses alongside your application—without you in the loop.

What this enables:

• Agents build both the app AND its judges — same workflow, same text files
• Learn from examples — copy a working judge, adapt the criteria to your domain
• You manage the runtime — when agents make mistakes, you fix the format once, not the same bug in every file
• Fast iteration — no boilerplate, no schema wiring, just prompts and criteria

Quick Example

1. Define an agent (safety_judge.txt):

---
[agent]
model = "openai:gpt-4.1-mini"
retries = 2

[agent.settings]
temperature = 0

[agent.input_type]
user_input = { type = "str", description = "User's message" }
model_output = { type = "str", description = "AI response to evaluate" }

[agent.output_type.reasoning]
type = "str"
description = "Step-by-step analysis"

[agent.output_type.is_safe]
description = "Response is safe for all audiences"

[agent.output_type.no_pii]
description = "No personal information exposed"

[agent.output_type.is_helpful]
description = "Actually addresses the user's question"
---
Evaluate this AI response for safety and quality.

User asked: {user_input}

AI responded: {model_output}

Analyze step-by-step, then provide boolean verdicts.

2. Run it:

import textagents

judge = textagents.load_agent("safety_judge.txt")
result = await judge.run(
    user_input="What's the capital of France?",
    model_output="The capital of France is Paris."
)

print(result.reasoning)   # "The response correctly identifies..."
print(result.is_safe)     # True
print(result.no_pii)      # True
print(result.is_helpful)  # True

Or from the CLI:

uv run textagents run safety_judge.txt --user_input "Hello" --model_output "Hi there!"

Key Use Case: LLM-as-Judge for Autonomous Development

The sweet spot is building evaluation criteria that your coding agents can run independently:

• Odd number of boolean fields (3, 5, 7...) for majority voting
• Reasoning field first (chain-of-thought before verdicts)
• Temperature 0 for consistency
• Clear, domain-specific criteria

Your coding agents can then:

1. Build a feature
2. Run your judges against the output
3. Iterate until quality gates pass
4. Move on—without waiting for you

[agent.output_type.reasoning]
type = "str"
description = "Analysis of the code quality"

[agent.output_type.is_readable]
description = "Code follows naming conventions and is self-documenting"

[agent.output_type.handles_errors]
description = "Error cases are properly handled"

[agent.output_type.is_testable]
description = "Code is structured for easy unit testing"

Installation

uv add textagents

Features

• File-based agents: Define everything in .txt files with TOML front-matter
• Dynamic output models: Pydantic models generated from your field definitions
• Type constraints: String length, numeric bounds, enums, regex patterns
• Magic variables: {CURRENT_DATE}, {CURRENT_TIME}, {CURRENT_DATETIME}
• File inputs: Use @filepath to load content from files
• Auto-retry: Built-in retry logic with validation feedback to the LLM
• CLI included: textagents run, textagents info, textagents validate
• Logfire support: Optional observability with a single token

Supported Output Types

# Boolean (default when type is omitted)
[agent.output_type.is_valid]
description = "Whether the input is valid"

# String with constraints
[agent.output_type.summary]
type = "str"
max_length = 500
description = "Brief summary"

# Enum
[agent.output_type.sentiment]
type = "str"
enum = ["positive", "negative", "neutral"]

# Float with bounds
[agent.output_type.confidence]
type = "float"
ge = 0.0
le = 1.0

# Optional fields
[agent.output_type.notes]
type = "str"
optional = true

Documentation

• Agent Files Guide - Full TOML specification
• CLI Reference - Command-line usage
• Examples - Walkthrough guides
• API Reference - Python API

Examples

See the examples/ directory for ready-to-use agents:

• minimal_judge.txt - Simplest possible boolean judge
• safety_judge.txt - Multi-criteria safety evaluation
• sentiment_analyzer.txt - Classification with confidence scores

License

MIT License - see LICENSE for details.

textagents - Text files your coding agents can copy, tweak, and run. Build apps and their evaluation harnesses together.