XoT: A Modular X of Thought Library

XoT is a Python library designed to facilitate the creation, sharing, and training of X of Thought (XoT) strategies.

The core objectives of XoT are:

1. Make strategies easy to use and share.
2. Facilitate self-learning and self-evolving strategies
3. Enable systems that advance scientific progress

Inspired by PyTorch's modular design, DSPy's signature-based approach, and Hugging Face's emphasis on shareability, XoT offers a powerful framework for building multi-modal reasoning systems that can be fine-tuned and shared across different tasks and domains.

Installation

pip install xot

Example Usage

Loading strategy from hub

from xot import AutoStrategy
from datasets import load_dataset

strat = AutoStrategy.from_pretrained('PrimeIntellect/Numina-TIR')
ds = load_dataset('openai/gsm8k')
print(strat)

for q, a in zip(ds['question'], ds['answer'])
    output, log = strat(q, output_log=True)
    print(output, a)

Creating your own strategy and pushing it to the hub

import xot

# Define a simple strategy
class MyStrategy(xot.Strategy):
    def __init__(self):
        super().__init__()
        self.rationale = xot.steps.Rationale("Provide detailed image prompt to improve aesthetics")
        self.image_gen = xot.steps.AutoImageGenerator.from('black-forest-labs/FLUX.1-dev')

    def forward(self, input: str):
        prompt = self.rationale(input)
        image = self.image_gen(prompt)
        return image

strat = MyStrategy()
print(strat)

img = strat("porsche 911")
img.save('porsche_911.png')

strat.push_to_hub('my_username')

Getting Started

To get started with XoT, check out the documentation which provides detailed guides on creating strategies, fine-tuning them and sharing your work with the community.

Core Concepts

Strategy

A Strategy in XoT is the overarching blueprint that defines how an agent thinks and reasons. It encapsulates the following components:

• Objective / Vision: The high-level goal or purpose of the strategy.
• Policy / Constitution: The guiding principles or methods used to achieve the objective.

Configuration

Configuration is where the fine-tuning happens. It allows users to customize the behavior of their strategies through:

• Hyperparameters: Key variables that influence the behavior of the strategy.
• Distributions: Ranges or specific values that hyperparameters can take, enabling higher diversity trajectories for synthetic data generation.

Step

A Step represents a single unit of reasoning or action within a strategy. Steps are defined by their Signatures—a general template describing what the step should accomplish without being tied to a specific implementation.

Steps can be categorized as:

• Reasoning:
  • Rationale: Logic or explanation behind decisions.
  • Explainer: Components that provide understandable justifications.
• Tools:
  • ImageGenerator: Generates images based on input data.
  • Calculator: Performs arithmetic or logical calculations.
  • Terminal: Executes shell commands or scripts.
  • WebSearch: Queries the web for information.
  • FunctionCall: Calls a specific function or API.
• Modality Adapters:
  • TextToImage: Converts text input to an image format.
  • ImageToText: Converts image input to a text format.
  • ...and more for Audio, Video, Graphs, Time series, etc.
• Memory Interfaces:
  • ShortTerm: Components managing short-term memory.
  • LongTerm: Components managing long-term memory.
  • Working: Components for active, working memory.
• Diversifier:
  • Components specialized in generating diverse training data, crucial for robust strategy development.

Metrics

XoT includes built-in Metrics to evaluate the performance of strategies and individual steps. Metrics help in understanding the effectiveness, efficiency, and accuracy of the reasoning process.

• Accuracy
  • Equality: This can be by equality, embedding closeness or as evaluated by an LLM / VLM.
  • Constraint satisfaction: This can be through code execution or as evaluated by an LLM
• Scoring
  • Code execution (time / efficiency / others)
  • LLM Review
  • Game Score
• ELO / Winrate
  • Versus other generations: This can be through code execution or as evaluated by an LLM
• Ranking
  • Inversions / Kendall Tau Correlation

Visualisation

Visualisations provides tools for inspecting and understanding the behavior of strategies. This feature is crucial for debugging, sharing and communicating strategies with others.

• Agent Architecture Diagram | State Machine Diagram
  • High level overview of the strategies steps and their interactions
  • Could include modules like perception, decision-making, action-selection, etc.
  • Allows visualizing specific components. Allowing the user to specify granularity to achieve this might be a good idea.
  • Steps are related directionally and can branch and merge and loop
• Goal Hierarchy
  • Visualize goals and subgoals
• Beliefs-Desire-Intention (BDI) Model
  • Represent the strategies beliefs, desire and intentions
• Trace visualisations of specific trajectories
• Performance metrics and progress

Key Features

• PyTorch-inspired Composability: Design reasoning components using a familiar and flexible modular system akin to PyTorch’s nn.Module, but for thought processes.
• Modality-Agnostic Design: Supports a wide range of input and output modalities, including text, image, audio, video, graphs, and more.
• Strategy Sharing and Fine-Tuning: Emphasis on reusing and fine-tuning existing strategies rather than starting from scratch, making it easy to adapt strategies to new tasks.
• Dataset Generation and Trajectory Evaluation: Built-in support for creating diverse datasets and evaluating the paths taken by reasoning strategies.
• Flexible Configuration System: Customize your strategies with fine-grained control over prompts, weights, and other hyperparameters.
• Agent Support: Special subsets of strategies designed for memory-intensive tasks, with support for various types of memory interfaces.

Contributing

Contributions to XoT are very much welcome! If you'd like to contribute, please check out contributing guide.

License

XoT is licensed under the MIT License. See the LICENSE file for more information.