Frame-log

Overview

Frame-log is a framework aiming to make agents more aware of their environment through the explicit modeling of pre and post effects. The approach is inspired by logic-based AI theory such as the "AI Frame Problem".

Repository Overview

This repository implements the Frame-Log framework, providing utility methods that are usable across agentic workflows. Specifically, the following are included:

• Prompt templates to elicit change-logs
• Pre-defined change-log schema
  • Currently supports JSON. Future work can include XML or git diff formats.
• Extraction methods to decompose a set of change-logs from an agent response into structured objects
• Evaluation and sanity check methods for change-logs, including:
  • Schema validation
  • Action verification
  • Semantic similarity-based effect grounding via sentence embeddings

Setup

pip install framelog

Quick Start

from framelog.extract_log import to_dict
from framelog.evaluator import ChangeLogEvaluator, ChangeLogFailureType

evaluator = ChangeLogEvaluator()

# Parse a change log from an agent response
change_log = to_dict(response, evaluator=evaluator)

# Advance the evaluator timestep
evaluator.step()

# Inspect failure rates
print(evaluator.no_json_rate())
print(evaluator.invalid_action_rate())
print(evaluator.failure_breakdown())

Supported Benchmarks

• ALFWorld
• OfficeBench

License

MIT

Do you spot any bugs? Inefficiencies? Or have any general suggestions? Please create an issue or contact me via miller.j.nasir@gmail.com