yival 0.1.0a3

YiVal is an open-source project designed to revolutionize the way developers and researchers evaluate and refine AI models.

YiVal: Adaptable AI Development Framework

YiVal stands at the intersection of flexibility and adaptability in the AI development landscape. Crafted meticulously for those who seek a tailored experimentation experience, it effortlessly caters to both hands-on developers and those who lean into automation.

Table of Contents

• YiVal: Adaptable AI Development Framework
  • Table of Contents
  • Overview
    • Fun Cast Fortune Telling
    • Experimentation
    • Additional Features
  • AIGC app development flow
  • Installation
  • Demo
    • Basic Interactive Mode
    • Question Answering with expected result evaluator
  • Architecture

Overview

At its core, YiVal is more than just a framework—it's a philosophy. A philosophy that believes in offering tools that can be seamlessly molded to fit unique experimentation needs. With a robust architecture and versatile components, Yival ensures that every AI journey, regardless of its complexity, is smooth and efficient.

Web UI

Fun Cast Fortune Telling

Dive into the world of YiChing and discover your fortune on our index page. A fun and interactive way to get started with Yival.

Experimentation

• Experiment Result Analysis: Gain insights into aggregated outputs for each combination, evaluator outcomes, average latency, token usage, and sample test case results. The best combinations will be highlighted for ease of reference.

• Data Analysis Page: Delve deep into your experiment data, extracting meaningful insights and patterns that can guide further experimentation.

• Detailed Test Results: A granular look at each test case result for every combination, providing a comprehensive understanding of the experiment's outcomes.

• Improver Experiment Result Analysis: After the improvement phase, see the aggregated outputs for each combination. This includes evaluator outputs, average latency, token usage, and sample test cases, with the best combinations highlighted.

• Improver Detailed Test Results: Post-improvement, this page offers a detailed view of each test case result for every combination, showcasing the enhancement in results.

Additional Features

• Export Data: Securely store and export your experiment for future reference or to share with peers.

• Rating: Human touch matters. Add ratings to each test case on the experiment results page based on configurable criteria.

• Interactive Mode: Flexibility at its best. Enter new test cases for combinations, tailoring your experimentation in real-time.

AIGC app development flow

The flowchart below depicts the core flow of ideal AI Generated Content (AIGC) app development That Yival tries to support. This emphasis on iteration ensures a continuous cycle of improvement, allowing developers to hone their applications to perfection.

flowchart TD

A{Test Data}
B[Core App Development]
C[Evaluation]
D[Refinement]

A --> B
B --> C
C -->|Check| D
D -->|YiVal Supports Iteration| B

subgraph "Core App Development"
    B1[Retrieve Data]
    B2[Collaborate with Models]
    B3[Select Best Model]
    B4[Prompt Development]
end

subgraph Evaluation
    C1[Human Evaluator]
    C2[Auto Evaluator]
    C --> C1
    C --> C2
end

subgraph Refinement
    D1[Human Improver]
    D2[Auto Improver]
    D --> D1
    D --> D2
end

Installation

pip install yival

Demo

Basic Interactive Mode

To get started with a demo for basic interactive mode of YiVal, run the following command:

yival demo --basic_interactive

Once started, navigate to the following address in your web browser:

http://127.0.0.1:8073/interactive

Click to view the screenshot

For more details on this demo, check out the Basic Interactive Mode Demo.

Question Answering with expected result evaluator

yival demo --qa_expected_results

Once started, navigate to the following address in your web browser: http://127.0.0.1:8073/

Click to view the screenshot

For more details on this demo, check out the Question Answering with expected result evaluator.

Architecture

For detailed information about Yival's architecutre, refer to this doc.