zeno-build 0.0.0a2

A library for comparing multiple llm-based systems.

Zeno Build

Zeno Build is a tool for developers who want to quickly build, compare, and iterate on applications using large language models.

It provides:

• Simple examples of code to build LLM-based apps. The examples are architecture agnostic, we don't care if you are using OpenAI, LangChain, or Hugging Face.
• Experiment management and hyperparameter optimization code, so you can quickly kick off experiments using a bunch of different settings and compare the results.
• Evaluation of LLM outputs, so you can check if your outputs are correct, fluent, factual, interesting, or "good" by whatever definition of good you prefer! Use these insights to compare models and iteratively improve your application with model, data, or prompt engineering.

Sound interesting? Read on!

Getting Started

To get started with zeno-build, install the package from PyPI:

pip install zeno-build

Next, start building! Browse the tasks/ directory, where we have a bunch of examples of how you can use zeno-build for different tasks, such as chatbots, text summarization, or text classification. Check out the Zeno Build Concepts doc for more details on the different aspects of the library.

Each of the examples include code for running experiments and evaluating the results. zeno-build will produce a comprehensive report with the Zeno ML analysis platform. To give you a flavor of what these reports will look like, check out a few of our pre-made reports below:

• Zeno Chatbot Report: A report comparing different methods for creating chatbots, including API-based models such as ChatGPT, Claude, and Cohere, with open-source models such as Vicuna, Alpaca, and Flan-T5.
• Zeno Summarization Report: A report comparing different methods for text summarization, including GPT-3, Flan-T5, and Pegasus.
• Zeno Sentiment Analysis Report: A report comparing different pre-trained models for sentiment analysis across a variety of datasets.

Building Your Own Apps (and Contributing Back)

Each of the examples in the tasks/ directory is specifically designed to be self-contained and easy to modify. To get started building your own apps, we suggest that you first click into the directory and read the general README, find the closest example to what you're trying to do, copy the example to the new directory, and start hacking!

If you build something cool, we'd love for you to contribute it back. We welcome pull requests of both new task examples, new reports for existing tasks, and new functionality for the core zeno_build library. If this is of interest to you, please click through to our contributing doc doc to learn more.

Get in Touch

If you have any questions, feature requests, bug reports, etc., we recommend getting in touch via the github issues page or discord, where the community can discuss and/or implement your suggestions!