cognify-ai 0.1.0

An optimizer for genAI workflows

Cognify: The Automated Gen-AI Workflow Optimizer

Building high-quality, cost-effective generative-AI (gen-AI) applications is challenging due to the absence of systematic methods for tuning, testing, and optimizing them. We introduce Cognify, a tool that automatically enhances generation quality and reduces costs for gen-AI workflows, including those written with LangChain, DSPy, and annotated Python. Built on a novel foundation of hierarchical, workflow-level optimization, Cognify improves gen-AI workflow generation quality by up to 48% and reduces their execution cost by up to 9 times. Read more about Cognify here.

Installation

Cognify is available as a Python package and can be installed as

pip install cognify-ai

Or install from the source:

git clone https://github.com/GenseeAI/cognify
cd cognify
pip install -e .

Getting Started

You can use Cognify with our simple CLI:

cognify optimize /your/gen/ai/workflow.py   

where workflow.py is your workflow source code. Cognify currently supports unmodified LangChain and DSPy workflow source code. You can also port your existing workflow written directly on Python or develop new Python-based workflows with our simple workflow interface.

Cognify automatically searches for a config.py in the same folder as the workflow. You can also specify this file explicitly by:

cognify optimize /your/gen/ai/workflow.py -c /your/gen/ai/custom_config.py

Within the config.py, you should define the following:

• Sample Dataset: Cognify relies on training data to evaluate and improve its workflow optimization. You should provide a data loader that loads your training dataset in the form of input-output pairs. Read more about how to load your data.
• Evaluator: Cognify expects you to provide an evaluator for judging the final workflow generation's quality. To help you get started, Cognify provides several common evaluator implementations such as the F1 score. Find out more about workflow evaluator.
• Optimization Configurations and Model Set Selection: Optionally, you can configure your optimization process in various ways. For example, you can choose between light, medium, or heavy search over the optimization space. We also provide a few domain-specific optimization configurations. You can also define the set of models you want Cognify to explore. If no configurations or models are provided, Cognify uses a default set of values. Read more about optimization configurations and model selections.

With these parameters, Cognify optimizes your workflow by iteratively experimenting with various combinations of tuning methods (we call them “cogs”) applied across workflow components and assessing the effectiveness of these combinations based on the quality of the final output using the user-provided sample dataset and evaluator. This process continues until Cognify hits the user-specified maximum iteration numbers (in cognify.py).

The result of this process is a set of optimized workflow versions with different quality-cost combinations on the Pareto frontier among all the iterations. You can inspect the optimizations applied in these output files You can evaluate these optimized versions with a test dataset:

cognify evaluate /your/gen/ai/workflow/optimization/output/path

You can also continue running Cognify with more iterations from these outputs using the -r or --resume flag:

cognify optimize /your/gen/ai/workflow.py -r

Follow our quickstart or read our full documentation to learn more.

• User Guide: A Cognify user guide consists of a quickstart and a step-by-step tutorial
• Fundamentals: Fundamental concepts about Cognify internals
• API Reference