Skip to main content

Manifest for Prompting Foundation Models.

Project description

Manifest

How to make prompt programming with Foundation Models a little easier.

Table of Contents

Install

Install:

pip install manifest-ml

Install with diffusion support:

pip install manifest-ml[diffusers]

Install with HuggingFace local model support:

pip install manifest-ml[api]

Dev Install:

git clone git@github.com:HazyResearch/manifest.git
cd manifest
make dev

Getting Started

Running is simple to get started. If using OpenAI, set export OPENAI_API_KEY=<OPENAIKEY> (or pass key in through variable client_connection) then run

from manifest import Manifest

# Start a manifest session to OpenAI - default `engine=text-davinci-002`
manifest = Manifest(
    client_name = "openai",
)
manifest.run("Why is the grass green?")

Examples

We have example notebook and python scripts located at examples. These show how to use different models, model types (i.e. text, diffusers, or embedding models), and async running.

Manifest Components

Manifest is meant to be a very light weight package to help with prompt design and iteration. Three key design decisions of Manifest are

  • All models are behind APIs
  • Supports caching of model inputs/outputs for iteration, reproducibility, and cost saving
  • Unified API of generate, score, and embed

Models

Manifest provides model clients for OpenAI, AI21, Cohere, Together, and HuggingFace (see below for how to use locally hosted HuggingFace models). You can toggle between the models by changing client_name and client_connection. For example, if a HuggingFace model is loaded locally, run

manifest = Manifest(
    client_name = "huggingface",
    client_connection = "http://127.0.0.1:5000",
)

If you want to use Cohere, run

manifest = Manifest(
    client_name = "cohere",
    client_connection = <COHERE_API_KEY>,
)

You can also just set export COHERE_API_KEY=<COHERE_API_KEY> and not use client_connection.

You can see the model details and possible model inputs to run() via

print(manifest.client.get_model_params())
print(manifest.client.get_model_inputs())

Global Cache

We support having queries and results stored in a global cache that can be shared across users. We treat inputs and outputs as key value pairs and support SQLite or Redis backends. To start with global caching using SQLite, run

manifest = Manifest(
    client_name = "openai",
    cache_name = "sqlite",
    cache_connection = "mycache.sqlite",
)

The cache will be saved in mycache.sqlite.

We also support Redis backend.

manifest = Manifest(
    client_name = "openai",
    cache_name = "redis",
    cache_connection = "localhost:6379"
)

As a hint, if you want to get Redis running, see the docker run command below under development.

Running Queries

Once you have a session open, you can write and develop prompts.

result = manifest.run("Hello, my name is Laurel")

You can also run over multiple examples if supported by the client.

results = manifest.run(["Where are the cats?", "Where are the dogs?"])

We support async queries as well via

import asyncio
results = asyncio.run(manifest.arun_batch(["Where are the cats?", "Where are the dogs?"]))

If something doesn't go right, you can also ask to get a raw manifest Response.

result_object = manifest.run(["Where are the cats?", "Where are the dogs?"], return_response=True)
print(result_object.get_request())
print(result_object.is_cached())
print(result_object.get_json_response())

By default, we do not truncate results based on a stop token. You can change this by either passing a new stop token to a Manifest session or to a run.

result = manifest.run(prompt, "Laurel", stop_token="and")

If you want to change default parameters to a model, we pass those as kwargs to the client.

result = manifest.run(prompt, "Laurel", max_tokens=50)

Local Huggingface Models

To use a HuggingFace generative model, in manifest/api we have a Flask application that hosts the models for you.

In a separate terminal or Tmux/Screen session, to load 6B parameters models, run

python3 -m manifest.api.app \
    --model_type huggingface \
    --model_name_or_path EleutherAI/gpt-j-6B \
    --device 0

You will see the Flask session start and output a URL http://127.0.0.1:5000. Pass this in to Manifest. If you want to use a different port, set the FLASK_PORT environment variable.

manifest = Manifest(
    client_name = "huggingface",
    client_connection = "http://127.0.0.1:5000",
)

If you have a custom model you trained, pass the model path to --model_name_or_path.

To help load larger models, we also support using parallelize() from HF, accelerate, bitsandbytes, and deepspeed. You will need to install these packages first via pip install manifest-ml[api]. We list the commands to load larger models below.

  • T0pp
python3 -m manifest.api.app \
    --model_type huggingface \
    --model_name_or_path bigscience/T0pp \
    --use_hf_parallelize
  • NeoX 20B (requires at least 60GB of GPU memory)
python3 -m manifest.api.app \
    --model_type huggingface \
    --model_name_or_path EleutherAI/gpt-neox-20b \
    --use_accelerate_multigpu \
    --percent_max_gpu_mem_reduction 0.75
  • Bloom 175B (requires at least 240GB of GPU memory)
python3 -m manifest.api.app \
    --model_type huggingface \
    --model_name_or_path bigscience/bloom \
    --use_bitsandbytes \
    --percent_max_gpu_mem_reduction 0.85

Embedding Models

Manifest also supports getting embeddings from models and available APIs. We do this all through changing the client_name argument. You still use run and abatch_run.

To use OpenAI's embedding models, simply run

manifest = Manifest(client_name="openaiembedding")
embedding_as_np = manifest.run("Get me an embedding for a bunny")

As explained above, you can load local HuggingFace models that give you embeddings, too. If you want to use a standard generative model, load the model as above use use client_name="huggingfaceembedding". If you want to use a standard embedding model, like those from SentenceTransformers, load your local model via

python3 -m manifest.api.app \
    --model_type sentence_transformers \
    --model_name_or_path all-mpnet-base-v2 \
    --device 0

Development

Before submitting a PR, run

export REDIS_PORT="6379"  # or whatever PORT local redis is running for those tests
cd <REDIS_PATH>
docker run -d -p 127.0.0.1:${REDIS_PORT}:6379 -v `pwd`:`pwd` -w `pwd` --name manifest_redis_test redis
make test

Cite

Please cite Manifest if you used it for any publications. Thanks!!

@misc{orr2022manifest,
  author = {Orr, Laurel},
  title = {Manifest},
  year = {2022},
  publisher = {GitHub},
  howpublished = {\url{https://github.com/HazyResearch/manifest}},
}

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

manifest-ml-0.1.3.tar.gz (49.3 kB view details)

Uploaded Source

Built Distribution

manifest_ml-0.1.3-py2.py3-none-any.whl (57.6 kB view details)

Uploaded Python 2 Python 3

File details

Details for the file manifest-ml-0.1.3.tar.gz.

File metadata

  • Download URL: manifest-ml-0.1.3.tar.gz
  • Upload date:
  • Size: 49.3 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.10.4

File hashes

Hashes for manifest-ml-0.1.3.tar.gz
Algorithm Hash digest
SHA256 0a45f7ac8bdbd8415c7f66c09ff3bb3e6a18c1d5b4cecc06d0abca89ecb96aa9
MD5 3bf8745d591477ac08b5259a1c970399
BLAKE2b-256 4d863e031c85fefdcb78b784aa551bc156ca16a179a42fb627032da963cedd0f

See more details on using hashes here.

File details

Details for the file manifest_ml-0.1.3-py2.py3-none-any.whl.

File metadata

File hashes

Hashes for manifest_ml-0.1.3-py2.py3-none-any.whl
Algorithm Hash digest
SHA256 500b1d2d2390e00888997325407dadfc666ffd651d998e95885d029d3f17d952
MD5 9f66e4a08bb922eb27fb1e8c74e66213
BLAKE2b-256 5aad42a57f69a8149c2e7a19579a0c2a325e23350d89b39d6e0a00e31d3b5416

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page