Chai Guanaco
Project description
Chai Guanaco is part of the Chai Guanaco Competition, accelerating community AGI.
It's the world's first open community challenge with real-user evaluations. You can submit any GPT-J based 6B models, it will be directly deployed on the Chai App where our over 500K daily active users will be providing live feedback. Get to top of the leaderboard and share the $1 million cash prize!
The Guanaco Guide
🥇 Evaluation & Prizes: Depending on the phase of the competition, a suite of user-level evaluation metrics will be used (i.e. thumbs up / thumbs down rate). Your model will be ranked in real-time compared with other models, you can view the leaderboard at anytime with the pip package
🕵️ Real-time user feedback: After your model is deployed, it will go through an safety + integrity checker, once passed, it will be deployed directly to our users who will provide written feedback that you can view via the pip package.
🤖 Model requirements: Currently, we support any models based off GPT-J 6B (i.e. 6B parameters with GPT2 tokenizer). All you need to do is push your model directly to huggingface. Support for more model types is coming soon!
⚙️ Sampling parameters: During submission, we allow for custom model generation parameters such as temperature. Once your model is deployed on our platform, it will be using the parameters you've provided to generate chat completions.
How Does It Work?
- The
chai_guanaco
pip package provides a way to easily submit your language model, all you need to do is ensure it is on HuggingFace 🤗 - We will automatically Tritonize your model for fast inference and host it in our internal GPU cluster 🚀
- Once deployed, Chai users on our platform who enter the arena mode will be rating your model directly, providing you with both quantatitive and verbal feedback 📈
- Both the public leaderboard and user feedback for your model can be directly downloaded via the
chai_guanaco
package 🧠 - Cash prizes will be allocated according to your position in the leaderboard 💰
🚀 Getting Started
Getting Developer Key
Join the competition discord, introduce yourself and ask for a developer key. Login-based authentication is coming next 🤗
Submitting A Model
Use pip to install the Chai Guanaco package
pip install chai-guanaco
Upload any GPT-J 6B based language model with a tokenizer to huggingface, i.e. EleutherAI/gpt-j-6b. Read this guide if you are unsure. Click the Use in Transformers button in huggingface to get the your huggingface model ID (i.e. "EleutherAI/gpt-j-6b")
To submit model simply run:
from chai_guanaco.submit import submit_model
model_url = "EleutherAI/gpt-j-6b" # Your model URL
developer_key = "CR_XXXX" # Your developer key
submission_parameters = {'model_repo': model_url}
response = submit_model(submission_parameters, developer_key)
submission_id = response['submission_id']
print(submission_id)
which outputs your submission id, unique to your model submission.
To verify the status of a submission, you can use the following command:
from chai_guanaco.submit import get_model_info
model_info = get_model_info(submission_id, developer_key)
print(model_info)
Once the status
field shows success
it means your model has been successfully submitted. A submission typically takes around 10 minutes for the tritonisation process to complete.
Getting User Feedback
Once your model has been submitted, it is automatically deployed to the Chai Platform where real-life users will evaluate your model performance. To view their feedback, run:
from chai_guanaco.feedback import get_feedback
model_feedback = get_feedback(submission_id, developer_key)
print(model_feedback)
Here, you will find a pandas dataframe with all the user feedback for your model, including the conversation each user has had with your model.
Getting Live Leaderboard
To see how your model performs against other models, run:
from chai_guanaco.feedback import display_leaderboard
display_leaderboard(developer_key)
which prints out the current leaderboard, with your models positions highlighted
Re-Submitting Models
Because it is a competition, you are allowed to test a single model at any given time. However, you can deactivate a model and submit a new one. To do this, simply run:
from chai_guanaco.submit import deactivate_model
deactivate_model(submission_id, developer_key)
Which will deactive your model, don't worry, all the model feedback will still be saved, it just means the model will no longer be exposed to users. You can then re-submit by repeating the model submission step.
Retrieve Your Model Submission IDs
In case you have forgotten your submission ids / want to view all past submissions, run:
from chai_guanaco.submit import get_my_submissions
submission_ids = get_my_submissions(developer_key)
print(submission_ids)
Here you will see all your model submission_ids along with their status, which is either failed
, inactive
or deployed
.
Advanced Submission
You can tweak the model generation sampling parameters by passing the parameters you want during model submission, for example:
from chai_guanaco.submit import submit_model
generation_params= {"temperature": 1.0, "top_p": 1.0, "top_k": 40, "repetition_penalty": 1.0}
submission_parameters = {'model_repo': model_url, 'generation_params': generation_params}
response = submit_model(submission_parameters, developer_key)
Resources
📒 Fine tuning guide | Guide on language model finetuning |
💾 Datasets | Curated list of open-sourced datasets to get started with finetuning |
💖 Guanaco Discord | Our Guanaco competition discord |
🚀 Deepspeed Guide | Guide for training with Deepspeed (faster training without GPU bottleneck) |
💬 Example Conversations | Here you can find 1000 example conversations from the Chai Platform |
⚒️ Build with us | If you think what we are building is cool, join us! |
🦙 Hosted & Sponsored By
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.