instruct-goose 0.0.1

Implementation of Reinforcement Learning from Human Feedback (RLHF)

InstructGoose

Paper: InstructGPT - Training language models to follow instructions with human feedback

Questions

• In the context of RLHF, how to calculate the $L_t^{V F}(\theta)$,
  • Like it’s a function of the PPO agent uses to predict how much reward it gets if generates the sequence?
• Does the RL model and the SFT model use the same tokenizer? Yes
• I don’t know how to returns the logit of the generation model

Install

pip install instruct_goose

Resources

I used these resources to implement this

• Copied the load_yaml function from https://github.com/Dahoas/reward-modeling
• Learned how to build a dataset to train reward model: https://wandb.ai/carperai/summarize_RLHF/reports/Implementing-RLHF-Learning-to-Summarize-with-trlX–VmlldzozMzAwODM2
• Learned how to add value head in PPO agent: https://github.com/lvwerra/trl