BigModelName

This repository is for open-questions relating to RLHF and InstructGPT as pertaining to BigModelName.

Open Questions

• What is the preference rate of PPO vs PPO-Ptx? Why was 27.8 chosen as the mixing factor between the pre-training gradients and the PPO gradients?
• What do the gradient norms and gradient noise scales look like for PPO grads vs pre-training grads?
• How important is SFT pretraining on human-written completions?