Spillway design vs. inoculation prompting

Inoculation prompting

mechanism: prompting

• requires the AI to attend to the inoculation prompt during training
• may degrade over the course of training — since the prompt always recommends reward hacking, RL may favor ignoring it
• some specific circuits that produced reward hacking might not have attended to that part of the prompt

Spillway design

mechanism: pre-RL priors

• pre-RL priors might shape generalization in a way that survives throughout RL
• might directly rewrite existing associations with reward hacking (like power-seeking)