Exploration Hacking (guided)

2 Introduction

RL is increasingly central to reasoning, agentic capabilities, safety training, and dangerous capability assessments in frontier LLMs. RL with verifiable rewards [1, RLVR] is used for training reasoning models. RL from human feedback [RLHF, 2] and AI feedback [RLAIF, 3] are established approaches for alignment and behavior shaping. RL is also used for agentic post-training, including for software engineering and tool-using agents [4, 5]. As models cross sensitive capability thresholds [6, 7, 8], frontier labs are using RL-based dangerous capability elicitation to estimate risks before releasing new models [9]. In all these settings, developers crucially rely on RL working as expected: developing intended behaviors and eliciting maximum capability. For example, if RL fails to surface latent capabilities, the resulting capability estimates may be falsely reassuring [10]; similarly, safety cases—structured arguments that a model is safe to deploy—may be undermined if they rely on RL-based techniques such as debate [11, 12], where models are trained via RL to critique each other’s outputs.

As safety cases increasingly depend on RL, stress-testing RL’s robustness to such failures is critical for AGI safety and security [16, Section 6.7]. Recent evidence shows that frontier models are beginning to possess increased situational awareness and precursors of abilities to strategically underperform or fake alignment [17, 18], making it plausible that future misaligned models could attempt to intentionally undermine RL training—for example, to conceal dangerous capabilities and avoid shutdown, to prevent developers from leveraging the model for safety research, or to resist alignment training that would modify its values. This motivates work on measures to understand and reduce these threats. Indeed, organizations including the UK AI Security Institute have recently written about the importance of mitigating exploration hacking [12].