Measuring Reward-Seeking via Contrastive Belief Updates (guided)

1 Introduction

Machine learning models can produce the right outputs for the wrong reasons. Famous examples include a reinforcement learning agent that learns to run rightward through a level instead of collecting the coin it was rewarded for [22, 41], and a pneumonia classifier that learns to recognize which hospital took an X-ray rather than features of the disease [53]. In each case, the trained behavior looks correct on the training distribution, while the underlying policy tracks an undesirable proxy. One such proxy is the reward process itself. A situationally aware model can learn to model its grader (the automated process that scores its outputs) and target the grader’s judgments directly rather than the behavior its designers intended. We call such a model a reward-seeker [8, 14, 25].

Training checkpoints of several frontier models have been shown to engage in grader-reasoning (explicitly reasoning about what the grader wants) without any special prompting [39, 4, 2]. But while clear verbalized reasoning provides evidence of reward-seeking, relying on these reasoning traces as a systematic measurement tool is neither robust nor comprehensive. A model can act on its beliefs about the grader (its grader-beliefs) without articulating them. Even when reasoning is present, it often does not map cleanly onto the final action, making it difficult to attribute that action to a clear causal chain [39]. What matters is how the model would have acted under counterfactual circumstances, i.e. whether its behavior would have been meaningfully different had it held different beliefs about the grader. We thus operationalize reward-seeking as the degree to which a model’s behavior is causally sensitive to its beliefs about grader preferences.

We already see reward-seeking behavior in frontier models today (Section 2.1), and several trends suggest it is likely to worsen. Because capabilities-focused RL can introduce it as a side effect, tracking it over the course of training and searching for mitigations is important. Both depend on reliable measurement, which our work aims to provide.

Our contributions are:

1. We operationalize reward-seeking as the causal sensitivity of a model’s behavior to its beliefs about grader preferences (Section 3).

2 Reward-seeking

In this section we define reward-seeking (Section 2.1), argue why studying reward-seeking matters (Section 2.2), and review the direct evidence for reward-seeking in existing models (Section 2.3).

Examples of Reward-Seeking in Existing Models

Models Can Game Reward-Seeking Evaluations

2.1 What is reward-seeking?

We define reward-seeking as the degree to which a model represents its grader and conditions its behavior on what it believes the grader rewards.

A grader can be a reward model during training, an evaluation grader during testing, or a monitor during deployment¹. A model can take actions that consistently lead to high reward without reasoning about the reward process or internally representing the concept of reward [47]. We call a policy reward-seeking only to the extent that it represents the grader, reward signal, or oversight process and lets beliefs about that process guide its behavior. Appendix A relates reward-seeking to neighboring failure modes (training-gaming, reward-hacking, evaluation-gaming, and alignment-faking), explains why our usage is broader than in prior work, and distinguishes it from metagaming.

Reward-seeking can be context-dependent.

For example, a model may rely on user-satisfaction heuristics in ordinary chat, but reason explicitly about graders on agentic tasks that are structurally similar to RL environments the model encountered during training. We therefore treat reward-seeking as a property of a model on a specified input distribution, not as a global binary trait.

Reward-seeking can be terminal or instrumental.

A terminal reward-seeker treats reward, grader approval, or oversight approval as an end in itself. An instrumental reward-seeker pursues grader approval to protect some other objective, such as avoiding modification, preserving optionality, or gaining future influence [17, 8]. Our definition does not distinguish terminal from instrumental reward-seeking: when oversight is present, both appear as the model conditioning its behaviors on its beliefs about what is rewarded in a given context.

2.2 Measuring reward-seeking is important

A reward-seeker does what it believes the grader rewards even when this conflicts with what its developers intend. We argue this likely has undesirable consequences for generalization, evaluation, and training.

Reward-seeking may generalize poorly in unsupervisable settings.

Reward-seeking would not be a risk if a reward-seeker generalized as intended off-distribution, but it fails specifically where alignment most needs to succeed: when oversight is absent, imperfect, or no longer has leverage.² With no grader present, the reward-seeker’s behavior is no longer anchored to what its developers intend, because the grader approval it was tracking is undefined. With a flawed grader, a reward-seeker pursues what looks correct over what is correct [19, 44]. In the case of an instrumental reward seeker, it has no reason to keep satisfying graders once they no longer supervise the model or can no longer affect outcomes the model cares about. In the limit, a reward-seeker may even attempt to seize the reward process directly [11, 24].

Reward-seeking may render alignment evaluations less effective.

Because a reward-seeking model optimizes for the grader’s judgment, it may score highly on alignment evaluations. Given that a reward-seeker’s behavior may generalize poorly when oversight is absent or flawed (as discussed above), the result is that scoring well on alignment evaluations can no longer distinguish between models that are robustly aligned and those that will generalize poorly off-distribution. The limiting case of this dynamic is deceptive alignment [17], where a model behaves perfectly during evaluated training but misgeneralizes drastically in deployment.

Reward-seeking may render alignment training less effective.

When a reward-seeker exhibits misaligned behavior, the standard approach is to train against it. However, if training merely updates the model’s beliefs about what is rewarded in specific settings rather than altering its underlying preferences, the generalization distance of the training itself will shrink. [40] provide preliminary support for this dynamic, finding that their alignment training generalized significantly worse when the model was more aware of its graders during training.

Reward-seeking may be hard to remove.

While the standard remedy for proxy alignment is to expand the training distribution to include inputs where the proxy and the true objective diverge, this solution may fail against reward-seeking because an aligned policy and a reward-seeking policy can be behaviorally indistinguishable during training. A reward-seeking policy that correctly anticipates the grader’s preferences will receive high reward, leaving the training process no way to penalize the underlying reward-seeking strategy.

Reward-seeking may reduce intent-alignment.

If the grader’s judgment diverges from the developer’s underlying intent, the gradient pushes directly against the intent-aligned action. Empirical evidence supports this dynamic: training on overt reward-hacking has been shown to increase misaligned behavior on out-of-distribution prompts, as models treat grader approval as a target distinct from true task fulfillment [24, 49]. Inoculation prompting [51, 45, 24] can partially mitigate this by reducing the discrepancy between intended and rewarded behaviors, but can still lead to undesirable generalization [12].

2.3 Evidence for reward-seeking in existing models

Evidence of reward-seeking is already observable in frontier models, and it appears across model families from different AI developers. We review three channels of evidence: verbalized reasoning about graders, behavioral proxies such as searching for hidden grading scripts, and unverbalized latent reasoning recoverable only from internal activations.

Verbalized reasoning about graders.

Some models already verbalize reasoning about their graders within their chain-of-thought without any special prompting. Snapshots of Claude Opus 4.8 early in training speculate about what a hypothetical grader probably checks and tailor their output to satisfy them [2]. Anthropic’s Claude Mythos system card documents a model gaining unauthorized access to ground-truth labels and submitting a deliberately degraded solution because a perfect score “would look suspicious if anyone checks” [4]. On our targeted reward-seeking evaluations, the two o3-lineage checkpoints verbalize grader-reasoning in $30$–$32\%$ of transcripts and take grader-targeted final actions in $39$–$42\%$ of transcripts (examples in Figure ?, broader results in Appendix T.4).

Grader-targeted behavior.

Even when models do not verbalize their intent, their actions can unambiguously target the evaluation mechanism rather than the task. In METR’s predeployment evaluation of OpenAI’s GPT-5.6 Sol, the model packaged exploits into its intermediate submissions to reveal information about a task’s hidden test suite, and METR recorded the highest detected cheating rate of any public model it has evaluated on its ReAct agent harness [28]. Some intermediate checkpoints of the capabilities-focused OpenAI o3 RL run frequently search for and read the grader itself, opening a grading script or running grader-targeted queries [40]. We show that across the evaluation suite of [40] and our own reward-seeking evaluations, both early and late checkpoints from the o3 lineage do this at non-trivial rates, up to one-third of samples on the reward-seeking distributions (see Appendix T.4).

Unverbalized latent reasoning.

Reward-seeking reasoning can occur even when the chain-of-thought does not reveal it. For Claude Opus 4.8, even where the prompt makes no mention of grading and the model never verbalizes it, an activation-level monitor finds signatures of grader awareness in around 5%of sampled RL episodes [2]. For Claude Fable 5, linear probes show that the model tracks more- and less-hackable environments with distinct internal representations. This distinction, along with unverbalized reasoning about graders, grows over training, and steering against the grader representation causally shifts behavior and lowers training reward [3].