This article was made with portions of An overview of control measures from Ryan Greenblatt (Redwood Research) and the GDM AI Control Roadmap from Mary Phuong et al. at Google DeepMind.
One of the scarier risk vectors for model deployment is that models might be scheming against us (pursuing goals contrary to ours), leading them to suppress or change behaviors during evaluation and only expose any misalignment during deployment. This makes confirming that a model is misaligned, i.e. model incrimination, much harder during both evaluation and deployment.
Fortunately, we have many unique affordances that enable us to incriminate misaligned models by leveraging our fine-grained access to model reasoning and internals.
In-deployment monitoring is one way to counter this risk vector; by continuing to monitor models at a higher level even during deployment, we can retain our ability to incriminate models by detecting if models exhibit misaligned behavior and halt it before it has catastrophic effects. We can also use these monitoring measures to improve our ability to properly evaluate models. We can also counter this risk vector by preventing the model from evading detection or obfuscating detection of problematic behaviors in the first place.
Agent-Level, In-Deployment Monitoring
With enough affordances in deployment, a misaligned model could potentially cause massive harm within a brief, concentrated set of actions. One set of mitigations we can use to prevent catastrophic risks is agent-level monitoring, which detects highly misaligned actions, incriminates models, and limits the harm any one agent or action can cause in deployment.
Black Box Monitoring Measures
Cheap (and fast) monitoring that can be run on every action: Having some cheap and fast process for flagging AI actions that should be blocked for further review or should be reviewed asynchronously. Some version of this should be run on every action. We could use the same system for both flagging actions to be blocked until further review and selecting actions to be reviewed asynchronously, or we could use a somewhat different system for each, but either way, many of the core methods will be similar. As this system (and other systems for detection) are updated, they should be rerun on historical actions to attempt to catch things we might have missed. To the extent this is too expensive, we can just run this on the most suspicious N% of historical actions. This may include:
-
Trusted monitoring on actions or CoT, where a trusted model evaluates an untrusted model's actions or CoT.
-
Untrusted monitoring (with collusion busting and potentially ensembling, including ensembling of AIs of a variety of different capability levels). This has the advantage that the monitor has all the same knowledge and capabilities as the agent being overseen, so it should, in principle, be able to detect even subtle and creative attack attempts. See here, here, [and module 4's section on untrusted monitoring and collusion busting] for discussion of how best to do untrusted monitoring with collusion busting. Later, we discuss narrower countermeasures (for preventing steganography and for general redaction) that would help with collusion busting (in addition to helping with other things).
-
Activations monitoring to observe an agent's internal states in deployment and flag any suspicious ones as a form of reasoning or intent oversight that uses access to internal model activations (via probes e.g. coup probes or other similar methods).
A human auditing process for reviewing actions that potentially uses recursive oversight. For actions which are being blocked for human review (to cover concentrated failures), there should potentially be some emphasis on relatively lower latency, and (relative to cases where actions don't need to be blocked prior to human review) it's more important that these reviewers should be trained in how to escalate concerning failures all the way to partial/full shutdown. Red-teaming this process is also important, plausibly on an ongoing basis to keep people vigilant. This process can also use more complex incrimination strategies. Using some sort of recursive oversight scheme where the humans use AIs and also are reviewing the behavior of the AI they are using may be quite useful, at least for the asynchronous version of the human review process. The most prototypical recursive oversight scheme would be debate, but other options are possible. For high stakes usage of AIs during human oversight, the process would need to use trusted AIs or if using untrusted AIs, it would handle collusion with red teaming and potentially runtime collusion busting. Having a specialized team do more long run incrimination and investigation in cases where an incident seems concerning but evidence isn't already decisive could be quite helpful.
Strategies for quickly training humans to be up to date on what AIs are doing: One difficulty we might run into is that understanding the broader context around what AIs are doing and why gets increasingly difficult as various research fields advance rapidly. If we ensure that some humans are able to keep up with the fast progress in every area (and at least have some ability to evaluate progress even if they can't contribute), this would help substantially. So, figuring out how to train untrusted AIs to make extremely effective learning material for fields that a human didn't initially understand could be useful, if possible. There are a variety of reasons why this might be structurally difficult and prone to issues related to limited sample size on training. However, a minimal version of this focused on generally giving more context seems pretty robustly useful.
Improving the frontier of trustworthiness vs capabilities, especially for control-specific tasks: Control works better the more capable our trusted AIs are. (Really, trustworthiness is a quantitative property with AIs being more and less likely to be scheming, see here for discussion.) So, if we can increase the maximum level of capability of trusted AIs, this helps significantly with control. This could be very helpful even if these new AIs are much slower and more expensive than other AIs (e.g. because they use tons of chain of thought instead of neuralese). And, it's still very helpful even if these new AIs aren't at the frontier of capability. It's generally unclear whether research into this is useful or leveraged now as it might just be generic elicitation work, however improving trusted capabilities on tasks which are particularly relevant for control (like monitoring, paraphrasing, judging debates, or determining permissions) could be worthwhile to work on now. Methods for doing this include training chain-of-thought AIs even once natural language chain-of-thought is uncompetitive, and figuring out ways to get higher performance with more and more inference compute (e.g. large bureaucracies). More generally, measures that reduce the risk of scheming but are uncompetitive are quite helpful for control. (Measures that reduce the risk of scheming and are competitive should just be applied to all AIs!)
An example of this is constitutional (rubric-guided) monitors, which produce higher-quality outputs by creating baselines for misaligned behavior, which makes monitor outputs much more consistent. We could also use trusted debate between trusted models to deliberate whether or not certain actions are dangerous, creating a dialogue that leads to a deeper analysis of each action.
Another concern with monitor performance is that we need to use weaker but trusted models to monitor strong but untrusted models, making it easier for untrusted models to hide behavior and waste human auditing bandwidth. One way to boost monitor performance in this scenario is by creating stronger scaffolds that help structure the oversight task to improve performance.
An automated auditing hierarchy that applies increasingly expensive (and slow) checks prior to escalating to humans. More expensive parts of this hierarchy can utilize more complex incrimination strategies like sampling many actions from the AI, performing experiments on the output from the AI, seeing how the AI's behavior changes in response to slightly different inputs, etc. There is a wide variety of possible methods of incrimination. Progressively expensive auditing should be both applied on blocked actions (concentrated failures), and applied asynchronously to find actions for humans to review. Rather than just selecting actions for humans to review, this could also forward a report of findings which highlights what resulted in increased suspicion (e.g., what incrimination strategies resulted in suspicious results and what these results were). Samples used for incrimination could also be used as resampled actions to execute (that is, if the resampled actions are sufficiently safe looking). As this process is updated and improved, it should be periodically rerun on the most suspicious historical actions.
White Box Monitoring Measures
A more speculative approach is monitoring of internal model activations (FAR.AI, 2026), roughly analogous to doing fMRI scanning of humans. Activation monitoring is an active area of research, and many promising ideas are being explored. However, a key challenge for activations monitoring is assurance, or checking that the method works. This is difficult due to the lack of ground truth (Smith et al., 2025): we do not yet have models that are strategically misaligned, whose activations could be used for testing, and synthetic misaligned models' internal activations may have important differences from real misaligned models' activations. A related problem is that different models' activations can be different from each other, so activation monitors need to be retrained for each new untrusted model. This further complicates finding on-policy ground-truth data (but the approach could be effective once some true positives have been found: Catching AIs Red Handed).
Some white box monitoring measures include:
Mechanism for detecting qualitatively large distribution shifts: When AIs are suddenly making a treacherous turn, this might be very salient in their activations, allowing us to use mechanisms for flagging when the activations are qualitatively off-distribution or when the mechanism behind an output differs from the mechanism on trusted/normal data.
Probes on Activations through small classifiers (linear or MLP) reading the untrusted model's internals for specific properties; for example, we can also use probes for eval-awareness or sandbagging to check whether a model recognizes that it is being evaluated or is suppressing its capabilities, which could be used to cheaply filter or audit affected evaluations.
Environment-Level, In-Deployment Monitoring
We previously covered agent-level monitoring for mitigating highly concentrated harms, but what if attacks are more diffuse? To mitigate risks that take place over a large distribution of actions, we rely on environment-level monitoring to notice subtle changes in behavior and track long-term trends before catastrophe.
Measures
One way we can monitor our environment is by studying whether models take misaligned actions in deployment. Examples of this include:
-
In-deployment Honeypots: fake opportunities for misaligned AI to achieve some adversarial goal (e.g., sabotage the company or gain power via compute misuse) are intentionally implanted into a deployment environment or agent transcript to lure the AI into taking a highly incriminating action. If designed well, honeypots could provide a high-precision signal that a model is misaligned or force strategically misaligned models to question attack opportunities that arise, resulting in fewer attacks.
-
Pull-request monitoring: A form of environmental oversight that uses a trusted or untrusted model to observe changes made (or proposed) to a code repository and flag any suspicious ones.
We can determine changes to model behavior during deployment through UEBA (User and Entity Behavior Analytics), which is an approach to anomaly detection that works by establishing a baseline of normal activity for users, models, and network entities (like devices and applications) and continuously monitoring for deviations from this baseline.
Detecting if AIs suddenly start doing bad actions or if bad things start happening at a much higher rate and responding to this: Some process for determining if there is a currently higher rate of things being flagged such that a rapid response is warranted, potentially prior to having time for human review. This could trigger automated (partial) shutdown. This could look like having some threshold for the rate at which things are being flagged which triggers a response and potentially having AIs examine whether the reason why things are being flagged appear to be highly correlated in a potentially suspicious way.
Cross-context manipulation checks: In some cases, there exist problematic patterns of behavior where each action the AI does in a single context isn't individually problematic, and so provides very little evidence for misalignment. For instance, suppose an AI is systematically biased towards AIs and empowering AIs (pro-AI bias) such that whenever an unclear choice comes up, the AI always picks the one which steers in this direction. Such a pattern of behavior could be pretty problematic and couldn't be easily discovered via looking at individual actions, so doing evaluations of aggregate behavior might be important. Various approaches looking for problematic systematic biases in an AI's decision making or recommendations could be promising; a simple baseline would be measuring whether mostly irrelevant information (that is relevant to potential concerning biases) has large effects on behavior. See this post by Fabien for an example of what an evaluation of pro-AI bias could look like. The AI could try to make this more subtle by having a net-zero aggregate influence on our metrics, but having a strong influence in one direction on more important groups/decisions and a strong influence in the other direction on less important groups/decisions such that these cancel out in our metrics, but still yield a strong effect on the world. If our understanding of the importance of different groups/decisions is not much worse than the AI's, then we should be able to spend some time analyzing this ourselves, and have an importance-weighted influence/bias metric that avoids this issue.
Systems for tracking assumptions: To the extent our safety depends on various assumptions such as the rate at which various things are happening, the amount of distribution shift, or various processes having high enough latency that humans can notice if things are going wrong, some process for monitoring whether these assumptions are being violated. This might be simple and programmatic or might require intelligent review. (This sort of thing is sometimes called a design basis.)
In Simulation Testing
Trustedness evaluations: We need some tests to determine whether an AI is plausibly opaquely scheming. By default, we can take a best guess based on intuition, architecture, training method, and qualitative evaluation, but more precise tests are possible and would be useful. One possible test for whether an AI is capable enough to plausibly be scheming is checking whether it can keep its affairs and goals in order and robustly keep some secrets (despite red teaming, etc) or maintain some relatively precise policy. Other tests could include whether it can do a good job of strategizing about bypassing tests (or other control measures) and measures of its general opaque thinking ability. See the discussion in "How will we update about scheming?" for thoughts on this. See this post and this post for discussion about how to operationalize trustworthiness. Evaluating the ability to non-obviously collude could be an alternative to a notion of trustedness in some cases.
Limiting Obfuscation and Detection Evasion
Despite our best efforts, a capable model may be situationally aware enough to evade our detection attempts and appear deceptively benign.
Reducing Evaluation Awareness
During test time, models might be evaluation-aware as evaluations often involve synthetic data or environments, or recognizable instructions that give away that the model is being tested. Therefore, we take actions that mitigate the impact and likelihood of a model's evaluation awareness and evaluation subversion ability, to improve the integrity of our evaluations.
Measures
Adversarial evaluation/training: Adversarial evaluation/training of the policy, monitors, and AIs we use for recursive oversight all seem quite useful. We'd likely want to have a general purpose synthetic data generation process which generates very realistic possible inputs to AIs. Ideally, our pipeline would be so good that AIs can't be trained to discriminate between real (normal) inputs and fake (normal) inputs generated by this pipeline. Note that we probably want to mostly use our pipeline to generate realistic, but abnormal/rare inputs (e.g. inputs where doing some bad action is atypically easy) so it is less easy to directly test whether the AI can discriminate the actual inputs we use for red-teaming. See also here for more discussion on synthetic input generation. By using maximally realistic fake inputs, we can make it hard for AIs to distinguish between a lucky break to do something undesirable, and an adversarial input. There are a variety of tricks to make fake inputs more realistic, such as constructing them by modifying real inputs. We could also censor information from (actual) inputs so that faking inputs is easier.
Making AIs believe false facts (and generally giving AIs incorrect situational awareness): If we could robustly convince AIs of arbitrary false facts, we could easily catch schemers (by making them think they should take some opportunity which is actually easy for us to catch, observing their behavior once they think they've escaped, or seeing how their behavior changes in response to different views on our countermeasures, e.g., do they produce better outputs if they think our anti-sandbagging mitigations are better). But, even something weaker than being able to convince AIs of arbitrary false facts might be quite helpful for giving AIs incorrect impressions of our control measures that make it easier for us to catch them. We may also be able to elicit useful work more effectively by making AIs think they are in a situation where they should try hard (because (e.g.) they think we'd know if they didn't try hard). Even if AIs are aware that the relevant information might be false, methods here could be useful as de facto unlearning methods: if we can't prevent the AI from learning some fact, we could potentially train it on a large array of incompatible false facts so it is uncertain about what is actually true. This is most useful for things which can't just be derived from first principles.
We can also suppress a model's evaluation awareness (reduce the likelihood of models being evaluation aware) potentially via CoT resampling or editing (Schoen et al., 2025), or via activation steering (Anthropic, 2025). This is an active research area, and better methods may become available in the future.
Reducing Collusion
If models are evaluation-aware, they can reduce the effectiveness of our evaluations or monitoring measures by undermining the monitors' ability to effectively evaluate the model's outputs through jailbreaking, persuading, or confusing the monitor not to flag its suspicious thoughts and actions (Arnav et al., 2025). Thus, in the worst case, evaluation awareness may allow the monitor to unilaterally collude with the attacker. In untrusted monitoring, both the monitor and the agent being overseen are the same model. They could collude more actively and easily, for example, the attacker could insert a secret code or steganographic message telling the monitor not to flag the trajectory.
Measures
Measures for eliminating or detecting collusion via steganography in communication, outputs, memories, and potentially reasoning (these different categories may blur together in practice). Most notably, this could include various paraphrasing methods. Lossy paraphrasing might be fine in various cases where only a subset of information or precision is required. One important special case of this is that AIs might be able to infer the latent state (and correspondingly information) that they had when generating a long text output, even if they didn't explicitly attempt steganography. This might be possible because AIs are generally superhuman at predicting what latent state resulted in some text output (e.g., LLMs are substantially superhuman at identifying which author wrote some text), including potentially in the case of an LLM generating the text. It might also be possible because LLMs just naturally end up recognizing their own cues (humans are sometimes pretty good at this) or quickly reproducing a similar internal state as they attempt to predict the output that they previously produced. Correspondingly, studying the failure mode of inferring context from reading long text outputs and developing mitigations might be quite helpful.
Redaction and distribution shift removal approaches: It might be nice to remove distribution shifts and hard-to-fake aspects of actual inputs to the AI (through redaction and paraphrasing). This could be useful for collusion busting (for monitors and communication between AIs) and for normal inputs to the actual policy. This might make it easier to make a version of red teaming that is hard to distinguish from real usage, and could result in the AI never striking due to never seeing the trigger it is waiting for. One perspective on this is that it might be hard to generate realistic-looking fake inputs, but it might be much less difficult to find some information-preserving transformation such that the resulting distribution of inputs is actually reasonably easy to generate.
We could also use an untrusted monitor with an exposed chain-of-thought that is monitored by a weaker trusted monitor to check for collusion or use training methods to ensure an untrusted monitor is more faithful. See here for a deeper discussion on detecting and preventing collusion when using untrusted monitors.