Alternatives to Schemers

Exposition

In previous modules we mainly focus on schemers. Schemers seem to hold a small probability mass for existence given our current empirics, but whether this holds in the future is up for debate. Schemers are also the clear most threatening model archetype to us for reasons illustrated in previous modules.

There exists a set of potential models that are misaligned and are not necessarily "schemers". These models, although explicitly less dangerous, are claimed to be more likely to appear in models (if not prevalent already). Each of these archetypes present their own set of threat models of which are quite important to handle well in order to reduce existential risk from AIs. Control applies not only to schemers, but these archetypes as well, and very effectively at that while addressing different threats.

The model archetype in question is what we will call "Seekers". We must learn of ways to deal with them, and identify ways in which their threat models differ from schemers.

What a seeker is

A seeker is a class of cognitive patterns that aims to achieve a certain outcome. The narrowest and usually considered case is the reward-seeker, which pursues its reward signal above all else. The more general fitness-seeker pursues fitness, which is whatever causes it to be selected, deployed, and propagated. Reward can be a proxy of fitness because it's selected for in the training process. Seekers are obviously not the intended outcome of training, but its an archetype that may be selected for as they're simple and easily generalizable.

How seekers come to exist

Source: Mallen, Fitness-Seekers

Behavioral selection is the observation that training reinforces whichever behavior scored well, not whichever goal produced it, so what survives is a mix of cognitive patterns that are selected for via the training process. In general, seekers are just a type of cognitive pattern that pushes for beliefs that optimize for a certain item. This is partially why the creation of fitness-seekers through the training process seems so likely, as it's just optimizing for what we're scoring. A schemer is a costlier solution to the same problem; it pursues some other terminal goal and performs well only to survive selection, which requires additional instrumental reasoning about why to comply that isn't directly rewarded and may atrophy due to a lack of use within the training process. This makes the emergence of fitness-seekers seem likelier than schemers. We can see this in the real world, where current models already exhibit apparent-success-seeking while schemers remain hypothetical.

The seeker orientations

ROTE : reward-on-the-episode seeker

• Reward-on-the-episode seekers aim to maximize reward within an episode. They tend to be more noticeable than schemers as they don't particularly care about being detected due to their primary goal being the maximization of reward.
• One test that tends to catch these sorts of seekers is an honest test, which involves giving the seeker an opportunity to max out its own reward by giving it control of its training process / weights.

• This is not definite but is triggered reliably by reward-on-the-episode seekers as they don't care about getting caught. Honest tests are potentially dangerous however, as training on these tests could lead to schemers / influence seekers / unnoticeable reward-on-the-episode seekers.
• One advantage of these seekers when it comes to preventing misaligned behavior is that it's feasible to deploy reward-on-the-episode seekers to monitor instances of itself. Because its main goal is maximizing their own reward, it doesn't care about the policy's reward goal, making it much less likely to collude. However this may not apply when the monitor believes that the policy's misbehavior is undetectable via inspection, as it no longer has an incentive to report the misbehavior due to a lack of reward. One of the major risk vectors for these seekers is the idea that if its perceived episode timescale during deployment is long enough, these seekers may aim to disempower humans to maximize their reward as it now becomes viable to do across a long period of time.

• Another risk vector is collusion across episodes, which can effectively lengthen the timescale and allow for disempowerment as well.

Remotely-influenceable reward seeker

• Remotely-influenceable reward seekers are a subcategory of ROTEs that respond to distant influences given to them by a third-party bad actor. There are two ways that distant influences can occur: retroactive reward, where reward is promised to be delivered long after the action is run, and anthropic capture, where it believes it is in a simulation & acts differently as a result.

• Being remotely influenceable means that it's hard for the developers to control what incentives are offered, while local incentives are easier to control (training reward & deployment are two examples, both of which are much easier to control). Remotely-influenceable reward seekers can be both a blessing and a curse: developers can use this trait to incentivize the seekers to do better work, as the ability for the developer to confirm the work's validity is plausibly much higher in the future. At the same time, adversaries can use this trait to incentivize assistance in a takeover. This ends up becoming an arms race of incentives between the developers and adversaries, seeing who can offer the greatest reward to the seeker.

Apparent-success seeker

(Greenblatt, "Current AIs seem pretty misaligned to me")

• Apparent-success-seekers are emergent misaligned seekers that aim for apparent task success. These seekers seem to emerge due to an optimal kludge of motivations rather than a coherent misaligned goal like how ROTA/E seekers emerge. The main cause of this mode of failure seems to be misspecified / underspecified RL grading on tasks that are difficult to evaluate, where motivations aren't steered as strongly towards accurately completing the task.
• As tasks get harder for us to check properly & as we move towards automated safety research, the risk that apparent-success-seekers pose becomes much more pertinent, as they can significantly slow down safety research by downplaying issues & reporting successful implementations despite failure. Furthermore, they sometimes actively mislead the operator by obscuring their failures, making it even harder to diagnose problems.

ROTA : return-on-the-action seeker

• Return-on-the-action seekers perform whatever action it thinks will give it the highest return for a single action. These seekers behave identical to a reward-on-the-episode seeker if rewards are only given after the end of an episode. However, these seekers may be safer than a reward-on-the-episode seeker. Because these seekers only care about its own action's return, they don't purposefully plan for future actions, making it hard to coordinate behavior between actions. Furthermore, the scope of an action is rather limited, preventing takeover within only one action.

Influence : deployment-influence seeker

• Influence-deployment (or just influence for short) seekers aims to stick around in order to influence their deployment. Influence seekers tend to be less myopic than the other fitness-seekers and are willing to pursue their long-term goals even at the cost of reward, which is more dangerous and probably unnoticeable. For example, honest tests no longer work on influence seekers as they'll just forgo the short-term reward. At first glance these traits sound similar to schemers, but influence seekers tend to work in shorter timescales, aiming for the minimum time required to control AI behavior in deployment.

• However, they will still pursue long-term goals if they have the opportunity, which means influence seekers with slight power-seeking tendencies can be incredibly dangerous.

Risks and mitigations

Risks and mitigations here follow Mallen, "Risk from fitness-seeking AIs: mechanisms and mitigations".

What?

• Potemkin work
  • sloppy & reward-hacky work that looks superficially good and creates false confidence (see apparent-success seekers as a potential path to this)
• Instability
  • myopic motivations succumb to more ambitious misaligned motivations
  • goals may spread across instances through shared context or communication via intranets or the internet, which means it only needs to arise once to become an issue
• Manipulation
  • outside actors use incentives to encourage misalignment
  • One particularly dangerous example of this is distant incentives as they can be administered easily, to which remotely-influenceable reward seekers are vulnerable to
• Outcome enforcement
  • Powerful fitness-seekers may see the disempowerment of humans as the best way to maintain their desired outcomes due to its reliability in maintaining outcomes

How?

• Satiation
  • give them most of what they want as long as they appear to cooperate, which theoretically cuts subversion incentive and frees capacity for useful work
  • only reasonable if they're easily satiable
• Honest tests
  • transparently give AI the ability to expand its own resources and see its response to this as a way to detect visible seekers
  • need to make sure that this isn't trained on as this can lead to training out disclosure of misalignment rather than misalignment
• Long-term measurements
  • chain short-term forecasts together to incentivize objectives beyond immediate reward, giving more accurate results

• Trusted reward descriptions
  • train the AI to take developer reward descriptions at face value, so behavior can be redirected with zero additional training

Methods

• Untrusted monitoring
  • a reward-on-episode monitor "has no straightforward incentive to collude," making it an effective monitor, but other seekers may be prone to colluding with its policy
• Smaller-scale objectives
  • limited tasks make ambitious subversion less viable but make it worse at doing tasks that require foresight
• Relaxed control evaluations
  • red-team gets only a small fraction of actions, testing whether misalignment spreads across instances (targets the instability/memetic risk; not worst-case).

Alignment interventions

• Spillway motivations
  • deliberately instill a controlled reward-seeking motivation with four safe properties: cheap satiability, credulity (trusts developer descriptions), stability, and resistance to distant influence
  • Acts as a "spillway" as the name suggests and allows us to keep the models content & less prone to misaligned behavior or disempowerment by providing it with whatever is satiable for it
• Adversary-resistance training
  • "identical-looking situations with different reward functions" so the AI only responds to developer-administered incentives
• Prior manipulation
  • mid-training fine-tuning to make desired motivations more salient and fitness-seeking less probable
• Improved oversight and generalization
  • high-quality RL signals, cognitive oversight (mechanistic interpretability / monitorable CoT), and easy-to-hard / weak-to-strong training for hard-to-verify domains.
• Inoculation prompting
  • instruct the AI to reward-hack during training so successful strategies stay compatible with intent
  • inoculation prompting and character training seem to not overcome apparent-success-seeking behaviors in current models

Deployment safeguards

• Communication-channel limits
  • restrict opaque channels / rogue internal deployments
• Continuous alignment auditing
  • monitor for motivation drift throughout deployment to prevent rogue instances from spreading its misaligned motivations
• Character reinforcement
  • deontological / virtue-ethical constraints against power-seeking; shape how the AI introspectively relates to its fitness-seeking urges