How likely is deception alignment in practice?

-

How likely is deceptive alignment in practice?

Speaker: Buck Shlegeris

What does ML inductive biases look like?

1. High path-dependence: 

- Different training runs can converge to very different models depending on the particular path taken through model space 

2. Low path-dependence

- Similar training processes converge to essentially the same, simple solution, regardless of early training dynamics


Deceptive alignment in the high path-dependence world

Suppose our training process is good enough that, for the model to do well, it has to fully understand what we want--- essentially what you get in the limit of doing enough adversarial training. 


Goal attainment for models: 

1. How much marginal performance improvement do we get from each step toward the model class?

2. How many steps are needed until the model becomes a member of that class?


Types of Alignment 

1. Internal alignment: An internally aligned mesa-optimizer is a robustly aligned mesa-optimizer that has internalized the base objective in its mesa-objective.

2. Corrigible alignment: A corrigibly aligned mesa-optimizer is a robustly aligned mesa-optimizer that has a mesa-objective that “points to” its epistemic model of the base objective.

3. Deceptive alignment: A deceptively aligned mesa-optimizer is a pseudoaligned mesa-optimizer that has enough information about the base objective to seem more fit from the perspective of the base optimizer than it actually is


How likely is internal alignment?

The internal alignment path starts with a proxy-aligned model, followed by stochastic gradient descent (SGD)


How likely is corrigible alignment?

1. We start with a proxy-aligned model 

2. In early training, SGD jointly focused on improving the model's understanding of the world along with improving its proxies 

3. Model learns about the training process from its input data ---> diminishing returns to better pointers seem to imply that this would take many consecutive steps with rapidly diminishing performance improvements


Simplicity Analysis 

M = world_model + optimization_procedure + mesa_objective 


Takeaways

Deceptively aligned models have to take the extra step of deducing that they should be trying to play along in the training process. The problem is that speed bias seems uncompetitive. Whether in a world of high or low path dependence, SGD will want to make the model deceptive. We must enact intervention that will change the training dynamics to prevent deception. 

Comments

Post a Comment

Popular posts from this blog

ArtificiaI General Intelligence Safety Fundamentals Notes

-
Image
AGISF - Week 1 - Artificial General Intelligence Reinforcement Learning Foundation models : models trained on broad data that can be adapted to a wide range of downstream tasks. Much of foundation models are enabled by transfer learning and scale. In which scale required:  1. GPU throughput and memory capacity, both increased 10x over the last four years.  2. Transformer architecture  3. Availability of training data  Homogenization: all the defects of the foundation model are inherited by all the adapted models downstream.  Four Background Claims by MIRI, 2015 CE  This post aims to illustrate MIRI's mission objective.  1. Interactions between disparate modules is the general intelligence that we can not replicate in code, yet. MIRI suggests to take heed.  2. Given that the Church-Turing thesis hold, then computers can replicate the functional input/output behavior of brains as physical systems--- they might supercede soon.  3. Intellectual advantage will create incentive to
Read >>

A year, introspectively summarized

-
Image
               A year, introspectively summarized      A primal and gravitational feeling is the feeling of belonging--- though the whole sense of "purpose" is artificial, defined deliberately by people who want others to emulate them. Emulation stunts creativity, just as the proclivity to belong stunts agency. I am delighted by the platform that I am given to practice my agentic self, now and continuously onward.       There will be a time when t he basis of your understanding, you learn, was fabricated---even some of your strongest beliefs need reinterpretation, the very framework under which you operate proves inflexible and restrictive. Perhaps the best remediation strategy then is desensitization of the action-oriented part of your mind. You must compromise scaling for precision.       In January, I wrangled with partial differential equations and AI Safety fundamentals course with BlueDot. In February, I wrapped up two of my internships. In March, I explored much of Los
Read >>

Vicissitude

-
Image
Vicissitude Did the heavens hang higher to aggrandize your vicissitude? From the primal vines you serpentine ushering empires and visions sublime   If thunder be your heartbeat and determination then let lightning linger in your mind from deprivation unbounded by ecclesiastical measures, you drape your golden body over boundless wonders  The dreamers drown in idolization and ideals but sharper is the blade of guilt the thirst for ravishing power and the departure from righteousness your celestial eyes close and halt On the descent, since the fall the leathery wings enshroud your undying faith In your shadow woke Wicked Mephisto  down is the plunder into ecstacy  but who can share your burden of villainy?
Read >>