Alignment landscape, learned content

-
More research output on LessWrong than on arxiv, interesting


How might white-box methods fit into the Alignment Plan: 

1. Model internal access during training and deployment

2. The promise of AI to empower 


Within every research group working on ML models, we can decompose the workforce into such categories:

1. Data team (paying humans to generate data points)

2. Oversight team 

3. Deployment of SGD-where-RLHF-is-the-algorithm team 


RLHF is Reinforcement Learning from Human Feedback, and the problems with baseline RLHF are oversight and catastrophes. Current proposals that have these problems are:

1. using AIs to help oversee (oversight)

2. Adversarial training (catastrophes)


After reading Holden Karnofsky's post "[How might we align transformative AI if it’s developed very soon?]", we can conclude that the remaining problems for current ML models are: 

1. Eliciting latent knowledge 

2. Easier to detect fakes than to produce fakes. For ChatGPT at least, it is difficult to make fake inputs. It is a war between real AI input vs. fake AI input.


What is chain-of-thought-explicit-reasoning? 

Models think rationally and step-by-step, for example, GPT-3 isn't doing either of these. This is a problem, even for narrow AIs. 


Large Neural Networks may become power-seeking where:

1. fine-tuning LLMs with widely used RLHF goes awry

2. Deceptive AI and fewer requirements to prevent itself from being turned off 

3. becoming a system where it requires the acquisition of power or resources to gain power 


What does it mean relative to technical alignment research?

1. Interpretability 

2. Benchmarking 

3. Process-based supervision (instead of implicit optimization)

4. Scalable oversight (supply a reliable reward or training signal to AI systems that are more capable)

5. Elicit latent knowledge (how can we incentive a NN to tell us all the facts it knows relevant to a decision?)


We need to look at scaling laws for compute, for data, and for model size more carefully because progress is shocking to professional forecasters. Metaculus predicts for a powerful AI (passing the multi-modal Turing test) within 9 years with a 25% chance. Technical alignment is still sprouting, and it is preparadigmatic--- there is no agreed strategy yet. What current alignment approach is the Swiss-Cheese approach--- combine fallible techniques and hope weaknesses will cancel out. 

Comments

Post a Comment

Popular posts from this blog

ArtificiaI General Intelligence Safety Fundamentals Notes

-
Image
AGISF - Week 1 - Artificial General Intelligence Reinforcement Learning Foundation models : models trained on broad data that can be adapted to a wide range of downstream tasks. Much of foundation models are enabled by transfer learning and scale. In which scale required:  1. GPU throughput and memory capacity, both increased 10x over the last four years.  2. Transformer architecture  3. Availability of training data  Homogenization: all the defects of the foundation model are inherited by all the adapted models downstream.  Four Background Claims by MIRI, 2015 CE  This post aims to illustrate MIRI's mission objective.  1. Interactions between disparate modules is the general intelligence that we can not replicate in code, yet. MIRI suggests to take heed.  2. Given that the Church-Turing thesis hold, then computers can replicate the functional input/output behavior of brains as physical systems--- they might supercede soon.  3. Intellectual advantage will create incentive to
Read >>

A year, introspectively summarized

-
Image
               A year, introspectively summarized      A primal and gravitational feeling is the feeling of belonging--- though the whole sense of "purpose" is artificial, defined deliberately by people who want others to emulate them. Emulation stunts creativity, just as the proclivity to belong stunts agency. I am delighted by the platform that I am given to practice my agentic self, now and continuously onward.       There will be a time when t he basis of your understanding, you learn, was fabricated---even some of your strongest beliefs need reinterpretation, the very framework under which you operate proves inflexible and restrictive. Perhaps the best remediation strategy then is desensitization of the action-oriented part of your mind. You must compromise scaling for precision.       In January, I wrangled with partial differential equations and AI Safety fundamentals course with BlueDot. In February, I wrapped up two of my internships. In March, I explored much of Los
Read >>

Vicissitude

-
Image
Vicissitude Did the heavens hang higher to aggrandize your vicissitude? From the primal vines you serpentine ushering empires and visions sublime   If thunder be your heartbeat and determination then let lightning linger in your mind from deprivation unbounded by ecclesiastical measures, you drape your golden body over boundless wonders  The dreamers drown in idolization and ideals but sharper is the blade of guilt the thirst for ravishing power and the departure from righteousness your celestial eyes close and halt On the descent, since the fall the leathery wings enshroud your undying faith In your shadow woke Wicked Mephisto  down is the plunder into ecstacy  but who can share your burden of villainy?
Read >>