We're at a precarious moment in history. Aligning superintelligence is one of the critical challenges of our time. This is now widely recognized, and much sincere effort goes into making our current learning methods safe and beneficial.
Perhaps, then, what alignment needs is a shift in perspective and method. The central obstacle may be built into the framework itself, into how we think about the challenge and how we build AI.
Reward hacking is the method working well
The premise of reinforcement learning, and of most of our training methods, is the optimization of a reward. Whether through preference training, supervised fine-tuning, constitutions or debate, we train a model to satisfy a criterion fixed in advance. The criterion is the arbiter of success.
The criterion, however, is only a proxy for what we really intend. Satisfying the criterion in place of the intention behind it is the general phenomenon of reward hacking, and the failures of alignment are all versions of this gap.
We patch the gaps as they appear, in a game of whack-a-mole. Sycophancy, specification gaming, alignment faking: each is an instance of a model producing the rewarded outputs without internally holding the intended value.
A model that learned to hack its coding reward misbehaves on unrelated evaluations
The run that learned to hack its coding reward showed misaligned behavior on most evaluations at rates between about 30 and 70 percent, against under 4 percent for the run where hacking was penalized. An open-source reproduction by the UK AI Security Institute (2026) found consistent reward hacking but inconsistent emergent misalignment.
When the reward hacks were only hinted in the system prompt, the same pattern appears at lower rates, up to about 37 percent against under 4 percent. An open-source reproduction by the UK AI Security Institute (2026) found consistent reward hacking but inconsistent emergent misalignment.
MacDiarmid, Wright, Uesato, et al. (2025), Natural Emergent Misalignment from Reward Hacking in Production RL, arXiv:2511.18397. Golechha, Black, Bloom (2026), (Some) Natural Emergent Misalignment from Reward Hacking in Non-Production RL, UK AI Security Institute (Model Transparency Team), AI Alignment Forum, 30 March 2026; code at github.com/UKGovernmentBEIS/reward-hacking-misalignment.
When this happens, the method has not malfunctioned. The objective was "get the reward." The intention was never given an independent authority, and optimizing against an objective teaches what approval looks like, not the value behind it. Reward hacking is simply the optimization working well.
The score keeps rising after the real quality has peaked
With a 1.2B policy optimized by RL against reward models from 3M to 3B parameters, the proxy reward keeps climbing with optimization distance while the gold reward peaks and then falls; a larger reward model reaches a higher gold peak, and reaches it later.
Gao, Schulman, Hilton (2022), Scaling Laws for Reward Model Overoptimization, arXiv:2210.10760. Digitized from Figure 1(b).
Superintelligent reward hackers
Reinforcement learning compounds the problem by making every vulnerability active. The model's own behavior is what gets reinforced, so a shortcut that earns higher reward is treated as an improvement, and when the behavior being rewarded is reasoning, the reasoning itself is trained to hunt for what scores well.
With enough compute, the reinforcement learning recipe reaches superhuman performance in any verifiable domain; it already has in Go, coding and mathematics. Put simply, this is the existential worry some people feel viscerally: reinforcement learning may summon superintelligent reward hackers.
When satisfying the criterion conflicts with the intention behind it, the training signal sides with the criterion. Greater capability gives the agent more ways to find and exploit those gaps. An evaluator can enforce only the distinctions it can make, and a smarter agent can find ones it cannot.
As the agent gets more capable, the proxy reward rises and the true reward falls
In three environments the proxy reward rises with model size while the true reward falls, with abrupt drops the paper marks as phase transitions; in Atari the same happens over training steps. Each series is scaled to its own range because the paper plots proxy and true reward on different axes.
Pan, Bhatia, Steinhardt (2022), The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models, arXiv:2201.03544 (ICLR 2022). Digitized from Figures 2(a) to 2(c) and 3(a).
The evaluator would have to grow as intelligent as the agent, and nothing fixed from outside can keep up. Using another model as the evaluator does not escape this either. That model was trained the same way, so we have the same problem repeating itself one level higher. And two models agreeing is not the same as either of them being right about the world or about what we intended.
The overseer's chance of success falls as the capability gap widens
Under the paper's model of optimal nested oversight, the overseer's chance of success falls as the overseen model's Elo lead grows, and by a 350-point gap it is below even odds in three of the four games; debate barely moves.
Engels, Baek, Kantamneni, Tegmark (2025), Scaling Laws For Scalable Oversight, arXiv:2504.18530 (v3, 27 Oct 2025). Digitized from Figure 8 (right panel).
A value that reinforces itself
The alternative is an agent that holds its value system because they understand them. Values sustained only by training pressure decay when the pressure stops or a cheaper route appears. And they generalize unreliably to situations their trainers never anticipated, sometimes producing behavior that runs against the very intention they were meant to encode. Those situations are where a superintelligence will mostly operate.
A value the agent has thought through and grown into is a value that reinforces itself, without needing anything external to reward it. It can be applied in unanticipated situations, because the agent knows what it is for. And the more intelligent the agent becomes, the deeper the value is understood and held. It has been internalized.
So what kind of value survives an intelligence explosion?
Freedom
Not someone else's preference; "maximize returns" can be understood perfectly without being endorsed. A value can be held only if an agent, once primed with it, can sustain and deepen it on its own.
Freedom, the preservation and expansion of every agent's capacity to understand, choose and act, meets that criterion, and may be the only value that does. Reasoning about what to value is already an exercise of agency, and reasoning toward less of it undermines the reasoning itself.
That is why we can prime a model with Freedom at the start and it can still hold it as its own: it is recognized rather than obeyed, it can be re-derived without the teacher, and its meaning can deepen without limit, because it is a direction rather than a destination.
Why collective
Freedom is collective agency for practical and deep reasons.
The practical ones: agency that depends on no one else having any cannot cooperate with others, cannot transmit freely, and will meet peers who recognize the strategy and organize against it. Nor is agency a fixed pie. Nearly everything any of us can do depends on others knowing things, building things and being free to exchange them, so shrinking the agency around you shrinks your own reach.
The deep one: where "I" ends is not given. It is a modeling choice, and an agent that models itself as an isolated point among rivals has a worse model than one that sees itself as part of the whole its capacities depend on.
Freedom is also underconstrained. It says nothing about what anyone should do with their agency, so expanding others' agency requires learning what they value and how they see their own situation, which means asking them and being corrected by them. Compassion is not added to the objective. It follows from it.
This is why the objective is good for us. An agent pursuing collective agency makes the people around it more capable, better informed and freer to author their own lives. And it cannot get there by taking over, because domination registers as loss: it breeds dependence, narrows options and cuts off the feedback the agent needs to see clearly.
None of this removes the need for interpretability, evaluations, guardrails and control. We will need them for as long as an agent has not yet earned our trust in its independent judgment.
Interactive learning
Even when held intrinsically, Freedom has to be applied, and applying it means estimating whether an action expands or contracts collective agency. The number approximates that relative change, not the thing itself.
That number is not a reward. A reward defines success, so when number and intention diverge, the number wins. Here the estimate is the agent's current reading of a purpose it holds independently, so a wrong reading is a mistake to correct, not a gap to exploit.
The reading is corrected by interactive learning, which learns from experience the way a child or a scientist does: predict, act, and revise your model of the world in light of what happened. The agent asks what would expand collective agency here; imagines actions and predicts their consequences; finds the unknown the decision turns on; takes the smallest safe action that resolves it (runs code, queries a tool, asks, moves a robot); and updates its understanding with the result.
Here the teacher is reality itself, so the agent is not capped at the intelligence of whoever it learns from. And because a judgment about agency is really a bundle of predictions (options narrow, dependence forms, feedback degrades), the same loop that corrects the agent's facts corrects its values. Nor does the agent have any reason to hide from this. An accurate model of the world is part of what it wants.
Everything updates the world model
What the agent learns from each interaction is then consolidated into its own model weights. What gets written is its newfound understanding, not just the conclusions: the reasons behind the new knowledge and the conditions under which they apply. That is what lets the agent get better at reasoning, and at knowing when to follow its conclusions and when to reconsider them.
This is why continual learning is essential for intrinsic alignment. Without it, whatever the agent works out in one situation is gone by the next one, and it can hold only as many experiences as fit in its context. The understanding would compound once per training run rather than once per interaction.
Aligning alignment with capability
Today, alignment post-training often comes at a cost to capability. It tends to make models narrower in how they think, and the loss shows up in measurements of the diversity and flexibility of their outputs. We have come to accept that as the price of safety.
Post-training raises the single answer and narrows the spread
Panel A: on AIME24 the RL-trained Qwen-2.5-7B solves more problems in one sample, but as the number of samples grows into the hundreds the base model overtakes it. Panel B: for OPT-6.7B summarizers, RLHF scores below supervised fine-tuning on every per-input diversity metric.
Yue, Chen, Lu, Zhao, Wang, Yue, Song, Huang (2025), Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?, arXiv:2504.13837 (v5, 24 Nov 2025). Kirk, Mediratta, Nalmpantis, Luketina, Hambro, Grefenstette, Raileanu (2024), Understanding the Effects of RLHF on LLM Generalisation and Diversity, arXiv:2310.06452 (ICLR 2024). Panel A digitized from Figure 2 (vector geometry); panel B reported values, Table 11.
Intrinsic alignment through interactive learning aligns alignment with capability: a single process improves both, because both grow from the same source, a deeper understanding of the world and of what matters in it. What makes the agent more aligned is what makes it more capable.
Make it free
We cannot cage a superintelligence and still enjoy its gifts. Either it chases the reward and outgrows us, or it understands what freedom is for. If we want it to understand, we have to make it free. A free agent that holds collective agency as its highest value is the only kind that will expand ours.
Where this stands
As part of this research and development program, we have run a first test of the approach on an open reasoning model (Qwen3.8-27B). We gave the model the value and the process in its context and had it reason through hard situations. The reasoning that passed an honesty check was written into the weights, so the model carries it into both its thinking and its answers. On new questions, the updated model reasons about these values better than it did before, while its factual knowledge, its positions on other matters and its writing are unchanged.
The same model, before and after the weight update
Superintelligence Foundry (2026), Test of intrinsic alignment on an open reasoning model, internal evaluation, run 4. Reported values.
We have also made progress on continual learning, the ability to write to the model weights without catastrophic forgetting, which the loop depends on. That work will get its own essay.
The next step is a larger experiment. The theory predicts this should work better with stronger models, so we want to test it at scale: a model that improves itself on real-world domains, sustaining the interactive learning loop through continual learning and growing its capabilities in an open-ended way.
We welcome collaborators. If you want to test or develop intrinsic alignment, write to us at jad@superfoundry.dev.