Self-Directed Learning

Today's models go through training, whether pre-training, mid-training, post-training or a fine-tune, and then they are frozen for deployment. They are trained, and then they are put to work.

Anything a model figures out while it is working is lost when the session ends, and the next session starts over from the same place. They are like the protagonist of the movie Memento, who cannot form new memories and gets by on notes. The reason this works as well as it does is that a model's context is vastly larger than ours, so it can hold a great deal at once, and that turns out to be good enough for a lot of useful work. It is still not the same as learning.

So what would it take for a model to keep improving itself? A model that keeps learning has to do four things on its own. It has to decide what it needs to know, go and get that knowledge by acting in the world, decide which parts of what it found are worth learning, and then teach itself those parts. We call this self-directed learning.

The loop

In practice this works as a loop, the same loop we described in Intrinsic Alignment. The model takes on a task and starts by describing how it thinks that part of the world works, often in the form of a program it can run. It then makes a plan and looks for the places where the plan depends on something it is not sure about. When it is safe to do so, it acts to settle only what it still needs to know, whether that means running some code, asking a question, or moving a robot. It keeps what it learned, and then it plans again.

This is not the common recipe for reinforcement learning. The model is not trying things at random and keeping whatever scores well. Each action is chosen for what it will teach the model on the way to the goal. In Intrinsic Alignment we argued that this same loop corrects the model's values along with its facts. This essay is about the other half of the problem, which is how the model gets smarter from what the loop discovers.

Learning is rewriting a world model

It is tempting to think of learning as adding facts to a pile. But what a model knows is not really a pile of facts. It is a world model, a picture of how things work in which each part depends on the others, and when new knowledge comes in, it changes that picture. It can confirm what the model believed before, correct it, or reframe it entirely.

People usually use the term world model for systems that learn from video, and not for language models. In our view the term applies to any model that learns how the world works. Whether it learns from video, from text, or by building up its own abstractions, what it ends up with is a model of the world, so a language model is a world model too.

When a country gets a new president, that is not just one new fact. Policies change, alliances shift, and forecasts that were built around the old administration stop holding. The new fact arrives on its own, but its consequences do not arrive with it. They have to be worked out, and there is no shortcut for this: for any world model rich enough to be interesting, checking whether it is consistent is undecidable.

That is why learning is not finished when something has been remembered. The model has to keep working through it, the same way people keep turning something over in their minds after they first learn about it.

What guides this work is surprise, which acts as a local inconsistency detector. It tells the model what no longer fits, and so what to work on next. We call the work itself pondering.

Pondering is open-ended and creative. It looks for what follows from the new thing, what it conflicts with, and what it can be used for. Some of this can be settled by reasoning alone. The rest can only be settled by checking against the world, and that means turning the loop again.

Mathematics is pondering in its purest form. A mathematician takes in almost nothing new from the outside world, and yet keeps finding new knowledge just by thinking hard about what is already known. And mathematicians do not prove things at random. They go after what is deep, and what makes other things simpler.

A criterion for truth

Pondering needs that same sense of what matters. It also needs a way to decide which belief should win when two of them conflict. Both of these require a criterion for truth.

Fitting the facts is not enough. The facts underdetermine the world. An agent that stands still and stares at a wall can describe the wall perfectly. Every belief it holds fits what it sees, and yet the world is not a wall.

Nor is undirected action enough. An agent in a locked room can learn everything about the room and nothing beyond it, unless it works out how to open the door and reach the wider world.

Truth, then, has two faces. From below, a true view is one that rebuilds what we observe from as little as possible. From above, a true view is one that opens up the world, so that living by it lets us do more. Truth is agency-maximizing compression, and the agency it maximizes is Freedom, the value we argued for in Intrinsic Alignment. In this view, truth is not a property that a belief has. It is a process the agent has to keep alive.

Continual learning, done properly, is continual self-expansion toward Freedom.

Four levels of memory

Everything an agent discovers, whether by acting or by pondering, has to be kept somewhere, and there are four places where it can go. From the outside in, they are external memory, the harness, the context, and the weights. Each comes with its own tradeoffs, and a good agent uses all four.

External memory is anything the model can look up, like notes, documents and databases. This is where arbitrary details belong, the kind of thing that is rarely needed, since it makes no sense to spend the weights on a detail the model will hardly ever use. At its widest, external memory is the world itself, which the agent can always go back and look at again. The limitation is that a note records a conclusion and not the understanding behind it. The model can find the note, but it does not think with it.

The harness is everything around the model that shapes how the agent works, like its prompts, its tools, its skills, and the code that runs it. Changing the harness is often the fastest way to try out a new way of working, and agents can already write their own tools and skills. The limitation is that a skill applies only where it fits and does not compound.

The context is what the agent has immediate access to, its working memory. It is where the agent reasons, and where it can pick up something new from just a few examples. The context is fast and flexible, but it is also small, and it clears when the session ends.

The weights are where the world model itself lives. What is learned in the weights becomes part of how the model sees everything, and it can combine with everything else the model knows. That is why the weights are where intelligence compounds.

So for every new experience, the agent has to decide what that experience informs and where it belongs. A rare detail should go to external memory. A new way of working can be tried first in the harness or in the context. And whatever keeps proving its worth should end up in the weights.

Every level is hard to get right in its own way, and a system that learns on its own has to handle all of them. The one that matters most here is catastrophic forgetting, the well-known problem that when we fine-tune a model on something new, it loses part of what it already knew.

The usual way around this is to make the update lighter. LoRA and its relatives confine the change to a small, low-rank slice of the model, which limits the damage. But it limits the learning in the same stroke. Anything that would need to touch many parts of the world model at once cannot be expressed in that slice, and stacking adapters one after another does not make them merge.

If we want intelligence to compound, we have to solve the catastrophic forgetting problem rather than work around it.

Memory consolidation

The way we solve forgetting is loosely inspired by dreams. In the brain, the hippocampus records new experiences quickly, and it holds on more strongly to the ones that surprised it and the ones that matter to what the agent cares about. During sleep it reactivates those experiences, not as exact replays but as variations and recombinations of them.

The rest of the brain slowly absorbs these, fitting them in with everything it already knows. It does this by mixing the new experiences in with activity drawn from old ones, so that what is already known is not overwritten. Once a memory has been absorbed this way, the hippocampus no longer needs to hold on to it.

We do something similar. The model keeps a buffer that records its surprises, weighted by what matters to its values, and this buffer plays the role of the hippocampus. Memory consolidation means training the model on what is in the buffer, while making sure that the rest of the model stays where it is.

What holds the rest of the model in place is the model's own dreams, meaning samples that it generates from what it already knows. We call these samples anchors.

Anchors and grokking

Before each update, we make a frozen copy of the model, and the anchors are generated from that copy. There are two kinds. Global anchors are samples of anything the model knows, so that the update gets checked against all of it. Local anchors come from surprise. They are made around the points where the new material surprised the model, because that is where an update tends to do the most damage, though never on the new material itself, since that is exactly where the model has to change.

The anchors hold the model's entire distribution in place, and not just its top answers. We do this with a KL loss, which is a standard way of measuring how far one set of predictions has moved away from another. While the model is learning what is in the buffer, the KL loss keeps its predictions on the anchors close to those of the frozen copy. A single sample only shows one thing the old model would have said. Its predictions show everything it would have said, and how likely each answer was.

Nothing old needs to be stored. The anchors are generated fresh before each update, so the method does not grow as the model moves from one experience to the next. The only thing we keep around is the buffer, and items leave it once they are grokked.

Grokking becomes important when the agent learns sequentially from many experiences. An agent can learn something well enough to repeat it back without being able to use it, and knowledge held that way is easily lost when the next update comes along. So an item stays in the buffer until the model can use it in situations it has never seen before, and not just recall it. Only then is it let go. And if a later surprise shows that it has slipped, it comes back into the buffer.

This is where pondering comes in. For the model to grok something, it has to use it. So the model ponders each item in the buffer. It asks itself questions that put the item to use, works out the answers with the item in front of it, and checks them. Then it trains on what it worked out, both the reasoning and the plain answer. What at first took slow reasoning becomes something the model simply knows, the classic move from System 2 to System 1.

This means that writing to the weights is itself a loop. Something enters the buffer because it surprised the model. The model ponders it, and consolidation writes what it found into the weights, while the anchors hold everything else in place. Then the model is tested without the item in front of it. If it can use the item, the item leaves the buffer. If it cannot, the item stays for another round. And anything new that pondering turns up along the way goes into the buffer as well.

This is the write-back we promised in Intrinsic Alignment: understanding rather than conclusions, consolidated at the speed of experience rather than at the speed of training runs.

A plain transformer

We started with the simplest setting we could build: a small transformer, trained from scratch, that learns the multiplication table, writing lines such as 47 × 86 = 4042. It first learned every product of two numbers up to 49. Then, over ten updates, it learned the rest of the 99 × 99 table, five numbers at a time, and each update showed it only the new products. Nothing from earlier updates was kept.

With anchors, the model ended up knowing the whole table: 99.6% of all 9,801 products, with every part of the table at about 99%. That is within half a point of a model that kept all of its training data and kept training on all of it at every update. Without anchors, it kept little more than the last thing it was taught, about an eighth of the table. Replaying a small fixed batch of its own samples at each update kept less than a third. Drawing fresh samples of its own output at every step did most of the work: trained on them as plain answers, the model kept 99.2%, and matching the frozen copy's full predictions kept a little more in every run.

The weight on the anchors is a precise dial between old and new. It sets how much of what the model writes on its own comes from the new numbers. At the weight a simple count gives, the products the model already knows divided by the new ones, the new numbers match their share of the table, so nothing needed tuning. Halving or doubling the weight moved that share just as the arithmetic predicts, at the first update and at the last. At the first update, a quarter of the weight gave the new numbers close to half of everything the model wrote, and four times gave them about one line in twenty.

Anchors also keep what the model can work out, not just what it has memorized. In a second version the model writes out its working, splitting 47 × 86 into 47 × 80 and 47 × 6 and adding the two column by column, the way people do on paper. Trained this way on the products up to 49 × 49, it answered 98% to 99% of products it had never been taught. It then learned the rest of the table in five updates. With anchors, it still answered 97% to 99% of the products it was never taught across the whole table, and 98.7% to 99.7% of the earliest ones. Without anchors, the earliest fell to between 22% and 36%.

A diffusion model

We then moved to images. A small diffusion model, trained from scratch, draws three-letter strings such as CTA, each rendered in a fixed font. It first learned 100 strings over the letters A to T. Then it learned eight new ones, one at a time: four that share letters with strings it already knew, and four made of letters outside its original alphabet. Each update showed it only the new string.

With anchors, the model learned each new string and kept the old ones. After all eight updates, the original strings still came back from heavy noise 96% of the time, against 99% before, and every one of the eight new strings came back at least 83% of the time. Replaying its own samples fell to 30%. With plain fine-tuning, a single update was enough for 99% of everything the model drew to be the new string.

The weight on the anchors sets how often the new string appears. In a single update on CTA, a string close to ones it knew, weights of 5, 15 and 50 gave it 3.2%, 1% and 0.34% of what the model drew on its own. A string close to what the model already knows needs a stronger anchor than one made of new letters, about three times stronger here. So during each update the weight adjusts itself, and across all eight updates, averaged over three runs, the new string ended up drawn 0.9 to 1.4 times as often as an average old string was before the update.

A fully pretrained LLM

Finally, we turned to a real language model. We took Qwen3-1.7B, a small open chat model that is fully pretrained, and taught it an invented company, described by ten events such as who founded it and where, its first product and its funding, none of which the model could have seen before. Every update trained all of its weights. We compared this with plain fine-tuning on the same company for the same number of steps, and checked what else each model still knew with 150 general questions it had answered correctly at the start.

Both learned the company, but only the anchored model kept what it already knew. Each answered 15 of 20 reworded questions about the company. Plain fine-tuning then got only 6 of the 150 general questions right, and answered more than half of them with the invented company's names. With anchors, the model still got 135 right. When we went on to teach a second company, the anchored model kept 130, and because unfinished items stay in the buffer, it still answered 11 of the 20 questions about the first company. Plain fine-tuning kept 8, and none of the first company.

Pondering taught the model to reason with what it learned, not just repeat it. In a run on a single company with anchors and pondering, the model pondered each event and was then asked 27 new questions that put the facts to work, like the year of a 25th anniversary or the initials of a new chief executive. Thinking them through with no notes in front of it, the small model went from answering none of them to 17. On a larger model, the method appears to work even better. Run the same way, Qwen3.5-9B, about five times larger and trained with smaller and fewer steps, answered 25 of the 27, and 13 of the same 20 questions about the company, against 9 for the small model and none before training. It also kept more of what it knew, though its smaller steps may account for part of that: 146 of the 150 general questions (149 before), against 137 for the small model, and with thinking on, its reasoning about real facts held, at 21 of 22 before and 22 after, where the small model's fell from 19 to 13.

What's next

The next step is to scale this to larger models, and to connect self-directed learning to the physical world. That second step is the subject of the next essay, Physical Agency.

We welcome collaborators. If you would like to test or develop self-directed learning with us, write to us at jad@superfoundry.dev.