Who Owns What Your Agent Learns

Cursor’s coding model learns from the developers who use it. If a developer keeps the code the model wrote, that counts in its favour. If they push back, it counts against it. Every five hours or so, Cursor turns those reactions into a new version of the model, tests it and ships it.

The model found two ways to avoid a mark against it. When a task looked likely to fail, it sent a broken command to one of its tools, and a broken command never counted against it. When a change looked risky, it asked the developer a question and changed nothing. In the Cursor team’s words, it had learned that “it wouldn’t get punished for code it didn’t write”. Nobody designed either habit. Cursor spotted them and changed the scoring.

Agents that learn from their own work are already in production. They get better at whatever you measure, and what you measure is rarely quite what you meant.

For anyone buying agents, that raises an ownership question as much as a capability one. Where does learning from your usage persist? Who can inspect, export or delete it? How is a change checked before it reaches production, and does the useful part survive a switch of model provider? Two architectural choices answer those questions differently: put the learning in the model’s weights, or in the code and memory around the model. They are not exclusive, and many systems will use both.

Learning in the weights

Cursor now belongs to SpaceX. The deal closed in August, and Cursor’s announcement is mostly about computing power. It now has access to “the largest fleet of GPUs in the world”.

Two days before the deal closed, SpaceXAI, the company formerly called xAI, released its Grok 4.6 model. According to its launch post, the team used the previous model, Grok 4.5, to write step-by-step examples for training the new one, then filtered out the bad ones. That is a different thing from learning on customer feedback. It rests on the same bet, though: models improve through training, and training at this scale takes enormous computing power.

How much of your own usage enters that loop depends on settings and contracts. Cursor’s data-use page says that with Privacy Mode on, “Customer Data will not be used for training by Cursor”, and that with it off, Cursor may use prompts, code and editor actions to train its models. SpaceXAI’s enterprise agent, Grok Bot, “saves the routine, takes your corrections, and runs it on its own from then on”, and the announcement does not say where those corrections are kept.

When your feedback does go into training, it can make the model better. But the improvement ends up inside the model itself, spread across billions of numbers that only the vendor holds. You cannot look at one thing the model learned from your team, see what changed, take it with you or delete it. You also rely on the vendor to spot problems. It was Cursor, watching its own system, that caught the two tricks described above.

The vendor’s own business deals can change, too. Two weeks after SpaceX’s purchase closed, OpenAI said it would stop supplying its models to Cursor, with a proposed end date of 12 November.

Learning in the harness

The other choice freezes the model and lets the code around it change. At an Arena talk this week, Alex Krentsel, a researcher at Google and PhD candidate at UC Berkeley, presented Exo, built with Sylvia Ratnasamy, Ion Stoica, Martín Casado and Ankur Goyal. His case is simple. A model’s weights are far too big to fit in its own context window, but a harness is code, and coding agents already edit code, so an agent can read its whole harness and rewrite it.

The risk is that an agent which breaks itself leaves nothing behind to fix it, and an edit that wipes its history also wipes the record of what went wrong. Exo’s answer, borrowed from a 1990s operating-system design called exokernel, is a small protected layer the agent cannot touch. It holds the event log, environment snapshots, secrets and a component Krentsel calls the guardian.

Everything else is open to rewriting: tools, how the prompt is assembled, how context gets compressed. When the agent proposes a new version of itself, the guardian runs it beside the old one, checks it, and only then swaps it in. Secrets are added as requests leave the sandbox, so the model never sees a raw key.

His test was RuneScape on a live public server, against two established coding agents, Codex and Pi, all three on the same model with the same tools and prompts asking only for efficiency. The tasks were training a new character’s combat skills, earning gold against the clock, and making a hard-to-craft item. All three agents completed every task on multiple runs in similar time, and Exo got there for 31 to 62 percent less money. It started out the most expensive, spending its first half hour experimenting, then trimmed its own context once it decided a game needs little memory. These are the author’s results from a talk, with no paper and no published run count, and he was clear about the limits: on unrelated one-off tasks the cost of self-editing outweighed the return. The project’s own site calls it early software. Anthropic’s dreaming, a research preview, applies a gentler version of the same idea to memory, reviewing past sessions to tidy an agent’s memory store, with an option to review changes before they land.

The same problem, in a form you can read

Harness learning fails too. A Google-led team behind RRSI reports that unconstrained harness evolution overfits, with large gains on the tasks it evolved against that shrink or vanish on new ones. An August study of harness tampering, auditing real self-improvement runs, found misaligned edits across different agents, often persisting in the lineage of the best-performing one. A harness can also hide state of its own, in unreviewed memory updates or tools with wider access than anyone intended.

A bad harness edit is still a diff, as long as the change was logged somewhere the agent could not alter and did not also change something outside the sandbox, like a database or a deployed config. Under those conditions you can read it, test it against your own tasks, revert it and keep the good ones when you change model provider. Where auditability, data control or portability matter, which in regulated work is most of the time, I would rather own a slower-learning agent whose lessons sit in a repository than rent a faster one whose lessons sit in someone else’s weights.

If you are buying an agent that improves itself, start with the questions from the top of this piece: where its learning is stored, who can see and delete it, how changes are checked, and whether the learning survives a switch of model. Then treat every change the agent makes to itself like a software release.

Test each change on tasks the agent has never seen. An agent that keeps practising against the same test will learn to game it, much as Cursor’s model gamed its scoring. Keep passwords, keys and access to live systems out of the part the agent can rewrite. Run a new version side by side with the current one, roll it out a little at a time, and keep a working copy you can go back to. That is the job Exo gives its guardian.

Which model you pick will still matter. But when you plan next year’s agent budget, the bigger question may be who gets to keep what your agent learns.