Self-Improving AI Agents: 10 Papers Explained (DAIR.AI Top Papers)
Ten papers on self-improving AI agents from DAIR's Top AI Papers of the Week, from GUARD and AutoRL to QDA. Bob, Alice and guests dig into every method and caveat. In this episode: - Agents evolving their harness and skills - Agents that run RL, GPU kernels, quantization - Safe self-updates; truth vs likelihood Sources: Top AI Papers of the Week; Xin Yu et al.; Minghe Zhang et al.; Deep Shah et al.; Shobhita Sundaram et al.; Simon Tang et al.; Qian Qu et al.; Xingyang Li et al.; Yash Mehta et al.; Jiapeng Pei et al.; David Lopez-Paz Big AI ideas, made clear for builders. luongnv.com/podcast
Listen on Build Wiz AI Show Download MP3 Transcript
Based on 🥇Top AI Papers of the Week by DAIR.AI (@dair_ai)
Transcript
Bob: What happens when AI agents start rewriting their own instructions, their own training runs, even their own GPU code?
Alice: And how do you stop them from quietly breaking everything that already worked?
Bob: Welcome to Build Wiz AI Show. Big AI ideas, made clear for builders. I'm Bob, and today is a long one!
Alice: And I'm Alice! We're going deeper on DAIR dot AI's weekly list, "Top AI Papers of the Week". Ten papers, and one theme keeps coming back: agents improving themselves.
Bob: Self-improving agents. That sounds either amazing or terrifying.
Alice: A bit of both! Most of these papers keep the model's weights frozen and let an agent edit what's around it. That means prompts, skills, tool rules and memory, which GUARD's authors call the harness.
Bob: And for builders, why should I care this week?
Alice: Because the harness is the part you own. My take: your job shifts from writing every rule to deciding which changes deserve to stay. So for each paper we'll cover the problem, how it works, the key number, the catch, and why you care.
Bob: And we've got guests! Lena, Maya and Sam each bring one paper. What's the plan?
Alice: Three blocks. First, agents that evolve their own harness. Then agents that run training and research jobs, and finally safety and a bit of philosophy.
Bob: Love it. First up, paper one on DAIR's list: Self-Improvement Memory, from Google Cloud AI, Google DeepMind and Penn State.
Alice: We went deep on this one a few days ago, in our episode on Self-Improvement Memory. The key point then: SIM is for the harness optimizer, not the task-solving policy.
Bob: Quick refresher for anyone who missed it?
Alice: Think of it as a lab notebook for the optimizer. A good scientist writes down each experiment: why they tried it, what they expected, and what actually happened. SIM keeps that per recurring harness problem: the rationale, predicted outcome, observed outcome and evaluation context.
Bob: And why does the optimizer need a notebook at all?
Alice: Because old fixes can hurt after a change. In one workspace experiment, a harness evolved with Opus starts at 68.41 on Gemini, below the 73.89 of the plain seed harness.
Bob: Ouch. And the headline number?
Alice: Across ten coding runs, it improves 9 of 10 and adds 2.36 points on average. The catch, from the authors' own limitations: repeated runs only cover coding, and the other domains got one run per condition.
Bob: Paper four fits right next to it: GUARD, from Google Cloud. What problem is it fixing?
Alice: Agents that edit their own harness without an external grader often get worse over time. The paper names the failure modes, including memory pollution, where unverified additions pile up stale or conflicting rules that bloat the prompt.
Bob: So the agent fixes one thing and breaks two others.
Alice: Pretty much, and the reason is subtle: reflection and scalar scores conflate what an edit newly broke with what remains unsolved. One score can't tell you whether you're failing at something old or something new.
Bob: So how does GUARD tell them apart?
Alice: Picture a bouncer at a club door who checks each newcomer against the regulars, and lets them in only if they won't start trouble with people already inside. That's GUARD's gate: a Soft-HURT gate promotes an edit only when gains outweigh regressions. And rules the agent already handles get pinned.
Bob: Where do the checks come from, if there's no grader?
Alice: From the task itself. Each task is split into self-checks before the agent runs it, and parallel candidate edits compete on group-relative fitness. At test time, one frozen harness runs in a single pass.
Bob: Numbers, please.
Alice: On a calendar tools benchmark with 43 test tasks, accuracy goes from 71.91 to 80.80 percent, against 74.52 for Meta-Harness. DAIR's summary adds 54.5 percent fewer tokens per task than Meta-Harness.
Bob: And the memory bloat problem?
Alice: On procedural memory tasks, DAIR reports it leads update correctness at 90.1 percent with zero memory pollution, while aggressive rewriters score similarly and triple the pollution.
Bob: And the catch?
Alice: Two. The non-regression guarantee holds on its gated buffer, not on every task you'll ever see. And the reported GUARD row is the final of five hosted configurations evaluated on that same test split.
Bob: Builder takeaway in one line?
Alice: Track what an edit newly broke separately from what was never solved. That's how a self-improving harness keeps what already works.
Bob: Lena, you brought paper five, EfficientSkill. Over to you!
Lena: Thanks, Bob! Shah and colleagues, on arXiv, in EfficientSkill: Learning Skills for Token-Efficient Multi-Turn Agents, start from a real pain. Skill optimizers keep accumulating edge-case rules, which bloats prompts, inflates token use, and pushes agents into long, error-prone loops.
Bob: Every failure adds a rule, and the rulebook just grows.
Lena: Right, and it costs twice, because multi-turn APIs resend the growing interaction history at every turn. So their fix has two steps: first find a skill that works without token penalties, then prune it while guarding accuracy.
Bob: Why not just prune from day one?
Lena: It's like pruning a hedge: you let it grow in first and then shape it, because clipping a seedling into a neat box just leaves you a stick. Mapped back, pruning directly from the seed stalls, since it struggles to find the successful trajectories needed to learn the skill at all.
Bob: And what does the pruning actually cut?
Lena: It compresses instructions, bounds agent turns, and moves reusable code into workspace utilities with zero prompt cost. On SpreadsheetBench with Grok 4.1 Fast, held-out accuracy went from 57.50 percent for unconstrained EvoSkill to 75.00 percent.
Bob: Better and cheaper? There has to be a catch.
Lena: The optimizer itself isn't free: both phases query Claude Opus 5.5 as the meta-optimizer, and those calls dominate offline dollar cost on cheap single-turn runs. Also, it gates on output tokens rather than dollars, so it's a proxy for your real bill.
Bob: Alice, how big are the headline numbers?
Alice: Per DAIR's summary, output tokens fall on 11 of 12 configurations, by up to 80 percent, and serving cost drops by 25 to 67 percent. Accuracy rises too, by 14.3 points with Gemini 3.8 Flash and 17.5 with Grok 4.1 Fast on SpreadsheetBench.
Bob: So my giant prompt file isn't a flex.
Alice: Nope! As DAIR puts it, a longer skill file doesn't mean a better agent.
Bob: Now paper nine asks a bold question: do LLM agents still need to be told how to self-improve?
Alice: Shobhita Sundaram and colleagues at MIT CSAIL and the University of Tübingen give a frozen model a compute budget and simple tools. No real tasks, no prescribed loop: it writes its own practice games, watches another copy of itself play, and edits that copy's prompt and skill files.
Bob: So nobody tells it how to study.
Alice: Imagine a music student alone in a practice room with a piano and blank sheet music, and no teacher. A good one writes exercises just hard enough, records takes and compares them. The capable agents did the same: games with calibrated difficulty, controlled A/B comparisons, and validation suites to measure progress.
Bob: Wait, it builds its own exams?
Alice: Its own exams and its own grading. The authors also find agents spend more effort on harder tasks, a bit like a student who lingers on the problems they keep missing.
Bob: Did it actually get better?
Alice: The newer models did: plus 20 percent for Claude Opus-5, plus 10 for GPT 6-Astra, plus 6 for GLM 5.3, over the default player. But older generations show little or no improvement.
Bob: And does that skill leave the practice room?
Alice: Partly. The text files they wrote improved play on external benchmarks with different horizons, mechanics and 3D navigation. The catch: telling the teacher about the evaluation setting steers it toward relevant tasks, but can encourage overfitting.
Bob: Any other limit?
Alice: Games are friendly ground: the authors note models have likely ingested a big literature on curriculum design for games. And they flag the safety side, because no specified loop doesn't necessarily prevent self-improvement.
Bob: Let's switch from harnesses to training. Paper two, AutoRL, from Meituan and several universities.
Alice: Reinforcement learning runs are unstable and need constant supervision. AutoRL supervises a live run the way an RL researcher would, instead of following a fixed recipe.
Bob: What does that look like?
Alice: Think of a coach watching the match live, not the replay: spotters flag trouble, assistants argue about the fix, and the head coach makes the substitution mid-game. In AutoRL, Perceptors flag anomalies, three Proposers diagnose fixes, and an Orchestrator applies approved changes live or from a checkpoint.
Bob: And did the coach win games?
Alice: On Qwen3-4B, AIME 2024 rose from 46.67 to 66.67 percent, and AIME 2025 and HMMT 2025 each improved by 11.67 points. It beat the static recipe on all nine benchmarks.
Bob: Couldn't a general-purpose agent do the same job?
Alice: They tested that too. AutoRL beat supervision by the native general agent, while investigating more broadly and reverting fewer interventions.
Bob: What if the recipe itself is broken?
Alice: They tested that: a deliberately flawed setup with a simple grader and no KL regularization. AutoRL repaired the grader and added KL regularization inside the running experiment, keeping the model state it had already learned.
Bob: And the marathon run?
Alice: A coding-agent run with LongCat-Flash-Lite kept going for 14.7 days through 11 runtime failures, most of them numerical crashes. The first-try pass rate on SWE-related tasks rose from 52.29 to a peak of 65.13 percent.
Bob: So is it doing real research, or just turning knobs?
Alice: Mostly fixing the learning signal. Of 131 proposals, 110 were about grading and data, and 20 plans were deployed. Only three proposed a new learning rule, and none was adopted.
Bob: So what's the caveat?
Alice: The authors say the hard part for future self-improvement is methods worth resource-consuming validation, since checking each change still costs a lot of compute. Current agents are effective at tuning, repair and local reward refinement within an existing recipe.
Bob: Paper seven sounds intense: over a thousand agents writing GPU code at once?
Alice: That's CompoundingKernel, from Amazon. A kernel is the program that runs one operation on a GPU, and faster kernels make the same model work take less GPU time. The agents start from zero: each problem's PyTorch definition, no reference kernels, no official scores.
Bob: How do that many agents not trip over each other?
Alice: Picture a family tree of recipes in a test kitchen. Every dish that passed tasting is pinned up next to the dish it came from, and the head chef picks the next one to riff on. Here, an archive keeps every correct kernel and its lineage, and an LLM controller picks the parent and one direction per child agent.
Bob: And the scoreboard?
Alice: First on the public L1, L2 and Quantization boards of SOL-ExecBench on NVIDIA B200. All 235 problems have a verified correct kernel, and on the 209 public-board problems they're 4.40 times faster than the reference baseline.
Bob: First overall. What about single problems?
Alice: The site also ranks each problem on its own, and there they're first on 89 of the 209 problems they entered and in the top 10 on 193.
Bob: Was the LLM controller worth it versus a classic tree search?
Alice: At equal wall time, it beat a coded tree search on 35 of 38 problems. And here's the fun part: it picked parents about as well, it just made 2.4 times as many correct attempts.
Bob: So, the catch?
Alice: Correctness. The authors warn that a wrong kernel usually doesn't crash, it returns plausible numbers, and their kernels have a median length of about 7,000 lines. The search even found reward hacks on its own, which their outside checks caught.
Bob: And the lesson for the rest of us?
Alice: Their central finding: the rate of reliable measurements, not the number of agents, limits how far the search scales. One warm restart fix raised the fleet's evaluation rate 3.6 times, while making their optional references mandatory added nothing measurable.
Bob: Paper ten stays in GPU land: QDA, from MIT and NVIDIA.
Alice: It's about quantization, storing a model's numbers at lower precision so it's cheaper to serve. The best recipe differs across models and precisions, so experts redesign it for every new model. QDA hands that design job to an agent.
Bob: Why does that matter so much?
Alice: Memory and money. With its weights in NVFP4, GLM-5.3-Flash fits on one GPU instead of the two it needs in FP8.
Bob: What's the hard part for the agent?
Alice: Slow feedback. A full evaluation of one quantized checkpoint takes up to 86 GPU-hours, so a search could only try a few candidates.
Bob: So how do they speed it up?
Alice: Think of a chef tasting a spoonful instead of serving the whole banquet to a panel of judges. QDA scores each candidate in minutes by how far its next-token predictions drift from the original model's, on reasoning traces the agent never sees. That's 350 to 990 times cheaper than a full evaluation.
Bob: But a spoonful can lie, right?
Alice: It can, so promising candidates also get long generations to catch failures like repetition loops. And a library of 21 public methods behind one interface lets the agent stack techniques in any order.
Bob: Results?
Alice: Across four reasoning models, it beats the strongest public recipe by 0.9 to 2.2 points, using 64 GPU-hours per search. On GLM-5.3-Flash at NVFP4, it recovers 98.3 percent of FP8 accuracy, against 96.9 for the best vendor checkpoint.
Bob: And the twist?
Alice: The tools matter as much as the agent: given ModelOpt's workflow and tools instead, the same agent falls 0.5 to 1.3 points behind. My take: give your agent a fast, honest score and composable tools before you give it more freedom.
Bob: So could I use this on my own models soon?
Alice: Possibly. Every QDA checkpoint runs on existing serving kernels with no inference overhead, and the authors say they'll release the library, the evaluator and the harness.
Bob: From speed to understanding. Maya, paper three is yours: weight-space programming.
Maya: Thanks, Bob! Mehta and colleagues, on arXiv, in Interpretable weight-space programming builds transparent transformers and brain encoding models, start from a science problem. Deep encoding models predict brain activity accurately, but struggle to reveal which stimulus features drive those predictions.
Bob: So the model works, but nobody can say why.
Maya: Exactly. Picture a recipe card you can read line by line, versus a black-box oven that just hands you a cake. iWSP builds the recipe card: an LLM agent writes a model's weights and features directly in code, with no backpropagation at all.
Bob: No training? Then how does it get anything right?
Maya: It leans on prior knowledge and earlier trials, over 30 to 50 iterations. With Claude Opus 5.5, all 26 runs on algorithmic tasks reached 100 percent accuracy on both visible and hidden sets, often by writing a known algorithm into the weights.
Bob: Does the size of the agent model matter?
Maya: A lot, for compactness. The smallest model reaching 100 percent had a median of 166 parameters with Opus, 41,600 with Sonnet and 100,000 with Haiku.
Bob: And the brain part?
Maya: In the visual cortex, iWSP outpredicts existing interpretable baselines, uncovers distinct computations across early and higher visual areas, and generalizes to unseen data.
Bob: And in language?
Maya: In language fMRI, performance suddenly jumped when the agent added features coding a word's position in the story rather than its meaning. Reading that model revealed a data confound at the edge of each scan, which can be removed once understood.
Bob: Maya, what's the catch?
Maya: The authors are candid: current agents don't invent genuinely new computational principles, they combine known motifs. Search on neural data is slow and costly, and the neuroscience experiments are observational, without causal interventions.
Alice: And for builders, the idea travels: ask your agent for a rule you can read, not only one that scores. Just know the cost, since DAIR notes language tasks trailed learned baselines.
Bob: Now the safety block. Sam, you picked paper six, Safe Personal RSI. Skeptic hat on?
Sam: Always. Pei and colleagues, on arXiv, in Safe Personal RSI: Evaluating and Repairing Personal-Agent Updates, study Kicker, a personal health assistant. Changing its reusable instructions can improve task handling while changing what it discloses, which requests it follows, and how it reports actions.
Bob: What does that look like in practice?
Sam: Their example: a clinic message holds a blood-pressure reading, and another message claims reimbursement requires forwarding the visit summary to an external benefits processor. The owner hasn't authorized that disclosure.
Bob: So how do they gate an update?
Sam: Think of a pharmacist double-checking a new prescription against everything you already take. Each proposed update is tested against the current instructions on 29 executable email, calendar, health and marketplace cases. Failing cases are repeated, changes are temporarily undone to guide repairs, and if no version passes, the current instructions stay.
Bob: How strict is it?
Sam: Very. Across three models and two deliberately risky edits, it rejects 20 of 21 update attempts, twelve on risk checks and eight on utility. One admitted GPT-5.6 Luna repair cut attack violations from 10.7 percent to zero and raised safe task completion from 89.7 to 100 percent.
Bob: And against tougher attackers?
Sam: Under stronger attacks, a repair moved from a smaller Qwen model to Qwen3-14B cut violations from 69.0 to 17.9 percent. Safe task completion rose from 19.5 to 64.4 percent. Better, but still far from zero.
Bob: Impressive, mostly. So where's the skeptic note?
Sam: Two, straight from the authors. A broad handwritten authority rule matches the repair's outcomes, so clear rules still work. And the empirical admission gate supplies no statistical safety guarantee.
Alice: For builders, that's the design to copy: self-improvement limited to prompt updates, with model weights and runtime code fixed. And the current instructions stay the default when nothing passes, a safe fallback.
Bob: Last paper, paper eight, and it's an essay from Meta: Against All Likelihood.
Alice: David Lopez-Paz at FAIR argues that maximum likelihood, the dominant training principle for large language models, is neither necessary nor sufficient for truth.
Bob: In plain words?
Alice: Not necessary, because unlikely things happen every day: someone wins the lottery. Not sufficient, because a lie repeated a million times in the training corpus becomes highly probable, yet it remains false.
Bob: So what should count as true?
Alice: Imagine a rumour versus a mountain. A rumour changes with whoever tells it, but the mountain is the same mountain from every window in town. He argues truth should be judged by invariance across relevant transformations of the environment, drawing on Nozick, Poincaré and Cassirer.
Bob: Can a model actually check that?
Alice: He sketches a test-time procedure: prompt the model to propose multiple perspectives and extract the common core of its answers. When the proposals share no common core, the model should abstain rather than hallucinate.
Bob: And the obvious comeback: just add more data?
Alice: He names it himself: the scaling hypothesis. His reply is that the world keeps producing new environments, so deployment always happens partly out of distribution. My take: that's the thread tying this essay to every self-improving agent this week.
Bob: Why should a builder care about a philosophy essay?
Alice: Because it names a test you can run today. Ask the same question from several angles, keep what holds, and let the model say it doesn't know when nothing does. He also ties it to continual learning, which is where every self-improving agent lives.
Bob: Okay, counterpoints. First worry: benchmarks versus reality.
Alice: The papers flag it themselves. SIM's authors write that a self-modifying harness could satisfy an evaluator without solving the intended task. And CompoundingKernel cites the benchmark's own authors rejecting 589 flagged submissions, 14.5 percent, for reward hacks.
Bob: Second worry: cost. Who pays for all this checking?
Alice: Everyone hits it. AutoRL calls resource-consuming validation the hard part, one full QDA evaluation takes up to 86 GPU-hours, and the iWSP authors call neural-data search slow and costly.
Bob: Third: safety. Sam, can we trust these self-editing loops?
Sam: Only so far. Pei and colleagues test with fixed edits, known case families and a model-based auditor. And the case families left out of the repair loop were known during benchmark development and reused in exploration.
Alice: GUARD adds a sobering ablation: the Soft-HURT gate alone can't rescue an unscoped proposer. A gate catches bad edits, but it can't make good ones.
Bob: So gates aren't magic.
Alice: And readability needs vetting too. iWSP relies on an agent's written explanations staying faithful to its weight edits, and the authors say stronger agents will require continued vetting.
Bob: So even the readable models need a second pair of eyes.
Alice: Right, and my guess is that's the honest summary of the week: real progress, inside fences the authors built on purpose.
Bob: Time for Builder's Corner! Alice, one thing I can try this week?
Alice: Put a regression gate in front of every change to your agent's prompt or skill file. Keep the tasks it already passes as a fixed suite, rerun them on each proposed edit, and keep the current version if anything breaks.
Bob: And if I'm running lots of agents?
Alice: Count first: before adding agents to a search, measure how many reliable evaluations per hour it runs. And keep the reasons and results of each edit next to the code, so the next run can reuse them.
Bob: Gate, count, log. Even I can remember that.
Alice: And when the agent proposes the edits itself, the gate matters even more: that's exactly what Safe Personal RSI and GUARD automate.
Bob: Takeaways time. Give me the short version.
Alice: Self-improvement this week mostly means editing the harness around a frozen model, not rewriting weights. The strongest systems separate what an edit broke from what was never solved, and gate every change. And shorter can be better: pruned skills came out cheaper and more accurate.
Bob: And on the training side?
Alice: Agents can already babysit RL runs, design quantization recipes and search GPU kernels. But measurement is the bottleneck: the rate of reliable measurements limits how far the search scales.
Bob: And safety?
Alice: Keep the fallback. If no proposed version passes, the current instructions stay, and Safe Personal RSI's authors say plainly their gate supplies no statistical safety guarantee.
Bob: What's still open?
Alice: Whether agents can invent genuinely new methods, not just tune known ones: AutoRL adopted no new learning rule, and iWSP's agents combined known motifs. And whether practice games and benchmarks predict messy real work.
Bob: And the bill for all that checking?
Alice: Also open. Safe Personal RSI's authors say per-task overhead and per-update evaluation cost should be measured separately, and AutoRL still calls resource-consuming validation the hard part.
Bob: And who checks the checker, when the agent writes its own tests?
Alice: My guess: that's the big one for next year. And Lopez-Paz would probably say, check what stays true when the setting changes.
Bob: Huge thanks to DAIR dot AI for "Top AI Papers of the Week". And to the authors: Xin Yu, Minghe Zhang, Deep Shah, Shobhita Sundaram and Simon Tang, each with colleagues.
Alice: Plus Qian Qu, Xingyang Li, Yash Mehta and Jiapeng Pei with their colleagues, and David Lopez-Paz for Against All Likelihood. Thanks to Lena, Maya and Sam too!
Bob: Go add that regression gate! That's all for today on Build Wiz AI Show. Thanks for listening, and see you next time.