WEBVTT

1
00:00:00.000 --> 00:00:05.419
<v Bob>11,000 pull requests in one month. Can that be anything but slop?

2
00:00:05.719 --> 00:00:11.776
<v Alice>One team says yes. And the share of those pull requests that get reverted? About half a percent.

3
00:00:12.076 --> 00:00:13.327
<v Bob>So what's the trick?

4
00:00:17.627 --> 00:00:23.510
<v Bob>Welcome to Build Wiz AI Show. Big AI ideas, made clear for builders. I'm Bob.

5
00:00:23.810 --> 00:00:35.278
<v Alice>And I'm Alice! Today's source is a talk called All Code Is Slop, Except Mine, by Lauren Tan, on the AI Engineer channel. She works on the Grok Bot team at SpaceXSI.

6
00:00:35.578 --> 00:00:39.766
<v Bob>I love that title. Everyone else's code is slop. Mine is art.

7
00:00:40.066 --> 00:00:56.299
<v Alice>Her thesis, plainly: the engineer's job now is to keep, or even improve, code quality while massively increasing productivity. The way there is trust. Not blind trust, but investing in your agents' environment so they do the right thing.

8
00:00:56.599 --> 00:00:58.004
<v Bob>Where does she start?

9
00:00:58.304 --> 00:01:14.681
<v Alice>Last month she shipped 3,000 pull requests to production, herself. Before agents, only outliers like Bret Taylor, who famously rewrote Google Maps in a weekend, worked at that pace. With agents, she says, anyone can be a coding machine.

10
00:01:14.981 --> 00:01:18.229
<v Bob>Okay. So where do you actually spend the effort?

11
00:01:18.529 --> 00:01:32.807
<v Alice>Picture an upside-down pyramid. The two biggest layers are deterministic: your code itself, then lint rules, type systems, compilers and tests. Next comes verification, to check that the agent understood your intent.

12
00:01:33.107 --> 00:01:35.542
<v Bob>And the smallest layer, at the tip?

13
00:01:35.842 --> 00:01:45.138
<v Alice>Prose. Rules, skills, your AGENTS.md file, the things that need judgment. Just a little of it, at the very end.

14
00:01:45.438 --> 00:01:48.229
<v Bob>Can you make that concrete for a total beginner?

15
00:01:48.529 --> 00:02:02.423
<v Alice>It's like a kitchen: a good layout and sharp knives do most of the work, tasting the dish is the check, and the recipe note on the fridge comes last. In software, the deterministic layers come first and written instructions last.

16
00:02:02.723 --> 00:02:07.155
<v Bob>Guilty. When my agent messes up, I just write a longer note for the fridge.

17
00:02:07.455 --> 00:02:18.749
<v Alice>And that's her key rule! Whenever you correct your agent, make it a deterministic check instead of another line in your rules. In code it's far more reproducible, so you can trust it more.

18
00:02:19.049 --> 00:02:21.948
<v Bob>Why does the codebase itself matter that much?

19
00:02:22.248 --> 00:02:38.226
<v Alice>Because agents copy it. Her image is a garden: code arrives like a continuous stream, and any workaround gets copied by the next agent because it looks like the established pattern. Weeds spread, and in code one bad pattern spreads everywhere.

20
00:02:38.526 --> 00:02:42.208
<v Bob>So keep the garden pristine, and agents copy the good stuff.

21
00:02:42.508 --> 00:02:45.884
<v Bob>Real examples, please. What did her team build?

22
00:02:46.184 --> 00:02:57.303
<v Alice>A client framework called Dune, a bit like Next.js for Electron apps. Its secret is the constraints and lint rules, which encode their lessons from building desktop apps.

23
00:02:57.603 --> 00:02:58.912
<v Bob>What kind of lesson?

24
00:02:59.212 --> 00:03:11.144
<v Alice>That anything needing nuance to use well often leads to slop. Her classic example is useEffect in React: it has its use cases, but it's often misused, and agents aren't good at it.

25
00:03:11.444 --> 00:03:14.901
<v Bob>The famous one. People joke you need a PhD to use it.

26
00:03:15.201 --> 00:03:25.891
<v Alice>So they banned it in product code, allowed only inside some of Dune's own APIs. The problem is categorically eliminated, and nobody has to catch it in code review again.

27
00:03:26.191 --> 00:03:37.438
<v Alice>It's like childproofing a kitchen: you don't explain the stove to a toddler, you put a lock on the knob. In code, you ban the risky tool in product code, so the agent can't misuse it.

28
00:03:37.738 --> 00:03:41.323
<v Bob>So the agent is the toddler. Rude, but fair.

29
00:03:41.623 --> 00:03:44.998
<v Bob>Then there's her take on comments. What's wrong with comments?

30
00:03:45.298 --> 00:04:01.878
<v Alice>Agents use comments to justify not doing the right thing. Her example, from Cursor: agents loved imperative timers, like setTimeout, as quick fixes, patching over a clash between Visual Studio Code's imperative style and React's declarative style.

31
00:04:02.178 --> 00:04:03.792
<v Bob>And timers are bad because?

32
00:04:04.092 --> 00:04:17.081
<v Alice>Tons of timers created and destroyed means major garbage collection, and lag. Worse, agents wrote comments justifying each quick fix, timers spread everywhere, and performance really suffered.

33
00:04:17.381 --> 00:04:20.583
<v Bob>So the comment was a permission slip.

34
00:04:20.883 --> 00:04:30.388
<v Alice>Exactly. So in Grok Bot they banned 99 percent of comments. They go stale, they hand the agent an escape hatch, and they aren't enforceable.

35
00:04:30.688 --> 00:04:33.982
<v Bob>99 percent! So what replaces them?

36
00:04:34.282 --> 00:04:41.060
<v Alice>Lint rules with very helpful error messages, telling the agent to fix the root cause instead of patching around it.

37
00:04:41.360 --> 00:04:56.485
<v Alice>Think of traffic: a comment is a sign saying please slow down, easy to ignore, while a lint rule is a speed bump. Nobody gets past without slowing. In code, comments aren't enforceable, and the rule's error message says why.

38
00:04:56.785 --> 00:04:59.556
<v Bob>Speed bumps for robots. I like it.

39
00:04:59.856 --> 00:05:08.805
<v Alice>Do it for every correction, and you have less to look out for in review, until there's nothing left to review and agents merge most of their own pull requests.

40
00:05:09.105 --> 00:05:14.020
<v Bob>The verification layer, then. What does code review even look like now?

41
00:05:14.320 --> 00:05:29.143
<v Alice>It's shrunk into looking at verification artifacts: videos, screenshots, diagrams. The basic one is runtime verification, which means the agent runs the code itself, with real debugging tools like the Chrome DevTools Protocol.

42
00:05:29.443 --> 00:05:39.924
<v Alice>And a feature map guides it. Imagine a site map of every feature, from the user's point of view, so the agent doesn't have to guess what you mean by the thingy in the left sidebar.

43
00:05:40.224 --> 00:05:44.447
<v Bob>The thingy in the left sidebar. I say that every single day.

44
00:05:44.747 --> 00:05:58.850
<v Alice>Then formal verification. They use TLA plus to model-check major user flows, like notifications across devices, and Lean to prove algorithms, like checking their custom virtualization library doesn't flicker or jitter.

45
00:05:59.150 --> 00:06:03.473
<v Bob>Runtime, formal... what's the difference, in plain words?

46
00:06:03.773 --> 00:06:21.492
<v Alice>Runtime verification is a test drive around the block. Formal verification is the engineer's math showing the brakes work on any road. In software, the agent runs the code for itself, while a proof shows the code always satisfies its spec, even on edge cases you didn't test.

47
00:06:21.792 --> 00:06:23.809
<v Bob>And does the math find anything?

48
00:06:24.109 --> 00:06:35.459
<v Alice>Quite a few bugs already, she says, but it's not a silver bullet, just another layer. They also do property-based UI testing with a library called Bombadil, which has caught a few UI bugs too.

49
00:06:35.759 --> 00:06:38.369
<v Bob>Lena, this is right up your alley.

50
00:06:38.669 --> 00:06:57.204
<v Lena>It is! Martin Kleppmann's post, Prediction: AI will make formal verification go mainstream, explains why it stayed niche. The seL4 microkernel was 8,700 lines of C, and proving it took 20 person-years and 200,000 lines of Isabelle proof.

51
00:06:57.504 --> 00:07:01.007
<v Bob>20 person-years. For one kernel!

52
00:07:01.307 --> 00:07:14.598
<v Lena>Kleppmann argues LLMs are getting good at writing proof scripts, and hallucinations matter less because the proof checker will reject any invalid proof. It's like a bouncer who checks every ID, so a fake never gets in.

53
00:07:14.898 --> 00:07:16.711
<v Bob>Love it. Any catch?

54
00:07:17.011 --> 00:07:27.237
<v Lena>He's careful: the hard part moves to the specification. How do you know the properties you proved are the ones you cared about? And he notes that Lean is where most of the action is.

55
00:07:27.537 --> 00:07:29.763
<v Bob>Isn't all this checking expensive?

56
00:07:30.063 --> 00:07:41.234
<v Alice>She quotes a developer who once couldn't trust her agents. Six months later, 70 percent of that developer's daily tokens go into verification, and Lauren says most of hers do too.

57
00:07:41.534 --> 00:07:44.735
<v Bob>70 percent on checking, not writing. Huh.

58
00:07:45.035 --> 00:07:59.991
<v Alice>None of these techniques are new, they were just too expensive, and agents cut that cost. It's like a building inspection that suddenly costs almost nothing, so you inspect every floor: a golden age of testing and verification, she says.

59
00:08:00.291 --> 00:08:01.762
<v Bob>So, the payoff?

60
00:08:02.062 --> 00:08:10.465
<v Alice>Last month the Grok Bot team shipped 11,000 pull requests to production, and they now cut a new stable version about once a day.

61
00:08:10.765 --> 00:08:14.322
<v Bob>11,000. Isn't that just chaos?

62
00:08:14.622 --> 00:08:27.672
<v Alice>That's the twist. In October about 227 people contributed at least one pull request, many of them drive-by contributors. Yet the share of PRs that get reverted went down, steady at about half a percent.

63
00:08:27.972 --> 00:08:30.662
<v Bob>Remind me what a revert actually is?

64
00:08:30.962 --> 00:08:40.166
<v Alice>Picture a busy restaurant where almost no plate gets sent back to the kitchen,. In code, a revert is a pull request that gets undone after it shipped.

65
00:08:40.466 --> 00:08:43.760
<v Bob>Maya, how does that square with the wider picture?

66
00:08:44.060 --> 00:08:58.842
<v Maya>It's a sharp contrast. Bill Doerrfeld at LeadDev reports on GitClear and GitKraken research into 623 million real-world code changes from 2023 to 2026. Duplication is up 81 percent.

67
00:08:59.142 --> 00:09:02.088
<v Bob>81 percent more copy-paste. Ouch.

68
00:09:02.388 --> 00:09:11.812
<v Maya>Reuse is down 70 percent, and GitClear found 47 percent more error masking: code that silently catches errors without evaluating their cause.

69
00:09:12.112 --> 00:09:16.382
<v Bob>Wait. That's the timers story again. Patch it, don't fix it.

70
00:09:16.682 --> 00:09:26.287
<v Alice>Exactly what her lint rules go after: fix the root cause, the real problem. It's like pulling weeds out by the roots instead of mowing over them.

71
00:09:26.587 --> 00:09:29.580
<v Bob>So should we all build a software factory tomorrow?

72
00:09:29.880 --> 00:09:43.136
<v Alice>Not yet, she says. First a lint rule to stop new anti-patterns, clean up the old ones, then a strong verification story. If your inner loop produces slop, automating it just produces more slop.

73
00:09:43.436 --> 00:09:54.776
<v Alice>It's like a bakery oven that burns one side of every loaf. Run it faster and you just get more burnt bread. In software, automating a sloppy inner loop just produces more slop.

74
00:09:55.076 --> 00:09:57.275
<v Bob>Burnt bread at scale. Got it.

75
00:09:57.575 --> 00:10:14.291
<v Alice>Once the foundations hold, the outer loop gets easier. Her example: on every release, an agent run uses the app like a real user and finds bugs. It opens a fix PR for each one and auto-merges it, as crazy as that sounds, in her words.

76
00:10:14.591 --> 00:10:19.452
<v Bob>My turn to push back. These are one team's numbers, and they're self-reported, right?

77
00:10:19.752 --> 00:10:32.486
<v Alice>Yes. The 11,000 pull requests and the half-percent revert rate are her team's own figures, and my reading is that the talk shows no outside measurement. It's also one product, an Electron desktop app.

78
00:10:32.786 --> 00:10:40.585
<v Bob>Back to your restaurant: a plate only gets sent back if someone notices. My take: a revert rate only counts the bugs somebody caught.

79
00:10:40.885 --> 00:10:48.602
<v Alice>I think that's fair. The talk doesn't say how many bugs shipped without a revert, and fewer reverts isn't the same as fewer bugs.

80
00:10:48.902 --> 00:10:52.324
<v Bob>Sam, you're our resident skeptic. What worries you?

81
00:10:52.624 --> 00:11:05.613
<v Sam>Security. Jens Wessling at Veracode wrote up their test of over 100 large language models. 45 percent of code samples failed security tests and introduced OWASP Top 10 vulnerabilities.

82
00:11:05.913 --> 00:11:08.987
<v Bob>45 percent? That's almost half.

83
00:11:09.287 --> 00:11:19.849
<v Sam>And newer models weren't better at it. They got better at code that works, not at secure code. Cross-site scripting defenses failed in 86 percent of relevant samples.

84
00:11:20.149 --> 00:11:26.792
<v Alice>And I'd say that's a gap in the talk. Its layers check behavior, and security scanning doesn't come up.

85
00:11:27.092 --> 00:11:30.904
<v Bob>What about auto-merging the fixes the bot found itself?

86
00:11:31.204 --> 00:11:41.940
<v Sam>I think that's the risky part. Wessling says to scan AI-generated code like any new app. A bot merging its own fixes skips the one person who might ask about security.

87
00:11:42.240 --> 00:11:51.700
<v Alice>Fair. Her answer is that the codebase is good enough that pull requests tend to be good by default. My take: that trust is only as good as the checks.

88
00:11:52.000 --> 00:11:55.677
<v Bob>Time for Builder's Corner. Alice, one thing to try this week?

89
00:11:55.977 --> 00:12:08.083
<v Alice>Take the next correction you give your agent and turn it into a lint rule. Write the error message so it names the root cause, the way Grok Bot's rules tell the agent to fix the real problem instead of patching around it.

90
00:12:08.383 --> 00:12:12.653
<v Bob>So the correction outlives the chat. A speed bump, not a sign.

91
00:12:12.953 --> 00:12:19.475
<v Alice>Right. Then clean up the old occurrences too, so the next agent copies the good pattern, not the weed.

92
00:12:19.775 --> 00:12:23.998
<v Bob>Takeaways. One: put the effort into code, not prose.

93
00:12:24.298 --> 00:12:36.183
<v Alice>Yes. Deterministic checks first, verification next, and rules as the thin top layer. Two: ban what agents get wrong instead of explaining it, like useEffect in product code.

94
00:12:36.483 --> 00:12:37.673
<v Bob>What's still open?

95
00:12:37.973 --> 00:12:46.283
<v Alice>Whether this holds beyond one team and one app. And Kleppmann's question: who checks that the spec says what you meant?

96
00:12:46.583 --> 00:12:51.793
<v Bob>And security. Would a lint-first codebase also cut that 45 percent failure rate?

97
00:12:52.093 --> 00:13:10.242
<v Alice>This episode is based on All Code Is Slop, Except Mine, a talk by Lauren Tan on the AI Engineer channel. Thanks also go to Martin Kleppmann for his post on formal verification. And to Bill Doerrfeld at LeadDev for the GitClear report, and to Jens Wessling at Veracode for the security results.

98
00:13:10.542 --> 00:13:16.936
<v Bob>Now go weed your garden. That's all for today on Build Wiz AI Show. Thanks for listening, and see you next time.
