Build Wiz AI Show

Factory Engineering: Build the System That Writes Code (Ben Holmes)

14 min · · English

Ben Holmes argues that software engineering is becoming factory engineering: building the system of AI agents that builds the product. We unpack what that loop needs and how to measure it. In this episode: - Product vs factory engineering, with human gates at spec and pull request - Cost-per-PR, cycle time and LLM-as-a-judge scores - A trial where developers felt faster but were slower Sources: Ben Holmes (@BHolmesDev); Birgitta Böckeler; Nathen Harvey; METR; Nathen Harvey and Derek DeBellis; Jeff Atwood Big AI ideas, made clear for builders. luongnv.com/podcast

Listen on Build Wiz AI Show Download MP3 Transcript

Based on Becoming a factory engineer by Ben Holmes (@BHolmesDev)

Transcript

Bob: What if your job stopped being writing code, and became building the factory that writes it?

Alice: That's today's claim: software engineering is becoming factory engineering. So who designs the factory, and who checks its work?

Bob: Welcome to Build Wiz AI Show. Big AI ideas, made clear for builders. I'm Bob, and today I'm the one asking the questions.

Alice: And I'm Alice! We're unpacking Ben Holmes's X article, "Becoming a factory engineer". His big idea: we went from writing code, to writing prompts, to engineering the system of agents that runs those prompts.

Bob: So, a factory. No forklifts, I'm guessing?

Alice: No forklifts. He calls it a systems design challenge, one that seems to have breathed new life into software development. Three guests and a few outside sources will help us test it.

Bob: Start me at zero. What changed in the day job?

Alice: Ben splits engineering into three steps: planning, implementation, and code review. Before agents, implementation was the bulk of the deep work. The time left over went to architecture discussions and code review.

Bob: And now the agents do the implementing.

Alice: Exactly, so the balance has flipped. And his blunt line: if you stop there, it feels like engineering was replaced with 20 percent meetings and 80 percent drudgery.

Bob: Twenty percent meetings? That's my calendar on a good week.

Alice: Picture a chef who used to cook every plate, and now only writes menus and checks orders all day. That's the trap he describes: the cooking moved to the agents, so the fun part looks gone.

Bob: But he doesn't stop there, right?

Alice: No. He says there are interesting engineering challenges worth solving at each step. Planning: when should the agent involve a human to draft a plan, and what goes into a good product or technical spec?

Bob: And the other two steps?

Alice: Implementation: how do we optimize the cost and quality of agent code, and what access do agents get across secrets, MCPs and skills? Review: how do agents verify their own work end to end, and assess the risk of a pull request to save humans time?

Bob: So the hard problems just moved up a floor.

Alice: That's his point: these are the systems questions engineers used to answer in syntax and abstractions, just at a higher level. Birgitta Böckeler, writing on Martin Fowler's site, frames the same challenge: to let coding agents work with less supervision, we need ways to increase our confidence in their result.

Bob: He turns that into two job titles. What are they?

Alice: Product engineering is building the product itself: deciding what needs to exist and how it should be architected. Factory engineering is building the system of agents that builds the product. It optimizes the implementation and review pieces through automation.

Bob: Wait. So factory engineers automate everything?

Alice: No, and he's firm on that. It means automating low-risk work, and only requiring human attention when absolutely necessary. Böckeler explains why people stay in the loop: human developers bring their skills and experience as an implicit harness to every codebase, and an agent has no organisational memory.

Bob: Give me a picture.

Alice: Think of a restaurant kitchen. The head chef decides the menu, and someone else designs the stations, the order tickets and the tasting checks, so most plates go out without the chef touching them. The menu is product engineering; that kitchen designer is the factory engineer.

Bob: What does a factory engineer actually get asked?

Alice: He gives three sample requests. One: "Our agent keeps adding test cases and code comments that it really shouldn't." Two: someone on call keeps copy-pasting error monitor reports into an agent and asks, could we automate this?

Bob: And number three?

Alice: The API bill skyrocketed after the latest model release: could some workflows move to cheaper models without sacrificing code quality? And his rule: the larger your product and the more agents you deploy, the more valuable factory engineering becomes.

Bob: Ah, the surprise invoice, everyone's favorite email. Okay, so what is a software factory, concretely?

Alice: Ben's definition: an automation loop around agents, designed to build better and better software over time. Ideally you set the product requirements and it runs without human interaction, but agents aren't at that level yet. So factory engineers build a factory around today's agents that automates as much as possible.

Bob: Where does this factory actually live?

Alice: In a shared setup that both the agents and the team can reach: usually a centralized system in the cloud or on your own compute, instead of local setups. Humans and agents can configure it: the environment, the agent's access levels, and the pipeline of steps it follows.

Bob: And where do the humans step in?

Alice: Picture a car assembly line that runs by itself, except at two stations where a person must sign off before anything moves on. In Ben's example pipeline, those two stations are the spec and the pull request.

Bob: Isn't "factory" just a new word for what people call a harness?

Alice: Close cousins. Böckeler defines a harness as everything in an agent except the model. She splits it into guides, which steer the agent before it acts, and sensors, which observe afterwards and help it self-correct.

Bob: And how does that fit Ben's two human gates?

Alice: Neatly. She writes that "a good harness should not necessarily aim to fully eliminate human input, but to direct it to where our input is most important."

Bob: Lena, you've been reading about the ground under all of this.

Lena: Right. Nathen Harvey and Derek DeBellis, on the Google Cloud Blog, announced the DORA report on AI-assisted software development. Its key insight: "AI doesn't fix a team; it amplifies what's already there."

Bob: So a shaky setup just gets louder?

Lena: They call platform engineering the foundation: 90 percent of organizations have adopted at least one platform. And a high quality internal platform correlates with being able to unlock the value of AI. Think of it as the concrete floor the whole line is bolted to.

Bob: Once the factory runs, how do you know it's any good?

Alice: Ben's first tool is spending metrics: track what agents cost you in token usage and on-device compute. Then translate raw numbers into real value, like cost-per-PR and cycle time, which he defines as initial prompt to release.

Bob: Cycle time. Isn't that a DORA thing?

Alice: A close relative. Nathen Harvey's guide on DORA's delivery metrics defines change lead time as the time for a change to go from committed to version control to deployed in production. Ben's cycle time starts earlier, at the first prompt.

Bob: And code quality? You can't put taste in a spreadsheet.

Alice: That's his second tool: LLM-as-a-judge scoring, for quality you can't check statically. Architecture quality, redundant or tautological tests, how efficient the agent's tool calls are. Another agent reviews past agent conversations to score them.

Bob: So how should I picture all that?

Alice: It's like a fitness tracker plus a coach. The tracker counts steps, cheap and constant; the coach watches the replay and tells you your form is off. In Böckeler's terms, the cheap counters are computational sensors, and the LLM judge is an inferential one.

Bob: But can a judge be trusted every time?

Alice: Böckeler is careful here. LLMs can partially address problems like redundant tests, but expensively and probabilistically, not on every commit. Cheap, deterministic checks like tests and linters are fast enough to run on every change.

Bob: Any warning label on all these metrics?

Alice: Two, from that DORA guide. Don't set a metric as the goal, because teams will try to game it. And don't pick one metric to rule them all: choose several, with a healthy amount of tension between them.

Bob: Last piece. Can the factory fix itself?

Alice: That's his third tool, self-improvement: once you track these metrics, make them agent accessible. His example is hypothetical: a redundant-tests scorer that does poorly on 60 percent of conversations. A self-improvement agent notices the trend and suggests changes to your skills and code architecture.

Bob: Sixty percent pointless tests. The robots are padding their homework!

Alice: In his made-up example, yes. Böckeler describes the same loop done by people: whenever an issue happens multiple times, improve the guides and sensors so it becomes less likely. And she adds that agents can help build those controls too.

Bob: And the fourth tool?

Alice: Benchmarking: test theories on how factory configurations change cost and quality. The biggest lever is model selection, but you might also cut stages or add and remove subagents. Treat each as a hypothesis, and change one variable at a time.

Bob: Why only one at a time?

Alice: Imagine baking bread and changing the flour, the oven and the timer all at once. If the loaf gets better, you can't say why. Change one variable in the factory and watch the metrics, after setting a baseline for current performance, which is step one in the DORA guide.

Bob: Time for pushback. Maya, you looked at what happens when experienced developers actually use these tools.

Maya: I did. METR, a nonprofit research group, ran a randomized trial with sixteen experienced open-source developers on 246 real issues from repositories they know well. Each issue was randomly assigned to allow or disallow AI.

Bob: And what happened?

Maya: With AI allowed, they took 19 percent longer. And the perception gap is striking: they expected a 24 percent speedup, and even afterwards believed AI had sped them up by 20 percent.

Bob: Slower, and they felt faster. Ouch.

Maya: To be fair, METR says it doesn't claim this holds for most developers; its setting is experienced people on projects they know well. But it does show that self-reports may be overoptimistic.

Alice: Which is exactly why Ben wants measured numbers, like cycle time from prompt to release, rather than a feeling of speed.

Bob: Sam, you think the whole factory image is off.

Sam: I do. Jeff Atwood, on his blog Coding Horror, asked "Is Software Development Like Manufacturing?" and he wasn't convinced. His line: "Variability is the enemy in manufacturing; in software, it's the reason we get up in the morning."

Bob: So every project is a one-off?

Sam: He says "Every worthwhile software development project is a custom one-off job for a unique problem." And he quotes Mary and Tom Poppendieck: "generating good software is not a production process; it is a development process."

Alice: Fair hit. Ben's answer is partly built in: he automates the low-risk work and keeps human gates at the spec and the pull request.

Bob: Lena, does more output from the factory mean more breakage?

Lena: It can. The DORA team found AI adoption still has a negative relationship with software delivery stability. Without robust control systems, like strong automated testing and fast feedback loops, more change volume leads to instability.

Alice: And one limit no factory removes. Böckeler notes that misdiagnosis, overengineering and misunderstood instructions aren't caught reliably. In her words, "Correctness is outside any sensor's remit if the human didn't clearly specify what they wanted in the first place."

Bob: Time for Builder's Corner. Alice, one thing to try this week?

Alice: Pull last week's agent pull requests and log two numbers for each: what it cost, and its cycle time from first prompt to release. That's Ben's cost-per-PR and cycle time.

Bob: Just two numbers?

Alice: Plus one judge score. Pick the behavior that bugs you most, say redundant or tautological tests, and have another agent score those past conversations. Keep a healthy tension between the numbers, as the DORA guide advises.

Bob: And then?

Alice: Change one thing, say a cheaper model on one workflow, and compare next week. One variable at a time.

Bob: Quick takeaways. Mine: the work moved up a level, from writing code to designing the system that writes it.

Alice: Mine: automate the low-risk work, and send human attention where it matters most, at the spec and the pull request.

Bob: And measure it: cost, cycle time and a judge score, not a gut feeling.

Alice: Because, as METR found, developers can feel faster while being slower.

Bob: What's still open?

Alice: Ben's own first question: when should an agent involve a human to draft a plan, instead of just building? The post asks it and leaves it open.

Bob: And on the measuring side?

Alice: Böckeler asks a good one: if sensors never fire, is that a sign of high quality or inadequate detection mechanisms? A quiet factory might be a great one, or a blind one.

Bob: A blind factory. Now that's a scary thought.

Alice: Thanks to Ben Holmes for his X article, "Becoming a factory engineer". And to Birgitta Böckeler on Martin Fowler's site, Nathen Harvey's DORA guide, and METR. Plus Nathen Harvey and Derek DeBellis on the Google Cloud Blog, and Jeff Atwood at Coding Horror.

Bob: That was factory engineering, from writing code to building the system that writes it. That's all for today on Build Wiz AI Show. Thanks for listening, and see you next time.