Software Factory Checklist: 10 Questions to Ask Before You Call It a Factory
Agents on every stage is not a factory
Every VP Engineering I met this quarter has a software factory. Almost none of them do.
What they have is agents on every stage of the SDLC: one that plans, one that writes code, one that reviews, one that deploys. That is real progress. It is not a factory.
I posted ten questions about this on LinkedIn. Within a day the post passed 800,000 impressions, and hundreds of engineers and engineering leaders weighed in. This is the longer version: a working definition of a software factory, the software factory checklist itself, the best pushback I got, and a 30-day way to start.
The short version
- A software factory is the system around the agents: what they may touch, what checks their work, and what happens when a run fails.
- The bottleneck is verification, not code generation.
- Start with one class of change, supervised, for 30 days. Expand by evidence.
What is a software factory?
A software factory is a delivery system in which AI agents plan, build, review and release software inside boundaries, checks and evidence that people define. The agents are the workers. The factory is everything around them.
That system has four parts:
- Boundaries. What agents may touch, and what they never touch without a human.
- Verification. Independent checks on the work at every stage, sized to the risk of the change.
- Safe failure. What happens at 3am when a run fails and nobody is watching.
- Shared context, rules and memory. What every agent knows about your architecture and standards, and the record of what was checked and why.
Software factory vs. agentic SDLC
An agentic SDLC puts an agent on every stage. People still carry the work between stages: every handoff waits for someone to check it, fix it and pass it on. The people are the pipeline.
In a software factory the checks are built into each stage. People set the boundaries and take the hard calls.
| Agents on every stage | Software factory | |
| Handoffs | A person checks and carries each one | Checked by the system, escalated by rule |
| Review | Same depth for every change | Tier set by blast radius |
| Standards | In documents and in senior engineers’ heads | Written as rules that agents and reviewers both use |
| Failed run | Someone gets paged to clean up | The system stops safely on its own |
| Evidence | Reconstructed after an incident | Recorded for every merged change |
| People’s job | Moving work along | Setting boundaries, making the hard calls |
A factory is a spectrum, not a switch
Most maturity models describe five or six levels of autonomy, from autocomplete to fully autonomous delivery. Itamar Friedman, Qodo’s CEO, lays one out in The Path to L4 Software Engineering. As Steve Jackson, a principal AIOps engineer, pointed out in the comments, plenty of real factories have gaps and are still factories.
The label is the problem. Teams say “factory” at level 2 and stop asking the level 3 and 4 questions.
Why verification is the bottleneck in a software factory
Generation got cheap. An agent writes a thousand lines of code in five minutes. Review capacity did not grow with it.
So the queue moves. Teams add agents to every stage and end up with the same review queue, only longer. By 2028, half of the enterprises that adopt autonomous coding agents without real-time guardrails will decommission or severely restrict them after a quality or security incident, according to Gartner’s How to Ensure Quality in AI-Generated Code (July 2026), as cited in What Every Software Factory Needs That No Model Will Provide.
Three rules follow from that:
- Verification is independent. The system that writes the code does not decide whether it ships. A model reviewing its own output shares its own blind spots.
- Verification runs at every stage. A single gate at the end of a stack of agent-written PRs is too late and too big to review well.
- Verification is proportional. A copy change and a payments migration do not deserve the same review.
Measure the factory by what it stops, not by what it ships.
The software factory checklist: 10 questions to ask your team
These are the ten questions I would ask my own team before we call anything a factory. Copy them, adapt them, challenge them.
Boundaries
1. Which one class of change goes first, and what is excluded?
Pick something narrow: dependency bumps, test additions, small bug fixes in one service. Trust is earned per type of change, not per agent. Each class gets its own acceptance criteria and its own evidence.
2. What can agents never touch without a human?
Auth, payments, migrations and CI config are the usual list. A simple test for anything else: if a PR revert does not undo it, a human signs off. As Anatoliy Kolodkin argued in the comments, the list has to be enforced outside the model: in access control, scoped tools and a pipeline that will not run without sign-off. A boundary that lives in a prompt is a suggestion.
Context and rules
3. Do agents see what a senior reviewer sees, or just the repo?
A senior reviewer knows the architecture, who owns what, which incidents this code caused and which other repos depend on it. Start narrow: the owners and incidents tied to the files in the diff. Relevance beats volume. This context has to be learned from the codebase and its review history, because nobody keeps it current by hand.
4. Which three review comments do we repeat every week, and why aren’t they rules yet?
Oleg Peker and Gregor Srdic both said they would start here. It is the cheapest step, and it pays off even if you never add an agent. Every repeated comment is a rule nobody wrote down. Once it is a lint rule, a test or a review rule, the agent gets that feedback from the system and not from a senior engineer.
Verification
5. Is the reviewer agent independent of the author, or the same model nodding at itself?
Roland Lopez, who leads data and platform engineering, made the case that independence has to be structural: a different model, different context, different instructions. Otherwise it is one opinion counted twice, and every record you keep inherits that flaw. Samuel Khemsi called the alternative “a very confident rubber stamp.”
6. Is the review tier set by the change’s blast radius?
Small, contained changes get automated checks. Changes that reach across services, data or money get a senior human. That only works if the system knows what a change touches, including the repos it does not live in.
7. When a run fails at 3am, does the system stop safely on its own?
Robert Mejlerö, a CTO, called this the honest test: a factory that needs someone awake to clean up after a failed run is still a demo. Define the safe state for each stage: stop, roll back, or hold and page.
Evidence and measurement
8. For any merged change, can we show what was required, what was checked, who approved, and under which policy?
Asaf Saar, CPO at Mend.io, noted that this starts before the agents run. Intent, constraints and acceptance criteria are written down first, so what was checked maps back to something a person agreed to. Then the audit trail is a byproduct and not a forensics project. Tsahi Rosenbaom added an eleventh check: the person who asked for the change can read the record and say “yes, that is what I meant.”
9. Are we measuring human attention per change, or just PRs merged?
PRs merged goes up by default once agents are in the loop. It tells you nothing. Lior Asulin asked how to measure attention without it turning into time tracking. You do not track time at all. The proxies are already in your Git data: review rounds per change, human comments per change, and how often a human overrides the agent. Watch two signals together: human review comments falling overall, and zero rubber stamps on high blast radius changes.
People
10. Who on our team is becoming the software factory engineer, and how are we training them?
The software factory engineer designs the boundaries, rules and checks that agents work inside. Itamar published a job description for the role. Antonio Agudo warned against making it one person. Someone has to lead this, but the way of working belongs to the team. Otherwise you have built a bus factor of one.
What engineering leaders pushed back on
The comments were better than the post. Six challenges, paraphrased here, changed how I would run the checklist.
“Tests alone will not hold it together.” Chris Cholette, SVP of Engineering at Navan, pointed out that an agent can turn a test green by creating a dependency the architecture forbids. The build passes and the architecture rots. His fix: enforce structural boundaries in CI first. The rest, in his words, is “prompt hope.” I agree, and the step before CI is the hard one. Someone has to know where the boundaries are. In most codebases they live in a few senior engineers’ heads, and they cross repos. An agent cannot respect a boundary nobody wrote down.
“No code should be out of bounds.” Ali Naqi Shaheen, co-founder of palgu, argued that a modern factory needs no human oversight by design. Some teams are there. For most organizations the technology is ahead of the trust. I would rather earn that autonomy one class of change at a time than assume it.
“Production is the only honest reviewer.” That line is from Scott Alexander, a site reliability engineer and founder. It is the most honest one. It is also the most expensive. I want most lessons learned before production, with production confirming that the reasoning held.
“Production is the only honest reviewer.”
Scott Alexander
“How do you know a reviewer reviewed and did not just approve?” A rubber stamp is not always bad. If the system can say with confidence that the blast radius is small, that approval should be automatic. That is attention saved for where it matters. The failure is a rubber stamp on a high blast radius change.
“Decide what counts as a one-way door.” Keiran Holloway, Chief Architect for AI at Rackspace, shared the rubric his agents apply: significant architecture or infrastructure changes, cost above a threshold, any security change that is not least-privilege, and anything a PR revert cannot undo. Soft guardrails sit in the agent. Hard ones sit in access control and the deployment pipeline.
“Go deep before you go broad.” Juan Vistro, an engineering leader, described the sequence: automate execution and handoffs for one bounded class of change, keep verification with people, then automate that verification step by step. Teams that go broad first end up with agents everywhere and the same review queue.
Keith Williams, Director of Software Engineering at Aura Global, said he would not call his setup a factory at all. It is closer to a garden shed. That was the most honest answer in the thread, and every factory started as one.
How to build a software factory: the first 30 days
Start with one class of change, supervised, for 30 days. Expand by evidence.
- Pick one class of change. Write down what is in it and what is excluded.
- Write the never-touch list. Enforce it in access control and CI, not in a prompt.
- Turn your three most repeated review comments into rules. Do this in week one. It pays off immediately.
- Put an independent reviewer on every change in that class. Once before the PR opens, and again at the PR.
- Run supervised for 30 days. A human still approves everything. Record review rounds per change, human comments per change, overrides and escaped defects.
- Expand by evidence. When people have stopped changing the outcome for that class, lower its review tier. Then add the next class.
The 30 days are less about the agent and more about finding the rules you were missing.
Where Qodo fits in a software factory
Qodo is the AI code review platform, the quality and governance layer between AI coding tools and production. In a software factory that is the quality layer, and it works on the middle of the checklist.
| Question | What Qodo does |
| 3. Context | Builds its understanding from your codebase and PR history. The Software Map shows how repos and services connect. |
| 4. Rules | Rules in Qodo are mined from your codebase and review history. Coding agents retrieve them through the Agentic Toolbox while they write. |
| 5. Independence | Review in Qodo runs separately from whichever agent wrote the code: on local changes before a PR opens, and again in the PR. |
| 6. Blast radius | The Software Map shows what a change touches across repos and services, so the review tier follows the blast radius. |
| 9. Measurement | Quality metrics for findings, fix rates and unresolved issues. |
It works across Git providers, on-prem and in air-gapped environments. Itamar covers the full picture in What Every Software Factory Needs That No Model Will Provide, and the Qodo 3.0 announcement lists what shipped.
Run the ten questions with your team first. If questions 3 to 6 are where you get stuck, that is the part we work on every day. Get started with Qodo.
Software factory FAQ
What is a software factory?
A software factory is a delivery system in which AI agents plan, build, review and release software inside boundaries, checks and evidence that people define. The term dates to the late 1960s and was later used for Japanese software factory programs and for US Department of Defense DevSecOps teams. Today it means agents doing the work inside a governed system.
Is a software factory the same as an agentic SDLC?
No. An agentic SDLC has an agent on every stage, with people checking and carrying the work between stages. A software factory builds the checks into each stage and escalates to people by rule.
Does a software factory remove human code review?
No. It moves human attention. People define the boundaries and the rules, and they review the changes with the largest blast radius. Low-risk changes are verified by independent automated checks.
What is a software factory engineer?
A software factory engineer designs and runs the system agents work inside: the boundaries, the rules, the review tiers and the evidence. It is a role for the whole team to grow into, led by one owner.
How do you measure a software factory?
Measure human attention per change, not PRs merged. Review rounds per change, human comments per change and human overrides of the agent are all in your Git data. Add escaped defects to see what the factory failed to stop.
Should we build or buy a software factory?
Build the factory around your own logic: your classes of change, your boundaries, your policies. Buy the components that are a product domain of their own, such as codebase context and independent review. Build vs Buy a Code Review Agent covers that decision for the review layer.
Where should a team start?
With one class of change, supervised, for 30 days. Turn the three review comments you repeat every week into rules, put an independent reviewer on every change in that class, and expand by evidence.
About the author
