Why AI Shouldn’t Test Its Own Code

[00:00:01] Wei-Wei Wu: I wouldn’t trust that same AI to write its own tests.

[00:00:04] Itamar Friedman: Welcome to The Agentic Review, the podcast where we explore what good code really means in the age of AI software development.

[00:00:13] Nnenna Ndukwe: I’m Nnenna Ndukwe, Developer Relations Lead.

[00:00:15] Itamar Friedman: And I’m Itamar Friedman, the co-founder and CEO of Qodo.

[00:00:19] Nnenna Ndukwe: So, let’s get into it.

[00:00:24] Itamar Friedman: Today, we’re joined by Wei-Wei Wu, co-founder and CEO of Momentic, an agentic quality platform for web and mobile software. Before founding Momentic, Wei-Wei worked across product engineering, front-end infrastructure, observability, reliability and developer productivity at Qualtrics, WeWork, Nashi and more.

[00:00:47] Nnenna Ndukwe: Wei-Wei argues that AI is making co-generation faster than humans can even verify it, that tests are becoming executable product specs, and that the QA job title may disappear even as quality work becomes more important. Wei-Wei, welcome to the show.

[00:01:04] Wei-Wei Wu: Yeah. Thanks for having me. Excited to be here.

[00:01:06] Nnenna Ndukwe: Is there anything that you’d like to add to the intro that we gave you? Things that are top of mind right now that you’ve been working on doing?

[00:01:12] Wei-Wei Wu: Yeah. I think top of mind is always, like, how do we help our customers build their own software factories? Everyone wants to build their own software factories to scale their own engineers. And I think a lot of the bottleneck is definitely part of the verification layer. The code output is cheap. How do we verify it correctly, accurately, at a cost that is reasonable?

[00:01:40] Itamar Friedman: I want to challenge that a little bit. You’re saying that everyone wants to build a software factory. Wouldn’t they just want to buy it? I mean, like, there are some product platforms out there giving you, you know, software factory as a service, a complete one. Why would I want to build one if I can buy a complete one?

[00:02:00] Wei-Wei Wu: That’s a great question. I do think this comes down to the age-old, especially in developer tools, age-old argument: build versus buy. I do think, as a software vendor, as a company selling software SaaS ourselves, I do think the argument is always like, it’s not necessarily, can you build it? You definitely could. Anyone could build their own software factory or any software, for that matter, given enough time and tokens. But it’s whether you should. Like, should you spend dedicated engineering hours maintaining something that maybe isn’t your core competency? And I do think today a lot of companies do buy this. And for example, like, a lot of people are using Claude Code; they’re using Codex or Cursor or whatever other agent coding tool that you’re using rather than trying to build your own. But there is also, I think a growing sentiment within the developer community to like have control over the software that they’re running. That’s why there’s open code, there’s open inspect, there’s a lot of open-source versions of closed-source software because I think a lot of engineers kind of want to know how the sausage is made. They want to be able to control it, customize it as much as possible. But I do ultimately think that it boils down to: as an engineering leader, do I want to spend a bunch of my engineers’ valuable hours and expensive tokens on building something that I could get off the shelf? Maybe, let’s say for the sake of argument, it’s 95% of the way there. Do I want to spend a few hundred grand or a few million dollars on that last 5%? I’m not sure. I think that calculus is up to each team.

[00:03:52] Itamar Friedman: Yeah, I’m with you. But maybe I’ll recap what I understand. Like, basically, we’re all on our journey to build our software factory, right? And then there’s a spectrum. Let’s take three points on that. One is I buy the software factory as a whole. I don’t know. Some companies is giving, “Oh! All you need to do is add this like bot to your Slack, and it will understand everything and complete everything for you, including production, root code analysis, whatever you want.” Then there’s the other side of, like, “Hey, I’m going to build on the model itself and do everything myself. The harness of all the generative workflows, all the quality workflows, all the context harness, all the optimization of the agent running and the governance. I’ll do everything alone.” There’s the middle of, like, I want to know how this sausage is made, but I’m not going to grow my cows or whatever or pigs or so. Like, I want to have the opportunity to mix and match and control everything. And I think like what I’m hearing is that, you know, probably we’re going to see like the major part of the market and the engineering that care about their work converging somewhere to the middle. Right?

[00:05:07] Wei-Wei Wu: I would definitely agree. It’s like you can mix and match any code and agent you want, any testing agent, any code review agent, any observability tool, whatever best fits your stack because you could do anything.

[00:05:21] Nnenna Ndukwe: I think, well, I have a major complaint right now with, like, AI for test generation. And I am not the only one who feels this way. And that is, it just seems like no matter how much I try to steer the quality of the tests it generates, there’s always some type of cheating. It feels like it’s just cheating to pass, like, create as many tests as possible that get the code to pass as opposed to generating very high-quality tests. But we are using AI to write the code and the tests. And I’m wondering from you, Wei-Wei, like, what do you think? Do you think that we’re allowing the same type of mistakes to appear in multiple stages of the SDLC because we’re using AI for multiple stages?
[00:06:08] Wei-Wei Wu: Yeah. That’s a great, great question. I think this was the topic of a very big discourse on Twitter maybe last week or two weeks ago, where a company deleted 800,000 lines of unit tests from their codebase. And I think this sparked, and I think the motivation was that the AI coding agents that they’re increasingly using to write their code is writing pretty useless unit tests, or it could be modifying the unit tests to get its changes through. And I do think, like, especially as models are getting better and better at writing software, the things that are very granular or easy to output – for example, unit tests are very easy to output, just like React code or Tailwind CSS in your codebase. It’s going to be really easy for the models to output that. And I think the value that the tests provide are much less so when it’s the same model that’s writing the code and writing the test. Because I do think there is incredible value in the concept of an independent verifier, whereas, like, before agents were writing the code, this was humans writing code. I push up a pull request. Another engineer on my team, I would review it based off of, say, the PR description or maybe the linked Jira ticket or something like that. And I do think we’re seeing that today with like AI code review platforms where it’s another AI reviewing the code you’ve written or even testing it, rather than, say, the same AI that is very prone to, I think we talked about that this. I think you talked about this with my co-founder at the WeAreDevelopers conference, where, like, it’s just context rot, right? So, it’s trying so hard. Let’s say 90% of the context is on implementing this feature, and then how is it able? There’s no way it can take a step back and write tests that are objectively correct because it just already has polluted context. So, I wouldn’t trust that same AI to write its own tests, and I wouldn’t want another – something that has clean context, that has a very clear understanding of, like, the product behavior or the expected behavior to actually test it instead.

[00:08:38] Itamar Friedman: Yeah. I hear from you that we are in an era that harness means a lot, despite models improving and having different harnesses that doesn’t have necessarily the same context; their process of collecting the context that fits for the task is very important. So, it’s not just like using a different model for doing the test instead of doing the coding, but rather having a different harness for doing the test versus the coding, for example.

[00:09:11] Wei-Wei Wu: Yeah. Yeah. Definitely. Definitely.

[00:09:13] Nnenna Ndukwe: I’m curious about what you think that means for quality for everyone who’s like participating in building or shipping software. What kind of responsibility do you think that puts on us all and the tools that we’re using to build software now? And we’ll get into the whole QA thing in a bit, but I’m kind of, like, leading up to that.

[00:09:34] Wei-Wei Wu: Yeah. For sure. I do think like the QA industry is changing very rapidly, whether they like it or not, because I think one of the big problems, big bottlenecks in the past, large organizations who have dedicated QA teams that are in charge of writing tests or manually testing. And in the age of AI agents, whenever you’re trying to release 10 times or a hundred times more code a day, relying on a separate organization or manual tester to test things is not going to scale. And as a result, I think this is kind of a forcing function for the shift-left methodology that I think everyone’s been talking about for the last 10, 15 years. But I think now there’s actually a forcing function because if you’re not shifting left, you are just bottlenecking your whole engineering and engineering productivity. So, I think with shift left, this also means that “quality is everyone’s responsibility.” I think a lot of engineering leaders are thinking about how can we encode quality into the definition of done for any ticket or any product feature or anything like that. And I think ultimately this is shared ownership. But that is also a hard problem to solve. How do you make sure that every engineer who’s creating, building new features, fixing bugs is properly testing it, properly reviewing it and things like that?

[00:11:09] Itamar Friedman: Do you think we can move from the, you know, reactive intelligence to the proactive intelligence? And with regard to what you said, like, what I’m hearing from you is that we always wanted to shift left QA and quality. We always wanted everyone, you know, to be responsible. But it was actually quite difficult to do that, considering that it takes so much time for a developer to finish a feature back in the day. And now that it’s faster, then actually, there is a forcing function to really do shift left and make the QA job title disappear. Let’s talk about it a little later, about what it means. But then you’re saying being disciplined about it is still not easy. So, you need to know how to do shift-left quality and how to use AI to shift left. And I’m wondering maybe the next step is not needing to know. Rather, you have, like, a proactive, intelligent QA swarm or whatever you want to call it, a harness that tells you, oh, here is how you could do it for this case or something like that.

[00:12:14] Wei-Wei Wu: Yeah. Yeah. I think the tooling and the harnesses and, like, agents that we build around it or integrate into the SDLC are definitely super, super important. I do think, like, a part of doing this proactiveness or almost like essentially, is making it, like, by default it’s kind of blocking PR checks that if they fail, I can’t merge. This means I either have to fix my code or fix the blocking PR check. And I do think a big half of this is what is the tooling that we’re building? What are the rules, and guardrails, and tooling that we’re building into the software development lifecycle, whether it’s stat linters, whether it’s code review, whether it’s testing – to essentially, not say force, but it is forcing engineers and anyone who’s contributing code to uphold that level, that quality bar, because if it’s something that’s opt-in that someone has to choose to do, I think that by default no one’s going to do it. No one wants to do want to remember an extra thing that they have to do and we have to make it automatic and as possible and easy to adopt.

[00:13:29] Nnenna Ndukwe: So, then beyond that, with not just the future of QA changing or I guess some of those responsibilities being dispersed across engineering, then what else do you think will change with jobs now in this AI era?

[00:13:44] Wei-Wei Wu: I do think the roles are the divide. The role definitions are, I think, very, very much blurring in the past that you would have QA that’s owning everything from writing automation, maybe even writing integration or unit tests, to manually testing, updating test case management software, TestRail – the OG test case management software. And I think, in response to AI coding tools just accelerating code output so much, we’re seeing a lot of that responsibility getting merged with the engineering role. But that doesn’t necessarily mean like every single QA is getting laid off. I think just with any role, like, you have to be able to adapt to market changes, right? You have to upskill yourself. And I think one of the things that’s pretty interesting is we see a lot of QA departments or organizations, rather than being their own vertical that’s like a sibling to appear to engineering, they actually roll up under developer experience or infrastructure because, ultimately, I think what this is shifting into is QA providing this valuable knowledge and context and building the tooling so that engineers can easily verify their own work or agents can independently verify their own work, rather than, say, a separate organization that just owns quality. Because we all know that. I think, well, I guess maybe we don’t know yet, but the market is responding. That is an incredibly inefficient way of building and shipping software. So, I do think that this is where we’ve seen roles and organization tree structures change over the past, you know, year and a half.

[00:15:39] Itamar Friedman: I have a different topic that I really love to hear your thoughts about. Once upon a time, some CTOs would say that testing is the best spec, right? It’s because it’s executable. Do you think this is true? Everyone agrees with that, but I think many would say that. And I’m wondering if this is still true. Like, considering what we said like 10 minutes ago, that people just deleted all their tests. Let’s admit it. Like, one of those that announced it was OpenClaw, right? Like, with all due respect, OpenClaw could delete all their tests. It’s not, like, they are a reliable transportation app or financial application or something like that. Okay, but having said that, I go back to my point. Like, my question: some CTOs, some tech leads would say that tests are the best, like, live spec. Do you think it’s still true with AI? Or maybe they go stale, and they go old, or they’re being cheated or something.

[00:16:43] Wei-Wei Wu: I think of it as like different levels of abstraction you have; for example, like, instead of manipulating the DOM and HTML by hand, I’m writing React code with like JSX. That gives me a nice abstraction over the underlying rendering and the DOM. And I do see that. I think about that the same way with tests as well. Rather than writing, say, a lot of very granular unit tests, say for a front-end feature or user-facing feature, I would rather have, like, maybe a higher-level end-to-end test. I think in the past this would be something like Playwright, where you’re encoding a lot of page specifics; you’re hard-coding locators, hard-coding CSS selectors. And I do think with AI there’s incredible value in even a higher level of abstraction. And that is pretty much just what a PRD or a product spec is. Like, this could be a Linear ticket, it could be a Jira ticket, it could be a Notion doc and/or, but ultimately, this is just a description of end-user expected behavior. How? Because ultimately, that is kind of the – no one wants to write tests for the sake of writing tests. Everyone wants to deliver value for their end customers. And I think the closer we can marry those two concepts, the faster we can iterate and the better experiences we can deliver for our users.

[00:18:17] Itamar Friedman: Yeah. I’ll have to double-click on that. But before that, maybe to give us more context, I’d love to learn about Momentic. What’s the input and output? Is the output of Momentic eventually a huge set of tests, or is it something else? And what’s the input? I think it’s very related to what we’re talking about. So, maybe it’s a good opportunity to introduce this.

[00:18:37] Wei-Wei Wu: Yeah. Yeah. So, Momentic. The core input is plain English instructions. And I think this can actually be at various levels of granularity. So, it can be plain English granular instructions, like individual clicks. So, for example, click on the submit button or something like that. But it could also be something that’s higher level, a goal. Say, like, I want to be able to log into Google with my username and password or something like that. And depending on what our customers care about, sometimes they care a lot about how individual steps are performed in sequence. And in those cases, they would use the granular steps. But if they just care about, say, a high-level goal being achieved, then they can just tell the agent, like, “Hey, this is what I care about. Go and execute it.” This is the test. And then I think one of the things that we’re thinking about is, like, is there a future without test scripts at all where this is just agents testing themselves and giving you reports? Because no one wants – I don’t want to spend my valuable Codex Max sub writing and generating a ton of unit tests or end-to-end tests. I just want to ship software with notebooks. And I think a big part of that is our motivation. That’s why we decided to build Mo. It’s like, can we build an agent that you can task with essentially bug bashing whatever part of the app you want it to, so that it’s able to just surface bugs for you and your agents to fix, rather than, say, spend tokens writing and maintaining whatever scripts you have, whether it’s Momentic scripts, whether it’s Playwright or Selenium.

[00:20:23] Nnenna Ndukwe: So, would you say this is an adversarial approach that you’re trying to embody? I guess the QA work that we used to have QA engineers focusing solely on. And is it multiple agents, or is it one, or how does that work?

[00:20:38] Wei-Wei Wu: I would say, definitely, I would categorize it as adversarial or exploratory testing because, like, I think for a lot of our customers, the workflow that they used to do would be before a big release or a big feature launch, they would have a big team of engineers or QA manually bug bashing their app, maybe on staging, and they’ll put together this database of, like, here’s all the bugs, here’s the different priorities, we need to fix these before the launch and things like that. But I think a big problem there is, especially in the world of AI, human attention and time are incredibly valuable, right? So, they’re realizing they’re spending so much time and resources and salaries on these bug bashes. And with these bug bashes, the whole point is, like, how do we find these hard-to-find bugs? Different permutations, different user journeys, different edge cases. And I do think agents are really good at that, especially on hosted infrastructure. As a single human, I could probably at most click once per second right through a user flow that I know well. Let’s say an agent is around the same speed, but I could spin up a thousand agents at the same time. And then spinning up, if hiring a thousand people to manually test my app is just not going to scale very well, it’s going to be very, very expensive. So, I do think agents give us a primitive that enables us to scale previously inefficient tasks and automate them pretty well as well. Because agents are great at computer use, and we’ve seen it do browser, mobile and things like that.

[00:22:18] Nnenna Ndukwe: When I would collaborate with, or I guess I used to work with QA engineers when I was, like, on a product engineering team. And so they would do that. Exactly what you’re saying in staging and kind of come back to myself and other engineers with some things that would need to potentially be fixed or validated. Again, different paths. And I’m wondering, like, I don’t know what fully encompasses their role, right? I know the ways in which to collaborate with them, but I don’t know all of the tools that they use and all of the different methodologies and, like, the background experience. And so I’m curious: does a tool that is designed to help with the QA process need to map through, or I guess mimic the same type of tools that humans needed for a proper QA?

[00:23:06] Wei-Wei Wu: I would say, like, not necessarily. I would say I think the thing that is incredibly valuable that a human provides is taste and judgment, especially when things are ambiguous. And I do think that’s like where the best QA teams that we work with are the ones that know every single thing about the product. They know every single nook and cranny, weird edge cases, like if you triple-click on this button, the whole page crashes or something like that. So, I do think that’s incredibly valuable. But I do think the tools are much less so because, like, tools are easily replicable. If I’m testing on a browser, I just need to open Chrome or Safari to be able to test it. And if I’m using, say, test case management software, TestRail, in case that’s just an API or an MCP server for an agent. So, I think the tools are much less important compared to, say, the context and knowledge that a human would have.

[00:24:05] Itamar Friedman: Did I get it correctly? I’d love to repeat what I hear is your vision for the QA and make sure I understand it correctly. So, basically, you’re saying that, as a developer or developer leader, I don’t want to deal directly with tests. I actually would appreciate that, like, a product, an AI product like an agent or a swarm, would have tests inside it, you know, in order to cost less for repeatability, whatever. But it’s like an intelligence problem. It’s an intelligence problem to, given, like, an exploratory, given a regression, given a request that I’ll give it, for an intelligent creature to be able to choose the right tests and create new ones and explore the mobile and website and whatever needed and period. And then my question is: is there, like, kind of, like, a confidence level, trust report, anything that, you know, eventually in that vision? Like, I’m saying, “Oh yeah, this was good. I can move on.” So, am I right about the vision? And how do you deal with the trust issue that, yeah, it did well?

[00:25:17] Wei-Wei Wu: Yeah. I think that vision is definitely super aligned. It’s just like, no, tests are just an implementation detail. The end goal is like higher quality, less bugs or like no bugs that are shipped to customers. And because I think a lot of times when we work with our customers on POCs with Momentic, there are usually, I would say, two axes that they’re measuring momentum against. It’s just like: how is this increasing our developer productivity, and then how is this increasing our product quality? So, what’s our defect escape rate for this particular area? What are the bugs that are caught by end users versus caught internally by QA or Momentic? And I do think that is the end goal because we work with some of our customers who have thousands and thousands of Selenium Suites. They have hosted infrastructure, Selenium Grid on their infra, and they still tell us, like, “Hey, it takes 8 hours to run all of this because we don’t have that much. We have limited parallelism.” And I think at that point it would actually be faster and more cost-effective to just run a bunch of agents and click through the app rather than just executing and fixing test scripts. Because when it’s taking that long, and there’s so much iteration and so much maintenance going on, you’re already like, there’s already so much turnover. You’re essentially wasting time maintaining the middleman, if that makes sense. You’re wasting time maintaining test suites rather than catching bugs. And I do think that that’s just a huge misallocation of budget and resources, in my opinion. So, I do think that that is where the future is shifting, but yeah. And then what was the second part of your question?

[00:27:07] Itamar Friedman: Like, what is the proof from the Mo, right, that, for example, you’re a Momentic agent, that hey, I finished the job, you can trust me. You know, like, let me give you like an example for a code review tool. Basically, we can break that question of trust into different categories. There is verifying the intent. There is verifying architecture. There is verifying maintainability, testability, compliance, a set of standards like rules and best practices that are accumulated. Each one of these categories is also quite complicated. So, you break it down another step, but then eventually all of that accumulates into kind of, like, a report. This works well. It’s like if you ask a random person, like, how many piano fixers are in New York? I don’t know. Like, how many gas stations are in California? You don’t know. But if you try to break it down to, like, how many people I know in New York that have a piano, how many blah, blah, you get a really good answer. So, the same thing with, like, review: for example, you break it down into different elements you want to review, and you check one by one; you get a report, and then, like, I’m wondering if that’s the same concept that happens with testing, or how would you break it down? Like, is it like, hey, here’s a common user behavior. Here are the edge cases. Like, is there any type of like reporting that helps the QA person or the developer to trust the agent work on the testing part?

[00:28:41] Wei-Wei Wu: Yeah. And I do think a core part of finding bugs is, like, reproducibility and the proof that comes with it. Because I think as an engineer, the worst thing that I get from support is, you know, or the most annoying thing is a bug report. But there’s no evidence. There’s no, you know, screen recording, or there’s no screenshots, or there’s just no evidence the bug was actually there other than someone saying it was. So, with Mo, one of the things we focused on a lot is, like, how do we reproduce everything? So, which, in our case, since it’s full end-to-end browser and mobile testing, is like, all right, what accounts or what apps are you running against? What is the exact URL, viewports, console and network logs throughout the whole bug repro? Every single step that the agent executes, with a live video recording and annotated so that you can see exactly where the bug occurred and all the different steps that happened. And I think our goal there is to have as much fidelity, or even higher fidelity, than, say, a human recording a Loom of a bug or something like that. Because then we have code-based context; we can tell you potentially even what commit or pull request actually caused this bug that’s deployed to production. And I think it’s also important to be able to associate what requests that are shown in the browser network requests and also track that back to, say, backend observability logs – Datadog and Honeycomb for the full end-to-end debugging workflow. Because ultimately, as an engineer, if I have to debug something, I’m going to look at Sentry, I’m going to look at Datadog, I’m going to look at all the different tools that’s available to me to fix and find this bug. And we want to provide all of that information with one click. So, every bug report must hit that standard, or else it’s just not very valuable because then if it doesn’t have all the evidence, then we’re going to be wasting human time, a human engineer’s or an agent’s time, on investigating and reproducing.

[00:30:53] Itamar Friedman: Yeah. Yeah. How hard is it to connect to all these signals, like, to Datadog, Sentry? Like, you know, for example, we’re talking about all the QA, like, a player QA person would do exactly what you said. Right? It sounds like, hey, the most annoying thing as a developer is getting a QA, like, line. You have a bug there. Okay. The next one is like, here’s the full description of reproducing the bug, right? I’m like, that’s how I appreciate QA people approaching me with issues. Then it’s like, okay, here’s a video or actually, like a Playwright or a script. It’s more than a text. Wow. And then you’re saying the most advanced QA people, like, they actually bring you the relevant logs and the relevant, like, so how do you build the AI that knows how to fetch that relevant information?

[00:31:46] Wei-Wei Wu: Yeah. That’s a great question. And I think there is, especially, for like observability platforms how I think of say if I was debugging something, I’m going to go into Datadog, I’m going to look for maybe a specific request ID or a specific log line. And I think one of the things that we gave Mo is the ability to connect to custom MCP servers. So, you’re able to connect your own Datadog instance, your own Sentry, your own Linear, and Notion to be able to fetch product context and help debug. So, then every single bug report is able to pull from all the data sources that you choose to connect. And of course there’s also stuff you might not want to connect, and that’s totally fine. But the more you connect, I would say the more context the agent has and the more value it would be for our customers. Just because it’s higher-quality bug reports. No one likes low-quality bug reports; they’re just noise.

[00:32:51] Nnenna Ndukwe: Yeah. I’m glad that you mentioned that because we just interviewed Liz Fong-Jones, from Honeycomb, talking about observability and what that means in the era of AI and how all of that data is extremely powerful that we can have and give to agents to figure out, “Hey, how can we actually prevent some of these types of bugs or issues in the future based upon any like incidents from the past or any of the discoveries that you know, Momentic would be able to provide us.” So, it’s great to hear that you’re thinking about all the integrations that can be useful.

[00:33:28] Wei-Wei Wu: And I do think that is like it’s super interesting with like how my personal interactions with like different observability platforms has changed. Like, when we first started Momentic, we were using Datadog, and I always thought their log search and dashboard UI was amazing, and I think they have great UX. But I think over the past 2 years, I mean, I’m not using the Datadog UI anymore. I’m just telling my agent to use the Datadog MCP or the CLI to fetch logs and analyze what’s going on, and set, maybe run it on a Cron or maybe on different alerts. All of the alerts and Datadog for us are configured by AI. It’s just based off of, like, the different ones using the Datadog, the observability platform’s UIs anymore. I’m just having agents do that for me because it’s going to be able to process a lot more information than I ever could, you know.

[00:34:23] Nnenna Ndukwe: Yeah. I think that kind of goes into another topic discussion which we won’t do here but I think of, like, what is the interface that we’re all like converging on. If we’re not looking at the UI and our agent, where are we reading the information that the agent has found? You know, like, where are we most likely to hang out, but yeah, very interesting topic. There was something that you brought up earlier that I’d like to bring to the forefront a bit more. And you mentioned things around metrics like defect density. I’m very curious to hear. I’ve had many conversations about how certain metrics matter less or more now in the agentic AI era for engineering organizations. Are there any numbers, any things that are part of this calculation that we should be considering or engineering leaders should be considering?

[00:35:15] Wei-Wei Wu: I think one that I hear a lot that I think is actually a very bad metric or is not a truthful metric is one metric that I hear a lot on LinkedIn and Twitter is change failure rate. So, this is a change or a commit that someone made that led to, say, an incident or a bug for a customer. And I think one of the things I hear very commonly or see on my socials is like, hey, we built our software factory, our engineers are shipping a thousand times more code, but our change failure rate stayed the same, which is a great win. And I think the percentage is very deceptive because, for an end customer, if I was an end customer of this software and their engineers are shipping a thousand times more commits and their change failure rate is staying the same, that means I’m also getting a thousand times more bugs. I’m getting an extremely worse user experience. So, that’s why I think we need to have metrics that actually align to customer value or customer feedback, because just tracking change failure rate to track code quality and product quality is actually not a good metric, because if I weren’t a customer and I had a thousand times more bugs in every single software I use, I’m probably going to be very upset. I don’t care how many commits your engineers are shipping and how often they’re failing. But I do think ultimately the engineering industry has to align on metrics that actually matter because I think we have all these DORA metrics, meantime to repair and cycle time, how fast it takes to ship a PR to production, deploy time and all these things that we track in DevOps. But I think, ultimately, we need to be able to align that with customer value, and if we can’t do that, then I don’t know, there’s not a good reason for us to track and improve on certain metrics.

[00:37:33] Nnenna Ndukwe: Itamar, I’d love to hear your perspective on that. Yeah, I agree. I think there definitely needs to be more conversations around that and a very clear connection to the customer value. Then there’s another hot take: what is it that people say, “Is it really a bug if, like, a customer hasn’t discovered it or run into it?” Which is, yeah, we’ll need to hear some of your hot takes, but Itamar, do you have any thoughts on any of this?

[00:37:58] Itamar Friedman: I think it sounds like the equivalent of, is it really, you know, a crime if nobody caught me? Right? Like, it sounds like the equivalent of that. Like, it depends. But seriously, I love the way you looked at it, Wei-Wei and like, we need to look at it also from the perspective of our customers. The thing that I’m looking at a lot is, like, when I’m updating Claude Code and Codex, like, Claude Code are doing that quite often. They’re saying how many features they, like, basically shipped in the latest version and how many bug fixes. And sometimes I’m seeing that like 100x on the bug fixes and it makes me think, you know, is it because they’re fixing much faster bugs these days or is it because there’s much more many bugs with the more features that they’re developing? And eventually, I think it boils down to how do I look, like, my feeling when I’m using the product? These are the numbers. They could have not shown it to us. Like, how do I feel about the product, how many times I have, like, critical issues, etc., while I’m using it. So, even if I have, like, 50 features more, right, and I’m using it for an hour and a half, how many times did I bump into a problem during that one and a half? Okay, they developed another 50 features. Now, I’m using it. Instead of an hour and a half, I’m using it for one hour and 45 minutes. Okay, how much bugs, like, am I encountering? So, I think we must look at that. And basically, in my opinion, like, the opportunity with AI is actually going down with the amount of bugs going down, with, like, increasing quality despite increasing speed. Like, I think velocity is a mix of speed and quality. They go together. And I think that’s what’s exciting about the next new generation of tools that are much more focused. They’re dedicated. They’re, like, all they care about is helping that angle of the velocity. And eventually, if we think about it, sometimes it’s like a riddle; we take it to the extreme. If there is a tool that promise you, I’m not saying that there is that the quality is 100%, whatever it means, then think about how fast you can move. But if there is a tool that tells you, “I can write a good level of code, like the speed of light.” Can you move as fast? No, you still need to verify, etc. So, to some extent, I think if we are coming from the side of the user, what do we care about, like, the value of the business, then I definitely think that we should check those metrics of how many bugs I have as a whole and how they’re going down, and not a fair amount of changes, and both are interesting, but we need to have them both.

[00:41:07] Wei-Wei Wu: Definitely. And I think people are realizing now that we’re a little bit under a year since Claude Code and Opus 4 took over, took the developer world by storm, and I think everyone is kind of finding out it’s like, oh, this is what happens when you ship 10 to 100x lines of code per day. Because I think a big part of that is, like, when code is cheap, and code is essentially free. Right? I’m just telling my coding agent to do it. Quality and taste are really the only thing that matters. And I do think, for example, Linear – that’s their whole brand, right? They build very tasteful, very high-quality software, and you can just feel it. You can just feel it by using the software. That’s like, wow, this is such an elegant experience versus where the future is going to be. The price of building software is extremely low, and I think there’s incredible value to it. I think quality and taste are a moat that is going. What is the best user or agent experience is what’s going to differentiate products from good products from bad products.

[00:42:24] Nnenna Ndukwe: I think you’re kind of getting into some of those hot takes. I would love to hear any more that you have surrounding that.

[00:42:31] Wei-Wei Wu: Yeah. I know, I think Lauren Tan Potato on Twitter, like one of the engineers on the bot team. I think she posts a lot about how she ships, like, what, 6,000 PRs a week or a month, or it’s some insane number. And I think a lot of people – there’s a lot of discourse on – like, oh, why PR accounts are a vanity metric. It’s not a real measure of productivity. And I do think, to some extent, that is true as an engineer. How, like, for example, when I was climbing the corporate ladder at the different startups I was working at, as an engineer, you’re not necessarily just measured on lines of code or the PR you’re creating; it’s around impact and how can you scale yourself across multiple teams. And yeah, I think a lot that’s also a lot of why I got into developer tooling and developer experience because that’s a way of scaling something I can do say if I decrease CI time by 50%, that’s going to benefit everyone on my team. But I do think in the age of AI, how much output an engineer has is actually a really good litmus test for how AI-pilled an engineer is, quote-unquote. Because that is an indicator of how are they approaching scaling their stem cells and their own output with AI. Because if I’m seeing, say, when I started coding, I was in college, I was using, like, Sublime Text. There was not even any autocomplete. I had to, like, memorized the whole TypeScript syntax and the JavaScript syntax.

[00:44:15] Itamar Friedman: Oh boy!

[00:44:16] Wei-Wei Wu: And as a hiring manager, every candidate we want to see, we give them a laptop with their coding editor of choice installed, whether it’s Cursor, Codex, or Claude, and we want to see how far can they get with AI help. Because I think that is just an indicator of how well, how good are they at scaling themselves with the tooling that they have. And because, ultimately, we expect AI to drastically scale their output. So, I think it’s not the only signal we look for, but it is definitely an important signal because if they’re still shipping the same number of PRs and having the same amount of impact, then I think that’s a red flag because, like, they’re just falling behind.

[00:45:07] Nnenna Ndukwe: Thank you so much, Wei-Wei, for your perspective on everything. And I’m very curious as we like close out this conversation, what is one change that an engineering leader should make in the next 30 days to really give engineers this ownership of quality that we’ve been talking about this episode?

[00:45:24] Wei-Wei Wu: I think one of the things that they should do, if they haven’t done already, is make every PR have a definition of done section. How did the engineers verify it? Because I do think the proof is in the pudding, and every engineer should be taking the time to ensure what they contribute is incredibly high quality and bug-free. Because ultimately, that is the only thing that matters, especially when writing or generating the lines of code itself is free or is your Cardiovier subscription, but essentially is free. So, I do think the onus is definitely on whoever wrote the code to be like, “Hey, I’ve spent the time to review this, and this is not slop, and test it.” And I think this is similar to how we approach written communication at Momentic. It’s one of our core rules. And I think Sophiebits Clay had a really good article on X about this. Where is essentially the TLDR is incredibly easy for AI to generate slop in both code and text. And you are being a very inconsiderate and lazy person if you’re just telling other people to review it. So, I think that we should take that approach to coding as well. Instead of having, I shouldn’t be subjecting my fellow engineers to review my slot code, I should be taking the time to make sure it’s actually very high quality, and it’s contributing to what I actually want to do.

[00:47:04] Itamar Friedman: Totally.

[00:47:05] Nnenna Ndukwe: Very well said. And then where can listeners follow your work? And yeah, keep up with what you’re up to and what you’re building.

[00:47:13] Wei-Wei Wu: For sure. Yeah. I try to be active on Twitter @x.com/wuweiweiwu.

[00:47:21] Nnenna Ndukwe: Thank you so much for joining us, and I think this was a much-needed conversation. We’ve kind of talked around topics that would concern QA, but we really haven’t been able to dive in deep. And so, thank you for sharing your perspective and what you’re building at Momentic, Wei-Wei.

[00:47:37] Wei-Wei Wu: Yeah. And thanks for having me. Really enjoy this conversation.

[00:47:43] Outro: If today’s conversation challenged how you think about AI and code quality, that’s the point. At Qodo, we believe that independent, context-aware code review with real rules as guardrails is how engineering teams maintain standards at scale. If you’re leading an enterprise team and want to see how intelligent AI code review can reinforce governance, visibility and accountability in your workflow, visit qodo.ai to learn how we help teams turn AI productivity into production-ready quality. And if you enjoyed this episode, subscribe, share it with your engineering leadership circle and leave us a review. Until next time, keep human in the loop and keep shipping.