AI has become one of the most genuinely useful tools in a software engineer's kit. It spots code smells, flags common vulnerabilities, suggests refactorings, explains unfamiliar code in seconds, and takes real drudgery off a developer's plate. Used well, it makes engineers faster. Used carelessly, it quietly replaces engineering judgment with statistical pattern matching — and organizations are starting to let that happen without noticing.
The problem was never the technology. It's what gets outsourced to it.
Large language models are trained on enormous volumes of public source code, which makes them genuinely good at recognizing what code looks like: common design patterns, language idioms, framework conventions, known vulnerability shapes, style inconsistencies. What they have no access to is why the software exists in the first place — who the customers are, what regulations apply, what operational constraints shaped a decision, what trade-off a previous team already made and documented nowhere but in a Slack thread from two years ago. That context never made it into the source code, so no amount of training on source code will teach a model to infer it.
Picture a banking application with a conditional that triggers extra verification above a certain transaction amount. A model reviewing that code might reasonably suggest simplifying the branching, extracting a strategy pattern, or improving readability — all technically sound observations about the shape of the code. What it has no way of knowing is that the threshold exists because of anti-money-laundering regulation, a fraud-detection requirement, or a central bank reporting obligation. To the model, it's just a conditional. To the business, altering it without understanding why it's there could mean real regulatory exposure. The hard part was never the language. It's the domain — which is exactly the thing a discipline like Domain-Driven Design was built to take seriously: Eric Evans's foundational argument, in the book of the same name, is that the real complexity in most software lives in the business domain itself, not in the code that represents it, and that the domain has to be modeled deliberately rather than inferred after the fact.
A troubling pattern has started showing up around AI-assisted code review. An organization hires a consultancy for its supposed expertise in some technology stack, expecting experienced engineers to actually look at the software. What sometimes happens instead: the source code gets fed into an LLM, the LLM generates review comments, pull requests get blocked until every comment is addressed, and a report gets delivered that looks comprehensive and professional without anyone involved having understood the business the code serves. Reviewing syntax isn't the same activity as reviewing software, even when the output is formatted identically.
This matters because every review — human or automated — can always find something. Another abstraction, another optimization, another naming tweak, another dependency bump, another architectural alternative. Software is never finished being improvable, which is precisely why engineering isn't about implementing every possible suggestion. It's about deciding which suggestions create meaningful value for this system, at this point in its life, given everything the reviewer knows that the tool doesn't.
No review — automated or otherwise — eliminates technical debt permanently, because debt isn't a symptom of carelessness. It's a consequence of software existing in time. Today's accepted practice is tomorrow's legacy pattern. Today's secure dependency is tomorrow's disclosed vulnerability. Expecting an AI review to produce "perfect" code misunderstands what software engineering actually is: an ongoing negotiation between competing pressures, not a one-time correctness proof.
The more dangerous failure mode isn't an AI making a wrong call — it's an organization that treats every AI comment as a mandatory action item regardless of whether it's right. Once that happens, developers stop asking is this recommendation correct and start asking how do I make the comment disappear. That's not engineering judgment. It's compliance theater, and it trains a team to optimize for passing a review tool instead of solving the actual problem in front of them.
It's worth being precise about what distinguishes this from a legitimate checklist. Atul Gawande's The Checklist Manifesto makes a strong case that checklists genuinely save lives in aviation and surgery — but the checklists that work encode hard-won expert judgment about what actually goes wrong, refined by people who understand the domain, kept deliberately short, and used to catch known failure modes rather than to generate an ever-expanding list of stylistic opinions. An AI-generated review that must be fully satisfied before merging isn't that kind of checklist. It's closer to what researchers studying human-automation interaction call automation bias — the well-documented tendency for people to defer to an automated system's output even when it's wrong or beside the point, especially under time pressure. A pull request gate that can't be reasoned with produces exactly the conditions where that bias thrives.
Nearly every meaningful software decision is contested by design: optimize for performance or readability, add another abstraction or leave the duplication, refactor now or ship the feature. The honest answer is almost always "it depends" — on business value, delivery timelines, regulatory exposure, operational risk, and cost of maintenance. None of that is recoverable from source code alone, which is exactly why the best code reviews have always been conversations rather than verdicts. A reviewer asks why something was built a certain way; the author explains that changing it would break an integration contract with hundreds of existing customers; the conversation resolves the question in a way no static analysis ever could. An AI can surface the "why did you do it this way" question. It can't have the conversation that follows, because the answer lives in institutional memory, not in the diff.
Knowing C# doesn't make someone a banking expert. Knowing Java doesn't make someone a healthcare expert. Knowing Python doesn't make someone an aviation expert. Real expertise is technical knowledge plus domain understanding plus operational experience plus historical context plus a feel for business priorities — and AI, today, is genuinely useful for exactly one of those five ingredients. That's not a small contribution. It's just not the whole job, and treating it as though it were is how organizations end up rejecting sound engineering decisions because a model disagreed with them for reasons that don't actually apply.
None of this is an argument against using AI in engineering workflows — it's an argument about where the authority sits. AI should surface possible issues, point out alternative approaches, flag security concerns worth a second look, and take repetitive review work off people's plates. It shouldn't become the final word on whether code is "correct," and a pull request shouldn't get rejected simply because a model disagreed with an implementation choice. It should get discussed, because the model noticed something worth a human looking at — which is a meaningfully different thing from the model having the final say.
The future of this work isn't engineers versus AI. It's engineers who stay responsible for the decision, using AI as one more input among several, because software was never just code. It's regulation, operational history, customer commitments, and years of trade-offs that never made it into any file the model could have trained on. AI can read your codebase fluently. It has no way of knowing your bank — and that gap is exactly why the judgment still has to belong to the engineer.