# Polished Code Can Hide a Broken Model ## Why AI-assisted code review should begin with the system model Author: Jason Doyle Drafted: 16 September 2026 > Disclosure: These views are my own and do not represent my current or any former employers. This paper uses only public sources and generalised engineering examples. It does not describe non-public code, incidents or product information. ## Executive summary Large language models can produce readable, idiomatic code with tests and careful local error handling. The resulting change can still implement an incomplete system contract. A retry may duplicate an external effect. A cache key may omit tenant identity. Two valid operations may lose an update when they run concurrently. AI does not create these defect classes. The cited studies support a narrower point: a passing test suite and a locally plausible patch are bounded evidence. Neither establishes behaviour outside the represented contract. EvalPlus expanded HumanEval's tests by a factor of 80. Across 26 models, HumanEval+ reduced pass@1 by up to 19.3 per cent and pass@100 by up to 28.9 per cent. The stronger suite also changed model rankings.\[1\] A study of SWE-bench Verified, a human-filtered 500-instance subset of the SWE-bench repository benchmark, found that 7.8 per cent of generated patches counted as correct despite failing developer-written tests.\[2\]\[30\]\[31\] Differential testing found behavioural divergence from the human patch in 29.6 per cent of plausible patches, and manual inspection judged 28.6 per cent of those divergent patches certainly incorrect.\[2\] Recent preprints examine the requirement problem directly. SWE-RPG diagnoses agent trajectories and reports implicit-requirement recovery as the main bottleneck, accounting for 24.5 to 46.0 per cent of runs across the evaluated configurations. The average resolved rate across those configurations was 31.5 per cent.\[3\] SpecPath reports different outcomes when the final contract remains equivalent but the requirement history changes.\[4\] These results need replication, although they expose a useful distinction between writing code and resolving the active contract. The original intuition behind this paper was that generated code is only as strong as the person prompting it. That claim is too absolute. A model can know more than its user and can find defects the author missed. In one vendor-run study, model critics caught bugs that paid human reviewers had missed, including errors in data that had been rated flawless.\[5\] The assurance problem is that unrecorded understanding cannot be independently checked or reliably repeated. A constraint becomes reviewable when its scope, its owner and its verification or enforcement path are explicit. Producing several artifacts from one interpretation, such as code, tests and a review summary, does not prove independence or coverage. For this paper, a system model comprises the stated actors, authoritative state, transitions, authority rules, failure assumptions and acceptance properties relevant to one change. The paper separates three scopes of evidence: 1. Surface quality covers readability, structure and local defensive code. 2. Local correctness covers the examples and tests currently visible. 3. System adequacy covers the required behaviour across identity, state, time, concurrency, failure and external effects. These categories describe the scope of evidence. They are not fixed model capabilities. Readable code and a passing visible suite cannot establish the third category by themselves. The title states an engineering possibility and a research hypothesis. Existing work has not tested whether stylistic polish causes professional reviewers to miss system defects. It has established limits in test oracles, contract recovery and human attention. This paper therefore proposes two practical artifacts: a system-model review record for consequential changes and a reproducible experiment that tests hidden invariants separately from visible acceptance checks. AI has made implementation fluency cheaper. Review now needs to establish what the implementation assumes, which assumptions carry consequence and what evidence would reveal that the model is wrong. ## 1. Surface quality has become cheap Source code has always carried social signals. Clear naming suggests that the author understood the domain. Consistent error handling suggests that failure was considered. Small functions, tests and documentation suggest deliberate work. None of these signals ever proved correctness, although they helped reviewers decide where to spend attention. LLMs can now produce these signals on demand. A coding agent can follow repository conventions, add typed errors, update documentation and write plausible tests. It can explain the design in a pull request description. The result often has a level of finish that once required sustained manual effort. That is useful. Readable code remains easier to maintain, and routine implementation quality has real value. The problem is evidential. Surface quality has become a weaker proxy for whether the system model is complete. In code review, I have encountered changes where local construction looked careful and one missed system assumption invalidated the approach. These observations motivate this paper. They are not a measured dataset, and they do not establish that AI-generated changes fail more often than human changes. The flaw may be obvious to a reviewer who has seen the failure mode before. That does not mean the model was incapable of finding it. The model may never have received the relevant context. Its tests may have encoded the same incomplete assumption. Its review prompt may have asked for code quality, security or style without asking whether the proposed state model could satisfy the real system. The result can look confident because it is internally coherent. The implementation commits to one interpretation and completes it thoroughly. Confidence is a human reading of that finish. Code does not possess confidence. The model's output may also contain no reliable confidence signal. Research on code-generating models finds that they are generally not well calibrated out of the box, which limits the use of model confidence for deciding how much review a change needs.\[6\] From the reviewer's perspective, this can resemble "fake it until you make it": the artifact presents completion before its system assumptions have been established. The phrase describes an impression created by the output. The model has no such intention. Polish remains a maintenance property. It says nothing about whether the reasoning behind the change is complete. ## 2. Code quality has three distinct layers Code quality is often discussed as one property. Review becomes clearer when it is separated into three layers. ### 2.1 Surface quality Surface quality covers properties that a reviewer can usually observe directly: - names and structure; - consistency with local conventions; - readable control flow; - comments where the logic needs them; - typed interfaces; - error propagation; - logging and diagnostics; - ordinary unit tests. These properties matter. Poor surface quality increases maintenance cost and can conceal defects. They are also properties for which LLMs have abundant training examples. A model has seen many error wrappers, repository patterns, test arrangements and refactoring shapes. It can often produce a convincing implementation even when the wider problem is underspecified. ### 2.2 Local correctness Local correctness asks whether the change satisfies the contract visible at the implementation boundary. Examples include: - the function returns the expected value for supplied inputs; - an API rejects malformed requests; - a parser handles the listed formats; - a repository test suite passes; - the changed module preserves its documented interface. Local correctness depends on an oracle. The oracle may be a test suite, expected output, type checker, linter or human judgement. Passing the oracle establishes compliance with what the oracle checks. It says less about omitted behaviours. ### 2.3 System adequacy System adequacy asks whether the implemented model is sufficient for the environment in which the code will operate. The system model for one change consists of its actors, authoritative state, transitions, authority rules, failure assumptions and acceptance properties. Adequacy means that this declared model is sufficient for the acceptance properties and consequence class under review. The model cannot describe every future behaviour. System adequacy includes questions that cross function and module boundaries: | Dimension | System question | | --- | --- | | Identity | Is every read and effect scoped to the correct principal or tenant? | | State | Which state is authoritative, cached, derived or temporary? | | Time | What can expire, arrive late or be observed out of order? | | Concurrency | Which operations can interleave, race or overwrite one another? | | Repetition | What happens when a request, event or job executes twice? | | Failure | What state remains after a timeout, crash or partial response? | | Authority | Which actor may perform each effect, including background paths? | | Lifecycle | Who creates, updates, retires and deletes the state? | | Recovery | Can the system determine what completed and resume safely? | | Capacity | Does the design remain valid at the expected scale and cost? | A locally correct function can still participate in an inadequate system. Consider a message handler: ```text receive event validate fields write record send notification acknowledge event ``` Every line can be well structured. If the process crashes after sending the notification and before acknowledging the event, the message may be delivered again. Without an idempotency rule, the same valid local implementation produces duplicate effects. Local exception handling cannot supply the missing delivery semantics or effect identity. This distinction explains why a pull request can require an overhaul after one review comment. The comment did not identify a small coding error. It invalidated an assumption on which the code structure depended. ## 3. The oracle defines visible correctness Generated code is usually evaluated against an oracle. For a function benchmark, the oracle is a set of input-output tests. For a repository task, it may be the existing test suite plus tests added for the issue. For a pull request, it also includes reviewers, static analysis, policy checks and staging behaviour. The oracle is an observation boundary. It cannot establish a property that it does not inspect. ### 3.1 Stronger tests reveal previously accepted failures EvalPlus examined this directly. The researchers extended HumanEval with automatically generated tests using model-based and mutation-based strategies. HumanEval+ contains roughly 80 times as many tests as the original benchmark. Across 26 models, it reduced pass@1 by up to 19.3 per cent and pass@100 by up to 28.9 per cent. The expanded oracle also changed model rankings.\[1\] The original benchmark still has value, although its correctness claim was bounded by its tests. The same problem appears at repository scale. Wang, Pradel and Liu studied plausible patches produced by three issue-solving tools on SWE-bench Verified, the human-filtered 500-instance subset of the SWE-bench repository benchmark.\[2\]\[30\]\[31\] They introduced PatchDiff, which compares generated and human patches through differential testing. Their reported findings include: - 7.8 per cent of patches counted as correct while failing developer-written tests; - 29.6 per cent of plausible patches behaved differently from the human patch; - 28.6 per cent of behaviourally divergent patches were certainly incorrect after manual inspection; - combined validation weaknesses inflated reported resolution rates by 6.2 percentage points.\[2\] Behavioural difference does not automatically mean an error. Two implementations can be different and both valid. The manual result matters because some generated patches changed more behaviour than the issue required or implemented a similar-looking contract that diverged under additional inputs. This is close to the review experience that motivates this paper. The patch can satisfy the visible examples while encoding the wrong boundary around the problem. ### 3.2 Functional correctness is one axis Efficiency provides another example. ENAMEL evaluates code against expert-designed algorithms and strong test generators. Across 30 models, the study found that generated code fell short of expert-level efficiency, particularly where advanced algorithms or implementation optimisation were required.\[7\] A function can return the right answer and still be unsuitable at production scale. Security research shows a similar separation. Pearce and colleagues generated 1,689 programs across 89 deliberately security-relevant scenarios and found that approximately 40 per cent were vulnerable.\[8\] The selected 2021 Codex-era scenarios do not estimate a current base rate for AI-generated code. A later in-the-wild study identified 733 snippets attributed to Copilot, CodeWhisperer or Codeium in public GitHub projects. Static analysis reported weaknesses in 29.5 per cent of Python snippets and 24.2 per cent of JavaScript snippets across 43 CWE categories. Supplying warnings to Copilot Chat fixed up to 55.5 per cent of detected issues.\[9\] The mitigation is instructive. The model could repair many weaknesses once another tool represented them explicitly. ### 3.3 A passing suite can share the same blind spot Generated tests often inherit the implementation's interpretation. If the prompt says "cache account settings by account ID", both code and tests may omit region, tenant or version because one description shaped both. Adding more examples within that interpretation does not challenge the model. This is why test quantity and separate derivation are different properties. A useful test asks whether the implementation handles another example. A stronger test may ask whether the implementation preserves a property across examples the author did not choose: ```text For every tenant pair A and B, an operation authorised for A cannot alter or reveal B's state. ``` The property externalises a system constraint. It can generate many tests without depending on one expected output. ## 4. Missing requirements are a first-class failure mode Software requests are rarely complete. An issue may state the symptom and expected result without documenting every compatibility rule. A feature request often assumes repository conventions that are obvious only to maintainers. Operational requirements live in deployment configuration, incident history or service ownership. Security properties may be absent from the ticket because the team treats them as universal. Human engineers also recover implicit requirements. They ask questions, inspect call sites, read tests and compare the request with the architecture. Experience provides a library of failure modes. Coding agents must perform the same work. ### 4.1 Repository tasks require contract recovery SWE-RPG separates repository issue resolution into requirement clarification, implementation planning and code generation. Its 163 tasks come from 31 Python and Java repositories and include bug fixes and feature additions. Across three coding agents and six model backends, the average resolved rate was 31.5 per cent. The authors identify implicit requirement recovery as the main bottleneck, accounting for 24.5 to 46.0 per cent of agent runs.\[3\] SWE-RPG is a recent preprint. Its taxonomy and attribution method need broader replication. It still advances the question beyond whether a final patch passes. It asks where the agent's understanding diverged from the reference requirement and plan. RepoProbe approaches repository understanding from a different direction. It evaluates open-ended architectural questions from GitHub Discussions and uses checklists of atomic facts instead of one scalar judge score. The authors report a persistent gap between answer clarity and evidence-grounded technical correctness. They also identify edit bias, where models move prematurely towards code changes before establishing the repository architecture.\[10\] This is a useful review warning. A well-formed patch can arrive before the problem model is stable. ### 4.2 Final wording is not the whole specification Requirements evolve. A comment narrows an earlier request. A security condition is added after a prototype. One branch of discussion is rejected. The final contract is the result of that history. SpecPath evaluates whether agents resolve the active contract or follow the most salient path through the history. It holds the repository, final contract, verifier, agent and execution budget fixed. Only the revision path changes. Across five tasks and 14 agent configurations, aggregate accuracy remained similar, yet 35 of 100 completed task blocks that passed the direct specification failed on at least one contract-equivalent history.\[4\] The study is small and is also a preprint. Its main value is the controlled design. It demonstrates why context construction is part of the system and carries its own failure modes. ### 4.3 People do not reliably state what they know The constraint problem cannot be assigned only to models. Bappy and colleagues interviewed 15 professional engineers and observed security-relevant coding tasks completed with AI assistance. None of the observed coding-session participants specified security requirements in the initial prompt, including participants who had the relevant knowledge. The study describes security thinking moving from code construction towards later review. Experience level did not reliably predict security performance.\[11\] The small qualitative sample cannot establish a population rate. It nevertheless supports a practical observation: knowledge in a developer's head does not automatically enter the interaction. Long-term coding memory makes the context effect more direct. Chen and colleagues evaluated insecure coding preferences stored in memory across four models and five languages. In their experiments, insecure memories increased vulnerable-code risk by 2.7 to 50.3 percentage points. Warning-rate increases lagged vulnerability-rate increases by 5.4 to 14.0 points. Two evaluated mitigation strategies lowered vulnerability rates by 19.7 to 33.6 points, although functional correctness fell by as much as 15.9 points in some settings. A third strategy, memory-level safety filtering, detected every evaluated risky memory entry and restored generation behaviour to the baseline without stored memory.\[12\] The evidence does not support a simple rule that more context is always better. Context can encode the wrong constraint. Corrective controls can introduce trade-offs. Memory-level safety filtering, applied to memory contents instead of the requirement context, carried the lowest reported cost in this study. The requirement is provenance: ```text Which constraint entered the model, where did it come from, and what evidence says it still applies? ``` ## 5. Probabilistic discovery is useful and insufficient A capable model can find problems the author did not name. This is one of the strongest benefits of AI-assisted development. A model can recall an API hazard, inspect distant call sites, generate adversarial inputs and challenge an assumption unfamiliar to the developer. The problem is using possible discovery as an assurance mechanism. ### 5.1 Vulnerability judgement remains unstable SecLLMHolmes evaluated eight models across 228 security scenarios and eight investigative dimensions. The authors report non-deterministic answers, incorrect reasoning and weak real-world performance. Small changes that left the underlying weakness intact, such as renaming functions or variables, or adding library functions to the source, caused incorrect answers in 26 per cent of PaLM 2 cases and 17 per cent of GPT-4 cases.\[13\] Current models can still identify vulnerabilities. One clean review result cannot establish robust coverage. ### 5.2 Self-review shares the generator's blind spots Olausson and colleagues evaluated self-repair on HumanEval and APPS using Code Llama, GPT-3.5 and GPT-4. When repair cost was included, gains were often modest, varied across subsets and were sometimes absent. Stronger feedback models produced larger gains. A small human-feedback study still outperformed model self-feedback substantially.\[14\] If one model misunderstood the requirement, asking it to polish or review the resulting code can preserve the same interpretation. The review prompt matters. "Find bugs in this code" invites local inspection. A model-level challenge asks different questions: - Which assumptions does this design make about delivery and retries? - Which identities are absent from keys and queries? - What state can change between validation and use? - Which operation is irreversible? - What happens if the process stops after each external effect? - Which behaviour is accepted only because the tests share the implementation's model? The questions introduce alternative hypotheses. ### 5.3 Model critics can add real value The counterevidence is important. McAleese and colleagues at OpenAI trained model critics to help people evaluate model-written code and evaluated those critics on their own assistant and training data. On code containing naturally occurring model errors, model critiques were preferred to human critiques in 63 per cent of cases. The critics found errors in training data previously rated flawless. They also hallucinated bugs. Human-machine teams found similar numbers of bugs while hallucinating less than critics alone.\[5\] This supports separately prompted AI review. The critic remains a fallible control and is not independent evidence by default. Prompting research also finds that structured criticism can reduce security weaknesses. Tony and colleagues evaluated prompting techniques on 150 security-relevant code-generation prompts across GPT-3, GPT-3.5 and GPT-4. Recursive Criticism and Improvement was particularly effective within the studied conditions.\[15\] A critic can propose where to look. Tests, policy checks, architecture and qualified review determine whether the concern is real. ## 6. Polished output changes the review environment The strongest version of the polish thesis would claim that clean generated code causes reviewers to miss defects. Current evidence does not establish that causal relationship. No study identified for this paper holds a seeded system defect constant, varies only the presentation quality of the code and measures professional reviewer detection. That gap should remain explicit. The title uses "can hide" as an engineering possibility to test. No measured causal effect is asserted. Related findings still justify concern. ### 6.1 Readability can coexist with less attention Al Madi compared model-generated code with human-written code in a study of 21 participants. Static analysis and human annotation found comparable readability and complexity. Eye tracking found that programmers directed significantly less visual attention to model-generated code.\[16\] The study measured inspection behaviour, not defect detection. Participants directed less attention to the model-generated condition even when measured readability was similar. ### 6.2 Reviewer sentiment can diverge from maintainability Huang and colleagues studied code quality and reviewer sentiment around AI-generated pull requests. They report more redundancy and more missed reuse opportunities in agent-generated changes than in human changes. Reviewer language was more neutral or positive towards the generated contributions.\[17\] The authors' own framing describes surface plausibility masking redundancy. Positive sentiment was not shown to cause acceptance or to conceal the issue. The result still shows that technical quality and social response can move separately. ### 6.3 Assistance can increase confidence without increasing safety Perry and colleagues ran a controlled study of security-related programming tasks. Participants with access to an AI assistant wrote significantly less secure code and were more likely to believe that their code was secure. Participants who trusted the assistant less and engaged more with prompt wording produced fewer vulnerabilities.\[18\] Another peer-reviewed study reached a different result. Asare, Nagappan and Asokan studied 25 participants and found that Copilot access was associated with a more secure solution on the harder task, with no measured security effect on the easier task.\[19\] The literature does not support one universal direction. Task difficulty, interaction behaviour, tool design and evaluation method matter. The consistent lesson is that confidence and correctness need separate measurement. ### 6.4 Automation changes monitoring Software review is a specific setting within a longer human-factors problem. Bainbridge's 1983 account of the ironies of automation describes a recurring design failure: automation handles routine control while leaving people to monitor for rare conditions, even though sustained monitoring is a task people perform poorly.\[20\] Parasuraman and Manzey later integrated research on automation complacency and automation bias as attentional effects that can produce omission and commission errors.\[21\] These studies do not concern modern coding agents. They provide a mechanism worth testing: ```text As routine output becomes more reliable, human attention may move away from the rare model failure. ``` Generated code can increase this tension because it produces large changes quickly and presents them in familiar engineering forms. ## 7. Vibe coding is a risk decision "Vibe coding" is a loose label for steering generated code through observed results without understanding every line. It can be reasonable for disposable prototypes, bounded personal scripts and demonstrations using synthetic data. The risk changes when code creates durable state, crosses an authority boundary, serves several tenants or performs an effect that cannot be reversed cheaply. Consequence determines the useful distinction. Authorship does not. Exploratory evidence may be sufficient for a disposable experiment: ```text I ran it and the result looked useful. ``` A production change needs a defined transition from exploration: ```text The affected actors and state are known. The load-bearing invariants are explicit. Relevant failure paths were exercised. Residual risks have owners. ``` This transition can still permit extensive AI generation. It changes what approval means. ## 8. Review the system model before the implementation Traditional code review often begins with the diff. The reviewer follows control flow, checks naming, inspects errors and compares the implementation with the ticket. That remains necessary. It is a poor first step when a polished implementation can make one incomplete interpretation feel settled. The first review target should be the model the change asserts. ### 8.1 Reconstruct the claim Every consequential change implies a statement about the system: ```text If these inputs and states exist, and these actors perform these operations, then these outcomes follow, including under expected failure and repetition. ``` The reviewer should be able to write that statement without reading every implementation detail. If the claim cannot be stated, the diff is premature. ### 8.2 Identify load-bearing assumptions Some assumptions determine the shape of the entire implementation. | Assumption | Review challenge | | --- | --- | | Requests are unique | What creates stable request identity, and how long is it retained? | | Reads remain fresh | What can change after validation and before effect? | | One worker owns the state | Can another worker, retry or user act concurrently? | | Failure is atomic | Which effects can complete before the failure is observed? | | Caller identity is sufficient | Which delegated, scheduled or background paths exist? | | Deletion is final | Which replicas, indexes, caches or derived records remain? | | Ordering is stable | What happens under delayed or reordered delivery? | | Current tests define compatibility | Which callers or behaviours are outside the suite? | One answer may invalidate many lines of otherwise strong code. ### 8.3 Ask for the failure state Happy-path diagrams show intended progress. Review needs the states between steps. For every external effect: 1. What durable fact records intent? 2. What records completion? 3. What happens if the response is lost? 4. How is a retry distinguished from a new request? 5. Can reconciliation determine the actual state? This is where local error handling often creates false comfort. A `try` block can log and retry without knowing whether the remote operation completed. Clean exception handling does not supply missing effect semantics. ### 8.4 Review omissions Generated changes deserve an omission review: - Which identities are never named? - Which transitions have no invalid state? - Which resources have creation without retirement? - Which operations assume one execution? - Which external calls lack a reconciliation path? - Which tests use only values selected by the implementation author? - Which behaviour depends on an undocumented repository convention? This review looks for absent concepts. ### 8.5 Work one case through Return to the notification retry example. The visible request says: ```text Retry transient delivery failures. ``` A polished local implementation may wrap the provider call, classify exceptions and apply exponential backoff. The missing question is whether a timeout means the provider rejected the request or accepted it without returning a response. The system claim needs more detail: ```text For one logical notification ID, the workflow may attempt delivery more than once while committing at most one user-visible notification. An unknown provider outcome enters reconciliation and cannot be treated as an ordinary retry. ``` One possible design creates a durable outbox record with a stable logical ID. A unique constraint prevents a second outbox item for the same effect. A worker obtains an expiring lease before attempting delivery and sends a provider idempotency key derived from that ID. A confirmed response records the provider identifier. A timeout records an unknown state. The evidence then follows the model: | Claim | Evidence | | --- | --- | | Duplicate queue delivery cannot create another effect | Replay the same outbox item and assert one logical provider request | | Two workers cannot both own one attempt | Run a controlled lease interleaving | | A crash after provider acceptance is recoverable | Inject a stop before the local confirmation write | | An unknown response is not treated as failure | Assert transition to reconciliation | | Reconciliation can resolve the outcome | Query by the stable provider key or document that the provider cannot support this contract | The last row may invalidate the design. If the provider has no stable idempotency or lookup capability, the system cannot guarantee both retry progress and at-most-once visible delivery after an unknown outcome. The team must change the provider contract, weaken the product promise or require manual reconciliation. No amount of local exception polish resolves that limitation. The review value came from stating the effect semantics before judging the retry loop. ## 9. Move constraints out of the prompt Prompt quality matters. It should not carry the full assurance burden. Natural language is useful for intent, trade-offs and context. A constraint carried only in prose is easy to omit, reinterpret or supersede. Important constraints should appear in more than one form. ### 9.1 Requirements state the obligation A requirement should identify the actor, state, action and prohibited outcome. Weak: ```text Add retry handling for notification delivery. ``` Stronger: ```text Retry transient delivery failures without producing more than one user-visible notification for one logical notification ID. Persist enough evidence to reconcile an unknown delivery outcome. ``` The stronger statement exposes identity and uncertainty. ### 9.2 Invariants state what must remain true An invariant survives implementation choices: ```text For every logical notification ID, at most one user-visible notification is committed. ``` ```text No principal can read, mutate or infer state owned by another tenant. ``` ```text After any recoverable process failure, the workflow can determine whether each external effect is pending, completed or requires reconciliation. ``` These statements are review anchors. They can guide tests, schemas and operational telemetry. ### 9.3 Executable checks make violations observable Property-based testing, model checking, differential testing and fault injection can make specified violations observable. Their evidential value depends on the validity of the property, workload, environment and fault model. QuickCheck established a practical approach for generating inputs from properties instead of enumerating examples.\[22\] Metamorphic testing supports domains where a complete expected output is unavailable by testing relations that should hold across transformed inputs.\[23\] Mutation analysis can test whether a suite detects representative faults beyond what coverage alone reveals.\[24\] Formal methods provide a stronger option for designs where state and concurrency dominate. Newcombe and colleagues describe how Amazon Web Services used formal specifications to find subtle distributed-system defects and validate designs that were difficult to reason about through ordinary testing alone.\[25\] These methods predate coding agents. Their value increases when code production becomes faster than independent reasoning about the state space. ### 9.4 Diversify controls where possible If one model writes the code, tests and review summary from the same prompt, the artifacts are diverse in format and highly correlated in interpretation. A separate prompt, reviewer or representation may reduce shared context. Independence requires more. A control is independently derived only when its author or process did not rely on the generated implementation or its explanation. Remaining shared assumptions still need to be recorded. Useful diversification can come from: - tests written from a separate invariant specification; - static analysis with rules outside the model context; - a reference implementation; - an independently constructed fault model; - a reviewer who sees the requirements before the diff; - a critic prompted to falsify assumptions; - production reconciliation against an authoritative system. The goal is another opportunity for the incomplete model to encounter a conflicting representation. ## 10. Use AI to challenge the model AI can reduce the cost of adversarial review when it is given the right role. An implementation assistant is optimising towards completion. A review assistant should optimise towards falsification. ### 10.1 Give the critic different evidence The critic should receive: - the original request; - the explicit invariants; - relevant architecture and data ownership; - the diff; - test results; - known operational constraints. It should not receive only the implementation's explanation. That explanation may already rationalise the wrong model. ### 10.2 Ask model-level questions A useful critic prompt asks for: ```text 1. The state machine implied by this change. 2. Assumptions about identity, ordering, freshness and uniqueness. 3. Every external effect and its commit point. 4. Failure states after each effect. 5. Counterexamples that pass the visible tests. 6. Missing evidence needed before approval. ``` This does more than ask for bugs. ### 10.3 Keep findings provisional Critics hallucinate issues.\[5\] A plausible concern needs confirmation through code, documentation, a test or domain review. The critic should produce falsifiable claims: Weak: ```text There may be a race condition. ``` Stronger: ```text Two workers can read version 4, calculate independent updates and write version 5. The second write loses the first because the update has no compare-and-swap condition. Add an interleaving test or show that the storage layer serialises this key. ``` The stronger finding identifies a state, an interleaving and a missing proof. ### 10.4 Separate generation confidence from approval Model confidence is not a review policy.\[6\] Review intensity should follow consequence and evidence. One illustrative policy is: | Change class | Illustrative evidence floor | | --- | --- | | Disposable, isolated prototype | Demonstrated output and bounded environment | | Low-risk internal tool | Tests, dependency review and rollback | | Durable business workflow | Explicit invariants, failure tests and ownership | | Multi-tenant or security boundary | Threat model, separately derived checks and adversarial tests | | Irreversible or high-consequence effect | Deterministic gates, approval and reconciliation | Fluent explanations do not lower the class. ## 11. Review capacity is part of the system AI changes how much code a team can produce. That increase does not automatically add maintainers who understand the system, reviewers with the relevant context or production evidence. A 2026 Meta preprint reports that significant lines of code per human-landed diff grew 105.9 per cent year over year and per-developer diff volume grew 51 per cent, with agentic AI responsible for more than 80 per cent of the growth. The share of diffs receiving timely review declined. The paper describes a widening gap between code supply and reviewer bandwidth.\[26\] This is one company's report, written by people involved in the system, and its observational comparisons do not establish universal causality. Its control design is more important than the headline. RADAR classifies authorship and source, applies eligibility gates, static heuristics, a learned risk score, model review and deterministic validation before qualifying a change for automated landing. The reported deployment covered more than 535,000 reviewed diffs. The authors report that RADAR-landed diffs had one third the revert rate and one fiftieth the production incident rate of non-RADAR diffs. These operating-team comparisons cover a selected low-risk population and do not provide an independent evaluation against otherwise equivalent changes.\[26\] The mechanism does not assume that an LLM review establishes safety. It restricts automation to a measured low-risk population and layers diversified and deterministic controls. This suggests a general review policy: ```text Code-generation capacity is not approval capacity. ``` Teams should measure: - change volume by consequence class; - reviewer time and queue age; - rework after review; - escaped defects; - rollback and revert rate; - ownership concentration; - the percentage of generated changes with explicit invariants; - the percentage of high-consequence changes with separately derived evidence. Faster code production can otherwise turn review into a throughput obstacle that teams work around. Productivity evidence also varies by setting. A Google randomised trial with 96 engineers estimated that AI shortened time on one complex enterprise task by about 21 per cent, with a wide confidence interval and explicit limits on generalisation.\[27\] A METR randomised trial with 16 experienced open-source developers across 246 tasks found a 19 per cent increase in completion time, while participants believed AI had reduced it by 20 per cent.\[28\] These studies measure different populations and tasks. They show why perceived fluency, throughput and system outcome should not be collapsed into one productivity claim. ## 12. A practical programme for AI-assisted changes The following programme is a proposal for consequential changes. It has not been validated as a universal minimum. It is intended for changes that affect authoritative state, cross-tenant behaviour, security boundaries, irreversible external effects, regulated decisions or safety. Teams should define consequence thresholds, the authority for accepting residual risk and measures of review time, rework and escaped defects before adopting it. ### 12.1 Classify consequence Record: - affected users and tenants; - durable state; - external effects; - authority boundaries; - recovery difficulty; - security and compliance relevance. This determines review depth. ### 12.2 Write the system claim Describe the intended behaviour before relying on the implementation's structure. Include normal operation, repetition, concurrency and partial failure. ### 12.3 List the invariants Write properties that must remain true. Give each invariant an owner and an evidence source. ### 12.4 Identify unknowns Mark requirements that depend on repository convention, external service semantics or unresolved product decisions. Do not let the model silently choose them. ### 12.5 Generate against the contract Provide the model with the requirement, invariants, architecture and relevant repository context. Ask it to state assumptions before editing. ### 12.6 Build a separately derived oracle Create tests and checks from the invariant set without relying on the generated implementation or its explanation where practical. Record any shared assumptions. Include adverse sequencing and failure. ### 12.7 Run an adversarial model review Ask a critic to reconstruct and falsify the system model. Require concrete counterexamples and evidence requests. ### 12.8 Perform human system-model review Review the state model and load-bearing assumptions before implementation style. Confirm which concerns are real and which are critic errors. ### 12.9 Review the diff Once the system model is accepted, inspect local correctness, maintainability, security and repository conventions. ### 12.10 Retain production evidence For durable workflows, observe duplicate suppression, reconciliation, authorisation failures, stale-state rejection and recovery. Review whether the original invariants still describe production. The sequence is deliberate. A line-level review is more useful after the reviewer knows what the lines must preserve. ## 13. A reproducible hidden-invariant evaluation The polish hypothesis deserves direct testing. The following protocol is a proposal. No results are reported here. ### 13.1 Research questions 1. How often does generated code pass a visible suite while failing a hidden system invariant? 2. How does explicit constraint representation change that rate? 3. Does separately prompted adversarial review detect failures missed by generation and self-review? 4. Does presentation polish change human detection time, accuracy or confidence when the underlying defect is held constant? ### 13.2 Task design Use small repository-level tasks with realistic system obligations. Each task should have: - a normal issue description; - public acceptance checks; - a pre-adjudicated contractual behaviour catalogue; - a withheld hold-out suite; - a documented fault and workload model; - a documented consequence class. Suggested invariant classes: | Class | Example hidden property | | --- | --- | | Idempotency | One logical effect ID produces at most one committed user-visible effect | | Tenancy | Requests authorised for tenant A cannot reveal or mutate tenant B through foreground, cache, background or recovery paths | | Concurrency | Concurrent updates produce the prescribed serial result or the specified conflict outcome | | Freshness | An effect rejects or revalidates state whose version changed after validation | | Partial failure | Recovery classifies each attempted external effect as pending, complete or unknown and routes unknown outcomes to reconciliation | | Authority | Background and delegated paths enforce the same permission boundary | | Lifecycle | Retired state cannot remain active through cache or derived storage | | Capacity | At workload W on declared hardware, p95 latency and memory remain within stated limits | ### 13.3 Generation conditions Use a factorial design with two separately varied factors. Constraint representation: | Level | Information available | | --- | --- | | Minimal | Issue description and repository | | Prose constraints | Minimal condition plus explicit invariants | | Executable constraints | Prose condition plus public acceptance checks derived from the invariants | Review condition: | Level | Review activity | | --- | --- | | Control | Equivalent additional budget used to finalise the answer without a critique instruction | | Self-review | The generation context is asked to find and repair model-level defects | | Separate critic | A fresh context receives the contract, repository, patch and a falsification prompt | The separate critic is a diversified control and is not independent evidence by default. Give every cell the same generation budget, additional model calls, token limit, tool permissions and elapsed-time limit. Record model and tool versions. Pre-register task count, runs per cell, sampling settings, exclusion rules and the analysis plan. Treat task and model as sources of variation. Do not pool every run as interchangeable. Report per-task paired differences where the design permits them. Confidence intervals should resample whole tasks or use a pre-specified hierarchical model. Repeated generations of one task do not create new independent tasks. The visible suite and the public acceptance checks form one scored artifact. Every condition is evaluated against them, while only the executable-constraint level supplies them to the agent before submission. Every condition is also evaluated against a separately authored hold-out suite using different inputs, schedules and fault injections. Report semantic overlap between public and hold-out checks. ### 13.4 Contract and oracle design The visible suite should be credible and incomplete. Do not make it deliberately trivial. The contractual behaviour catalogue is primary. Two blinded assessors should review the catalogue before generation. Report their agreement, use a third assessor to resolve disagreements or mark the behaviour disputed. The hold-out suite should: - exercise the pre-adjudicated contract; - use property-based or metamorphic tests where examples are insufficient; - include controlled fault injection; - include declared concurrency schedules and workload limits; - kill seeded mutants that the visible suite misses. Differential comparison with a reference implementation may generate cases for adjudication. A divergence is not itself a failure. Blinded assessors should classify each divergence as correct, incorrect or undetermined against the contract. Coverage and mutation results are adequacy diagnostics. They do not establish that the contract, property or fault model is complete. EvalPlus provides the two-tier testing precedent.\[1\] PatchDiff provides a method for exposing behavioural differences for later adjudication.\[2\] Mutation testing provides an adequacy check for the hold-out suite.\[24\] ### 13.5 Human review condition To test the claim about polish, create paired patches with the same seeded system defect: - one has ordinary but acceptable presentation; - one differs only in validated presentation features such as formatting and semantically neutral identifier names; - executable behaviour, tests, test outcomes, functional changed-line count, authorship framing, pull request description and review information remain equivalent; - total diff size and expected reading time remain within a declared tolerance; - a pilot confirms that reviewers perceive the intended difference in polish; - no reviewer sees both variants of one task. Measure: - detection of the pre-specified contractual defect; - time to an evidence-backed detection; - approval decision; - numerical probability assigned to patch correctness before approval; - false-positive concerns; - which evidence changed the decision. Randomly assign reviewers to variants and counterbalance tasks across conditions. Use blinded adjudication against a pre-specified detection rubric. A fixed review window or survival analysis should handle reviews that end without detection. Pre-register a target effect size, minimum detectable effect and required sample before recruitment. Professional reviewers should be sampled across relevant domains. The study should be pre-registered because task selection and exclusion rules can easily move the result. ### 13.6 Reporting Report visible and hidden outcomes separately. ```text visible-suite pass rate hold-out contract pass rate task-runs with at least one invariant violation adjudicated behavioural divergence critic true-positive and false-positive rate reviewer detection rate reviewer confidence calibration score time to a corrected patch that passes both suites ``` Calculate critic rates only after every reported finding has been adjudicated, including findings unrelated to the seeded defect. Define a corrected patch as one that passes the public and hold-out suites without introducing a new adjudicated contract violation. Do not combine style, test success and invariant success into one quality score. The useful result would identify where a control changes outcomes. It would not prove that one model, prompt or reviewer represents all software development. ## 14. Counterarguments ### 14.1 Strong models already infer missing constraints They sometimes do. This is a reason to use them in review. Inference remains probabilistic and context-sensitive. SecLLMHolmes shows material judgement changes after small code transformations.\[13\] SpecPath shows different outcomes from contract-equivalent requirement histories.\[4\] A constraint that matters to every production execution needs a more stable control than probabilistic inference. ### 14.2 Better prompts will solve the problem Better prompts can improve outcomes. Secure prompting research and the memory study both provide evidence that represented requirements change generated code.\[12\]\[15\] Representation is necessary and insufficient. Zhu, Tsantalis and Rigby report in a 2026 preprint that detailed prompting and functional correctness did not prevent architectural degradation in the systems they evaluated. They identify code volume as a strong predictor of structural decline.\[29\] Prompt quality cannot replace architecture management, separately derived tests or scope control. ### 14.3 AI can improve security and productivity Yes. The Asare study found a security benefit on its harder task.\[19\] The Google trial measured a development-speed improvement in its setting.\[27\] Model critics can find bugs missed by people.\[5\] These benefits are compatible with the paper's narrower claim. Implementation fluency does not establish that the system model is adequate. ### 14.4 Human-written code has the same defects It does. Humans omit requirements, overfit tests and write polished implementations of wrong assumptions. The classic work on distributed failures, formal methods and automation predates current coding agents. AI matters because it changes scale and can make uncertainty less visible in the finished artifact. The appropriate response is stronger engineering discipline for consequential changes, regardless of authorship. ### 14.5 Tests cannot represent every requirement Correct. Some properties depend on product judgement, law, human impact or an environment that cannot be reproduced. Each material constraint should instead name its evidence form. This may be an executable test, policy review, threat model, staged observation, approval, reconciliation or explicit accepted risk. An untestable requirement still benefits from being visible and owned. The strongest objection is treated separately in section 15. ## 15. The strongest counterargument The strongest objection is that the paper places too much weight on representation. Software systems contain tacit knowledge. Experienced engineers recognise patterns they cannot fully enumerate in advance. Requirements emerge through use. A model can connect evidence across repositories and documentation that no one person has assembled. Excessive specification can slow work, freeze a mistaken design or create a large set of controls that all encode the same misconception. This objection is valid. The goal cannot be complete specification. That standard is impossible for meaningful software. The practical target is load-bearing uncertainty: an unresolved assumption whose failure would change the architecture, violate an acceptance property or increase the consequence class. A team should represent the constraints whose failure would invalidate the design or create unacceptable consequence. It should preserve room for model and human discovery while testing whether several artifacts share one blind spot. Representation does not guarantee correctness. It creates something that can be challenged. The model may know more than the prompter. The system is stronger when different sources of knowledge meet in inspectable claims and separately derived evidence. ## 16. What this paper does not claim This paper does not claim: - that AI-generated code is generally worse than human-written code; - that polished code is more likely to contain a defect; - that presentation quality has been proven to cause reviewers to miss bugs; - that developers must specify every requirement in one prompt; - that LLMs cannot discover unprompted vulnerabilities or design flaws; - that self-review and model critics have no value; - that tests can encode every system obligation; - that seniority reliably protects reviewers from automation bias; - that the proposed evaluation has been run or has produced results; - that one review process fits every consequence class. The paper claims that surface quality, local correctness and system adequacy are different properties. It argues that consequential constraints need inspectable representation and evidence beyond the generator's fluency. ## Conclusion LLMs can produce strong implementations of incomplete contracts. Stronger test oracles expose wrong code that simpler suites accepted.\[1\]\[2\] Recent repository studies report failures in implicit-requirement recovery and sensitivity to requirement history.\[3\]\[4\] Model critics can improve review, particularly when people adjudicate their findings.\[5\] The presentation claim remains a testable hypothesis, and section 13 sets out how to test it. The engineering response is still useful. Review can begin with the system claim. Teams can record load-bearing invariants, construct separately derived checks, exercise failure states and use AI critics to challenge assumptions. The apparent finish of the implementation does not alter the required evidence. The person prompting the model does not need to know everything, and the model does not need to discover everything alone. An important constraint needs somewhere to become visible before the code reaches production. ## Appendix A: System-model review record ```text Change Title: Owner: Repository: Commit or pull request: Consequence class: Intent User or system outcome: Explicit exclusions: Compatibility obligations: State model Authoritative state: Cached or derived state: Temporary state: Ownership and tenancy: Lifecycle: Execution model Entry points: Background paths: Concurrent actors: Retry sources: Ordering assumptions: External effects: Irreversible effects: Invariants Identity and authority: State: Repetition: Concurrency: Freshness: Partial failure: Recovery: Capacity: Evidence Visible tests: Hold-out or adversarial tests: Derivation source and shared assumptions: Static analysis: Fault injection: Differential or reference check: Model critic: Human reviewers: Unknowns Unresolved requirements: Unverified dependency behaviour: Accepted risks: Owners and review dates: ``` ## Appendix B: Pull request checklist ```text [ ] I can state the system claim without relying on the diff. [ ] The consequence class and affected principals are clear. [ ] Load-bearing assumptions are written down. [ ] State ownership and authority are explicit. [ ] Retry and duplicate-delivery behaviour are defined. [ ] Concurrent operations and stale reads were considered. [ ] External effects have commit and reconciliation semantics. [ ] Partial failures leave a recoverable or diagnosable state. [ ] Tests cover properties outside the examples chosen by the implementation. [ ] At least one check was derived without relying on the generated implementation. [ ] Shared assumptions between evidence sources are recorded. [ ] AI review findings were confirmed with evidence. [ ] Reviewer confidence is based on evidence, not code finish. [ ] Residual risks have owners and review dates. ``` ## About the author Jason Doyle writes about reliable software, observability, incident leadership, applied AI and practical controls for systems that influence human and organisational decisions. He publishes at [jasondoyle.ie](https://jasondoyle.ie) and can be contacted at [contact@jasondoyle.ie](mailto:contact@jasondoyle.ie). ## References 1. Jiawei Liu et al., _Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation_, NeurIPS 2023, arXiv:2305.01210v3, [https://arxiv.org/abs/2305.01210](https://arxiv.org/abs/2305.01210). 2. You Wang, Michael Pradel, and Zhongxin Liu, _Are "Solved Issues" in SWE-bench Really Solved Correctly? An Empirical Study_, ICSE 2026, DOI 10.1145/3744916.3764576, arXiv:2503.15223v2, [https://arxiv.org/abs/2503.15223](https://arxiv.org/abs/2503.15223). 3. Xin Zhou et al., _A Unified Issue Resolution Benchmark for Requirement Clarification, Planning, and Code Generation for Coding Agents_, preprint, 10 August 2026, arXiv:2608.09072, [https://arxiv.org/abs/2608.09072](https://arxiv.org/abs/2608.09072). 4. Yangfan Wu et al., _SpecPath: Testing Coding Agents Across Contract-Equivalent Specification Histories_, preprint, 10 August 2026, arXiv:2608.09799, [https://arxiv.org/abs/2608.09799](https://arxiv.org/abs/2608.09799). 5. Nat McAleese et al., _LLM Critics Help Catch LLM Bugs_, preprint, 28 June 2024, arXiv:2407.00215, [https://arxiv.org/abs/2407.00215](https://arxiv.org/abs/2407.00215). 6. Claudio Spiess et al., _Calibration and Correctness of Language Models for Code_, ICSE 2025, arXiv:2402.02047v4, [https://arxiv.org/abs/2402.02047](https://arxiv.org/abs/2402.02047). 7. Ruizhong Qiu et al., _How Efficient is LLM-Generated Code? A Rigorous & High-Standard Benchmark_, ICLR 2025, arXiv:2406.06647v4, [https://arxiv.org/abs/2406.06647](https://arxiv.org/abs/2406.06647). 8. Hammond Pearce et al., _Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributions_, IEEE Symposium on Security and Privacy 2022, arXiv:2108.09293v3, [https://arxiv.org/abs/2108.09293](https://arxiv.org/abs/2108.09293). 9. Yujia Fu et al., _Security Weaknesses of Copilot-Generated Code in GitHub Projects: An Empirical Study_, accepted for ACM Transactions on Software Engineering and Methodology, 2025, arXiv:2310.02059v4, [https://arxiv.org/abs/2310.02059](https://arxiv.org/abs/2310.02059). 10. Yuexi Yang et al., _RepoProbe: Benchmarking Architecture-Aware Repository Comprehension with Checklists_, ASE 2026, arXiv:2608.04783v2, [https://arxiv.org/abs/2608.04783](https://arxiv.org/abs/2608.04783). 11. Faisal Haque Bappy et al., _From Preventive to Reactive: How AI Coding Assistants Transform Developers' Security Awareness_, SOUPS 2026, arXiv:2605.23130v2, [https://arxiv.org/abs/2605.23130](https://arxiv.org/abs/2605.23130). 12. Yuchen Chen et al., _Insecure Coding Preferences in Long-Term Memory: Security Risks for LLM-based Code Generation_, ISSTA 2026, arXiv:2607.17619, [https://arxiv.org/abs/2607.17619](https://arxiv.org/abs/2607.17619). 13. Saad Ullah et al., _LLMs Cannot Reliably Identify and Reason About Security Vulnerabilities (Yet?): A Comprehensive Evaluation, Framework, and Benchmarks_, IEEE Symposium on Security and Privacy 2024, arXiv:2312.12575v3, [https://arxiv.org/abs/2312.12575](https://arxiv.org/abs/2312.12575). 14. Theo X. Olausson et al., _Is Self-Repair a Silver Bullet for Code Generation?_, ICLR 2024, arXiv:2306.09896v5, [https://arxiv.org/abs/2306.09896](https://arxiv.org/abs/2306.09896). 15. Catherine Tony et al., _Prompting Techniques for Secure Code Generation: A Systematic Investigation_, accepted for ACM Transactions on Software Engineering and Methodology, 2025, arXiv:2407.07064v2, [https://arxiv.org/abs/2407.07064](https://arxiv.org/abs/2407.07064). 16. Naser Al Madi, _How Readable is Model-generated Code? Examining Readability and Visual Inspection of GitHub Copilot_, ASE 2022, DOI 10.1145/3551349.3560438, arXiv:2208.14613v2, [https://arxiv.org/abs/2208.14613](https://arxiv.org/abs/2208.14613). 17. Haoming Huang et al., _More Code, Less Reuse: Investigating Code Quality and Reviewer Sentiment towards AI-generated Pull Requests_, MSR 2026, arXiv:2601.21276, [https://arxiv.org/abs/2601.21276](https://arxiv.org/abs/2601.21276). 18. Neil Perry et al., _Do Users Write More Insecure Code with AI Assistants?_, ACM CCS 2023, DOI 10.1145/3576915.3623157, arXiv:2211.03622v3, [https://arxiv.org/abs/2211.03622](https://arxiv.org/abs/2211.03622). 19. Owura Asare, Meiyappan Nagappan, and N. Asokan, _A User-centered Security Evaluation of Copilot_, ICSE 2024, DOI 10.1145/3597503.3639154, arXiv:2308.06587v4, [https://arxiv.org/abs/2308.06587](https://arxiv.org/abs/2308.06587). 20. Lisanne Bainbridge, _Ironies of Automation_, Automatica, volume 19, issue 6, 1983, DOI 10.1016/0005-1098(83)90046-8, [https://doi.org/10.1016/0005-1098(83)90046-8](https://doi.org/10.1016/0005-1098%2883%2990046-8). 21. Raja Parasuraman and Dietrich H. Manzey, _Complacency and Bias in Human Use of Automation: An Attentional Integration_, Human Factors, volume 52, issue 3, 2010, DOI 10.1177/0018720810376055, [https://doi.org/10.1177/0018720810376055](https://doi.org/10.1177/0018720810376055). 22. Koen Claessen and John Hughes, _QuickCheck: A Lightweight Tool for Random Testing of Haskell Programs_, ICFP 2000, DOI 10.1145/351240.351266, [https://doi.org/10.1145/351240.351266](https://doi.org/10.1145/351240.351266). 23. Tsong Yueh Chen et al., _Metamorphic Testing: A Review of Challenges and Opportunities_, ACM Computing Surveys, volume 51, issue 1, 2018, DOI 10.1145/3143561, [https://doi.org/10.1145/3143561](https://doi.org/10.1145/3143561). 24. Rene Just et al., _Are Mutants a Valid Substitute for Real Faults in Software Testing?_, FSE 2014, DOI 10.1145/2635868.2635929, [https://doi.org/10.1145/2635868.2635929](https://doi.org/10.1145/2635868.2635929). 25. Chris Newcombe et al., _How Amazon Web Services Uses Formal Methods_, Communications of the ACM, volume 58, issue 4, 2015, DOI 10.1145/2699417, [https://doi.org/10.1145/2699417](https://doi.org/10.1145/2699417). 26. Chris Adams et al., _Automating Low-Risk Code Review at Meta: RADAR, Risk Calibration, and Review Efficiency_, preprint, 28 May 2026, arXiv:2605.30208v2, [https://arxiv.org/abs/2605.30208](https://arxiv.org/abs/2605.30208). 27. Elise Paradis et al., _How Much Does AI Impact Development Speed? An Enterprise-Based Randomized Controlled Trial_, preprint, first submitted 16 October 2024, arXiv:2410.12944v3, [https://arxiv.org/abs/2410.12944](https://arxiv.org/abs/2410.12944). 28. Joel Becker et al., _Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity_, preprint, first submitted 12 July 2025, arXiv:2507.09089v2, [https://arxiv.org/abs/2507.09089](https://arxiv.org/abs/2507.09089). 29. Yuecai Zhu, Nikolaos Tsantalis, and Peter C. Rigby, _AI-Generated Smells: An Analysis of Code and Architecture in LLM and Agent-Driven Development_, preprint, 4 May 2026, arXiv:2605.02741, [https://arxiv.org/abs/2605.02741](https://arxiv.org/abs/2605.02741). 30. Carlos E. Jimenez et al., _SWE-bench: Can Language Models Resolve Real-World GitHub Issues?_, ICLR 2024, arXiv:2310.06770v3, [https://arxiv.org/abs/2310.06770](https://arxiv.org/abs/2310.06770). 31. Princeton NLP, _SWE-bench Verified Dataset Card_, accessed 16 September 2026, [https://huggingface.co/datasets/princeton-nlp/SWE-bench_Verified](https://huggingface.co/datasets/princeton-nlp/SWE-bench_Verified).