{"id":7067556,"date":"2026-09-26T15:19:43","date_gmt":"2026-09-26T15:19:43","guid":{"rendered":"https:\/\/peraltafinancing.com\/business\/internet-business\/best-ai-models-for-coding-we-tested-7-on-real-production-work\/"},"modified":"2026-09-26T15:19:43","modified_gmt":"2026-09-26T15:19:43","slug":"best-ai-models-for-coding-we-tested-7-on-real-production-work","status":"publish","type":"post","link":"https:\/\/fivemor.com\/?p=7067556","title":{"rendered":"Best AI models for coding? We tested 7 on real production work"},"content":{"rendered":"


\n<\/p>\n

\n
\n
\n

\n Friday September 25, 2026 <\/p>\n<\/p><\/div>\n<\/p><\/div>\n

\n \"What\"What <\/div>\n

Seven AI models read the same ticket and reached the same answer. They were all wrong.<\/p>\n

The ticket said add a 30-second timeout, so they wrote a 30-second timeout. The engineer who\u2019d shipped this task wrote 20 instead.<\/p>\n

They\u2019d read the codebase, which already defined a 20-second timeout for that type of service because the system caps how long any single request can wait. A 30-second timeout would never have fired.<\/p>\n

That information was sitting right there in the code. The AI models had access to it, but none of them looked.<\/p>\n

That was one of 12 pull requests (PRs) in an experiment we ran. We spent $1,024 letting seven models attempt work our engineers had already written, reviewed, and shipped to production. We reset each codebase to the commit before the engineer started, gave the models the same Jira tickets, and let them work alone with no human help.<\/p>\n

Five of the seven finished within four hundredths of a point of each other. But the scores don\u2019t tell the full story. The timeout was the most obvious example of a shared blind spot. It wasn\u2019t the only one.<\/p>\n

What we tested and how<\/h2>\n

We picked 12 PRs from six different codebases, all shipped to production. The tasks ranged from small (4 files, 33 lines) to large (21 files, 2,497 lines), and covered ordinary engineering work: a new API endpoint, a ticketing-system update, a payment provider safeguard, a frontend CTA behavior change, template version bumps.<\/p>\n

Each model got the task description straight from the Jira ticket, unedited, and worked alone with no human help. No rewriting to make it clearer or more model-friendly, no follow-up questions answered, no \u201ctry again.\u201d They also couldn\u2019t access our internal tools and never saw the engineer\u2019s solution.<\/p>\n

All seven models ran inside Claude Code with identical settings across all 84 runs, routed through nexos.ai<\/a>. The only thing that changed was which model answered.<\/p>\n

For scoring, we used GPT-6 Astra, a model that wasn\u2019t part of the test. It compared each model\u2019s work to the engineer\u2019s solution, scored the match from 0 to 1, and wrote out its reasoning on every run.<\/p>\n

Here\u2019s what came back.<\/p>\n

\n\n\n\n\n\n\n\n\n\n\n
Model<\/strong><\/td>\nMatch to engineer\u2019s work (0\u20131)<\/strong><\/td>\nCost\/run<\/strong><\/td>\nTime\/run<\/strong><\/td>\nTokens\/run<\/strong><\/td>\n<\/tr>\n
Kimi K3<\/td>\n0.88<\/td>\n$8.42<\/td>\n17.8 min<\/td>\n6.38M<\/td>\n<\/tr>\n
Claude Opus<\/td>\n0.87<\/td>\n$24.99<\/td>\n7.6 min<\/td>\n4.84M<\/td>\n<\/tr>\n
Claude Sonnet<\/td>\n0.86<\/td>\n$16.69<\/td>\n12.8 min<\/td>\n7.09M<\/td>\n<\/tr>\n
Qwen 3.8-Max<\/td>\n0.86<\/td>\n$11.83<\/td>\n25.2 min<\/td>\n6.48M<\/td>\n<\/tr>\n
GLM Flash<\/td>\n0.83<\/td>\n$7.20<\/td>\n12.7 min<\/td>\n8.54M<\/td>\n<\/tr>\n
DeepSeek V4 Flash<\/td>\n0.74<\/td>\n$12.96<\/td>\n9.2 min<\/td>\n3.08M<\/td>\n<\/tr>\n
GPT Luna<\/td>\n0.67<\/td>\n$3.26<\/td>\n15.5 min<\/td>\n2.54M<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/figure>\n

Five models finished with almost the same score<\/h2>\n

Kimi K3 scored 0.88. Claude Opus scored 0.87. Sonnet and Qwen 3.8-Max both hit 0.86. GLM Flash came in at 0.83. Five models, five different companies, separated by four hundredths of a point.<\/p>\n

The gap only opened up at the bottom: DeepSeek V4 Flash scored 0.74, GPT Luna 0.67.<\/p>\n

Based on what we saw, Kimi K3 is a solid pick if you want a single winner since it produced the best answer on 8 of the 12 tasks. But on everyday maintenance work in a mature codebase, these five models perform about the same.<\/p>\n

We saw a similar pattern when we benchmarked four AI models on creative and analytical tasks<\/a> \u2013 the newest model wasn\u2019t always the best.<\/p>\n

Where they differ is speed. Opus finished in 7.6 minutes per task on average and was fastest on 8 of the 12 tasks. Kimi K3 took 17.8 minutes. Qwen 3.8-Max took 25.2 minutes, and its slowest single run lasted an hour and a half.<\/p>\n

Speed isn\u2019t fixed the way pricing is. It depends on provider load, network path, and spare capacity. We measured each model on a different day within a single week. That means a big gap like Opus at 8 minutes vs Qwen at 25 is real. But if two models are a couple of minutes apart, that could easily flip on a different day.<\/p>\n

Whether speed matters depends on how you\u2019re using these models. A developer watching a progress bar feels every extra minute. A batch job running overnight that someone reviews over coffee the next morning? The slower, cheaper model is fine.<\/p>\n

Since five models scored about the same, the quality gap between them isn\u2019t big enough to drive the decision. You\u2019re really choosing based on speed, cost, and how easily they fit into your existing setup.<\/p>\n

Teams spending weeks evaluating which model writes the \u201cbest\u201d code might get more out of that time by improving their prompts, tooling, and review process instead.<\/p>\n

\n
\"Scatter\"Scatter