An experiment has found that in coding work, an artificial intelligence model with a lower token price can be more expensive in actual revision costs. The result suggests response length and task success rates should be assessed alongside simple API pricing.

According to online media outlet Gigazine on Sept. 22 local time, a research team at AI monitoring and evaluation company Fiddler said that in a coding test comparing Anthropic's Claude Haiku 4.5 and Claude Sonnet 5, the cheaper Haiku used more money to produce a single success.

The test covered 5 tasks to fix bugs in a Python program. The researchers ran each task 3 times, forming a set of 15 runs, and conducted 3 sets per model. That was 45 runs per model, 90 runs in total.

In each run, the AI model issued commands such as checking files, modifying code and running tests. The setup executed only 1 real command per model response. The run ended when the model declared the fix complete, and it stopped after up to 8 responses even without such a declaration.

The researchers focused not on token prices themselves but on the total cost needed to obtain 1 successful bug fix. They validated fixes using separate, undisclosed tests and counted only those that passed as successes. They then divided the total cost, including failed attempts, by the number of successes to calculate cost per success.

The gap was large. In 45 runs, Sonnet cost $0.45 in total, while Haiku cost $4. The cost per success by set was also 8.4 to 14.1 times higher for Haiku than for Sonnet.

Token pricing alone showed the reverse. Haiku's per-token price for both input and output was about half of Sonnet's. In actual work, Haiku generated far more tokens, erasing the advantage of its lower price.

The biggest cause was response length. The median number of output tokens was 16 for Sonnet, compared with 361 for Haiku. The researchers explained that Haiku tended to generate in advance the results of commands that were not actually executed.

Instead of waiting for actual execution results, it first wrote expected outcomes and then generated the next commands and results in sequence. As a result, content not executed in the real environment was still counted as output tokens, increasing costs.

In one run, the problem appeared in an extreme form. Haiku generated 60 commands and hypothetical execution results in a single response and used a total of 21,492 tokens before reporting the fix complete.

But in the test environment, only the first of the commands included in the response was actually executed. The remaining 59 commands were not executed and remained only as text. Even so, all of that content was included in output costs.

Long responses also raised later costs. Because the environment re-entered past conversation content each time, a single excessive output led to more input tokens in the next interaction.

In an example presented by the researchers, input tokens in the fifth interaction were 9,930 for Haiku and 1,220 for Sonnet. A single longer response sharply increased the input volume in later dialogue.

Further analysis showed the cost gap for Haiku became clearer depending on response length. The per-run cost in the short-response group was $0.00235, lower than Sonnet's overall average of $0.00244. The per-run cost in the long-response group was $0.01693, 7.2 times the short-response group.

That is, the issue was not Haiku's lower per-token price itself, but that the cost structure can change sharply if the model generates long responses not needed for the actual task.

The researchers said the result does not mean the same outcome appears in all coding work. They explained results may vary depending on the type of task, the execution environment and how the model is used. They stressed that when adopting AI models for coding work, users should not compare only input and output token prices. They said similar tasks should be run, success rates and response volume measured together, and cost per success compared including failed attempts.

The experiment in particular shows that criteria for selecting models can change in AI agent-based coding environments. One unnecessarily long response can increase not only output costs but also later input costs in subsequent dialogue.

The analysis said the price competitiveness of coding AI depends less on how much is paid per token than on how much it costs to complete real work successfully. Even if a cheaper model is chosen, overall operating costs could be higher if excessive responses and low task efficiency repeat, making cost measurement in real work environments likely to become more important.