[DigitalToday reporter Chi-gyu Hwang] "The latest models are not needed for enterprise work AI agents."

Daniel Kornum (대니얼 코넘), who leads forward deployed engineers at Varick, which supports large companies in adopting AI agents, stressed this in a recent post on social media platform X (Twitter) titled "You Don’t Need Better Models". He said most operational tasks such as finance, human resources, sales and procurement can be handled even with models that lag 1 to 2 generations behind the latest.

He cited as an example a hypothetical company where 40 employees manually check 30,000 invoices received from clients each month and process payments.

An agent reads invoice PDFs, matches them against purchase orders and receiving records, assigns accounting accounts and records them in an ERP system. "Do you need the world's best model to read a PDF and match three documents?" he asked.

If employees and the agent each process the same invoices for a month and the results are compared, outcomes differ on 1,800 of 30,000 invoices, or about 6 percent. In 700 cases, the problem was a lack of context. The agent did not know information such as billing methods sent by clients via email two years earlier or payment terms that existed only in employees' heads.

In 500 cases, the cause was exceptions not found in work documents, and in 350 cases it was system data errors. In 150 cases, the agent's result, not the human's, was correct. Model error was limited to 100 cases, or 0.3 percent of all invoices. Even then, only about half of those could be fixed with the latest model.

The remaining 1,550 cases caused by missing context, exceptions and data errors cannot be solved by switching models, but are problems that require understanding workflow and filling in missing context.

Switching models also comes with costs. He said, "If you use 40,000 tokens per invoice, that is 1.2 billion tokens a month. With a $2-per-1 million-token model and a $10 latest model, monthly costs are $2,400 versus $12,000. If you expand the work to 50 tasks, the annual difference is $5.8 million." He added, "A new model should be adopted only when it solves problems the existing model cannot, or when it delivers the same performance at a lower cost."

He said the burden is also large because the agent must be revalidated from scratch every time the model is changed. He said, "Each model differs in output format and tool use, so every time you switch you have to fix prompts and test again. If you keep up with new models released every quarter, you spend 8 weeks a year on this per task. During that time, mistakes unrelated to the model have no choice but to be left unattended."

Kornum called as alternatives for setting a model by task, building an in-house evaluation set with 500 to 1,000 real invoices including exceptions, and placing an intermediate layer between the model and the AI agent to avoid dependence on a specific vendor.

He also said, "An agent that knows how a company works beats an agent with the latest model that does not, even if you go back to last year's model," adding, "The real competitors to watch are not the ones that use the latest model first. They are the ones already using agents for 20 tasks even with models released last year."