Chip companies including Nvidia are presenting new performance evaluation standards for the era of AI agents. It is difficult to gauge how much work an AI agent actually completes through multiple steps, and how accurate the results are, using only theoretical compute speed or a single benchmark score. The industry says performance figures must be read alongside test conditions, and that humans should directly review outputs.

Jeff Yen (제프 옌), Nvidia APAC technical marketing director, explained benchmarking methodology in a separate session at an RTX Spark product briefing on Nov. 6. Yen said, "Traditionally benchmarking has always been about speed, but for AI you need to look at speed and accuracy together." He added, "No matter how fast it is, it is useless if it produces wrong answers or unwanted generation results," stressing that "accuracy is equally important."

Games and general applications are still evaluated using performance indicators such as frames per second, latency and image quality, but AI needs different standards, he said. Yen also recommended that games be measured in gameplay scenes to better reflect gamers' real experience.

Yen pointed to the pace of technological change behind the shift in AI evaluation standards. "As AI develops rapidly, model parameters, benchmarks and best practices keep changing," he said. He added that local AI, where AI runs directly on a PC, is drawing more attention than ever as new attempts pour in.

The scope of what is being evaluated is also widening. There are already established benchmarks for computer vision, image and video generation, and large language models. In addition, demand is rising for agentic benchmarking methods that measure how much AI agents get done in real workflows, he said.

Other chip companies have also described changes in performance evaluation for the same reasons. A common metric used to describe AI chip performance is TOPS, a theoretical figure that calculates the number of operations a neural processing unit can handle per second at peak clock speed. Chip companies say that even if this number rises, it does not translate into real AI performance without memory bandwidth and software optimisation.

Qualcomm sees it as difficult to fully verify platform improvement effects with benchmark results alone because AI performance varies depending on end-to-end optimisation, software and the types of models being run, and it is using external benchmarks as well. According to Netherlands-based Traverse Research, Qualcomm is currently running the company's GPU and AI benchmark Evolve internally.

Arm has also released a separate CPU performance evaluation methodology for agentic AI. Fujitsu, Meta and OpenAI supported the development of the methodology. Arm's standard is how many tasks it completes while maintaining response speed. If throughput falls as utilisation rises, the number of tasks actually completed does not increase even if theoretical processing capacity grows, it said. Arm set out principles of real agent use cases, repeatable measurement and validation of final task results.

Nvidia also recommends benchmarks that follow real workflows. Yen said benchmarks should resemble real applications and cited MLPerf Client and Agent perf local as benchmarks to watch. The agentic benchmark in MLPerf Client 2.0 is a short test consisting of 4 prompts and simulated tool calls. AgentPerf-local from Artificial Analysis uses more than 160 prompts for the baseline run alone and calls real tools to test the entire system.

Yen cited memory capacity and input and output context as factors that determine agent performance. Memory capacity determines what kinds of models can be loaded. He said Nvidia set separate evaluation standards for memory capacity and input and output context for RTX Spark, the company's first system-on-chip platform it has disclosed, because it has broad use as an agent.

Ironically, the importance of people in performance measurement has grown. Yen said, "Whether it's image generation or video generation, or agentic benchmarking, human review makes a big difference." It is difficult to determine from measurements alone whether images or videos were generated as intended, or whether an agent properly completed a task, he said.

He described this as based on Nvidia's internal experience. "Even models with similar names can differ in output quality depending on the quantisation method, and the runtime backend also affects accuracy," he said, adding, "We make that mistake too." That means people must confirm which model was actually used in testing before speed and accuracy metrics can be trusted.

That is why, he said, accuracy test results must also be considered for a fair comparison. He said image and video generation in particular is the area with the least testing that measures accuracy numerically, so human review remains useful. Arm's inclusion of validation of final task results among its methodology principles is in the same context, reflecting an intent to include in performance whether tasks were correctly handled to the end.