[DigitalToday reporter Chi-gyu Hwang] "What determines AI model performance is not parameter count alone. You have to look at data volume, where computing resources are used, and who runs the model under what conditions as well."

Jie Tang (탕 줴), a Tsinghua University professor and founder of AI startup Z.ai, shared this view recently on social media platform X (Twitter) while discussing changes around AI model scaling laws. He said the judgment was a conclusion reached after trial and error.

In a 2020 paper, "Scaling Laws for Neural Language Models," OpenAI researcher Jared Kaplan (재러드 카플란) presented results indicating parameters should be increased much faster than data, and the industry rolled out large models such as GPT-3, Gopher and MT-NLG.

In 2022, a DeepMind team led by Hoffman re-analysed 400 models and reached the opposite conclusion to Kaplan’s team. It found it was more efficient to train about 20 tokens per parameter, and that as computing resources increase, parameters and data should be scaled at the same rate. This was the so-called Chinchilla law.

On this, Tang said, "The initial calculation error grew as computing resources grew. The trillion-parameter models that came out then were the result of the most inefficient design."

Debate over scaling laws continued even after the Chinchilla law. Tang said the Chinchilla law assumed a model that is trained once and evaluated. He said today’s AI models are called billions of times a day, and most of the lifetime cost, meaning the total cost from a model’s release through continued use until it is retired, comes from the inference process.

If inference costs are included, the conclusion changes. Tang said, "It becomes advantageous to keep models smaller and set training periods much longer. That is why Llama2-7B and Gemma2-9B trained up to 290 tokens and 889 tokens per parameter."

Mixture of Experts (MoE) architectures also had a major impact on changing performance standards. Tang said total parameter count indicates how much a model knows, while the number of active parameters and effective depth determine how far the model can carry a long chain of reasoning to the end.

The 20-to-1 ratio suggested by the Chinchilla law does not apply to MoE. Tang said, "For memorisation-heavy tasks, increasing parameters was more advantageous, while for reasoning-heavy tasks, increasing data was more advantageous. Follow-up research showed that keeping the token ratio fixed while increasing only total parameters instead reduced reasoning ability. Increasing the number of experts activated raised reasoning ability."

The principle that total parameters determine knowledge and active parameters and effective depth determine reasoning power was applied as-is to Z.ai’s GLM model design.

He said, "The ability to find vulnerabilities does not come from memorising how many CVEs you know, but from whether you can carry a 20-step chain of reasoning through to the end without losing it. That kind of ability does not arise just by increasing parameters."

Tang said Z.ai’s recently released GLM-5.3 model was a stage to verify this. He said, "We kept the base model, architecture, total parameters and active parameters identical to GLM 5.2 and increased only long-horizon reinforcement learning (RL) environment training for a month. The result was clear." He added, "This time we only adjusted post-training. That is because it still had the most room for improvement, not because other factors are finished. There are still parts we have not touched, including base model size, pre-training data and computing resources per forward pass." He continued, "There is no need to turn every dial at once. The next dial to turn is likely to be different from the last. Scaling is not over yet."