[Digital Today reporter Jae-won Choi (최재원)] A study has found that AI usage costs can differ by up to several dozen times depending on which language is used for the same question. It said languages with less training data than English require more computing resources, creating cost gaps among users.

Gigazine, a Japanese media outlet, reported on Oct. 27 that CLEAR, a research group on AI language equity, recently analysed token-processing efficiency by language and described the phenomenon as a “Tokenizer Tax.”

Large language models (LLMs) operate based on tokens that split sentences into units such as words, characters and bytes. Many AI services charge fees based on the number of tokens processed. The issue is that in languages with less training data, sentences can be broken into overly small pieces, increasing token use.

Researchers compared the sentence, “People should benefit from AI regardless of the language they use.” The result was 13 tokens in English, 26 in Malayalam used in India, and 78 in Amharic in Ethiopia. If the fee per token is the same, an Amharic user would pay 6 times as much as an English user.

The phenomenon was not simply due to differences in sentence length by language. In the data used for the study, Myanmar Shan had 42 percent more characters than English, but the token-use gaps seen in some languages were far larger than that.

The gap by language was more pronounced in language communities with a lower share of internet use. There were also significant differences in token use even among the top 20 most-used languages online. When the analysis expanded to about 200 languages, the gap widened to as much as 17 times depending on the model.

The researchers pointed out that “this imbalance does not stop at being a simple cost issue.” As token use increases, processing takes longer and the amount of documents an AI can remember or analyse at one time shrinks.

CLEAR warned in particular that “the cost burden could become even heavier in areas lacking computing resources and internet access,” adding that “technical inequality by language could limit participation in the AI economy.”