AI & Enterprise
AI costs rise with non-English input, token gap up to 17-fold
AI usage costs can differ by up to several dozen times depending on the language used for the same question, a study found. Languages with less training data than English may require more computing resources, creating cost gaps among users. CLEAR, a research group on AI language equity, called the phenomenon a “Tokenizer Tax” after analysing token-processing efficiency by language. In one example, a sentence required 13 tokens in English but 78 in Amharic. Wider analysis showed gaps of up to 17-fold depending on the model.
