A Comparative Study of Tokenization Efficiency in English, Hindi, Marathi and Romanized Hinglish Across OpenAI Tokenizer Generations

Large language models read text as small pieces called tokens. The number of tokens decides how much text fits in a model’s context window and how much a token-billed service charges. Recent studies show that Indian languages need many more tokens than English, and that Hinglish, which is Hindi written in English letters, creates its own tokenization problems. However, to our knowledge, Marathi, Hindi and several spellings of Hinglish have not been compared side by side as tokenizers have improved. We built 99 parallel sentences in English, Marathi, Hindi and three Hinglish spellings, and counted their tokens with four OpenAI tokenizer encodings, from older to newer. On the oldest encodings, Marathi and Hindi needed 7.4 and 7.6 times as many tokens as English. On the newest, o200k_base, they needed 1.80 and 1.54 times as many. Hinglish needed 1.8 to 2.2 times as many tokens on the oldest encodings and 1.5 to 1.9 times on the newest, so it was much cheaper than Hindi on older tokenizers but not on the newest. Different Hinglish spellings of the same sentence changed token counts by 26% to 31% on average. These results show that the cost of writing in Indian languages depends strongly on the tokenizer and on the script and spelling a user chooses. The data and code are available at https://github.com/samruddhibelsare-max/hinglsh_tokenization_study.