NVIDIA Indic Pretraining: The 2.8T-Token Corpus and Open Translation Recipe

NVIDIA and AI4Bharat's October 16 Indic pretraining announcement gives developers a concrete workflow to inspect while a much larger dataset is still on the way. Nemotron-pretraining-Indic-v1 is described as a forthcoming corpus of about 2.8 trillion tokens across 22 Indian languages. The generation plugin is already public.

That distinction matters for anyone planning an experiment this week: the code can be examined now, while the announcement still describes the corpus as releasing soon.

What is available now

The corpus contains native-script text for all 22 languages and romanized text for twelve. These are translations of English source material, including technical content and question-answer documents. Native script describes how the text is written, not where it originated. A romanized version can also express the same content as its native-script counterpart, so the headline token count does not represent that many tokens of independent source material.

The published translation plugin makes the implementation easier to assess. It supports separate translator and evaluator model settings, preserves earlier attempts in translation and evaluation histories, and offers language-specific handling.

Its recommended configuration uses Gemma-4-31B-it for both translation and evaluation across twelve languages. The other ten use Bodhan AI's Indic Translate for translation, with Gemma evaluating the result. Santali can use overlapping document windows; Manipuri passes through Bengali script before a deterministic conversion to Meetei Mayek.

Configure repair explicitly

There is a practical setup detail worth catching before a team copies an example. The configuration reference sets the default number of rewrite attempts to zero. A rewrite threshold of 75 therefore does not, by itself, turn on repair.

Enabling windowed translation also forces rewrite attempts to zero because rewriting the complete merged document would undo the reason for windowing. Teams adapting the recipe need to choose those settings deliberately and define which scored outputs their training pipeline will accept.

What the training results establish

The training results give this work some evidence beyond a data-volume claim. NVIDIA's Nemotron 3.5 technical report describes a 100-billion-token continued-pretraining stage for Super, with 10 billion tokens allocated to a mixture of synthetic and natural Indic text.

The checkpoint improved by 4.98 to 9.82 percentage points across five reported Indic benchmarks. This supports the combined training recipe. It does not isolate how much improvement came from the synthetic corpus, and benchmark language coverage varies.

The same report quantifies an engineering tradeoff. Optimizations reduced median translation cost from 0.835 to 0.335 GPU-hours per million English input tokens on the reported twelve-language benchmark. That roughly 60% reduction applies to translation. Evaluation and rewriting add cost.

A team budgeting a domain-specific dataset should measure the complete path from source document to accepted training record, including rejected attempts, before extrapolating a translation throughput number.

Evaluate accepted data, not just translation speed

For builders, I would start with a small set of documents whose meaning native-language reviewers can check. Translating and judging with the same model creates a risk that the judge accepts its own mistakes.

Technical documents also need checks for missing equations, altered code and incomplete paragraphs. Keeping the source identifier makes those checks easier, but duplicate identifiers and semantically repeated material are different problems.

The useful next experiment is a controlled training comparison: hold the token budget fixed, change the synthetic-data share, and inspect results separately for each language and script. Related translations should stay together when constructing held-out splits.

Naturally authored Indic material remains important for local knowledge and everyday language use. This release gives teams a way to study those choices, with enough implementation detail to understand what they are changing.

I'm Paras. I work at the intersection of AI, engineering and distribution.

I help technical founders with launches and storytelling, turning complex products into a clear reason for people to try them.

Book a 30-minute call.