Az Nvidia az IIT Madras kutatóival és az AI4Bharat laborral közösen bejelentette a Nemotron-pretraining-Indic-v1 nevű nyílt adatbázis közelgő kiadását, amely mintegy 2,8 trillió tokent tartalmaz India 22 hivatalos nyelvén. A projekt célja a nyelvi modellek fejlesztésének felgyorsítása azokon a nyelveken, amelyekhez eddig kevés strukturált digitális tartalom állt rendelkezésre.
A csomag nemcsak az adatbázist, hanem az annak előállításához használt GPU-gyorsított adatgeneráló rendszert is tartalmazza. A folyamat angol nyelvű forrásokból készít magas minőségű fordításokat a NeMo Data Designer keretrendszer és a Gemma-4-31B-it modell segítségével, miközben automatikusan ellenőrzi és javítja a fordítások minőségét, megőrizve a kódokat és matematikai képleteket.
A feldolgozott anyagok között mind a 22 nyelv saját írásrendszerével készült adatok, valamint 12 nyelv latin betűs változata is megtalálható. Az így létrejött szintetikus adatokat már felhasználták a Nemotron 3.5 Super VL modell továbbképzéséhez is.
Az eredeti szöveg (Hugging Face)
What is in the corpus? From English source documents to Indic training data The twelve-language workflow The ten-language workflow What changed after Indic continued pretraining? Broader Indic capabilities in Nemotron 3.5 Super VL Start building Load the corpus Adapt the generation recipe Scope and limitations Further reading and acknowledgments Native-script data for 22 languages, romanized data for twelve, and a reusable GPU-accelerated pipeline for translation, quality assessment, and repair.
Authors: AI4Bharat — Mohammed Safi Ur Rahman Khan, Mitesh Khapra, Anoop Kunchukuttan. NVIDIA — Niket Agarwal, Raviraj Joshi, Anusha Kamath, Markus Kliegl, Yev Meyer, Sriharsha Niverty, Kiran Praveen, Utkarsh Vaidya.
We're excited to introduce Nemotron-pretraining-Indic-v1 approximately 2.8 trillion tokens across 22 Indian languages, releasing soon alongside the open, GPU-accelerated data generation pipeline used to create it. These are the 22 languages listed in India's Constitution: the first languages for 97% of Indians, or nearly 1.2 billion citizens at the 2011 Census. Developed by NVIDIA with IIT Madras, India's premier engineering institute, and its AI4Bharat lab, this release gives model builders a substantial new resource for Indic pretraining. The pipeline captures the full generation recipe, including prompts, LLM workflows, and quality checks, so developers can adapt it to their own source material and generate more data for their needs.
AI4Bharat, IIT Madras' Indian language research lab, builds open-source datasets and models for Indian languages. Its resources, such as Sangraha, an open pretraining corpus for 22 Indian languages, have helped lay the foundation for open Indic language-model research. Our joint open-source release represents a substantial expansion of the open synthetic Indic training data that will be available to the community. It brings a broad mix of curated English material, including mathematics, code, and question-answer content, into Indian languages, giving teams more training material for building models that can explain technical concepts, work through problems, and answer questions in those languages. Every record traces back to a curated source document, and each output is checked against its input, so the synthetic text stays grounded in real information.
Mitesh Khapra, who heads AI4Bharat and made the 2025 TIME100 AI list, has named the core problem: Indian language technology trails English because "we do not have enough data for Indian languages." We're answering that call with more data, plus an open recipe for generating more.
The corpus also accounts for how people write. It includes native-script data for all 22 languages and romanized text for twelve, spanning 34 language-script configurations. Our workflows combine document-level translation, quality assessment, and targeted repair, with checks to preserve content such as equations, tables, and code. This data complements naturally authored Indic text, which remains essential for local knowledge and cultural context.
We're sharing the open generation recipe so the community can not just reproduce the data but take this work further. Teams can use the corpus directly, adapt the prompts and quality checks, or bring their own source documents to the workflow. By opening up both the data and the process, we aim to make large-scale Indic data generation more accessible and enable more developers to contribute to stronger models across Indian languages.
Continued pretraining for Nemotron 3.5 Super VL added 10 billion Indic tokens, drawn from a twelve-language subset of this corpus and natural Indic text. Our technical report on the model covers the full story. We plan to build on this corpus in future Nemotron models.
Figure 1. The corpus contains 2,601.0 billion native-script tokens and 224.6 billion romanized tokens. Counts use the Gemma-4 tokenizer. Document records include translations and script variants of shared sources; they
