🔍 Read the full analysis: Open ASR Leaderboard Adds A New Chapter With Its First Global South Language on ThorstenMeyerAI.com
TL;DR
The Open ASR Leaderboard has introduced two new evaluation sets, Monsoon en-IN and Monsoon hi-IN, covering Indian English and Hindi. This marks the first inclusion of a Global South language, expanding the benchmark’s diversity and scope. The new sets enable detailed error analysis across varied populations, addressing biases in speech recognition models.
The Hugging Face Open ASR Leaderboard has officially added two new evaluation sets — Monsoon en-IN for Indian English and Monsoon hi-IN for Hindi — making Hindi the first Indic and first Global South language to appear on the multilingual leaderboard. This development broadens the scope of the benchmark, which previously included only European languages, and aims to improve speech recognition models for diverse populations.
The new sets are designed with a focus on speaker diversity and real-world variability, as highlighted in the original analysis of the benchmark’s expansion. They include recordings from a total of 4,888 speakers across India, capturing a wide range of demographics, accents, and environments. Each set offers a public split for self-scoring and a private split to prevent model overfitting, with the clips derived from spontaneous, unscripted conversations.
Specifically, the Indian English public set contains 5.62 hours of audio from 1,444 speakers, while the private set has 5.58 hours from 1,405 speakers. The Hindi public set includes 1.33 hours from 468 speakers, and the private set comprises 4.47 hours from 1,571 speakers. The recordings were collected across hundreds of districts, using contributors’ own devices and environments, ensuring the data reflects real-world acoustic conditions.
Beyond basic transcription, each clip records 12 speaker attributes such as age, gender, occupation, education, income, device used, and geographic location. The Hindi transcripts incorporate a lattice structure to account for spelling variations, addressing language-specific normalization challenges. This comprehensive metadata allows for detailed bias and performance analysis across different population segments, similar to the insights discussed in the original analysis.
Expanding ASR Benchmarking to the Global South
The addition of Hindi and Indian English to the Open ASR Leaderboard represents a significant step toward inclusive and representative benchmarking of speech recognition technology. By incorporating languages spoken by over half a billion people, this move addresses a long-standing gap in ASR evaluation, which has historically focused on European languages. It provides a market signal for developers to improve models for Indic languages, potentially leading to more equitable AI systems.
Furthermore, the detailed speaker metadata enables researchers and developers to analyze model biases related to region, age, gender, device, and socio-economic status. This approach aims to foster the development of more trustworthy, fair, and robust ASR systems that perform well across diverse populations, reducing disparities highlighted by prior research on racial and demographic biases in speech recognition.
speech recognition microphone for Hindi
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Background on ASR Benchmarking and Language Gaps
The Hugging Face Open ASR Leaderboard has been a key benchmark for assessing speech recognition models, primarily focusing on European languages like English, French, and German. Prior to this update, it lacked representation from languages spoken in the Global South, where speech recognition technology often underperforms due to data scarcity and linguistic complexity.
Previous research has highlighted disparities in ASR accuracy across racial, gender, and regional lines, with models often performing poorly on non-Western accents and dialects. The introduction of diverse datasets with rich speaker metadata aims to address this imbalance by providing a more nuanced understanding of model performance across different populations.
The new Indian language sets are part of ongoing efforts to diversify training and benchmarking data, recognizing that language-specific challenges—such as spelling variation and code-switching—must be explicitly addressed to improve global applicability of speech AI.
“The new datasets are designed to push the development of models that perform reliably across different demographics and acoustic conditions, addressing biases in current systems.”
— Hugging Face spokesperson
Indian English transcription software
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Unresolved Questions About Dataset Impact
It remains unclear how existing top-performing models will perform on these new datasets, especially given the limited hours of Hindi data. The stability of model rankings on the Hindi set, which contains only 1.33 hours of audio, has not been established. Additionally, the effectiveness of the lattice normalization approach for Hindi, compared to traditional methods, has not been demonstrated with published results. Whether leaderboard participants will analyze and report disaggregated results based on the new speaker attributes is also still to be seen.
As an affiliate, we earn on qualifying purchases.
Next Steps for Benchmarking and Model Development
The public splits are now available for self-scoring, and the private splits will be used for official rankings in upcoming leaderboards. Researchers and developers are expected to evaluate their models on these new datasets, providing insights into performance across Indian English and Hindi speakers. Future work may include expanding the datasets further, increasing hours of audio, and conducting detailed bias analyses. Additionally, benchmark results on existing models will clarify how much progress is needed for equitable ASR across languages and populations.
multilingual speech recognition tool
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Key Questions
Why is adding Hindi to the leaderboard significant?
Hindi is spoken by over 500 million people, making its inclusion essential for developing more equitable speech recognition systems that serve a large, diverse population.
How do the new datasets differ from previous ones?
They focus on speaker diversity, spontaneous speech, and real-world acoustic conditions, with detailed metadata to analyze biases and performance across different demographics.
Will current models perform well on these new datasets?
This is still unknown; the datasets are small, and baseline results have not yet been published, so model performance remains to be evaluated.
What challenges does Hindi pose for ASR systems?
Spelling variation, code-switching, and diverse dialects make Hindi a complex language for ASR, requiring specialized normalization approaches like the lattice method used here.
What are the implications for future ASR research?
The datasets set a precedent for including more Global South languages and for detailed bias analysis, encouraging the development of fairer, more inclusive models.
Primary source: Hugging Face · via ThorstenMeyerAI.com
