uzasr 0.2.1: our language model had seen the benchmark — a correction
We found that 20% of our Uzbek ASR benchmark had leaked into the KenLM text corpus. Here is how we found it, measured it and fixed it, and why honest WER is 11.5%, not 10.2%.
- WERuzasr-large10,2%11,5%
- WERuzasr-base10,8%12,2%
In short. A day after 0.2.0, we found that our language model had been trained on text that overlapped 62 of the 311 benchmark clips. The published 10.2% WER was optimistic by about 1.3 points. We rebuilt the LM with n-gram decontamination, re-tuned it, and corrected every public number. Honest WER is 11.5% for uzasr-large and 12.2% for uzasr-base. The acoustic model was not affected.
How we found it
Our LM pipeline removed each benchmark sentence and the 25 lines around it, but only from one source. A code review asked whether the other sources needed the same filter. We didn't want to argue about it, so we checked empirically: we searched every training text for word 10-grams from the benchmark references.
Some appeared verbatim, in 64 lines of an IT-lecture dataset and 22 lines of a podcast dataset. The same YouTube recordings had been published in more than one open dataset, so the benchmark text reached us through a side door. About 0.6 hours, or 20% of the benchmark, was affected.
How we measured the damage
We split the benchmark into the 62 affected clips and the 249 clean ones, and scored each decoding path separately:
| Path | All 311 | Clean 249 | Affected 62 |
|---|---|---|---|
| greedy (no LM) | 12.2% | 11.9% | 13.3% |
| leaked LM (0.2.0) | 10.2% | 11.5% | 5.8% |
| clean LM (0.2.1) | 11.5% | 11.4% | 11.9% |
The table answers two questions:
- Did the acoustic model memorise the benchmark? No. Without an LM, the affected clips are harder than the clean ones (13.3% vs 11.9%).
- Did the LM memorise it? Yes. On the affected clips, error fell from 13.3% to 5.8%, a gain that does not exist on unseen audio.
The clean LM is also slightly better than the leaked one on genuinely unseen clips (11.4% vs 11.5%). Users lost nothing; only the headline number was wrong.
The fix
- Standard n-gram decontamination, applied to every source. Any line sharing a 10-gram with the benchmark is removed, along with its ±25 neighbours in the same file. After the fix, 0 of 9,828 benchmark 10-grams remain in the corpus.
- Same filter for training audio. The data preparation step now applies the same filter, so future training sets are clean by default.
- Re-tuning. The clean LM "knows" less, so its optimal weight dropped from 0.5 to 0.2. Both model sizes use the same settings.
- Corrections everywhere. The model card, benchmark card and site were corrected the same day. The leaked LM was removed from the main branch. The 0.2.0 tags stay frozen for reproducibility and are marked as superseded.
Results (benchmark v2)
| Model | 0.1.0 greedy | 0.2.0 as published | 0.2.1 corrected |
|---|---|---|---|
| uzasr-large | 12.2% | 10.2% | 11.5% |
| uzasr-base | 13.9% | 10.8% | 12.2% |
Why we publish this
Data leakage between train and test is one of the most common ways speech benchmarks mislead, and it is rarely announced. We would rather show a higher number we can defend than a lower one we can't. Every number on this site is reproducible from the tagged weights and the public benchmark.