Benchmark v3: re-checking the reference by ear — uzasr-large at 8.1% WER

In short. We re-checked our Uzbek spontaneous-speech benchmark by listening to the audio wherever our model and the reference disagreed. 144 of 311 clips received corrections. On the corrected benchmark (v3), uzasr-large-v0.3.0 scores 8.1% WER / 2.5% CER, and uzasr-base scores 10.5%. We re-scored every model, published v2 and v3 side by side, and documented a bias in the method that favours our latest model.

Why the reference needed a second look

The error analysis of v0.3.0 sorted every error by type. Near-misses (one or two letters) made up 44% of WER. Reading through them, three kinds kept appearing:

1. The reference itself was wrong. The machine transcript behind the benchmark misheard a word, and our model heard it correctly. For example, raykun was actually raykom, and aytamiga was maktabiga (as in "Presidential school").
2. Two correct spellings of one word. Examples: stol/stul, mersedes/mercedes, bloger/blogger, and dialect forms such as manga/menga.
3. Genuine model errors. These are the ones we want to count.

v2 counted all three as errors. Only the third kind says anything about the model.

What we did

• Collected every disagreement. The model and the reference disagreed on 814 word pairs. We extracted all of them.
• LLM pre-screening. An LLM sorted each pair as "reference wrong", "model wrong", "spelling variant" or "unsure", using text only. This only set priorities; it changed nothing in the benchmark.
• Human listening. A native speaker listened to the audio for 374 of the 450 "reference wrong" and "unsure" pairs, using a small review tool that plays exactly the disputed span. For each pair they ruled whether the reference was right, the model was right, both were wrong (with a typed fix), or the audio was unintelligible.
• Outcome.
  • 246 rulings for the model and 100 for the reference.
  • 15 cases where both were wrong.
  • 10 unintelligible cases, which were left unchanged.
• The LLM was not trusted alone. In 8 of its 92 "reference wrong" calls, the listener found the reference correct. Text alone is not enough, which is why every change to the benchmark comes from a human ruling.
• Rebuilt the benchmark. Corrections were applied to the original text with its capitalisation and punctuation preserved. Every rebuilt clip is checked automatically against the intended correction. Only "model was right" and typed fixes changed the text.

Results

Third-party baselines were re-run on the same audio, and their per-clip predictions are now published too. They gain much less from v3 (GigaAM-large 1.7 points, Whisper 0.4), as expected: the corrections were found through our model's disagreements. On v3, uzasr-large makes 43% fewer word errors than untuned GigaAM-large and 4.7× fewer than the Whisper turbo Uzbek fine-tune.

* Re-run with the same code, weights and library versions. The original August run, whose predictions were not kept, gave 39.6%. The 0.6-point drift comes from the runtime environment; the untuned GigaAM-large reproduced its old v2 number to within 0.05 points.

The bias, stated plainly

We only re-checked spots where v0.3.0 disagreed with the reference. So v3 fixes exactly the errors that hurt v0.3.0 most, and other models benefit less:

• v0.3.0 gained 2.6 points.
• Our other models gained 1.4 to 2.1 points.

The ranking did not change, but the gap between our models is somewhat overstated on v3.

How we will reduce it. Every new model's own disagreements go through the same listening review before its number is published. The benchmark gets cleaner with each model instead of being tuned to one.

We also publish:

• Both versions. final_etalon.jsonl (v3) and final_etalon_v2.jsonl.
• Every change. changes_v3.jsonl holds before/after text per clip, and adjudication_v3.jsonl holds the individual rulings.
• The scoring script, so anyone can check our numbers or score their own model on either version.

Also fixed in this update

• Quotation marks. Our text normaliser turned quotation marks ('Tolibon') into the Uzbek tutuq apostrophe. That created phantom tokens in scoring and glued quotes onto words in about 2% of training lines. Quotes are now dropped; an apostrophe is kept only between two letters.
• Filler sounds tracked separately. Hesitations such as a, aa, mhm are written in the reference but often omitted by models and by most training transcripts. We now track filler recall as its own metric. For expressive speech it matters, and we do not want it hidden inside WER.