uzasr-base v0.3.0: the 220M model drops below 10% WER
We replayed the uzasr-large training recipe on the 220M model. Error fell from 10.5% to 9.4% WER on benchmark v3, decoding got faster, and the small model gained twice as much as the large one.
- WERuzasr-base10.5%9.4%
- CERuzasr-base3.3%3.0%
- Speeduzasr-base×126×168
In short. uzasr-base-v0.3.0 makes 9.4% word errors on benchmark v3, down from 10.5%. CER fell from 3.3% to 3.0%, and throughput rose from 126× to 168× real time. The cheaper tier now matches what uzasr-large-v0.2.1 scored on the same benchmark. No new data or methods were needed: we replayed the v0.3.0 large recipe on the 220M model in a single overnight run.
What changed
The steps are the same as for the large model, applied to the 220M model. Each one was measured on its own (benchmark v3, whole-clip decoding):
| Step | Greedy | With LM |
|---|---|---|
| v0.2.1 (starting point) | 12.3% | 10.5% |
| + larger LM (10.2M sentences), no training | 12.3% | 10.2% |
| + domain training on spontaneous speech | 11.3% | 9.7% |
| + agreement corpus (~250 h cross-checked YouTube) | 11.1% | 9.4% |
- Language model swap. Changing only the decoder's language model, from 387k sentences to 10.2M, gave −0.3 points with no training.
- Domain training. We ran 3,000 steps on conversational audio with 19% read-speech replay, using the same mix as the large model. This was the biggest single gain: −1.0 point greedy.
- Agreement corpus. The same ~250 hours of YouTube speech where two independent transcripts agree. It gave another −0.3 points with the LM.
- Beam 64 instead of 256. Same accuracy, faster decoding.
What we learned
The small model gains more from the same data. The same two training stages cut greedy WER by 1.3 points on the 220M model, versus 0.7 on the 600M. A smaller model has more to learn from in-domain speech before it runs out of capacity. That makes the cheap tier a better product than its parameter count suggests.
Validation still can't pick checkpoints. On the 300-sentence validation set, scores bounced between 14.1% and 15.1% with no link to benchmark results. As with the large model, we ship the final step instead of the "best" validation checkpoint.
Results (benchmark v3, production path)
| Version | Decoding | WER | CER | Speed |
|---|---|---|---|---|
| uzasr-base v0.2.1 | KenLM 387k sentences, beam 256 | 10.5% | 3.3% | 126× |
| uzasr-base v0.3.0 | KenLM 10.2M sentences, beam 64 | 9.4% | 3.0% | 168× |
| uzasr-large v0.3.0 | for comparison | 8.1% | 2.5% | 163× |
Measured on the production worker in float32 with 25-second chunking. The model fits in 1.4 GB of VRAM. On benchmark v2, the same model scores 11.3%, down from 12.2%.
A note on fairness
Benchmark v3 was corrected by listening wherever uzasr-large-v0.3.0 disagreed with the reference, as described in benchmark v3. The new 220M model has not had that listening pass on its own disagreements yet. If anything, its 9.4% is therefore slightly pessimistic.
Next
- A listening pass on this model's own disagreements, so the benchmark keeps improving with every model.
- More agreement-corpus hours from new channels. That data has helped both model sizes.