CosyVoice2 vs Chatterbox Multilingual in Production: Word Error Rate in Eight Languages

• Fixed sentences per language, written the way the app actually sends them (with accents and native punctuation), synthesized from the same reference voice clip.
• Each clip transcribed by the speech recognizer the product uses in production (faster-whisper small, on GPU).
• Word error rate (WER) between the input text and the transcript; per character for Chinese and Japanese. 0% means every word came back exactly.

Round-trip WER measures intelligibility, not naturalness. It is also only as good as the recognizer, which is why section 3 checks the recognizer separately.

* The French bake-off input had its accents stripped by mistake; the transcript was correct apart from a hallucinated trailing phrase. Re-measured in production with proper input: 0%. CosyVoice2 stayed badly wrong on French even with accented input — “J’ai pensé à toi toute la journée et tu me manques beaucoup ce soir” came back as “J’ai hésité à tout, la journée est tout, mais manque, beaucoup s’y soit”.

Figures for later additions and re-measurements, including Portuguese, are kept current on Languages on Lovora, measured.

A high round-trip WER can mean bad audio or a weak recognizer. To separate the two, the same CosyVoice2 clips were transcribed by faster-whisper small on GPU and by large-v3 (int8) on CPU. Both models made the same errors — both heard “mi mangi” for the Italian “mi manchi” — so the speech itself was wrong.

large-v3 on CPU took 13–21 s per clip against 0.2–0.7 s for small on GPU, with no quality gain on this audio. For a voice call, which needs recognition in about a second, the smaller model on GPU was the right choice and stayed.

Counted across all stored conversations at the time (181 chats, 2,236 messages), 39.6% of messages were in languages where CosyVoice2 was degraded. French alone was 26.5% — the second language of the service. The weak languages were not an edge case; they were two messages in five.

The isolated benchmark predicted about 40% slower audio; production came in at about 25% slower. A response cache for repeated phrases absorbs part of the difference. Running both engines for per-language routing would have needed about 8 GB for speech alone and did not fit.

Licence mattered as much as quality. XTTS-v2 covers a similar language list but is released under a non-commercial model licence, which rules it out for a paid product.

• Separate process and environment. Chatterbox required a newer major version of the transformers library than the running stack; installing it system-wide would have broken the existing services. It runs as its own service behind the gateway, so an out-of-memory error or a decoder hang in speech cannot take chat or recognition down. Rollback is one environment variable.
• The looping decoder. Occasionally the model speaks the whole sentence correctly and then repeats it (seen in French, three times over). The guard: if the audio is more than 2.2× the duration the text warrants, synthesize once more and keep the shorter take.
• Memory creep. After about 200 generations, PyTorch’s caching allocator had grown from 20.1 to 21.9 GB of 22.5 GB. Releasing the cache after each synthesis costs a few milliseconds and held usage flat across long runs.
• Invalidate everything keyed to the old voice. Cached clips and pre-rendered voice previews were keyed without the engine; both needed an engine version in the key, or users would have kept hearing the old voice.

• Small sentence sets per language (a handful each), one reference voice, one recognizer. The differences are large enough that the conclusion is not in doubt; the exact percentages for the stronger languages are not precise to the digit.
• WER says nothing about prosody, emotion or accent. Those were judged by listening, not measured.
• Measured on the model versions available in September 2026 (Chatterbox package 0.1.7).