Speech Recognition Evaluation: Use WER Without Missing Critical Errors

Word error rate is a useful measure of transcription mistakes, but it is not a complete measure of whether a transcript is safe or useful. Calculate it consistently, inspect the errors that matter to the task, and test realistic audio. A lower average WER can still hide a wrong amount, name, or negation.

New speech models and demonstrations make comparison tempting. The relevant question is not which demo sounds most convincing, but which system handles your speakers, microphones, language mix, and output workflow. Build a small, documented test set before making a product decision.

Calculate the basic metric correctly

The Hugging Face WER metric card defines word error rate using substitutions, deletions, and insertions relative to the reference word count:

WER = (substitutions + deletions + insertions) / reference words

If a reference contains 100 words and the aligned prediction has six substitutions, three deletions, and two insertions, WER is 11%. This is an illustrative calculation, not a model benchmark. Insertions can make WER exceed 100%, so do not cap the value merely because it looks unusual.

Alignment matters. Counting different words by position is not equivalent to finding the sequence alignment. One omitted word can shift every later position while creating only one deletion in a correct alignment.

Define the reference before comparing models

Use reviewed transcripts created from the actual audio. Record speaker labels, unintelligible spans, language changes, and the chosen treatment of disfluencies. A reference that silently corrects a speaker's grammar may unfairly penalize a model that transcribes the speech accurately.

Decide whether the task needs verbatim speech or readable edited text. These are different outputs. If the product promises verbatim transcription, do not remove repetitions in the reference. If it promises edited notes, evaluate semantic faithfulness separately from literal word matching.

Have a second reviewer inspect a sample and resolve disagreements. Keep the reference revision stable while comparing candidates. Changing a difficult reference after seeing one model's output introduces bias into the evaluation.

Normalize without hiding important information

Specify case, punctuation, whitespace, number formatting, abbreviations, and Unicode handling. Apply the same transformation to reference and prediction. Preserve both raw and normalized text so investigators can understand what the score excludes.

For example, “twenty five” and “25” may represent the same number in a meeting transcript. But normalizing every number away would conceal an error from 25 to 250. Keep a dedicated numeric-accuracy check even if the WER normalization expands number formats.

Language-specific tokenization needs care. Space-delimited words are not equally meaningful across every writing system. Where word segmentation is ambiguous, document the segmentation rule and consider character error rate as an additional view. Do not compare a word-based score in one language directly with a differently tokenized score in another.

Build slices that reflect real audio

Include quiet speech, office noise, overlapping speakers, low-quality microphones, accents, short commands, technical vocabulary, and code-switching where relevant. Track audio length and speaker distribution. Avoid letting one long, easy recording dominate the aggregate result.

Keep a clean holdout set separate from tuning examples. Record model version, decoding configuration, language settings, processing location, and any prompt or vocabulary hints. A system given customer-specific names should not be compared silently with one that was not.

Report overall WER and per-slice results. A good average can coexist with poor performance on the language or device your users rely on most. Small slices also have uncertainty; describe the sample size instead of treating a single percentage as universal truth.

Add critical-error checks

Identify words whose mistakes have disproportionate consequences: names, account identifiers, quantities, dates, addresses, medication names, and negations. Use domain reviewers for consequential contexts. This guide is an evaluation workflow, not authorization to rely on automated transcripts for medical or legal decisions.

Consider the difference between “ship tomorrow” and “do not ship tomorrow.” Only a few words change, but the operational meaning reverses. Evaluate action instructions and speaker attribution separately. A correct sentence assigned to the wrong speaker can also mislead the reader.

For developers, compare sanitized transcript snippets using tools from the Duck Cloud catalog. A text diff helps review changes, but it does not calculate alignment-aware WER or verify the audio's meaning. Do not upload confidential recordings or transcripts to an unreviewed tool.

Evaluate the complete workflow

Measure processing latency, cost, retry behavior, correction time, and export quality alongside transcription accuracy. Test what happens when audio is empty, damaged, too long, or in an unsupported language. A confident invented transcript from silence is a different failure from an ordinary spelling error.

Choose the system that meets the task's quality and operating requirements, not merely the lowest published WER. A useful evaluation combines consistent scoring, representative audio, critical-error review, and transparent limitations.

Check segment boundaries

Review long recordings at segment joins. A system can duplicate or omit words where audio is split even when each short segment sounds accurate. Include speaker changes and pauses near those boundaries, and score the final assembled transcript rather than only the individual chunks.