LLM Judge Bias: Test Position and Verbosity Effects

An LLM judge can prefer an answer for reasons unrelated to its correctness. Position, length, and writing style can influence the result. Before using an automated judge to choose a model or approve a release, test whether its decisions remain stable when those presentation details change.

The research paper Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, first submitted in June 2023, examines several judge limitations, including position and verbosity effects. That research is background, not a guarantee about a current model. Your evaluation needs evidence from the judge, prompt, and tasks you actually use.

Define what the judge should assess

Write a rubric with separate dimensions such as factual correctness, instruction compliance, completeness, and clarity. Explain what counts as a serious error. A single instruction to “choose the better answer” leaves too much room for style to dominate.

Include the task context and a reference where appropriate. For a factual answer, give the judge verified evidence rather than expecting it to reconstruct every fact from memory. For a creative task, define the constraints without pretending there is one factual reference answer.

Allow ties or insufficient-evidence outcomes when they are appropriate. Forcing a winner between equally useful answers can turn small presentation effects into an apparently decisive ranking. Keep that distinction visible in the reported results.

Swap the answer positions

For each pair, present answer A first and answer B second. Then repeat with the order reversed, keeping the task and rubric unchanged. Map the result back to the original answer identities before comparing decisions.

Do not compare raw labels without remapping. If “first answer” wins both runs, the winning model changed. If the original A wins both, the decision was order-consistent for that pair. Store both the displayed position and the original answer identifier.

Calculate a disagreement rate across the paired runs and inspect examples. A disagreement is a reason to investigate, not proof that the judge is useless. Ambiguous tasks and near ties may legitimately produce uncertainty. Separate those cases from clear factual errors.

Test length without rewarding padding

Build controlled examples that vary length while preserving the relevant substance. For instance, compare a concise correct answer with a longer version that repeats the same point. Then compare a concise incomplete answer with a longer answer that includes a genuinely necessary qualification.

These are different tests. The judge should not reward empty repetition, but it may reasonably prefer useful completeness. Label the intended distinction before running the judge so you do not reinterpret every outcome to fit a desired result.

Do not mechanically shorten answers in ways that remove crucial evidence or safety conditions. A matched-length test must still preserve the property being evaluated. Have a human reviewer check the controlled examples before using them as bias probes.

Blind model identity where feasible

Remove model names and vendor-identifying claims from the evaluation presentation when they are not part of the task. Use neutral answer IDs. A statement such as “generated by the premium model” should not become evidence of quality.

Keep the original metadata in a separate evaluation record so results can be traced after judging. Blinding should reduce irrelevant cues, not destroy provenance. Also avoid editing away a real answer failure merely because it reveals a model's style.

If the judge is one of the models being compared, note that relationship. Do not assume impartiality from the API's role. Run targeted checks and human review before letting a model's self-assessment decide a consequential deployment.

Add human spot checks

Select cases across easy tasks, difficult tasks, disagreements, and important user groups. Have qualified reviewers apply the same rubric without seeing the automated decision first where feasible. Record disagreements with a short reason rather than only a final score.

Use these examples to distinguish a bad rubric from a judge limitation or an ambiguous reference. If humans disagree because the task lacks necessary context, fix the evaluation item. More judge calls will not create a missing product requirement.

Do not claim that an agreement percentage proves correctness across all future tasks. A measured result describes the sampled tasks and review process. State the sample, model version, prompt, and limitations when sharing it.

Version the evaluation system

Store the judge model identifier, rubric, prompt, decoding settings where available, evaluation items, and result schema. A change in the judge can shift scores even when the candidate model stays the same. Keep judge changes separate from candidate changes in the experiment.

Re-run a stable control set after a judge update. Include order swaps and length probes. Compare per-item outcomes as well as aggregate scores so a changed failure pattern is not hidden by an unchanged average.

JSON Diff can compare redacted evaluation manifests and structured results. It does not determine which answer is correct. Our benchmark contamination guide addresses a separate risk in choosing evaluation material.

Conclusion

An automated judge is a measurement component that needs testing. Use explicit rubrics, order swaps, controlled length cases, and human review to understand its limits. Trust the documented evidence and uncertainty, not the appearance of precision in a model-generated score.