Local LLM Chat Template Mismatch: A Debugging Guide

A local chat model that repeats your question or speaks as the user may have a formatting problem rather than a reasoning problem. Before changing sampling settings, inspect how your application turns message objects into the token sequence the model receives. A model swap can leave the old chat template behind even when the API request still looks valid.

This guide focuses on that integration boundary. It doesn't assume that every poor answer is caused by templating, and it doesn't require retraining a model. The objective is to find a small, reproducible difference between a working reference request and your application's request.

Separate the message list from the rendered prompt

Your application might send this familiar structure:

{"messages":[{"role":"user","content":"Reply with the word ready."}]}

That JSON is an interface representation. The inference backend still has to translate roles, message boundaries and the start of the next response into the model's expected format. According to the Transformers chat-template documentation, different chat models can require different control tokens. A valid message list therefore isn't proof of a correct model input.

Use the JSON Formatter to inspect a synthetic request. Check whether your adapter sends a message array, a preformatted prompt, or both. Formatting helps reveal the structure; it cannot determine which template belongs to your model.

Capture a minimum failing conversation

Remove retrieval, tools, memory and long system instructions. Ask for a short answer with an obvious success condition. Then test a two-turn exchange where the second question depends on the first answer. If the single-turn case works but the conversation fails, inspect how prior assistant messages are terminated.

Record these items in your debugging note:

• Model identifier and exact revision.
• Tokenizer identifier and exact revision.
• Serving engine and version.
• Active template or template source.
• Whether the server or client applies formatting.
• Generation settings and the synthetic message list.

Don't use a real customer conversation as the minimum example. An invented exchange is easier to share and avoids carrying sensitive context into logs.

Compare against the model's supported path

Run the same synthetic messages through the model's documented reference integration. Keep the checkpoint and tokenizer identical. Compare the rendered text and, when possible, the token IDs. Text inspection can hide differences in special-token handling, so a visually similar prompt isn't conclusive.

In Transformers, apply_chat_template provides a model-aware formatting path. Its generation-prompt option can mark the beginning of an assistant response, depending on the template. If you render text first and tokenize later, verify whether special tokens would be added again. The documentation warns that duplication can damage the input.

The Text Diff can compare two sanitized rendered prompts. Pay attention to the final boundary, not only the opening system message. The last few tokens often explain why a model continues the wrong speaker.

Look for double formatting

A common migration mistake is manually wrapping text in role markers while also sending it through a server that applies its own template. Another is using a tokenizer configuration from a related base model rather than the instruction-tuned checkpoint.

Draw the actual request path in a short note: frontend, adapter, server, tokenizer, model. Assign exactly one component responsibility for templating. Make every other component preserve the chosen message representation.

Check empty assistant messages, trailing whitespace and unsupported roles. Also check whether a middleware layer converts all content to one string. The application may preserve JSON syntax while losing the separation between instructions and prior responses.

Treat tool conversations as a separate test

After ordinary chat works, add one harmless read-only tool. A calculator-like function with a small input and deterministic output is enough. Test the tool request, the tool result and the final user-facing response as separate transitions.

Don't assume that a template which supports plain chat supports your engine's tool-call representation. Verify the expected function names, argument representation and tool-result association. Preserve tool-call identifiers where the integration requires them.

Compare the old and new synthetic request objects in JSON Diff. If the adapter turns an argument object into a quoted JSON string, a model-formatting investigation may actually uncover a request-contract bug.

Change one variable at a time

Keep a small test matrix: working reference path, current application path, corrected template path. Hold sampling parameters constant. Otherwise, a lucky answer can look like a fix.

Include a short answer, a two-turn dependency, a long instruction and a harmless tool exchange. Run each several times if output is nondeterministic. Save the input and the observable acceptance condition rather than relying on a screenshot of one successful response.

If formatting matches but quality remains poor, investigate model suitability, truncation, quantization and decoding. See quantized-model memory planning for a separate deployment constraint. Avoid editing templates merely to compensate for an unrelated failure.

Conclusion

Debug a chat-template mismatch by comparing the exact input path with a supported reference. Match the checkpoint and tokenizer, apply formatting once, inspect final role boundaries and add tool interactions only after ordinary chat passes. A small synthetic conversation gives you stronger evidence than a long prompt full of unrelated instructions.