Small vs Large AI Models: Route Requests by Task

The largest available AI model is not automatically the best choice for every request. Classification, extraction, short rewriting, search query generation, and routine code transforms may work well on a smaller model with lower latency and cost. Complex planning or ambiguous analysis may justify a more capable model. Model routing turns that observation into an explicit product decision instead of an informal collection of exceptions.

Why this decision matters

Routing can improve responsiveness and control spending, but a careless router creates a new failure layer. If it sends a difficult case to a weak model, users receive a confident low-quality result. If it escalates everything, the system keeps the complexity without earning savings. The design must be grounded in task-specific evaluations, not vendor labels such as small, reasoning, or flagship. It must also respect data residency, context size, tool compatibility, availability, and the consequence of a wrong answer.

A practical workflow

1. Name the task classes. Separate activities with different success definitions: intent classification, structured extraction, summarization, drafting, coding, planning, and tool use. Avoid a single “easy or hard” label for an entire product.
2. Create quality gates. Build representative test sets for every class. Decide what counts as acceptable before comparing models: exact fields, grounded claims, executable tests, reviewer ratings, or a safe abstention.
3. Choose the cheapest qualified route. Select the smallest or least expensive model that consistently clears the gate for that class. Treat latency, input and output pricing, retries, and supporting infrastructure as part of cost.
4. Escalate from observable signals. Use missing required fields, low confidence from a separate validator, excessive context, failed tests, policy risk, or explicit user intent. Do not ask the first model to grade itself as the only escalation signal.
5. Keep a stable fallback. Define behavior for rate limits, outages, changed model versions, and an unavailable preferred region. A fallback must pass its own tests; a different API name is not proof of equivalence.

Work through a realistic example

A document service receives invoices and free-form vendor emails. The extraction route first checks file type and language. A smaller model handles clean digital invoices and must return a schema that passes validation. Scanned pages, handwritten notes, unusual currencies, or conflicting totals are escalated to a multimodal model and then reviewed. A separate summarization route may use the smaller model for short emails but choose a larger context model for a long contract. The same input can therefore take different routes depending on the requested job.

What to measure and record

Measure quality and economics together. For each route, record task class, selected model and version, input size, validation result, escalation reason, latency, retries, and estimated cost. Compare the router with simple baselines: always-small and always-large. A router is useful when it maintains the required quality while improving latency or total cost. Watch tail latency, not only the median, because multi-step escalation can make difficult requests much slower. Review differences by language, document type, customer segment, and risk level to detect uneven performance.

Common traps

• Routing by prompt length alone: A short request can require difficult reasoning, while a long document may need a simple field lookup.
• Using the first model as judge: Self-reported confidence is often poorly calibrated. Prefer validators, tests, rules, or a distinct evaluation step.
• Silent behavior changes: Providers may update aliases. Pin versions where possible and rerun the route suite before adopting a new snapshot.
• Ignoring escalation cost: A cheap first attempt followed by an expensive retry can cost more and respond slower than choosing the qualified model immediately.

Review questions

• What quality threshold must each task class meet?
• Which failures can a validator observe before the result reaches the user?
• Does routing behave consistently across languages and long inputs?
• What is the maximum acceptable end-to-end latency after escalation?
• Can a user or operator force a safer route for a high-consequence task?

A 30-day implementation plan

Begin with one bounded case and an owner who can make a decision. The first milestone is name the task classes. Write down the current state, the intended result, and the evidence that will count as complete. Keep the initial scope small enough to review in one working session, but realistic enough to expose operational friction.

During the second week, run the workflow with a colleague who did not design it. Ask them to answer: “What quality threshold must each task class meet?” Record where they need undocumented knowledge, which data is unavailable, and which step depends on a person or system that has no backup. Fix those gaps before increasing volume or authority.

By the end of the month, repeat the process under a failure condition related to routing by prompt length alone. Compare the observed result with the original acceptance criteria, assign unresolved actions, and set the next review date. Preserve the decision record beside the operational documentation. A modest control that is used, measured, and improved is more valuable than an ambitious design that exists only in a policy file.

Put the result into routine operations

Store routing policy outside individual prompt files so it can be reviewed and changed deliberately. Release changes behind a small traffic percentage and compare them with the current policy. Preserve the chosen route in request logs and user-visible audit records when it affects important work. Revisit thresholds when prices, models, or product expectations change. When a task repeatedly escalates, consider routing it directly or redesigning the workflow around deterministic software. Sometimes the correct optimization is less model use, not a more elaborate router.

Related Duck Cloud reading

Use LLM Evaluation Regression Tests to qualify each route.

Conclusion

Good model routing is a qualification system. Define the job, test candidate models, choose the least costly qualified path, escalate on observable evidence, and keep verified fallbacks. That produces a faster and more resilient product without pretending that every request has the same difficulty.