AI Model Selection: Which Model Belongs in Which Part of a System?

Support ticket sorting and a difficult code investigation place very different demands on an AI model. Sending both to your most expensive option might be quick to build, but you pay for that convenience on every routine request.

Choosing models becomes interesting once you look beyond the chatbot window. The same application might transcribe speech, search company documents and generate an answer. Each stage can fail differently. Spending more on the final language model won’t retrieve a missing document or repair a broken integration.

Start With the Job, Then Choose the Model

Model rankings tell you something about broad capability. They won’t tell you how your app behaves when an order lookup times out or an incoming message uses a term your classifier hasn’t seen before. Write down what a successful result looks like and collect examples that reflect the actual inputs.

Some jobs are already covered by ordinary code. The database should decide whether an order exists. The authorization service should decide whether somebody can cancel it. AI can help interpret a customer’s request, provided the software checks the answer before anything changes.

Other tasks benefit from stronger reasoning. Tracing a bug across several files or comparing contradictory documents can justify a slower model. For straightforward classification or extraction, a smaller one may perform just as well at lower cost. Test the difference. A cheaper response loses its appeal when staff must keep correcting it.

Many applications will use more than one model. Speech recognition, image understanding and language generation involve different work. A local model may also make sense when the device needs to function offline. Treat model selection as a set of decisions tied to particular tasks, not as a single winner for the whole product.

Finding Evidence Is a Different Kind of Intelligence

A technical manual may contain thousands of pages, but an answer depends on a particular passage. Retrieval chooses which material the language model receives. If the search step misses an exception in the manual, even an excellent model may give the wrong answer with complete confidence.

Embedding models turn text or other supported input into patterns of numbers that help a search system find material with related meaning. They’re part of the search process, not the part that writes the final answer. OpenAI’s embedding guide provides a more technical explanation for readers who want one.

A search model can find related passages without requiring the same wording. It may also confuse two facts that sound similar but differ on one crucial detail. An exact code or version number may be easier to find with an ordinary keyword search. Many systems combine both approaches.

Embeddings can also be used with different types of media in systems designed for that purpose. A picture and a sentence may be searchable through related representations. That helps locate possible evidence, but it doesn’t guarantee the model will interpret the result correctly.

Our Retrieval Layer article explores what happens when the correct document never makes it into the model’s context. That failure can survive an upgrade to an otherwise more capable AI model.

Routing Questions to the Right Model

Model routing is the decision about which model receives a request. For predictable jobs, the application can make that choice through ordinary code. Speech goes to a model that can handle audio. A task involving sensitive records must stay within the environment approved to process them.

Other requests are harder to classify. A lightweight model may decide whether a question needs a stronger one. UC Berkeley’s RouteLLM research explores how to do this while managing costs. Its savings came from the tested workloads and models. Your application’s requests could produce very different results.

Escalation works better when the system recognizes a concrete problem. Invalid JSON, a missing required field or a source that fails verification gives you a reason to retry with a different model. Confidence expressed in the generated answer is a poor substitute for checking the output.

For user-facing products, a quick or thorough mode can be useful when it changes something meaningful. Record which path each request took and what happened. Otherwise, it’s difficult to explain why the same question became more expensive without getting a better answer.

When the Provider Goes Down

Routing for capability and switching providers after an outage solve different problems. A backup model should meet the same output requirements and follow the same data access rules. Don’t assume two providers handle tools or structured responses identically because both accept text prompts.

The dangerous case involves actions. If an order cancellation times out after the remote system processed it, retrying through another model could cancel the same order twice. The application needs to check the operation’s actual state before repeating it. That’s a software reliability problem, and the model doesn’t get to decide the permissions.

Some models also allow developers to spend more computing effort on a request. That setting can improve harder answers and make easy ones slower. Compare actual completion time and corrected results before turning it up across the board. Our agent architecture guide goes deeper into the tools and permissions around the model.

A Coding Model Needs to Understand the Project

Generated code can look excellent in isolation and break a working application. A bug fix might change an assumption shared by five other files. A test suite may pass locally while the deployment fails because an environment variable or external service behaves differently.

Don’t judge coding models only by how neatly they write a function. Give candidates unfamiliar codebases, ask them to investigate a bug and check whether their edits preserve existing behavior. Include the time engineers spend reviewing, correcting and testing the work. If they have to rebuild the change afterward, the fast draft wasn’t fast.

Context windows add another complication. A large window lets you send more code or documents, yet the model may miss a crucial line buried among unrelated files. Retrieval that selects the relevant material can be more dependable than pasting in the entire repository. Test whether the model used the information it received, especially when the answer depends on one easily overlooked exception.

Different Inputs Create Different Constraints

Five seconds can be acceptable for a detailed written explanation and painfully slow during a spoken conversation. Voice applications need to deal with pauses and interruptions. Evaluate them with real audio, including background sound and people speaking over one another. Fast text generation on a benchmark doesn’t establish that the whole voice experience works well.

Images need their own tests. Reading a crowded screenshot, recognizing an object and generating a picture use different capabilities. For image understanding, try the kind of photographs or diagrams customers will supply. Tiny writing and poor lighting can expose weaknesses that clean demo images won’t.

Content moderation often benefits from a dedicated classifier. It can flag a message for review, but the application still needs rules for what happens next. A false positive can shut out a legitimate user. Policy decisions should not rest on a confidence score without knowing how often the system makes mistakes.

Local models deserve consideration when a device needs to work without a connection or when data must remain in a controlled environment. Hardware limits are real, and the team must maintain the model and software it runs on. The correct choice depends on the task and the constraints, not the size of the model alone.

Look for Problems Outside the Model

When an assistant gives an outdated answer, check the knowledge base it searched. The model may have received an old document and summarized it perfectly. Swapping in a more capable language model won’t correct the missing source.

Output validation and tool permissions also belong in the surrounding application. If the response must include a valid order number, verify it against the system that owns those records. A model’s persuasive explanation doesn’t grant access to somebody else’s account.

The Model Families Available in October 2026

Model names and availability move quickly. These examples reflect provider offerings checked on October 13, 2026. They show the kinds of choices teams face and aren’t a permanent ranking of models.

• OpenAI. GPT-6 Astra represents the highest-capability tier in the current lineup. GPT-6.1 Sol offers another option for demanding work with attention to efficiency, while Luna is positioned for faster, more bounded tasks.
• Anthropic. Opus, Sonnet and Haiku offer different balances of capability and speed. Anthropic’s Haiku 5.5 release illustrates its smaller, efficiency-focused tier.
• Google. The Gemini family includes different performance tiers and specialized capabilities. The choice changes with the kind of input and how quickly the result is needed.
• Locally hosted models. These can be useful for offline tasks or environments where the software needs to run under direct control. The model must still fit the available hardware and pass the application’s tests.

For up to date model capabilities, start with the providers’ current documentation rather than a permanent ranking in an article. The OpenAI model selection guide and Google’s Gemini catalog are examples. Product names and limits change quickly, so check the details at the time you build.

The useful insight survives those changes. A model family gives you candidates to investigate. It doesn’t settle the job each candidate should perform in a real system.

Compare Models With Work You Can Check

Published benchmarks are useful for narrowing the field. They don’t know your customers’ documents or your application’s failure modes. Select a sample of requests the system really receives, remove private information where necessary and keep some awkward cases that have defeated earlier versions.

Give the candidates the same information and instructions. Where possible, have a reviewer judge the answers without seeing which model produced them. An answer that sounds confident but misses the exception in a contract should fail that test. For tasks involving retrieval, check whether the answer cites the right passage and whether the passage supports the claim.

Track the outcome of the whole job. Include the time spent retrieving records, repeating failed operations and correcting responses. API prices matter, but paying less for a generated answer means little if an employee must repair it every time. Keep a record of the failure cases so the next release can be tested against them.

Check the integration too. Providers can differ in how they call tools and handle structured output. A version upgrade may change behavior even when your prompt stays identical. Pin model versions where supported and rerun the same tasks before switching.

What I’d record for every candidate: the jobs it completed correctly, the mistakes that required intervention, end-to-end time and cost, and whether it followed tool permissions. That gives the team a defensible reason to select a cheaper model for some requests and a stronger one for others.

What I’d Leave to Ordinary Software

When a database already knows the answer, read the database. Keep permission checks and irreversible actions in code that can be tested. Let models handle the language and interpretation they do well, then check the parts where mistakes would matter.

Before choosing a model, I’d want the team to show me a few requests it completes correctly and a few it still gets wrong. Those failures tell us where another model might help and where the surrounding software needs work. CMX’s AI automation and integration work focuses on that wider system.