The Practical Case for Self-Hosting Open Models

By now, you’ve probably heard that open models have caught up with proprietary ones. I think that claim needs a closer look.

Look at Xiaomi’s MiMo-V2.6-Pro. The open-weight model scored 46 on Artificial Analysis’ Intelligence Index, just two points behind OpenAI’s GPT-6 Sol at max effort.

Yet the measured cost per benchmark task was $0.13 through Xiaomi’s API, versus $1.06 through OpenAI’s. That’s close overall performance at roughly one-eighth the API cost.

So why spend engineering time running open models yourself when you could use a hosted API?

Well, the shift toward open-source models hasn’t got much to do with them being dramatically smarter. It’s just that they’re perfectly capable of doing all that work.

Add better cost control, data residency, and freedom from vendor lock-in, and the case for open source becomes pretty compelling.

Where open models fit in an agent

Frontier models still take the gold medal for the hardest reasoning. But your agent doesn’t need that level of reasoning on every call.

Say a customer sends an agent an invoice and asks why a payment hasn’t appeared. The agent may use OCR to read the scan, an embedding model to search payment policies, a reranker to choose the right passage, and an extractor to pull the invoice number. Only then does it need to answer the customer’s question.

Teams can put high-volume, everyday work on open models and reserve frontier-model calls for cases that need them. Retrieval, extraction, classification, and routine generation are candidates, provided the model passes tests on your data.

What pushes teams to make the switch

These are the four reasons why teams usually switch to open-source models:

Cost is often the top reason.

With a closed API, you pay for every token, whereas a GPU in a self-hosted setup is a fixed cost whether it’s busy or not (excluding electricity, a nearby sea, etc.)

However, keep in mind that self-hosting isn’t automatically cheaper.

It becomes more affordable when your volume is high and steady enough to keep those GPUs busy doing the math. In practice, utilization is the main factor when deciding whether to go fully self-hosted.

I’d count successful tasks, not just tokens or GPU hours. If a cheaper extraction model gets a field wrong and you have to run the document through a frontier model anyway, the first call didn’t save you much.

Add the time spent operating the system, too. Someone will have to deal with slow requests, failed deployments, and the occasional model that won’t fit where you expected.

Data control is sometimes a hard requirement.

With closed APIs, your prompts and documents are processed by another company under its terms.

Major providers do offer controls. OpenAI, for example, says API data isn’t used to train its models unless you opt in. For plenty of applications, the provider’s controls and a contract are enough.

Still, if you handle regulated data or run in an air-gapped network, keeping inference in your own cloud may be a requirement. You also get to decide where requests are logged and how long those logs stay around.

Of course, that means your team has to set those rules up and make sure they work.

Model control is another reason I find interesting.

You can pin a model version while you test its replacement. That’s useful when an agent depends on a particular output format or when a small change in retrieval quality sends it the wrong documents.

But swapping models is rarely as easy as changing a name. Change an embedding model, and you’ll usually need to rebuild your search index.

Change a generation model, and you need to check its prompts, tool calls, and structured output again. You have more choices, but you’re also the one testing them.

When This Move Isn’t for You

Open models are not universal solutions. A closed API is still a better choice when:

• A task needs the top reasoning tier
• Your demand is small, new, or spiky
• You need broad multimodal work
• You want the newest model without deploying it yourself
• Your team has no resources to run and maintain the infrastructure

Measure twice, cut once. And I’m not just being metaphorical here. Measure for real, then let those measurements drive your decisions. If it turns out you really need frontier models, use them without giving it a second thought.

For retrieval, I usually check whether the right document appears in the results, then whether the reranker puts the useful passage near the top.

For extraction, compare the returned fields with the source file. Use your own documents, especially the messy ones.

A benchmark score won’t tell you whether a model understands your product names or reads your PDFs correctly.

Hybrid Pattern

You don’t need to abandon closed APIs entirely. You could split the workload depending on the task at hand.

• Keep high-volume, low-difficulty work on open models you control.
• Pay for frontier-model tokens only when the value is obvious.

A good place to start experimenting with local models is the retrieval and extraction layer. These steps are easier to check than an agent’s final answer, so you can see fairly quickly whether the model is good enough.

Imagine a support agent reading an email with an attached invoice. An open model could extract the invoice number; another could find the relevant account notes.

The frontier model can handle the actual reply if the case needs judgment. If the invoice number is missing or the search results look weak, send that case down a different path.

I recommend you track how often that happens. If most requests end up at the frontier model anyway, you’ve added work without cutting much of the bill.

Superlinked’s SIE vs. OpenAI breakdown and its walkthrough of show what this kind of split can look like.