The AI model router is the unexpected infrastructure story of the summer.
The headlines, in case you haven’t been following along: Stripe is acquiring OpenRouter for over $7 billion. Ramp opened its internal router to the public — on router.com, a domain that does not come cheap — claiming a 30% cut in its own LLM costs. When a payments giant and an expense platform both decide routing is their business, a category has officially arrived. On the more traditional tech side, Fireworks launched Nexus, a difficulty-aware router that steers routine coding work onto open-weight models. Self-hosters have LiteLLM, Portkey, and Kong's AI gateway.
Underneath the froth sits a very sound idea: most AI requests don't need a flagship frontier model, so in theory all you need to do is put a smart router in front of your traffic, send the easy work to cheaper models, and you can watch the bill come down (ideally also while being able to monitor and control usage and performance — although that’s easier said than done).
We think the idea of a router is right. But a router on its own isn’t a complete solution to the tokenomics crisis that is on every enterprise CIO’s mind. And for regulated and sovereign teams it can leave the biggest problem untouched entirely.
The easy solution: hosted routers
If your data already lives comfortably in the cloud and your only concern is the bill, a hosted router might be all you need to make some big savings: change one base URL, get one endpoint between your users and the models, and you’re golden.
But. (You knew there was a ‘but’.)
You’re still paying for every input and output token. Every request the router handles still goes to a model running on a metered GPU, whether it’s in one of the frontier labs, in a hyperscaler, or a neocloud. The router downshifts you from expensive models to cheaper ones, which is real money — 30% by Ramp's numbers — but the meter itself keeps ticking up. And for workloads like agentic coding, your token consumption will be relentless.
Perhaps more worryingly, with a hosted router sitting between you and hosted models, your data still leaves the building to be processed, and what’s worse it now transits two external parties instead of one: the router's infrastructure, then the model provider's. Prompts, context windows, source code, the lot. If you're carrying data residency obligations, sector regulation, or sovereignty requirements, the hosted router arguably makes your situation more challenging.
Self-hosting the router puts you in the platform business
The obvious answer is to run the router yourself. LiteLLM, Portkey, and Kong all support this mode, and now your routing logic lives inside your perimeter. Progress.
Except the router is the top layer of a bigger system. Underneath it you now own a stack running on Kubernetes, token metering, quotas, observability, and all the Day-2 operations to keep the stack running with production-grade service levels. Do you want to be in the business of building something so important in a DIY mode?
Note too that a self-hosted router with no local serving layer beneath it still routes every request straight back out to hosted APIs. Again, that’s restricting you to metered models and leaving you subject to sovereignty concerns. And if you do bake in local serving (which we recommend, as we’ll explain in a sec), now you need to run and maintain vLLM or equivalent, all the NVIDIA or AMD drivers, etc too.
The third option: local routing and local-first inference as a pre-baked supported stack
PaletteAI Inference Launchpad is our answer to this conundrum.
It's a pre-validated, turnkey solution that sits on one or more GPU server nodes — your own, or the AMD Instinct Coder bundle with Supermicro if you'd rather buy it as one system — and makes it a self-contained local-first private inference platform.
It includes a smart and granular classifier and router, but more importantly it actually runs models locally: your choice of certified open models (we use GLM 5.2 by default just like Fireworks do) or you can ‘bring your own’. All this with tuned serving configurations for maximum concurrency performance: think out-of-the-box baked-in vLLM recipes, that have been put through their paces by our engineers.
By default all requests run truly locally, airgapped, for watertight sovereignty, but you can turn egress on and burst to a frontier model of your choice when needed for burst capacity or for more difficult requests.
Routing, metering, quotas, quality enforcement and a full audit trail ship in the image, managed as one product through Day 2, with technical support from us.
By swapping pricey frontier models for meterless on-prem inference, we can claim really big token cost savings, easily beating what the hosted routers alone can claim. You can also place your owned server colo-style in a neocloud and still gain massive savings, without the need to feed the box yourself.
So your CFO is happy because you made big savings on the token bill.
Your CISO is happy because you stopped data leakage.
And most amazing of all, your developers are happy because they don’t have to make any changes: the endpoints are OpenAI-compatible, so Claude Code, Cursor, Open Code, and the rest work unchanged.
Fair warning that this is a slightly bigger day-one decision than pasting an API key into a router dashboard. If you’re buying hardware, there’s some capex and racking-and-stacking. But we know it’s worth it, both for the magnified savings and the control.
So what should you do?
If your token spend is modest and your data can live in the cloud, use a hosted router. It's a fine tool, the savings are real, and you'll be running by Friday.
If your bill is big enough to hurt, or your prompts carry things that can't leave — customer data, source code, anything regulated — then a hosted router alone isn’t enough, and you probably don’t want to build your own local inference stack to accompany a self-hosted router.
Next steps? Run your own numbers in our TCO calculator, or if you want the fuller local-first argument, the launch post — Bring AI inference home from the frontier — is the place to start.