Did you miss our webinar with TeraSky on the real economics of production AI? No judgment. It was EMEA-timed, which for half our audience meant it clashed with sleep, and no webinar can compete with that. You can watch the recording on demand here.
Meet the speakers (and the man failing to herd them)
Guiding us through the session were Pedro Oliveira, Principal AI Architect at Spectro Cloud, and Scott Rosenberg, Lead Architect in the CTO Office at TeraSky, a long-standing partner of ours.
Pedro works hands-on with deployments from the largest AI factories to single-node appliances, and has spent the last few months benchmarking inference on more GPUs than is probably healthy. Scott leads TeraSky's platform engineering and cloud native practice, is active in the Kubernetes project, and coined "shift down" as the antidote to shift left. He's opinionated. He'll tell you so himself.
The two of them last did a webinar together about a year ago, and the feedback was good enough that we had to run it back. My job was to get them through three topics in half an hour. Reader, I did not.
GPU utilization: the problem everyone stopped talking about
We opened with the numbers everyone has seen: single-digit utilization in clusters full of very expensive accelerators. Pedro thinks the topic is fading from the conversation, and he was blunt about why. "I don't think it's dying off because it's not important anymore. I think it's dying off because everyone has to focus on all these different things and AI throws a million at you every week."
His root cause is that Kubernetes ("the runtime for AI and I think that's pretty much settled") only recently learned to treat a GPU as a first-class resource. Dynamic resource allocation (DRA) changes that, alongside vendor features like NVIDIA's multi-instance GPU (MIG).
Scott, who spends a lot of time upstream, gave the 30-second version of DRA: it does for accelerators what CSI did for storage and CNI did for networking. A driver describes what devices exist and what they can do, the scheduler hands out slices, and nobody has to hard-code NVIDIA or AMD specifics into Kubernetes. It matters more every quarter, because "it's not just GPUs anymore, it's specialized processing units for AI directly." NPUs, APUs, whatever ships next.
The other half of the problem is human. Developers naturally oversize resources for their apps. Scott's impression of the average request: "My app needs the full GPU, please." CPU and memory have autoscalers to correct for that (Karpenter, Cluster Autoscaler, KEDA, take your pick). GPUs don't obey. "Can I please have another node with four H200s? Good luck. Yeah, you can get that in 60 days." So people squat on capacity they aren't using, because releasing it means someone else grabs it.
Scott's answer is to stop treating a GPU as belonging to one use case. Serve the priority workload, then let a gang scheduler backfill the idle half with classification jobs and the unglamorous batch backlog. "Because I'm paying for that GPU anyway." For the practical version, Pedro has written a guide to secure self-service GPU sharing with MIG and DRA, four tenants to a card.
Shelfware: the servers that arrived before the models were ready
I asked about the loading-dock-to-live problem, and Scott confirmed it's real. "We see customers that bought GPU servers that are just sitting there, not even plugged in or anything."
His explanation was more interesting than "they bought too much." Those servers were ordered in the first flush of the AI boom, before Kimi K3, before the GLM models, before even gpt-oss-120b. Many teams ran a proof of concept against the open-weight models of the day, compared them with frontier, and walked away disappointed. "They joined too early." The hardware stayed in the rack gathering dust while everyone went back to the ever-improving frontier API.
That comparison is now stale. The gap between open-weight and frontier is smaller every month, and plenty of enterprises are still paying frontier prices for models a generation or two old. If your POC was a year ago, run it again. (We've written about what it takes to make on-prem AI work this time round.)
Tokenomics: the bill that hits you in the face
Pedro's field report: "You mention tokens and frontier models, their eyes widen." Everyone chased the 10x productivity curve for their engineers with agentic coding, and budgets got spent 10x faster too. He compared it to the cloud bill shock of five or six years ago.
Scott quoted a Gartner prediction that by 2027 the average developer's AI usage will cost more than the developer. But he was just as worried about waste on the demand side. The typical enterprise rollout is "here, take a Claude sub. Go." Then people hear about skills, ask the model to write one, and load a badly written skill and a chatty MCP server into every session. "It's bloating their context and costing huge amounts."
I confessed to burning an alarming number of tokens the week Astra launched, by leaving it on its Ultra setting for a day. Which rather proves the point. You can't rely on users to optimize their own usage; that's a platform team's job.
Pedro's condition for local inference getting a second hearing was speed. "The time to value has to be almost instantaneous, because the time to value with frontier models is instant." Fast for the platform team to stand up, and invisible to the developer who doesn't want to reconfigure their coding agent every time the model behind it changes.
Shift down: make model choice nobody's problem
In our community we’ve spent a lot of time talking about ‘shift left’, and how it puts way more responsibilities in the hands of poor, overworked devs. In an AI world, shift left hands the model decision to the developer: Fable, Opus, Sonnet, a local model, whatever. Scott instead prefers to talk about moving responsibility into the infrastructure platform: shifting down. That hands the decision to an intelligent router that looks at each request and picks the model that fits, inside guardrails the ops team set once. "The developer doesn't know hardware, and the team doesn't know the application." Let the platform, which knows both, decide.
Scott added a distinction between organizations that have to self-host (regulation, data residency, an existing data center) and organizations that choose to. The first group can't burst to models hosted outside the perimeter; whatever they bought is what they have. The second gets the best of both if the router isn't locked to one ecosystem, sending the hard problems to frontier and everything else local. Some tasks still deserve the biggest model money can buy. Most don't. His words: "You shouldn't be using Fable for most things."
Without getting salesy, that's the design behind PaletteAI Inference Launchpad: a router, a tuned local serving stack, and metering and quotas in one package. We've published what happened when we ran it on our own engineers, and why a hosted router on its own won't fix the bill.
Performance tuning: let the model breathe
There’s one more lever to the economics of AI that doesn’t get talked about so much, but we tried to squeeze it in at the end: and that’s performance tuning. You can tune how a model is served, and the tuning changes the economics. Pedro's ROI framing is simple: on-prem inference pays back on how many concurrent users you can serve from one node without them noticing, "because if it feels degraded, if it feels sluggish, then they just won't use it."
Months of benchmarking across AMD and NVIDIA hardware taught his team that public benchmarks are the wrong shape for enterprise decisions. "We have to look at the traffic patterns of 50 developers and their agents over a day. That's very different from hammering eight GPUs with 100 streams all at once." Mixture-of-experts models like GLM 5.3 behave differently under bursty agent traffic than under steady prompting, and there are a lot of knobs to turn. Pedro's method: "let the model breathe," watch how it behaves, then tune.
Scott's warning was about who does the tuning. "You need a doctorate to understand these things when you go down the open source world." vLLM alone has over 100 flags. Bedrock and Azure AI Foundry hide all of that behind "deploy Kimi for me." Self-hosting hasn't had that button until recently, which is the gap Inference Launchpad's tuned recipes exist to fill. His summary of what the market needs: "the simplicity of frontier models with the flexibility of the open source CNCF ecosystem."
One thing I added. When a hyperscaler tunes its serving stack for better model performance, the savings land on its P&L: more users per GPU means more profit. When you tune your own models on your own infrastructure, the benefits are all yours. And a bonus hot take from Scott: plenty of small language models run happily on CPUs. For the overnight job that classifies emails, don't buy a GPU at all.
Monday morning advice
I asked each of our opinionated speakers for one thing a platform engineer should do first thing next week.
Pedro: find out how much more you can get from the GPUs and models you already have, through utilization and tuning. The headroom is bigger than most teams assume.
Scott: don't DIY it. "Just going and thinking, oh yeah, I can DIY this, is a quick recipe for failure." Whether it's a vendor, an SI or both, find a partner who lives in this space, because by the time you've become the expert, the field has moved on.
Watch the webinar and take the next step
The full conversation runs well past its 30-minute slot (sorry) but I promise it’s worth your time. Watch the recording here.
Then pick your next move:
- If idle GPUs are your problem, read Pedro's guide to GPU sharing with MIG and DRA.
- If the token bill is, explore PaletteAI Inference Launchpad.
- If you're stuck between proof of concept and production, TeraSky's GenAI Anywhere framework and Applied AI Center of Excellence are built for exactly that.
- For the bigger picture on matching each workload to the right infrastructure, start with our definition of hybrid AI.