.png)
The token tax hockeystick
Goldman Sachs Research forecasts that AI token consumption will grow 24-fold by 2030, to 120 quadrillion tokens per month. KPMG's Q2 AI Pulse survey found that only 26% of organizations have real-time visibility into what running AI costs them.
As we explored in Cheaper tokens, bigger bills: per-token prices fell roughly tenfold in a year, yet enterprise AI spend still more than tripled.
Coding agents are perhaps the biggest offenders for burning tokens. An agent that inspects files, holds long context, calls tools and retries failed tasks consumes tokens at a pace no chat window ever did.
You might not get trillion-dollar monthly bills from the frontier labs yet (unlike last week’s AWS glitch) but the numbers still hurt.
The local alternative
Big beefy frontier models like Sol and Fable are powerhouses, but you don’t need to use them for every prompt. Nor is your alternative simply to fall back to the likes of Sonnet and Haiku.
The latest crop of open-weight models are competitive with the best from the big hosted labs — for example, China’s Moonshot AI literally just released Kimi K3, the largest open-weight model ever published and a genuine Fable competitor.
In many cases, you can run these open models yourself, on your own hardware, and skip the token tax (almost) entirely.
That’s exactly what we’re solving for with the PaletteAI Inference Launchpad, released today. It’s a turnkey software appliance that makes a GPU server into a governed, production-ready local inference platform.
Let’s walk through.
The fast lane to local inferencing
We ship PaletteAI Inference Launchpad as a bootable image. That one artifact contains the whole software stack: an immutable Linux OS (Kairos on Ubuntu 24.04), Kubernetes, the vLLM inference runtime, intelligent routing, token metering and an operator console.
Install it on an off-the-shelf GPU server (an 8-GPU box with NVIDIA or AMD GPUs from the likes of Supermicro is a typical home) and you have a working inference platform, without spending a quarter assembling, integrating and supporting those layers yourself.
Importantly, it's fully standalone: you don’t need to use our PaletteAI management platform, nothing phones home, and you can even run it totally air-gapped if your environment demands it (of course that means no access to the frontier labs).
At launch we certify GLM 5.2, DeepSeek v4 Pro and Kimi 2.7 as local models. Certified means we've tested the model on specific GPU configurations rather than assuming it works from public benchmark — and the appliance smoke-tests every deployed model before it takes real traffic from your developers.
Point your coding agents at it and go
Every model on the appliance is served through an OpenAI-compatible endpoint. Whether they use Claude Code, Cursor, OpenAI Codex or OpenCode, your devs can keep the tools they've already adopted. The only thing that changes is where inference runs and who can see the prompts and context.
The Inference Launchpad routes each request based on rules you set: by client, task type, data sensitivity or cost. Work involving private code or regulated data can stay on the box. Tasks that need a frontier model can get one, through an approved egress path that's metered like everything else. Egress is denied by default: no client reaches an external provider until an operator (that’s you) switches it on.
Which brings us back to Kimi K3, Inkling and GPT-5.6 Sol — and whatever hits Huggingface next week, or the week after that.
With a policy-based gateway approach like the Inference Launchpad, adopting new models just means a routing change. K3's open weights make it a candidate to serve locally, hardware permitting (2.8 trillion parameters is not a casual commitment). Frontier models like Sol are just a new egress target behind policy.
Out of quota? You get a 429
Every request belongs to a client: a coding assistant, an internal chatbot, a data pipeline. Each client holds its own API tokens and its own quotas, enforced across requests, tokens and dollars over rolling windows from one second to one day.
A client that exhausts its budget gets HTTP 429 until the window rolls over — that means there’s definitely no surprise invoice at month end. And because every request is metered per client and per model, chargeback becomes possible: each team can see what its AI usage costs while it's happening.
Find us in San Francisco this week, or Vegas next month
We're launching PaletteAI Inference Launchpad at AMD Advancing AI in San Francisco, this week (July 22–23). The venue is fitting: AMD Ventures joined our recent Series D, and the appliance is validated to run the AMD stack end to end — GPUs, GPU Operator, ROCm — alongside NVIDIA. If you're at Moscone, come find us and bring your inference bill. We enjoy a challenge.
Two weeks from now, August 4–6, we'll be at Ai4 at The Venetian in Las Vegas. And if conferences aren't your thing, talk to us directly: tell us what you're running, and we'll show you what it looks like running at home.