Cloud Providers Are Preparing to Cannibalize Their Own LLM Inference

A surprising amount of generative LLM work comes down to one small decision. Which queue should receive this ticket? Should this post pass moderation? Which tool should the agent use? How relevant is this document? Developers request JSON, restrict the possible answers and turn off reasoning, yet still buy token generation to get a single choice. Decision models compete directly for that spending. Taking over the most routine, frequently repeated calls would be enough to redraw part of the market.

We explored the setup in our earlier piece on Jev: application state and typed questions go in; choices, scores and probabilities come out. TypeSafe calls Jev a System One Model and says it evaluates questions in parallel rather than generating an answer token by token. The application gets a value its code can branch on immediately. That becomes particularly attractive when the branch runs millions of times and nobody needs a written explanation.

“Jev evaluates all declared questions in parallel and returns typed answers plus probabilities directly.” — Vercel

Open-weight alternatives followed, including Laya and Kev. Their architectures differ: Laya uses an encoder with a separate decision head; Kev uses LoRA adapters and a pointer head on Qwen. Both produce bounded decisions and probabilities directly. Self-hosting becomes another option alongside choosing an API vendor: a recurring classification task can move off someone else’s generative service and onto a company’s own server. The savings depend on hardware, operating costs and workload, but the migration path already exists.

The same product idea has now reached major providers. Alibaba Cloud offers decision-model-preview, with a System One API for classification, scoring and yes-or-no decisions. Its listed uses include ticket routing, moderation and agent action selection. OpenAI announced its Decisions API at DevDay 2026: Luna receives text or image context and answers developer-defined questions with a finite set of possible responses. It was in limited preview at announcement. Whatever the underlying architectures, the commercial target is clear: short, bounded decisions for software, offered through a dedicated API.

TypeSafe has put an aggressive price tag on that proposition: $0.042 per million input tokens, with free output, plus a claimed latency of 70–500 milliseconds. The economic case does not require beating every LLM on every benchmark. A narrow call that meets the required quality threshold at a lower price and latency is a sensible candidate for migration. The useful comparison is cost per usable decision, including mistakes, retries and escalation to a stronger model.

There are two ways this can eat into cloud inference spending. The first is direct substitution: a decision model takes over classification, ranking or verification previously handled by a generative LLM. The second is routing: a cheap decision layer determines which model should receive the actual task. In an AWS Builder example, Jev classifies the request, ordinary code selects a model, and that model writes the response. The example demonstrates the architecture; the business case is to reduce expensive calls while preserving acceptable quality across the system. One small classifier can reshape the budget for a much larger stream of work.

Classifiers, scoring models and routers existed long before Jev. The appeal of this wave is flexibility: developers can ask questions about varied text inputs through a familiar API, change the available answers and use one model for several kinds of decisions. General-purpose LLMs made that logic easy to prototype. Decision models are competing for the next stage, when the prototype works, traffic grows and paying for general-purpose capability at every step starts to hurt.

The available analysis does not yet put a separate number on how much cloud LLM inference has already moved to Jev-like services. The incentives nevertheless point to a messy fight, particularly over the hyperscalers’ expensive-model business.

If a customer switches from an expensive generative API to a cheap decision API within the same cloud, the provider keeps the customer but earns less from the existing set of tasks. Its serving costs can fall too, so a smaller bill alone says little about margins. If the customer runs open weights on its own hardware, the cloud can lose that workload altogether. Renting GPUs from the same provider keeps spending in the cloud, but moves it to another billing category, with less compute needed for the task. Decision counts, compute consumption and inference revenue stop moving in lockstep.

Lower prices can also expand demand. Fast, inexpensive decisions become worth making in places previously served by crude heuristics: checking an agent’s output more often, evaluating more candidates or filtering every step of a process. Expensive generation gives up some existing tasks while the total number of machine decisions grows. For cloud businesses, the question is whether those additional calls make up for lower spending on the old ones. A billion cheap decisions can look spectacular in usage statistics and considerably less impressive on the revenue line.

Offering these APIs is a rational way for Alibaba and OpenAI to lead their own cannibalization. Keeping every classification task on a general-purpose LLM gets harder when a specialized service—and downloadable weights—are available next door. The obvious migration target is the high-volume layer of short decisions currently buried inside generative inference. Some will stay with larger models because the quality requirements or context demand it. The rest will need a better reason to keep paying for the ability to write prose. Cloud providers are already offering another way to buy the decision.