Inworld opens Inference with caching and per-request service tiers

Inworld opened Inworld Inference to developers on October 14, bringing hosted open models, prompt caching, service tiers and failover into one inference service. For teams generating NPC dialogue or interactive fiction during play, the practical choices are how much repeated context can be reused and which calls need a fast response. The launch announcement describes both.

An existing game customer illustrates the workload. Inworld's Latitude case study says Voyage runs roughly 20 AI tasks per turn alongside story generation. Its deterministic World Engine manages skill checks, HP, inventory and persistent memory, with Inworld-hosted models serving engine tasks, stories and dialogue. The vendor reports more than 25 million inference calls in six weeks. Those are customer figures supplied by Inworld, not a cost or performance forecast for another game.

Reuse the context, generate each reply afresh

Prefix caching reuses computation for an exactly matching prompt prefix, such as stable system instructions and conversation history. New input still needs processing, and each reply is generated afresh. Inworld says its router balances available cached context against server load when assigning a request.

The caching documentation exposes prompt_tokens_details and cached_tokens in usage responses so developers can inspect actual reuse. Cache rules, lifetimes and read/write charges vary by provider.

For Gemma 4 26B, the launch's Default rates match the current model page: $0.07 per million input tokens, $0.035 for cached input, and $0.34 for output. The lower cache-read rate applies to the input that hits the cache; uncached input and generated output remain separate costs.

Give replies and background memory different budgets

The announcement lists four tiers, selectable per request or as a default for an API key or workspace:

• Flex: lower-cost work that can be retried later, such as summaries or backfills. If capacity is busy and the call cannot start, it fails promptly without billing that request.
• Default: standard serving for routine chat, classification and decisions.
• Priority: priority scheduling, with availability commitments and latency targets for the relevant model and service.
• Fast: realtime decoding for latency-sensitive generation.

The tiers use the same model weights; availability and commitments depend on the model and service. A game could configure a waiting player's reply separately from a background memory summary, with retry handling for unavailable Flex capacity.

The speed, cache-hit and success-rate figures in the launch artwork describe particular models and traffic. Its “nearly 8×” claim concerns Gemma 4 26B time to first token. Inworld explicitly describes that comparison as separate measurement snapshots, not a matched-workload benchmark. It does not establish that every NPC exchange will improve by that amount.

Switch models when the first token takes too long

The existing request-level routing guide supports fallback models and a first-token deadline, with a minimum timeout of 300 ms. If the primary model misses it, the router cancels that attempt and tries the next model. Response metadata lists the attempts. The player's total wait still includes the first timeout plus the fallback's own response time.

Requests use the OpenAI-compatible Chat Completions format with an Inworld endpoint, model ID and API key. A direct call does not require a saved router. The quickstart keeps the full API key on the server; browser and mobile clients use backend-issued tokens or a server proxy.

For a first dialogue prototype, record calls per turn, cached input, player wait time and actual token charges before choosing tiers for the rest of the game. The Inworld AI tool page covers its runtime integration direction; the Narrative, Dialogue & NPCs category lists related tools.