Mental Models for Cloud LLM Inference

Deploying fine-tuned models to the cloud requires a complete inversion of the training mindset. During training, the goal is total hardware saturation: batching datasets to fill GPU memory and maximize compute utilization. In production cloud inference, the workload is unpredictable, concurrent, and highly latency-sensitive. A model that runs instantly on a local workstation can choke in a production cluster when hit with parallel user queries. Scaling cloud inference demands an architectural paradigm built around dynamic batching, virtual memory management, and bandwidth constraints.

The Latency-Throughput Trade-off

At the hardware level, generating a single token requires reading the entire model weight matrix from the GPU’s High Bandwidth Memory (HBM) to its SRAM processors. This makes inference fundamentally bound by memory bandwidth, not compute. When a single user queries a model, the GPU sits mostly idle, waiting for the slow memory bus to fetch weights. To reclaim efficiency, cloud runtimes use dynamic batching, grouping incoming requests into a single forward pass. This introduces a physical trade-off: a request must wait in a queue for a few milliseconds to form a batch with other requests. We trade individual query latency for massive parallel throughput. If the queue delay is set too high, the system feels sluggish; if it is set too low, the GPU runs under-utilized, driving up the cost per query on expensive instance runtimes like an A10G instance costing around ₹80 per hour.

PagedMemory and the KV Cache

The hidden bottleneck of cloud inference is not the model weights, but the intermediate states of active generations. For every token generated, the model must store the Key and Value matrices of all preceding tokens in the self-attention layer to avoid recomputing them. This Key-Value (KV) cache grows dynamically with batch size and sequence length. In standard execution environments, memory for this cache must be allocated contiguously. Because generation length is unpredictable, runtimes are forced to pre-allocate maximum-length blocks of memory, leading to severe virtual memory fragmentation. Runtimes like vLLM bypass this by implementing PagedAttention, partitioning the KV cache into small, non-contiguous physical pages. By mapping logical token addresses to physical pages in a virtual memory table, we reclaim up to 96% of wasted VRAM, permitting larger batch sizes and higher concurrency before the system runs out of memory.

Speculative Decoding as a Hardware Conduit

Because memory bandwidth dictates token generation speed, running a large model is structurally slow. Speculative decoding bypasses this by using a tiny draft model to predict future tokens, which the larger target model verifies in a single parallel step. In our architecture, the fine-tuned 270M Gemmaiku model functions as the draft engine, generating a sequence of tokens at high speeds on cheap hardware. The larger base model verifies the draft sequence in one GPU cycle. If the draft model’s predictions match the base model’s probability distribution, the system generates multiple tokens in a single step, bypassing the sequential memory-read bottleneck. The success of this system relies entirely on the draft model’s alignment with the target’s output style; if the draft model wanders, the verification step rejects the sequence, and the system falls back to slow token-by-token generation.

The Cold-Start Transport Tax

In serverless cloud deployments, scaling to zero saves capital but introduces severe boot bottlenecks. When a request hits a cold GPU node, the system must pull the model weights from network storage, transfer them to host CPU memory, and copy them over the PCIe bus to GPU VRAM. For a 7B parameter model, this requires transporting roughly 14 gigabytes of data. Over a standard 10 Gbps network interface, this network-to-disk hop takes at least 11 seconds before the model can execute its first floating-point operation. We cannot treat model weights like lightweight code binaries. To mitigate this transport tax, weights must be cached locally on the GPU host node’s NVMe drive, using memory-mapped files to bypass the CPU and stream weights directly into the GPU’s memory space at the speed of the PCIe bus.