Training a custom model is only half the problem. Serving it profitably is the other half. This post walks the stack from the top down.
What a LoRA is, and why it changes the economics
A finetuned model does not have to be a full copy of the weights. Low-rank adaptation starts from a simple observation. The change finetuning makes to a weight matrix carries far less information than the matrix itself. So LoRA freezes the base matrix and learns the update as a product of two thin matrices . and have rank typically 8 to 64 against hidden dimensions in the thousands, so the trainable parameters shrink by orders of magnitude. The bases we serve carry hundreds of gigabytes of weights, while an adapter weighs in at megabytes.
Thinking Machines' LoRA Without Regret pinned down when a small adapter matches full finetuning in quality. Apply adapters to all layers, not just attention, and give the adapter enough capacity for the dataset. Inside that regime, which covers most post-training, LoRA reaches the same final performance with the same sample efficiency. For reinforcement learning the capacity bar drops even further, since policy gradient feeds the model so little information per episode that even rank 1 matches full finetuning. Training economics improve too. LoRA training needs barely more memory than inference, where full finetuning needs an order of magnitude more accelerators, and a few-megabyte adapter is quick to checkpoint, load, and move between machines.
But the property that matters for serving is simpler than any of that. Every custom model runs on the same frozen base and only adds a few megabytes of delta on top, which is what makes it possible to serve many custom models from one shared deployment.
One base, many finetunes
Deploying one custom finetune on its own hardware makes no economic sense. Two forces decide the cost of serving, and a dedicated deployment loses on both. First, the cost that matters is per token, not per hour, and what you pay per token depends on how full the GPU runs. The base weights only hit their low per-token floor when they process a large batch every step. One finetune serving one workflow cannot fill that batch, so the GPU idles between requests and the effective cost per token climbs to several times its floor. Second, agent traffic is bursty. It spikes in business hours and collapses overnight. A dedicated deployment is fixed capacity. You reserve your own GPUs for the model, pay for them by the hour around the clock, and have to pick the size up front. Size it to the average and the peak swamps it; size it to the peak and it pays for idle hardware the rest of the day.
The first force is where LoRA changes the picture. Because every finetune is the same frozen base plus a few megabytes of delta, the serving system loads the base onto the GPU once and runs many finetunes on top of it. Requests for different adapters batch into the same forward pass. The expensive base matmuls run once over the whole batch, and LoRA-aware kernels such as S-LoRA and Punica gather each token's adapter and add its small delta in place. Adapters wait in host memory and page onto the GPU only while in use, so one resident base serves hundreds to thousands of adapters with small overhead.
The more requests share one base, the cheaper each token gets, and a finetune running alone on its own GPU never gets that discount. Our serving is built around this. We run two shared base models, a small one for high-throughput work and a large one for the hardest tasks. Your finetune becomes an adapter on whichever fits the job, and all tenants share these two bases, so every one of them gets the economies of scale a large MoE needs.
Where LoRA serving breaks on large MoE models
The serving architecture described above works well when the base model is dense, meaning every parameter participates in every forward pass. The strongest open base models today are instead large mixtures of experts, which route each token through a small subset of the network, and on these models current multi-LoRA serving stacks have real gaps. We ran multi-LoRA serving benchmarks on Qwen3.6-35B-A3B, a 256-expert open MoE, under vLLM to map where the problems sit.
Enabling LoRA moves the engine off its fastest MoE kernel paths, so some of the cost lands on every request even before an adapter is active. Adapter capacity is also lower than the small file sizes suggest, because the engine pre-reserves expert-layer LoRA buffers on every GPU for the worst case rather than for the adapters actually loaded. The batching benefit that the economics rest on erodes as a batch spans more distinct adapters, and it degrades sharply once the working set outgrows the GPU's adapter slots.
There is a workaround, but it costs model quality. Adapters restricted to the attention layers avoid the expert-layer machinery and serve well. However, LoRA Without Regret shows that adapters need to cover all layers, including the MoE blocks, to match full finetuning. Those expert-layer adapters are the ones today's engines struggle to serve, and vLLM does not support them in combination with expert parallelism at all. In practice, vanilla LoRA serving on a large MoE forces a choice between adapter quality and serving cost.
We are working on closing these gaps at the kernel level so that full-coverage adapters serve at base-model speed through our product. A dedicated post will follow with the exact numbers, the issues behind them, and how we address each one.
Why we still need autoscaling
Multi-tenancy fixes utilization; autoscaling fixes the second force, bursty traffic. The serving pool adds replicas as inference load rises and drops them as it falls, tracking real demand instead of provisioning for peak. Bursts from different workflows are uncorrelated, so combined load on a shared pool is far smoother than any single tenant's, and the pool holds high steady utilization without a swamped queue at the peak or idle GPUs at the trough. Autoscaling is not optional for a productionized custom model. It keeps the pool efficient, since capacity follows demand instead of paying for idle GPUs at the trough. And it protects latency, since new replicas come up before the queue backs up at the peak.
Work with us
Your finetuned model, served on a shared autoscaling pool sized to your traffic, on a small or large base depending on the job. You bring the data and the workflow. We handle training, serving, and keeping you on the best model over time. If that is the system you want, reach out at hello@constructlabs.com.