Research note
Multi-LoRA serving and routing
Many adapters, one base model, one router.
Once a team starts training, adapters multiply. One for reading inspection reports, one for writing SQL against a particular warehouse schema, one tuned on a quality team’s review notes. Each is small. Serving each on its own copy of a base model is not, and most of those copies would sit idle. This note is about serving many low-rank adapters on one base model, and about deciding which adapter (or which model) should handle a given request.
Why adapters are cheap to keep
A LoRA adapter freezes the base weights and learns a low-rank update beside them: for a weight matrix of size d by k, it trains two thin matrices of rank r, so the trainable parameters drop from d times k to r times (d plus k). At the ranks we use, an adapter is a fraction of a percent of the base model.11Hu et al., 2021. The update is BA with B in d by r and A in r by k, added to the frozen W. Storing a hundred of them costs less than storing a second base model.
The catch is compute, not storage. Merging an adapter into the base weights makes inference free of overhead, but then that copy of the model serves only that adapter. Keeping adapters separate lets one copy serve many, at the price of an extra small matrix product per layer per request.
Batching requests that want different adapters
The interesting systems problem is the batch. Throughput on a GPU comes from batching many requests through the base model together. If every request in the batch wants a different adapter, a naive loop over adapters throws that away.
Two published designs shaped how we think about this. Punica introduced a kernel that applies a different adapter to each row of a batch in one launch, so the base pass stays shared and the adapter pass is gathered rather than looped. S-LoRA added memory management on top: adapters live in host memory and are paged onto the GPU in a unified pool alongside the attention cache, so thousands can be registered while only the active ones occupy device memory.22Chen et al., 2023 (Punica); Sheng et al., 2023 (S-LoRA). Both report throughput far above serving merged copies when adapter popularity is skewed.
The practical lesson is that adapter popularity is very uneven. A few adapters take most of the traffic; a long tail is used a few times a day. A serving pool that keeps the head resident and pages the tail on demand fits that shape well. Cold-start latency for a tail adapter is the visible cost, and it is usually acceptable for workflows that run in the background.
Routing: which adapter, which model
Having many adapters creates a decision on every request. A planning step may need the strongest general model; a table lookup may be handled well by a small model with the SQL adapter; a classification of defect text may need only the smallest model with a domain adapter. Sending everything to the largest model is simple and expensive.
Our router is a small classifier over the step type, the workspace and a short summary of the input. It predicts, for each candidate (model plus adapter), the probability that the step passes its checks, and picks the cheapest candidate above a quality threshold set per workflow. The threshold is the knob the workflow owner controls: a month-end close workflow sets it high, an exploratory search sets it lower.
Training the router needs labelled outcomes, and the platform already produces them: every step in a graded run has a pass or fail from the evaluation suite. Routing research elsewhere has used preference data the same way to learn when a cheaper model suffices.33Ong et al., 2024 (RouteLLM) train routers from preference data between a strong and a weak model.
Evaluate each adapter on its own
An adapter that improves average quality can still regress on a slice. We grade each adapter against its own slice of the gym and against a shared regression set, and an adapter is only registered for routing after it passes both. The routing table records which adapter version served which step, so when a reviewer corrects an answer we know exactly which weights produced it.
Adapters are cheap to keep. The discipline is in knowing which one answered, and whether it should have.
A version change to an adapter is treated like any other change to a workflow: it is proposed, graded, and approved by the workflow owner before traffic moves. Rolling back means pointing the routing table at the previous version, which is still in the pool.
Open questions
- Composing adapters. Some requests want two skills at once, such as a schema adapter and a domain adapter. Summing low-rank updates works sometimes and interferes other times. We do not yet have a reliable rule.
- Router drift. As adapters improve, the router’s predictions go stale. We retrain it on a schedule; retraining on every adapter change would be better and is more expensive.
- When not to train. Many gains we first attributed to an adapter came from a better instruction or tool description. That question has its own post.