Question
How should an LLM serving layer route requests when cache reuse and load balancing point in different directions?
Takeaway
The central tradeoff is not simply routing to the least-loaded worker. Prefix locality can make future requests cheaper, but only if the load model does not create new hotspots.
Follow-up
- Compare against a naive least-load balancer.
- Track when prefix affinity hurts latency.
- Note the metrics needed for a reproducible experiment.