Back to Papers

Reading notes

CacheRoute: Prefix-aware Routing for LLM Serving

Reading note stub for routing requests in ways that improve KV-cache reuse while balancing load.

ENreading note
llm-servingsystemskv-cache

Reading Brief

Paper Snapshot

Reading stage
reading
Importance
high
Venue
Reading queue
Authors
Unknown
Add the BibTeX, APA, or short citation once the canonical source is confirmed.

Open paper source

Key ideas

  • Route similar prompts together to reuse KV-cache state.
  • Cache affinity must be balanced against live worker load.

Contributions

  • Frames prefix locality as a routing signal for LLM serving.
  • Suggests that serving efficiency depends on future request structure, not only current latency.

Limitations

  • Needs comparison against least-load and random routing baselines.
  • Requires workload traces to avoid overfitting to synthetic prompts.

Follow-ups

  • Reproduce the routing policy on a small serving simulator.
  • Measure when prefix affinity creates hotspots.
Paper source

Question

How should an LLM serving layer route requests when cache reuse and load balancing point in different directions?

Takeaway

The central tradeoff is not simply routing to the least-loaded worker. Prefix locality can make future requests cheaper, but only if the load model does not create new hotspots.

Follow-up

  • Compare against a naive least-load balancer.
  • Track when prefix affinity hurts latency.
  • Note the metrics needed for a reproducible experiment.