Goal
Add a CacheRoute-level primitive for proactively replicating KV cache from one LMCache node to another.
The intended use case is a cluster with large host memory but limited inter-node bandwidth. Instead of waiting for a request to arrive and then fetching KV cache remotely, CacheRoute should be able to use idle network bandwidth to copy useful KV cache into the target node's local DRAM ahead of time.
Node A L1
|
| background / idle-bandwidth replication
v
Node B L1
Later, when a request reaches Node B:
GPU miss
↓
local LMCache DRAM hit
↓
reuse KV directly
This keeps cross-node KV transfer out of the request critical path.
LMCache PR LMCache/LMCache#5089 already provides most of the required data-plane mechanism through the Coordinator cache-move API. With keep_source=true, it can effectively perform L1-to-L1 replication.
CacheRoute therefore only needs to expose a logical replication operation and map it to the LMCache Coordinator API.
Future work
Once the replication primitive exists, CacheRoute can add a policy that decides what to replicate, where, and when, based on signals such as:
- cache reuse frequency;
- saved prefill cost;
- available DRAM;
- current network utilization;
- KV size.
The first step is only to provide the replication mechanism. Automatic bandwidth-aware placement can be implemented separately.
Goal
Add a CacheRoute-level primitive for proactively replicating KV cache from one LMCache node to another.
The intended use case is a cluster with large host memory but limited inter-node bandwidth. Instead of waiting for a request to arrive and then fetching KV cache remotely, CacheRoute should be able to use idle network bandwidth to copy useful KV cache into the target node's local DRAM ahead of time.
Later, when a request reaches Node B:
This keeps cross-node KV transfer out of the request critical path.
LMCache PR
LMCache/LMCache#5089already provides most of the required data-plane mechanism through the Coordinator cache-move API. Withkeep_source=true, it can effectively perform L1-to-L1 replication.CacheRoute therefore only needs to expose a logical replication operation and map it to the LMCache Coordinator API.
Future work
Once the replication primitive exists, CacheRoute can add a policy that decides what to replicate, where, and when, based on signals such as:
The first step is only to provide the replication mechanism. Automatic bandwidth-aware placement can be implemented separately.