Skip to content

[KDN] Add background KV cache replication between LMCache nodes #190

Description

@grubwithu

Goal

Add a CacheRoute-level primitive for proactively replicating KV cache from one LMCache node to another.

The intended use case is a cluster with large host memory but limited inter-node bandwidth. Instead of waiting for a request to arrive and then fetching KV cache remotely, CacheRoute should be able to use idle network bandwidth to copy useful KV cache into the target node's local DRAM ahead of time.

Node A L1
   |
   | background / idle-bandwidth replication
   v
Node B L1

Later, when a request reaches Node B:

GPU miss
  ↓
local LMCache DRAM hit
  ↓
reuse KV directly

This keeps cross-node KV transfer out of the request critical path.

LMCache PR LMCache/LMCache#5089 already provides most of the required data-plane mechanism through the Coordinator cache-move API. With keep_source=true, it can effectively perform L1-to-L1 replication.

CacheRoute therefore only needs to expose a logical replication operation and map it to the LMCache Coordinator API.

Future work

Once the replication primitive exists, CacheRoute can add a policy that decides what to replicate, where, and when, based on signals such as:

  • cache reuse frequency;
  • saved prefill cost;
  • available DRAM;
  • current network utilization;
  • KV size.

The first step is only to provide the replication mechanism. Automatic bandwidth-aware placement can be implemented separately.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions