NetKV: Network-Aware Decode Instance Selection for Disaggregated LLM Inference

arXiv:2606.03910v1 Announce Type: cross Abstract: Disaggregated LLM inference forces the KV cache to traverse the datacenter network before decoding begins, so transfer time enters directly into the Time to First Token (TTFT) budget. Current schedulers route on compute load and prefix-cache locality alone, ignoring the topological distance and...

Læs hele artiklen hos kilden.