HPE puts storage to work on AI’s expensive memory problem

HPE is giving its Alletra Storage MP X10000 another job: holding inference state that would otherwise be evicted from expensive GPU memory and recalculated later.In testing validated by Kamiwaza using NVIDIA H200 GPUs, the Kamiwaza AI platform, and HPE Alletra MP X10000, the solution achieved up to 20× faster time to first token and 17× higher effective inference throughput versus running without KV cache offload.The GPUs did not become cleverer. They had less repeat work to do.The results move a storage decision directly into the discussion about AI economics. Model pricing gets plenty of attention. Paying expensive accelerators to process the same context repeatedly deserves some too.Especially as enterprises move from chatbots...

Read the full article at the source.