Long conversations and autonomous workflows create a storage problem disguised as model intelligence. A chatbot may answer one question and discard its working state. An agent that uses tools across sessions must retain large amounts of context and make that history available quickly enough to guide its next action.
Transformer models avoid recomputing every previous token by storing earlier states in a Key-Value cache. As context windows grow toward millions of tokens and many agents operate concurrently, that cache expands into an infrastructure class of its own.
Keeping it entirely in high-bandwidth GPU memory is expensive. Moving it to general-purpose shared storage introduces latency and power overhead. NVIDIA’s Inference Context Memory Storage platform proposes an intermediate layer designed specifically for this derived, fast-moving data.
Context Is Valuable Without Needing Permanent Durability
KV cache is essential to immediate inference performance, but it is not equivalent to a customer record or compliance archive. The underlying conversation and tools may require durable retention; the calculated model states can often be recreated and do not need extensive replication or metadata services.
Existing architecture forces active context through a hierarchy from GPU HBM to system memory and eventually shared storage. Once it reaches the slow tier, millisecond latency can leave costly accelerators waiting for data. Power is consumed maintaining general storage guarantees that the cache may not require.
Infrastructure teams therefore need a new classification: ephemeral but latency-sensitive. Treating context according to its actual lifecycle can reserve HBM for immediate computation and durable systems for information that genuinely must survive failure.
ICMS Inserts Flash Directly Into the Compute Pod
NVIDIA’s ICMS creates what it calls a G3.5 tier—Ethernet-attached flash positioned between host memory and shared storage. BlueField-4 data processors manage context without relying on the host CPU, providing petabytes of shared capacity per pod.
Relevant cache can be prestaged back into GPU memory before the model needs it. NVIDIA claims up to five times higher tokens per second for long-context workloads and five times better power efficiency than traditional methods. These are vendor claims that production deployments will need to validate across different models and retrieval patterns.
Spectrum-X Ethernet supplies low-jitter connectivity, while NVIDIA Dynamo and the NIXL library coordinate movement between tiers. DOCA treats KV communication as a dedicated resource. Storage vendors including Dell, HPE, IBM, Nutanix, Pure Storage, VAST Data, and WEKA are developing BlueField-4 systems expected in the second half of the year.
Memory Placement Becomes an Orchestration Decision
Adding a tier does not automatically improve performance. Software must know which context will be needed, move it early, and place work close to the cached state. NVIDIA Grove uses topology-aware orchestration to reduce unnecessary transfers across the fabric.
Capacity planning also changes. More usable inference can fit within a rack, extending facility value, but higher compute density can increase cooling and power-distribution demands. Savings at the storage protocol level must be evaluated against the entire pod.
Security and tenancy matter when many agents share a memory pool. Organisations need isolation, deletion rules, and observability that show where sensitive context moved. Ephemeral should not mean ungoverned.
Benchmarking should use real agent behaviour rather than a single long prompt. Concurrent sessions, tool calls, cache reuse, eviction, and failure recovery determine whether the new tier reduces latency under production pressure. Average throughput can conceal the pauses that make an interactive agent feel unreliable.
Agentic AI forces the data centre to remember differently. Selecting the fastest GPU is no longer sufficient when that processor waits on growing histories. A specialised context layer could decouple agent memory from HBM cost, but the lasting advantage will come from classifying, placing, and protecting each form of data according to what the workload actually needs.
