Securing high-bandwidth memory (HBM) alone is increasingly insufficient to meet AI server memory demand, shifting the memory industry’s focus from expanding bandwidth to boosting resource utilisation. A multi-layer memory structure that uses CXL, PIM and high-bandwidth flash (HBF) together has emerged as an alternative.

Industry data show that only about 50 to 60 percent of memory installed in AI servers is used for actual computation. The remaining capacity stays fixed in each server and cannot be moved for use by other servers. As memory expansion continues to lag demand, investment is shifting from increasing supply volumes to technologies that reclaim this idle resource.

The situation of having surplus memory yet still being short stems from server architecture. Increasing memory capacity and bandwidth requires adding memory channels to the CPU, which raises both engineering complexity and cost. That is why overprovisioning memory in each server has become common practice to prepare for workload fluctuations. With no way for one server to use idle capacity in a neighbouring server, nearly half of the resources can remain tied up across an entire data centre.

This is also not a phase that can be filled by volume. The number and capacity of high-bandwidth memory (HBM) that can be attached to a single GPU are physically limited, and increasing stacking layers raises process difficulty and production costs. Industry sources say SK hynix and Micron face constraints in securing factory sites needed for additional expansion, while Samsung Electronics is reported to be continuing HBM mass production without fully stabilising yields for 1c-nanometre DRAM. As room to increase supply narrows, improving utilisation of already installed memory has emerged as an alternative.

Demand-side change is faster. Yuanta Securities said that when performing AI tasks, the share of time a GPU spends on actual computation is only about 20 to 30 percent, with the remainder spent moving data and waiting. That reflects a sharp rise in how often intermediate data, such as prior conversations and judgment results, are stored and retrieved as the AI industry’s centre of gravity moves from training to inference and then to agentic AI.

The first route to higher memory utilisation to enter commercialisation is Compute Express Link (CXL). CXL ties together memory scattered across servers through an open high-speed interface so it can be shared dynamically as needed. CXL 2.0, released in 2021, supported memory pooling, and CXL 4.0, released in November 2025, presented 128GT/s speeds based on PCIe 7.0 and bandwidth of more than 1.5TB/s per single logical link. Market researcher Yole forecasts the CXL market will grow to about $15 billion by 2028, and expects a meaningful market to begin forming as next-generation CPUs from Intel and AMD provide full support in 2026.

The two South Korean companies are focusing more on operating software than hardware. Samsung Electronics is pushing mass production of a 256GB CMM-D based on CXL 2.0 and will provide an orchestration console to manage memory resource allocation. SK hynix has commercialised 96GB and 128GB CXL 2.0 memory solutions and presented a full-stack strategy combining Niagara 2.0 software with CMM-Ax computational memory. In the switch segment, Panmnesia has announced plans to supply customers in the second half of next year, and Marvell moved toward vertical integration by acquiring XConn for $540 million in February 2026.

This trend is difficult to read as a structure in which a specific product replaces HBM. The 'tiered memory' presented by SK hynix at an FMS 2026 keynote starts from the premise that a single memory type cannot handle the challenges of the agentic AI era. It envisions a multi-layer structure in which HBM handles bandwidth, CXL provides capacity and reduces total cost of ownership (TCO), and PIM, which puts compute functions into memory, shares responsibility for power efficiency.

PIM, which puts compute functions into memory, is still at the validation stage. Samsung Electronics has said that applying PIM to HBM and installing it on an AMD GPU accelerator card doubled performance and cut energy consumption by 50 percent. SK hynix introduced the GDDR6-based accelerator AiM and the combined card AiMX. Software support, including compilers, still remains, limiting server deployment to specific workloads.

SK hynix presents 'tiered memory'... single memory has limits

HBF stacks NAND flash vertically like HBM to place large volumes of data close to the GPU. Response speed is more than 1,000 times slower than DRAM, but it is optimised for moving large amounts of data at once, drawing attention for its potential in inference-focused environments. SK hynix and SanDisk unveiled the first standard specification through OCP at FMS 2026 in August. Based on a 2-stack configuration, it offers up to 512GB of capacity, 0.4 to 3.0TB/s of bandwidth by grade, adopts a UCIe interface, and Google and Tenstorrent joined the consortium.

Commercialisation timing is converging around 2027. SK hynix and SanDisk plan to release samples in the second half of this year and then introduce the first AI inference system equipped with HBF in early next year. Samsung Electronics set a target of full-scale mass production in 2027, leveraging its No. 1 position in NAND flash.

This technology shift is expected to extend to back-end processes and testing. Yuanta Securities analysed that adoption of a new memory structure could drive changes in specifications for test equipment and consumables such as sockets, boards and pins, which could help improve average selling prices (ASP) for related companies.

Ultimately, a key hurdle is whether the supply-chain ecosystem can accept these memory technologies. The industry sees adoption could spread more slowly if Nvidia does not actively embrace CXL. HBF also faces challenges including durability issues, with NAND endurance of about 100,000 cycles, and access latency. An industry official said, "Separate from hardware competition, spreading the operating system that supports this across the ecosystem is a different issue."