Nvidia has introduced a new memory architecture called NVHBM, a customized High Bandwidth Memory design that the company says can deliver substantially more bandwidth, lower power consumption, and larger usable silicon area for AI accelerators. The announcement came in a blog post published on the same day as Nvidia's quarterly earnings call, underscoring the strategic importance of memory in the company's AI platform efforts.

NVHBM is not being sold as a standalone memory product. Nvidia does not manufacture memory chips; that job will fall to one of the big three DRAM suppliers: Micron, SK Hynix, or Samsung. Nvidia's role is to design the architecture and define the interface in a way that its partners can implement. The new memory is positioned as an element of NVLink Fusion, Nvidia's broader effort to enable hyperscalers and AI companies to build custom silicon while connecting into Nvidia's rack-scale infrastructure.

Amazon's Annapurna Labs will be the first announced partner to work with Nvidia on NVHBM. Annapurna Labs designs custom silicon for Amazon Web Services, including the Graviton processors and Trainium and Inferentia AI chips. This suggests NVHBM could eventually find its way into cloud data centers where AWS both competes with and relies on Nvidia's GPU-based offerings for different customer workloads.

What makes NVHBM different

At its core, HBM is a type of memory that is placed very close to an accelerator chip. Rather than sitting in DIMM slots on a motherboard, HBM dies are stacked vertically and mounted beside or on top of the processor using advanced packaging. The vertical stacking increases density and allows a very wide interface between memory and the accelerator. Data can travel in parallel across thousands of tiny wires, providing far more bandwidth than conventional DRAM running over a memory bus.

The main difference between NVHBM and standard HBM is where the memory controller resides. In a conventional HBM system, the memory controller is built into the accelerator's compute die. That controller and the physical interface, or PHY, consume silicon area and require numerous I/O connections. Nvidia's design moves the memory controller into the HBM stack's base die, which also contains the logic needed to manage the stack. By integrating these elements into the memory package, Nvidia claims it can reduce interface circuitry on the XPU and give chip designers more breathing room.

In a conventional memory subsystem, the base die under the stacked DRAM dies is mostly responsible for routing signals and performing simple buffer functions. Nvidia is essentially turning that base die into a more intelligent component by embedding a custom memory controller and high-speed PHY into it. That shift may seem subtle, but it has major implications for accelerator design.

What Nvidia claims NVHBM delivers

Nvidia is never shy about making bold claims, and its NVHBM announcements are no exception. The company says the new architecture can provide, compared with standard HBM4E, up to 30% greater memory bandwidth, up to 15% lower HBM power consumption, up to 25% more area on the XPU compute die, up to 67% less PHY and supporting interface area, and up to 80% more usable silicon across the overall layout.

• Up to 30% greater memory bandwidth than standard HBM4E
• Up to 15% lower HBM power consumption
• Up to 25% more area on the XPU compute die
• Up to 67% less PHY and supporting interface area
• Up to 80% more usable silicon across the overall layout

To understand why these numbers matter, consider how much of a modern AI accelerator chip is actually devoted to compute. Large matrix multiplication engines, attention blocks, and data movement circuits occupy a significant fraction of the die. Memory controllers and high-speed PHYs add yet more area. By moving those components into the memory base die, Nvidia is essentially giving chip builders more space to add compute engines or cache. This could be as valuable as raw bandwidth improvements, because in a fixed package footprint, a design that saves area can add more AI performance without making the chip larger or more expensive.

Nvidia also says that when the bandwidth increase, additional compute-die area, and lower memory power are considered together, the combined architectural changes can produce approximately a 30% improvement in end-to-end XPU performance. That is a significant number in the data center market, where AI accelerators are often constrained by memory bandwidth and power.

Why Nvidia is developing NVHBM

NVHBM is tied to Nvidia's ambitions beyond selling complete GPUs. Many hyperscalers and large AI companies want to design their own chips, either to reduce cost, increase power efficiency, or differentiate their workloads. Nvidia has been building NVLink Fusion as a way to let those companies integrate their custom XPUs with Nvidia's networking, switching, and management software. By offering a standardized memory design, Nvidia can make it easier for custom silicon to use the same high-bandwidth memory stacks across multiple suppliers.

In the blog post, Nvidia wrote: "Nvidia is establishing a standard NVHBM implementation, available from multiple memory providers. This reduces the engineering effort required to integrate and qualify memory across multiple suppliers — giving NVLink Fusion customers a faster path for bringing custom AI chips to market."

The qualification process is one of the slowest parts of deploying new memory. Memory vendors must ensure that their products meet reliability, power, thermal, and signaling requirements for a given processor or accelerator. By designing a standard NVHBM implementation that multiple memory makers can produce, Nvidia aims to avoid the need for each vendor to develop its own proprietary interface. This could make it easier for a custom AI chip designer to switch between Micron, SK Hynix, and Samsung without redesigning the chip.

Memory market context

HBM has become one of the most sought-after components in the AI industry due to the explosion of generative AI and large language models. Memory suppliers have struggled to keep up with demand, with HBM packages often sold out well in advance. Nvidia's move to design its own HBM flavor adds a new layer to its already complex relationships with memory manufacturers. Samsung, SK Hynix, and Micron all make HBM for Nvidia's current GPUs. Nvidia is not about to start manufacturing those chips itself, but it is increasingly behaving like a memory architecture company rather than just a GPU buyer.

The move also puts Nvidia in more direct competition with companies that have already invested in custom memory. Samsung and SK Hynix, for example, have both developed customized high-bandwidth memory solutions and are working with multiple chip designers. Nvidia's NVHBM, though, could set a de facto standard because of its influence over the AI accelerator market. If NVHBM is adopted broadly by custom XPU vendors through NVLink Fusion, memory makers could be forced to produce more specialized stacks to remain aligned with Nvidia's roadmap.

The announcement may also be seen as a response to AMD's and Intel's ecosystems. AMD has its own memory hierarchy for AI accelerators, while Intel is pursuing advanced packaging and memory integration with its foundry business. Nvidia's strategy is to define the memory architecture at the system level, even when the memory is not made by Nvidia. This gives the company leverage over issues such as acceleration, interfaces, and packaging that are increasingly critical to AI performance.

How NVHBM fits into AI chip roadmaps

For custom AI chip developers, NVHBM could alter the way they plan future accelerators. If Nvidia succeeds in creating a standard HBM implementation with an embedded memory controller, then XPU designers can allocate more power and area to compute logic. For example, a cloud giant like Amazon might use those extra transistors to build more matrix math cores, bigger on-chip SRAM caches, or specialized sparsity support for recommendation models.

Those benefits are potentially more important than simply increasing raw memory bandwidth. Freeing silicon area means accelerator designers can add more compute engines, cache, specialized AI functions, or other workload-specific circuitry without increasing the package footprint. The power savings come from both lower HBM power and reduced I/O driving power, since the controller and PHY are now in the memory stack rather than far away on the accelerator die.

There are also engineering hurdles. Moving the memory controller off the XPU and into the HBM base die requires advanced packaging and TSV designs, and Nvidia’s custom controller must work with the standard DRAM dies made by the memory vendors. Thermal management becomes more complicated as well, because the base die is now doing more active logic work and generating more heat in a part of the package that is traditionally cooler.

Given the claimed bandwidth and power improvements, NVHBM would be especially valuable for power-constrained data centers. AI facilities are increasingly limited by electricity and cooling capacity. A memory architecture that cuts power consumption by 15% would let hyperscalers put more accelerators into an existing rack without exceeding thermal limits. Combined with more compute area, that can yield a meaningful performance-per-watt gain even before the software stack is optimized.

Nvidia is positioning NVHBM as another component of NVLink Fusion, rather than as a separate memory product. That means a customer building a custom XPU with an Arm CPU core or a custom accelerator for inference could have that XPU plugged into Nvidia's networking fabric, while also using Nvidia's memory interface design. This is part of a broader shift by Nvidia from selling just chips to selling rack-scale systems and design blueprints. The company has made clear that it wants to provide not only GPUs but also the infrastructure glue that allows other companies’ chips to coexist in the same data center.

Amazon's early involvement is a strong signal that NVHBM may see commercial deployment before long. Annapurna Labs has become a key part of AWS’s strategy to differentiate its cloud infrastructure. If NVHBM helps Annapurna deliver more efficient custom AI chips, it could give AWS an advantage over other cloud providers that rely more heavily on off-the-shelf accelerators.

Memory manufacturers will need time to integrate Nvidia’s custom logic into their HBM production flow. HBM stacks are already among the most complex memory products manufactured today, with multiple layers of DRAM and a logic base die. Adding a custom controller to the base die increases complexity and may affect yields. Nvidia, however, has substantial leverage. It is likely the largest buyer of HBM in the world, and its designs influence the entire AI supply chain.

The first NVHBM-equipped systems could be built around partner XPUs rather than Nvidia’s own GPUs. By making its memory design broadly available through NVLink Fusion, Nvidia is inviting the very companies that might one day be competitors into its ecosystem. The promise of standard HBM implementations from multiple vendors may be the incentive those companies need.

Whether NVHBM will become an industry standard or remain closely tied to Nvidia remains to be seen. The success of the architecture will depend on execution from memory suppliers, customer demand from custom chip makers, and Nvidia's ability to deliver the promised bandwidth and silicon-area improvements in real products. Nvidia has a history of pushing its own interfaces and packaging standards into the data center market, and NVHBM is yet another example of that strategy.

Source: Network World News