The center of gravity in AI silicon is moving from training to inference, and the chip industry's design priorities are moving with it. The evidence in this cycle points one way: memory bandwidth, power delivery and system-level integration now determine whether AI hardware is useful, while raw compute throughput has become a necessary but insufficient measure. That shift rewards different designs, different vendors and different deployment sites than the training era did.
The workload split has flipped
SiliconANGLE reports that inference now drives two-thirds of AI workloads, a figure that reframes how the industry thinks about where chips go and what they must do. Training clusters were built around massive, centralized facilities and the interconnect needed to keep thousands of accelerators in conversation. Inference is a different problem: it is distributed, latency-sensitive and closer to the user, and it is paid for token by token rather than in multi-month training runs.
The consequence, as SiliconANGLE describes it, is a shift in AI spending from giant training clusters toward smaller, quickly built sites. That is not a marginal change in procurement patterns. It changes what a datacenter is for, what it must contain and how quickly it must be stood up. Modular data centers are emerging as a natural home for these workloads, according to SiliconANGLE's reporting, because they can be built and provisioned on a shorter cycle than the large campuses of the training era.
For US chipmakers, that means the addressable design win is no longer just the hyperscaler's flagship cluster. It is also the smaller, faster-built site, and the constraints there are different.
Power, cooling and memory become the design brief
Once inference dominates, the questions that shape a chip change. SiliconANGLE notes that power, cooling and memory supply now shape where tokens get generated as much as raw chip performance does. That is a striking claim about the semiconductor industry, because it demotes the metric that has dominated AI chip marketing for years.
The logic is straightforward when stated plainly. A training run can be scheduled around available power; an inference service must respond when a user asks. If the memory subsystem cannot feed the compute logic quickly enough, the chip sits idle regardless of how many operations it can theoretically perform. Cooling becomes a first-order constraint because inference capacity is deployed where it is needed, not only where water and substations are abundant. Power delivery becomes a design problem at the rack and site level, not only at the package level.
This is a systems argument, and it favors vendors who can supply more than a die. SiliconANGLE quotes Satyam Srivastava saying that AI infrastructure has become a systems game, and that winners combine memory, interconnect and software into cohesive designs. A chip that is excellent in isolation but awkward to integrate is a weaker product than one that arrives with its memory and software story already resolved.
Memory bandwidth is the new battleground
The clearest illustration of the shift is the memory question. SiliconANGLE reports that D-Matrix is answering inference's memory bandwidth problem with Raptor, a second-generation chip that stacks 3D DRAM beneath its compute logic. As SiliconANGLE puts it, memory bandwidth decides whether hardware generates tokens quickly or sits idle.
That is a design bet with real consequences. Stacking DRAM beneath compute logic is an attempt to shorten the distance between memory and processing, addressing the bottleneck that inference-heavy workloads expose most sharply. It is also a manufacturing and packaging commitment, not a software workaround, which tells you where the company believes the constraint sits.



