Xscaping Limits: The Hard Ceiling Facing AI Brains For Human Intelligence
By Vivek Raghunathan, CEO and Co-Founder of Xscape Photonics
There is an unquestionable quest for understanding human intelligence. And AI data centers are the
current physical manifestation of the human brain. Naturally, human brain performance sets the true
limit for inference hardware in AI data centers. Let’s try a thought experiment to uncover the hardware
limitations of machines and understand if the efficiency of human intelligence can ever be mimicked in a
robot…
Take a human Grandmaster’s brain. It uses about 20W to play a complex strategy game. Today’s state-of-the-art AI inference systems (or AI brains), performing a similar task, consume about 1 MW of total
power. This is a 50,000X reduction in power consumption or efficiency improvement at the hardware level - assuming the reasoning algorithms are in place.
The AI industry has prioritized the understanding of what AI brains can unlock over their efficiency. As a result, building AI applications on existing hardware and data center ecosystems to go-to-market faster and increase revenue becomes the natural path of adoption. Once a market is validated, aggressive business growth is a natural trajectory that fuels the growth of the underlying hardware platform. As a result, building more data centers for more tokens became prevalent. Growth of an inefficient foundational platform is bound to hit scaling and physical limits on every front.
We have started to see those limits manifest even before the applications have started scaling. Within a year of ChatGPT’s launch, there was a widespread concern about the trajectory of data center power consumption. While there are hardware innovations that are being actively deployed to make the AI brain more efficient and reduce the slope of progress towards hitting a power limit, there is a consensus that power remains the true limit of AI scaling. I couldn’t agree more given the 50,000X discrepancy between the human brain and an AI brain. Moving forward, efficiency should be the main metric of performance tracking in the quest for human intelligence.
Human brains consume 75-80% of power in processing and communication. The remaining 20-25% is reserved for maintenance and housekeeping. A study published in PNAS found that communication consumes 35 times more energy than computation in the human cortex. Moving signals across the
brains vast network of wires (axons) and gates (synapses) is the primary driver of your brains power bill, far outweighing the cost of the internal logic within a single neuron. Similarly, today’s AI brains spend around 85% on processing and communication. The remaining 15% is spent on maintenance
(cooling and distribution loss). However, the key contrast to human brains is that computation for AI brains consumes 4-5X more power than communication. There is a strong case to be made that future inference brains for reasoning and planning will require more investment in communication fabric over computation if we strive to mimic human behavior.
An ideal of system of interconnected processors can behave as one large processing unit if the speed of information processed is equal or less than the speed of information communicated across the processors. However, current AI brains have been designed to maximize utilization of the most
expensive asset - compute processing units. Investments in technology innovation (Moore’s law) have been focused on driving down unit economics of expensive assets (i.e., the processor). Communication technologies have been treated as an overhead that needs to be minimized to maximize processing assets. The current communication fabrics of AI brains is extremely expensive and slow if it needs to scale to connect multiple processing units. The result of imbalanced investment and processor-driven technology development choices is an AI brain where a lot of extremely fast processors are waiting on extremely slow inefficient communication medium and this imbalance is speed is as high as 1,000X. The productivity of today’s AI brain system is determined by the speed of communication within processors. Adding more inefficient AI systems is the only option to meet the productivity needs of the end user in the near term. This results in an incrementalism mindset that pushes the limits of all the underlying technologies of an inefficient system and puts us in an unsustainable, unviable growth trajectory.
For instance, more processors require more power and more cooling and more power-hungry communication fabrics. More investment in incremental innovations increases switching costs to more efficient foundational system implementations that are critical to realize human intelligence. An incremental ROI mindset can result in an irreversible path to a different destination.
50,000X efficiency discrepancy can be solved with 2X improvements over 16 yrs and trillions of investments. Or it can be solved now by rewiring the AI brain with the right foundation.
There is a need to Xscape the incremental scaling limits. This starts with reimagining the fabric.
Contributors: Karthik Vaithianathan, Sander Arts, and Wireside