Maia 300: Microsoft Bets the AI Race on Tokens per Megawatt

Microsoft used to buy its intelligence by the GPU. On October 12 it said it intends to build it instead. The company unveiled Maia 300, the third generation of its in-house accelerator, and described it in unusually blunt terms: "the primary AI inference system for Microsoft products and services," scaled to "multi-gigawatt deployments starting in 2027," according to a post by cloud chief Scott Guthrie on the official Microsoft blog.

The headline number is not FLOPS. It is 40% more tokens per megawatt than leading GPU-based systems, and up to seven times the rack-scale token throughput of the previous generation. Microsoft is measuring itself in the unit that actually shows up on a data center's utility bill, and that is the clearest signal yet of where AI infrastructure competition has moved.

What Microsoft Announced

Maia 300 is an inference-first design aimed at the workloads that are growing fastest: long-context reasoning, latency-sensitive conversation, and what Guthrie calls "sustained, multi-step agentic tasks." The hardware reflects that: eight HBM4 stacks delivering 18 TB/s of peak memory bandwidth inside a 950-watt envelope, a multi-chiplet architecture that separates compute, memory-controller and I/O functions, compute chiplets built on TSMC's N2P node, and support for enhanced MXFP4 precision that shrinks the memory footprint of both computation and KV-cache.

At rack scale, Microsoft packs 96 accelerators into 167 kilowatts, with 27.6 TB of HBM capacity and 1.7 PB/s of aggregate bandwidth. That is more chips in less power than Nvidia's current GB300 racks, which hold 72 GPUs at roughly 142 kilowatts. Microsoft also claims its dataflow architecture treats networking as an extension of the chip, using Ethernet-based transport to scale from one rack to datacenter scale.

The generational jump is real on paper, but the caveats matter. Maia 300 does not enter production until 2027, and Microsoft declined to give a firm timeline for gigawatt-scale rollout, telling CNBC it wants "optionality" as models change. The 40% figure is a projection against GPU systems Microsoft has not identified. Evercore ISI analysts estimate Maia will account for only about 10% of Microsoft's AI chip spending by early 2027.

The current generation is further along. Maia 200 is running production workloads for Microsoft 365 Copilot, with GitHub Copilot and Microsoft Foundry to follow by year end, serving OpenAI models, Microsoft's own MAI models and open-source families. Microsoft says Maia 200 delivers up to 1.7x more tokens per watt than leading GPU infrastructure for built-in AI in Teams, Copilot and Cowork.

One asymmetry stands out. Unlike Google's TPUs or Amazon's Trainium, Maia cannot be rented by outside companies. "Nobody has tested them outside of Microsoft yet," Moor Insights and Strategy chief executive Patrick Moorhead told CNBC. That makes the efficiency claim self-reported, and it makes Microsoft the sole beneficiary of its own silicon advantage.

Why the Rack Became the Unit of Competition

Microsoft was not alone in rethinking the box. On the same day, AMD introduced the Helion agentic AI rack, an Open Compute Project design that can be configured as a CPU-only system carrying up to 32,768 cores for hyperscale agent execution, or as an all-in-one rack mixing EPYC CPUs with Instinct MI350P GPU nodes. Nvidia and Cisco announced a new pluggable optics form factor designed to push AI factory fabrics to 400 Gb/s per lane and 12.8 Tb/s per module. Three vendors, three announcements, one message: the sellable unit is no longer a chip, it is a powered, cooled, networked rack measured in tokens.

Power is why. UBS argued this month that electricity, transmission and data center capacity, not silicon, now set the pace of AI growth, with 57 GW of capacity needed in 2027 against roughly 38 GW of sites with confirmed power. The Intel and Broadcom chief executives warned the same week that a capital crunch could become the industry's next bottleneck.

Agentic workloads sharpen the pressure. A chatbot request is a burst; an agent run is a long-lived session that rereads context, calls tools, retries steps and keeps KV-cache alive. Efficiency per megawatt therefore compounds across every agent step, which is exactly the metric Microsoft chose to sell and the reason AMD built a rack with an integrated token router that spreads model calls across frontier and open-source models.

What This Means for Builders

If Microsoft's numbers hold, the marginal cost of an agent step keeps falling on Azure and on any provider that follows the same curve. But the benefit lands first with whoever owns the rack. Hyperscalers are vertically integrating silicon, networking and software, and the external price of inference will be set by their internal economics, not by neutral hardware markets.

Three practical consequences follow. First, treat tokens per megawatt as the leading indicator to watch, not benchmark scores: it predicts pricing, rate limits and capacity long before a price list changes. Second, nothing here removes the case for model and provider heterogeneity. Maia is closed to third-party rental, and the reported talks about Anthropic renting Azure capacity on Microsoft silicon show that even labs with their own chip programs keep several suppliers in the mix. Keep provider identifiers in your logs and fallback paths tested.

Third, plan for capacity, not intelligence, as the binding constraint. When power and grid interconnection decide how much inference exists, the scarce resource for an agent fleet is allocation, not model quality. Microsoft will publish more detail at Ignite next month, and the comparable Nvidia Rubin and AMD Helios racks will keep the tokens-per-megawatt scoreboard crowded. The company that wins it will not be the one with the fastest chip, but the one that serves the most agent work from each watt it can buy.