Microsoft's Maia 300 Publishes Rack Numbers for Its Own Inference Silicon

TL;DR

• Microsoft Azure Maia 300 is an AI inference accelerator, the third generation of Microsoft's own chip family for running large models inside its datacenters, announced on 12 October 2026.
• The chip is a multi-chiplet package with compute chiplets on TSMC's N2P process, memory-controller and I/O functions on separate dies, eight HBM4 stacks providing 288 GB and 18.0 TB/s of peak bandwidth inside a 950 W TDP, and support for enhanced MXFP4 and sub-3-bit storage formats.
• The rack holds 96 accelerators inside a 167-kilowatt envelope with 27.6 TB of HBM in the announcement (27.7 TB in the deep dive's rack table) and 1.7 PB/s of aggregate memory bandwidth, which doubles the accelerator density of Maia 200 per Microsoft's published rack table.
• A two-tier Ethernet scale-up fabric joins up to 128 racks into a superpod of 12,288 accelerators, using electrical links inside the rack, optical links between racks, a bandwidth split that job packing can move and hot-spare capacity that schedulers take from another rack.
• Microsoft expects 40% more tokens per megawatt than leading GPU-based systems and up to 7x the rack-scale token throughput of Maia 200, so the unnamed baseline sits on the cross-vendor claim while the Maia 200 ratios stay checkable.
• Capacity planning changes shape around these figures: more tokens per watt, twice the accelerators per rack, a liquid-cooling loop that carries more than 80% of the rack's thermal load, and a power envelope that decides where capacity can land.
• Maia 200 already serves Microsoft production workloads including Microsoft Copilot and is deploying across the United States, Europe and Australia, with GitHub Copilot and Microsoft Foundry following by the end of 2026.

Microsoft built Maia 300 as a rack product

The announcement of the third Azure Maia generation arrived on 12 October 2026 with a system-level frame. Scott Guthrie, executive vice president for Cloud + AI, described Maia 300 as "a complete intelligent system designed for the new types of AI workloads emerging today: Long-context reasoning, latency-sensitive conversational experiences and sustained, multi-step agentic tasks", and set the deployment plan alongside it. Microsoft will scale the Azure Maia family to multi-gigawatt deployments starting in 2027 and make it the primary inference system for Microsoft products and services, with Maia 300 production deployments beginning that year.

That ordering carries information. A chip announcement that leads with a power envelope and a rack count is a statement about datacenter capacity.

The generational story matters more than the launch itself. Maia 200 is already serving production workloads including Microsoft Copilot, and Microsoft says GitHub Copilot and Microsoft Foundry follow by the end of 2026, with OpenAI models, Microsoft's own MAI models and leading open-source model families all running on that silicon across the United States, Europe and Australia.

The multi-chiplet package and its HBM4 memory

Inside the package, Microsoft separates functions across dies. Compute sits on chiplets built on TSMC's N2P process, while memory-controller and I/O functions occupy dies chosen for bandwidth, connectivity and cost, an arrangement Microsoft's Azure hardware team describes as allowing each function to use the technology best suited to its role.

The memory side is where the arithmetic for inference gets decided. Eight HBM4 stacks give the accelerator 288 GB of capacity and 18 TB/s of bandwidth, which Azure's architecture post reports as 2.5 times the memory bandwidth, 33% more capacity and roughly twice the HBM bandwidth per watt of Maia 200. Alongside the memory change, Microsoft moved the package from a uniform memory organisation to a locality-aware one across chiplets, so data placement follows the compute that consumes it.

SiliconANGLE's reading of the same package is worth keeping in the margin: the 288 GB matches the capacity of NVIDIA's Rubin accelerator, while memory bandwidth lands about 20% lower. Two vendors converging on the same capacity and diverging on bandwidth per accelerator describes the design space a buyer is actually choosing inside.

Ninety-six accelerators inside a 167-kilowatt envelope

The rack is the unit Microsoft chose to publish. Maia 300 integrates 96 accelerators across 24 four-accelerator trays inside a 167-kilowatt envelope, with a host processor, memory, storage and cooling in each tray. Folding the host into the accelerator tray is the detail that doubles accelerator density relative to Maia 200, because the separate host trays disappear from the floor plan.

Read the HBM bandwidth and rack power rows of that table together. Rack HBM bandwidth grows by a factor of 5.1 while rack power grows by 2.6, which the table expresses as 1.95 times the HBM bandwidth per watt, close to the ratio Microsoft states per accelerator in its deep dive: approximately 2 times the HBM bandwidth per watt of Maia 200. A team that reasons in tokens per watt finds that number more useful than any peak figure, because the power envelope is what a datacenter bills and what a facility has to cool.

Cooling is the second half of that envelope. Liquid cooling carries more than 80% of the rack's thermal load from accelerators, voltage regulators and switch ASICs, and the design connects either to facility water or through a heat-exchanger unit for air-cooled halls. Those two options decide which buildings can host the capacity at all.

A superpod of 12,288 accelerators over Ethernet

Scale-up networking is where Maia 300 departs furthest from a conventional GPU cluster. Microsoft's AI network controller runs custom AI transport over Ethernet, and the topology is two-tier: a half-switched arrangement inside the rack where each accelerator divides its connectivity between direct links to a partner in its tray and switched paths to the other 94, all of it electrical, and a second optical tier of switches that connects up to 128 racks into a superpod.

The engineering choice worth naming is that the bandwidth split between in-rack and cross-rack traffic stays open after the system is designed. It becomes a parameter set by job packing, and an accelerator can drive up to 750 GB/s to a peer elsewhere in the superpod while only a subset of accelerators travel cross-rack. Tensor parallelism can stay inside one rack while expert and pipeline parallelism span racks, and schedulers can absorb a failed tray by reaching into hot-spare capacity in a neighbouring rack.

Our reading of that arrangement is straightforward: this is a scheduling capability expressed as a network design. Capacity that used to sit stranded inside one rack boundary becomes usable by a job packed elsewhere, which raises the effective utilization of the same silicon.

The 40% tokens-per-megawatt claim and its unnamed baseline

Microsoft's efficiency headline reads as follows: Maia 300 should deliver 40% more tokens per megawatt compared with leading GPU-based systems, and up to 7x the rack-scale token throughput of Maia 200 in a 96-accelerator configuration. Maia 200, meanwhile, is credited with up to 1.7x more tokens per watt than leading GPU infrastructure for built-in AI in Teams, Copilot and Cowork.

The two halves of that sentence carry different weights. The 7x rack-scale figure and the 2.5x performance-per-watt figure are comparisons inside Microsoft's own product line. The 40% figure crosses a vendor boundary, and Microsoft's post names no counterpart system, no model, no sequence length and no batch configuration for it. The Verge reported the same figure on the same day with the same absence.

We treat cross-vendor numbers as the starting point for a question. A useful question asks which baseline the tokens-per-megawatt comparison used, on which model, at which context length, and whether the serving stack was tuned for both systems. The answer decides whether the number transfers to your workload. This is the same discipline we apply to our own sovereign infrastructure and cloud exit work for European teams, where a published efficiency ratio has to survive contact with a real power contract and a real rack count.

Capacity planning around a rack power envelope

For anyone buying inference, the announcement is useful as a planning template even when the silicon never reaches your estate. Capacity now arrives denominated in racks and megawatts, and the arithmetic runs in this order: tokens per second required, divided by tokens per watt at the context length your workload uses, giving a power figure, which then meets a cooling limit and a facility limit.

Two of Microsoft's own numbers change how that arithmetic behaves. Fewer racks carry the same load when each rack holds twice the accelerators, and the bandwidth-per-watt improvement moves more of the bottleneck onto cooling and power delivery. The rack as a power contract is the frame we use for that shift, and hyperscale rack designs keep confirming it.

Liquid cooling deserves a line of its own in any plan. A rack that removes more than 80% of its thermal load through liquid needs either facility water or a heat exchanger.

Maia 200 already serves production workloads

The installed generation tells a reader what Microsoft's inference roadmap looks like in practice. Maia 200 runs Copilot workloads today, with GitHub Copilot and Microsoft Foundry following by the end of 2026, and the same silicon hosts models from OpenAI, from Microsoft's MAI team and from open-source families.

Running one accelerator family across three model provenances is the less visible achievement in this announcement. For an operations team, a fleet that serves several model lineages on one serving path reduces the number of hardware-specific tuning tracks it maintains, which is the same portability question we answer in self-hosted platform work, where the goal is keeping the model layer swappable while the serving layer stays stable.

Microsoft says more detail follows at Ignite 2026 in November. Until then, the published rack table and the two architecture figures are the parts of this announcement that a technical reader can check.

Sources

Source: Microsoft to expand Azure Maia system to multi-gigawatt deployments — The Official Microsoft Blog, blogs.microsoft.com, 12 October 2026, retrieved 2026-10-12 (Scott Guthrie, executive vice president Cloud + AI; the Maia family scaling to multi-gigawatt deployments starting in 2027 and becoming the primary inference system for Microsoft products and services; Maia 200 serving Microsoft production workloads including Microsoft Copilot, with GitHub Copilot and Microsoft Foundry to follow by end of year, running OpenAI models, Microsoft MAI models and leading open-source model families, deploying across the United States, Europe and Australia at up to 1.7x more tokens per watt for built-in AI in Teams, Copilot and Cowork compared with leading GPU infrastructure; the description of Maia 300 as "a complete intelligent system designed for the new types of AI workloads emerging today: Long-context reasoning, latency-sensitive conversational experiences and sustained, multi-step agentic tasks"; the expectation of "40% more tokens per megawatt compared to leading GPU-based systems" and up to 7x the rack-scale token throughput of Maia 200 in a full 96-accelerator configuration; a software-defined dataflow architecture with AI-assisted kernel optimisation; eight HBM4 stacks providing 18.0 TB/s of peak memory bandwidth within a 950-watt TDP; a multi-chiplet architecture separating compute, memory-controller and I/O functions with compute chiplets on TSMC's N2P process node; enhanced MXFP4 among the low-precision formats; an Ethernet-based custom AI transport; a rack of 96 accelerators within a 167-kilowatt power envelope with 27.6 TB of HBM capacity and 1.7 PB/s of aggregate HBM bandwidth; production deployments beginning in 2027; further detail promised at Microsoft Ignite 2026).

Source: Deep dive into Azure Maia 300 architecture — Azure Blog, azure.microsoft.com, 12 October 2026, retrieved 2026-10-12 (Rani Borkar and Saurabh Dighe; the move from uniform memory access to a locality-aware non-uniform organisation across chiplets; enhanced MXFP4 and sub-3-bit storage formats reducing footprint and data movement; the rack doubling to 96 accelerators and the scale-up domain growing to a superpod of 12,288 accelerators; the multi-chiplet architecture increasing compute FLOPs by 36%, adding 2.5 times HBM bandwidth and providing 14% more scale-up bandwidth; eight HBM4 stacks with 288 GB of capacity and 18 TB/s per accelerator, 2.5 times the memory bandwidth, 33% more capacity and approximately 2 times the HBM bandwidth per watt compared with Maia 200; 96 accelerators across 24 four-accelerator trays within a 167-kW rack, doubling accelerator density by eliminating separate host trays, with six high-radix switch ASICs in three liquid-cooled switch trays; a half-switched in-rack topology of direct links and switched paths using electrical links, with an optical second tier connecting up to 128 racks; an accelerator driving up to 750 GB/s to any peer with the in-rack and cross-rack bandwidth split set by job packing; tensor parallelism staying within a rack while expert and pipeline parallelism span racks; schedulers using hot-spare capacity in another rack; liquid cooling removing more than 80% of the rack's thermal load, with direct facility-water connection or a heat-exchanger unit for air-cooled datacentres; a published table listing pod size 48 to 96, HBM bandwidth 341 to 1,728 TB/s, HBM bandwidth per watt 5.3 to 10.6 GB/s/W, HBM capacity 10.4 to 27.7 TB, FP4 performance 485 to 1,324 peta-operations per second and rack power 64 kW to 167 kW, with "up to 7 times greater rack-scale token throughput" and "more than 2.5 times the performance per watt" relative to Maia 200).

Source: Microsoft's new AI chip makes it less reliant on Nvidia's GPUs — The Verge, theverge.com, 12 October 2026, retrieved 2026-10-12 (third-party reporting of the same announcement: the description of Maia 300 as Microsoft's primary AI inference system, the quoted 40% tokens-per-megawatt expectation and the 7x rack-scale figure against Maia 200, and the note that Maia 200 was announced in January 2026 and already runs production workloads).

Source: Microsoft debuts multichiplet Maia 300 inference accelerator — SiliconANGLE, siliconangle.com, 12 October 2026, retrieved 2026-10-12 (third-party analysis used here for one comparison: four computing chiplets on a two-nanometre node flanked by eight HBM4 modules with 288 GB of combined capacity, the same capacity as NVIDIA's Rubin accelerator with about 20% less memory bandwidth; trays of four accelerators in liquid-cooled racks of 96; superpods of 12,288 accelerators linked by a custom Ethernet-based network stack that can reallocate bandwidth between in-rack and inter-rack traffic).

Source: Microsoft Azure Maia 300 — Official Microsoft Blog announcement and Azure architecture deep dive, published 12 October 2026, retrieved 2026-10-12, with third-party figures from The Verge and SiliconANGLE the same day. Figures attributed to Microsoft are the company's own published claims; the images are two official Microsoft figures and one official hero photograph, plus one Netics editorial diagram.