MetaRoCE Puts the RDMA Decision at the NIC for AI-Scale Ethernet

TL;DR

Meta has introduced MetaRoCE, a clean-sheet RDMA transport for AI workloads on commodity Ethernet. Its central design choice is to make the NIC the system’s traffic interpreter: paths carry their own telemetry, packets can be sprayed across them and arrive out of order, and loss recovery is handled without asking the fabric to provide PFC-based losslessness. Meta says the transport supports native multipathing, per-path congestion control, receiver-driven fair-share hints, and a single connection that separates application streams from network paths. In a comparison on a 64-node AMD GPU cluster running RCCL collectives, Meta reports higher throughput and lower flow completion times than RoCEv2, graceful behavior at packet-loss rates that degrade RoCEv2, linear scaling across four- and eight-plane topologies with up to 4,000 concurrent connections, and autonomous recovery from simulated plane failures. The release matters beyond Meta’s benchmark because the specification, Linux software reference implementation, and compliance suite are being opened through the Open Compute Project. Our reading: the real proposal is not “a faster Ethernet.” It is an attempt to make endpoint software carry the complexity that large AI fabrics cannot safely centralize in switches.

The bottleneck Meta is trying to move

Meta frames the problem in familiar AI-infrastructure terms: training and serving frontier models depend on moving data between GPUs without wasting compute cycles. Collective operations such as all-reduce and all-to-all synchronize thousands of accelerators, so the slowest transfer sets the pace for the job. During inference, communication between distributed model shards affects response time for hundreds of millions of users, according to Meta’s source.

At that scale, a network hiccup changes meaning. A delayed packet is not merely one delayed packet when a collective is waiting for its slowest participant. A fabric policy that protects ordering by pausing traffic can also turn a local problem into a wider scheduling problem. Meta’s argument is that the transport should be designed around the conditions large Ethernet networks actually face: multiple paths, variable congestion, loss, and distances that grow as clusters spread across data centers and regions.

Meta has already argued that RoCE can support distributed AI training at scale. MetaRoCE is the next step in that direction, but it does not simply add another tuning layer to standard RoCE. The release describes a transport designed from the ground up for Ethernet at million-GPU scale.

The NIC becomes the system’s traffic interpreter

The most useful sentence in the announcement is also the simplest: the fabric sees packets, but the NIC sees intent. Standard architectures place considerable responsibility in switches, expecting ordered delivery and using Priority Flow Control to help maintain a lossless behavior. MetaRoCE instead decomposes a connection into fine-grained logical paths. Each path has real-time information about round-trip time, ECN state, and utilization.

The division matters because a switch sees a packet in isolation. The endpoint knows which stream generated it, which path carried earlier packets, what acknowledgments arrived, and whether a delay looks like congestion or failure. MetaRoCE uses that context to make decisions at the edge. This is not a claim that switches become irrelevant. The fabric still supplies ECN marking and ECMP. The claim is that the fabric should offer those primitives while the NIC decides how to use the available paths.

For operators, this is a more honest placement of complexity. The trade-off does not disappear; it becomes software and NIC state that can be observed, tested, and updated independently of every switch configuration.

Out-of-order delivery is a feature, not an exception

MetaRoCE sprays packets across many paths and treats out-of-order arrival as normal. Each packet carries its destination, so data can be written directly to the final memory location as it arrives. Meta says that removes the need for a reorder buffer and avoids head-of-line blocking.

The detail about message semantics is important. Writes carry their destination in each packet. Sends carry the match to a posted receive buffer, allowing a Send to land correctly even when messages ahead of it have not arrived and without a round trip to discover the destination. Collective libraries can therefore use two-sided messaging where it fits instead of reducing every operation to a Write.

Our analysis is that this is the protocol’s clearest departure from the old mental model. Ordered delivery is convenient for the fabric, but it can make one missing or delayed packet hold back unrelated data. Direct placement changes the unit of waiting from “everything behind the gap” to “the specific data that is still missing.” That is a meaningful architectural choice for collective traffic, where independent pieces of work are often moving at the same time.

Multipathing with failure in the model

Each MetaRoCE connection receives first-class paths and sprays traffic packet by packet. A distinct UDP source port supplies ECMP entropy, and the NIC can change it to move traffic away from a bad route. On multiplane fabrics, the NIC selects the plane rather than relying on the fabric to infer the application’s preference.

The transport keeps a window and round-trip estimate for every path. Meta says that lets it distinguish congestion from failure and rebalance explicitly: a hot or broken link should slow one path instead of stalling the whole connection. That is a stronger operational promise than simply saying a network is multipath. Multipath only helps if the system can identify which path is unhealthy and stop treating it as representative of the connection.

Meta’s own validation includes simulated plane failures, where traffic redistributes without application involvement or operator intervention. This remains a vendor-reported result, not an independent production guarantee. Still, it identifies the right test: not peak throughput on a clean topology, but whether recovery stays local when one part of the fabric degrades.

Loss tolerance without PFC

MetaRoCE assumes Ethernet can be lossy. It does not ask the fabric to provide pause frames or PFC-based losslessness. Each path has an ordered sequence, so a gap in its 256-bit selective acknowledgment bitvector is treated as evidence of loss rather than reordering. Meta says the missing packet can then be retransmitted precisely, on the path that lost it, as soon as the gap appears.

This design is attractive because PFC is not free operationally. Removing it from the transport’s requirements does not remove the need to understand queueing, congestion, or buffer behavior, but it changes who reacts. The NIC must interpret acknowledgment state correctly, maintain per-path recovery, and avoid turning a burst of loss into an uncontrolled retransmission storm.

Implementation quality becomes decisive here. A specification can define the behavior; production systems still need instrumentation, interoperability tests, and failure drills. Meta’s decision to publish a compliance suite is therefore more than an open-source footnote. It is part of the argument that endpoint complexity can be made repeatable across vendors.

Congestion control from both directions

MetaRoCE combines sender-driven ECN-based additive-increase/multiplicative-decrease with receiver-driven fair-share rate hints. Windows exist per path as well as per connection. A congestion mark trims the affected path and steers subsequent packets toward clearer paths. In acknowledgments, the receiver returns the share of inbound bandwidth allocated to the sender, so the sender can approach an appropriate rate directly rather than searching for it.

Meta says incast resolves in one or two round trips, with better fairness and lower tail latency. That is a source claim, and the exact result will depend on workload and topology. The design direction is nevertheless coherent: sender feedback reacts to what the network marks, while receiver feedback represents the capacity being shared at the destination.

For an infrastructure team, this is a reminder not to evaluate congestion control only with average throughput. Tail latency, fairness among senders, and behavior during incast are the practical tests. In AI clusters, a transport that is slightly less impressive at the median but avoids long stalls may deliver more usable work overall.

Open tooling is the adoption test

Meta says it is contributing the full protocol specification to OCP, alongside a reference implementation and compliance test suite. Meta and AMD have already implemented the protocol on Pensando programmable NICs, while additional vendor implementations are underway. The software reference, libsoftmetaroce, runs on commodity Linux over standard UDP sockets without specialized hardware and is intended to serve as a behavioral model for silicon development.

The release also preserves a familiar application boundary. Existing RDMA Verbs APIs and software stacks are expected to work mostly unchanged; enhanced capabilities such as multiplane support use extension APIs. Meta plans to present the specification, a DPDK-optimized implementation, and its production compliance framework at the October 2026 OCP Global Summit.

Netics’ judgment is conditional. Open publication lowers the barrier to experimentation, but it does not automatically produce a multi-vendor ecosystem. Buyers and operators should ask whether NIC implementations agree under loss, whether telemetry is exposed at useful resolution, whether collective libraries need extensions, and whether troubleshooting tools can explain a single path’s behavior. The compliance suite can make those questions testable; it cannot answer them in advance.

The road ahead is also broader than scale-out. Meta lists scale-up latency inside a rack, scale-across jobs spanning buildings thousands of kilometers apart, and storage or key-value-cache incast as future work. Those are different regimes, not one simple roadmap checkbox. A transport that handles one well may need different tuning and fairness mechanisms in another.

For practical infrastructure and AI-systems analysis, visit the Netics homepage.

Sources

• MetaRoCE: A New RDMA Transport Built for AI-Scale Ethernet — Engineering at Meta, August 24, 2026. Primary source for all protocol, benchmark, implementation, roadmap, and OCP claims in this draft.

Source: Meta Engineering primary-source release, published August 24, 2026.