Ultra Ethernet for Scalable AI Network Deployment - Cisco Blogs

Skyler: Hey, welcome back to Line Rate — the show tracking the hardware, standards, and companies wiring together the AI datacenter. I'm Skyler Monroe, and if it moves bits between accelerators, racks, or datacenters, we're probably obsessing over it here. Big thanks to our sponsors: Ada, who helps companies actually integrate A I into their real-world workflows, and Zen Semiconductor, an umbrella company with a portfolio of businesses across the silicon industry, including Ada on the A I deployment side. Alright — today we've got a loaded episode: Ultra Ethernet is showing up everywhere, Meta is pulling back the curtain on a gigawatt-scale cluster architecture called Prometheus, a northern Italian municipality just told a datacenter to take a hike, and Huawei is making some very loud noises about open-sourcing an interconnect that wants to replace, well, everything. Let's get into it. Skyler: So first up — Ultra Ethernet. We've got two pieces this week, one from Cisco and one from Nokia, and I want to take them together because honestly they're telling a consistent story, and it's a story worth paying attention to. Skyler: Quick context for anyone who's newer to this space: the Ultra Ethernet Consortium — UEC — launched in 2023 as an industry push to make Ethernet a first-class citizen for A I and HPC workloads. The idea is that standard Ethernet, as great as it is for the web and enterprise, has some real friction points when you're running distributed training jobs across thousands of GPUs. Things like how it handles congestion, packet loss, and retransmission just don't play nicely with the tight synchronization requirements of collective operations — your AllReduce, your AllGather, that kind of thing. Skyler: What UEC is building is a stack of enhancements — transport layer changes, new reliability semantics, multipathing, and receiver-side optimizations — that sit on top of Ethernet's physical layer. So you keep the economics and ecosystem of Ethernet, but you get behavior that's much closer to what InfiniBand has been delivering for years in HPC clusters. Skyler: Now, the Cisco piece is essentially a deployment-focused deep dive. Cisco is making the case that U E C is ready — or getting ready — for production-scale A I networking, and they're walking through how the spec maps to actual cluster topologies. They're emphasizing things like their transport protocol work, how you handle out-of-order delivery without melting your latency, and how you scale congestion control when you've got tens of thousands of endpoints all doing synchronized communication. Cisco is obviously a major networking incumbent with a massive install base, so them putting serious marketing weight behind Ultra Ethernet signals that they see it as the path to staying relevant in the A I fabric wave — a wave that, frankly, InfiniBand and proprietary fabrics have been eating their lunch on. Skyler: Nokia's piece is more forward-looking, talking about the roadmap and where the consortium is headed. Nokia is leaning into the idea that A I networking demands are creating a genuine inflection point — that the scale of next-generation clusters just can't be served by bolting on fixes to existing Ethernet or relying on vertically integrated proprietary solutions. They're also touching on the multi-vendor interoperability angle, which is genuinely one of UEC's strongest value propositions. If you're a hyperscaler or a large enterprise, you don't want to be locked into one vendor's fabric stack. Open standards give you leverage. Skyler: Here's my honest take though: U E C is still a spec in motion. The transport layer work is real and technically interesting, but we're not yet at the point where you can walk into a vendor and order a fully U E C compliant switch with a fully U E C compliant NIC and expect everything to just work end to end. The interop testing infrastructure, the full stack integration — it's still maturing. That said, the direction is right, and having Cisco and Nokia both out here publicly championing the technology tells you that major players are betting on it. Watch this space — 2025 and 2026 are going to be telling years for whether U E C can actually displace InfiniBand in new A I cluster builds. Skyler: Alright, next story, and this one is genuinely fascinating if you care about what it actually takes to build at the frontier. Meta published a detailed engineering post about something they're calling Prometheus — their backend aggregation architecture for gigawatt-scale A I clusters. And I want to spend some real time on this because it's one of the more technically substantive public disclosures we've seen from a hyperscaler on cluster networking at this scale. Skyler: So first — gigawatt scale. That is not a typo. Meta is talking about clusters that consume on the order of a gigawatt of power. For reference, that's roughly the output of a small nuclear power plant, or a city of maybe 700,000 to a million homes. The compute density implied by that number is staggering, and it creates networking challenges that just don't exist at smaller scales. Skyler: What Prometheus is addressing is the backend network — meaning the fabric that connects GPUs to each other for training, as opposed to the frontend network that handles storage and general data movement. At gigawatt scale, the backend fabric is the thing that determines whether your cluster actually trains efficiently or whether you're leaving enormous amounts of compute on the table waiting for data. Skyler: Meta's approach here is centered on what they're calling backend aggregation — essentially, how you aggregate traffic from thousands of GPU nodes in a way that minimizes congestion, maximizes bandwidth utilization, and keeps the collective communication operations that training workloads depend on moving at line rate. They're talking about topologies, about how they think about bisection bandwidth, about how they handle failures in a cluster where some number of nodes going down at any given time is just a statistical certainty, not an edge case. Skyler: One of the things I find most interesting is the scale at which failure tolerance has to be designed in. When you're running a cluster this large, you're not thinking about whether a node fails — you're thinking about how many nodes are failing right now and how your training job keeps going anyway. That changes your topology design, your routing logic, your redundancy strategy, fundamentally. It's a different engineering problem than a thousand-node cluster, and Meta is being pretty candid about how they've had to rethink assumptions at each order-of-magnitude scale jump. Skyler: The aggregation piece specifically is about how you build the switching fabric to handle the traffic patterns of collective operations without creating hot spots. A I training generates extremely bursty, highly synchronized traffic — everyone needs to do an AllReduce at roughly the same time — and if your fabric isn't designed for that, you get congestion cascades that destroy efficiency. Meta's architecture is trying to solve this with smart topology choices and aggregation points that can absorb and smooth that traffic pattern. Skyler: For the industry, this disclosure is valuable on a few levels. It gives network equipment vendors a concrete sense of what hyperscalers are actually demanding. It gives fabric architects at other companies a reference point. And it validates that the networking layer — not just the compute layer — is a genuine differentiator at frontier scale. The companies and standards bodies working on A I fabric need to be solving for this class of problem, not yesterday's HPC cluster. Skyler: Quick story now, but one worth noting because it illustrates a constraint that's going to shape datacenter geography for the next decade. A municipality in northern Italy just rejected an 80 megawatt datacenter proposal, citing conflicts with local environmental policy. The specific details are a bit thin, but the headline is clear: communities are starting to push back on large datacenter builds, and it's not just about power consumption — it's about water, land use, local grid impact, and frankly, political will. Skyler: Eighty megawatts is not a small facility, but it's also not the gigawatt-scale stuff we were just talking about. The fact that an 80 megawatt project is getting rejected on environmental grounds tells you something about how the regulatory and community landscape is shifting in parts of Europe. We've already seen similar dynamics in Ireland, the Netherlands, and Singapore — places that became major datacenter hubs and then started hitting ceilings because local infrastructure and political appetite both ran out roughly simultaneously. Skyler: For the A I infrastructure buildout, this is a real constraint. The power requirements for frontier training clusters are growing faster than the regulatory frameworks and physical infrastructure to support them in many traditional datacenter markets. That's pushing development toward places with more permissive environments, cheaper land, and — critically — access to power sources that can be framed as clean or at least lower-emission. Think Texas, parts of the Middle East, certain locations in Southeast Asia. Skyler: It also drives interest in things like small modular reactors and dedicated power generation, because if you're going to build a gigawatt-scale facility, you increasingly need to think about bringing your own power rather than relying on the grid. That's a wild sentence to say out loud but here we are. The physical infrastructure story for A I is as much about power and real estate as it is about chips and fiber, and this Italian rejection is a small but emblematic data point in that larger story. Skyler: Okay, last story, and honestly this one dropped my jaw a little when I read the details. Huawei is planning to open-source something called UB-Mesh — their datacenter-scale interconnect architecture — and the framing is extraordinarily ambitious. The tagline from the reporting is essentially: one interconnect to rule them all, designed to replace everything from PCIe to TCP/IP. Skyler: Let me sit with that for a second. PCIe — the bus that connects your CPU to your GPU, your NIC, your storage — and TCP/IP — the protocol stack that runs essentially the entire internet and most datacenter networking — are two very different animals operating at very different layers of the stack. Claiming you're going to replace both with a single unified interconnect is either visionary or marketing, and the truth is probably somewhere in between. Skyler: Here's what's technically interesting about UB-Mesh as described. The architecture is apparently designed to provide a unified addressing and communication model that spans from chip-to-chip connectivity within a server all the way up to datacenter-scale communication between racks and clusters. The pitch is that by having a single coherent protocol across all those scales, you eliminate the translation overhead and protocol boundary friction that you currently pay every time data moves from one domain to another — from PCIe domain to Ethernet domain, for example, or from your accelerator's memory space to the network. Skyler: That's actually a legitimate technical problem. Every time data crosses a protocol boundary, you're paying some cost — whether that's latency, CPU cycles, memory copies, or just complexity in the software stack. If you could genuinely unify those layers with a clean abstraction, there would be real performance benefits. The question is whether that's achievable in practice without creating new bottlenecks or losing the specialization that makes each of those protocols good at its specific job. Skyler: The open-source angle is strategically interesting. Huawei has been operating under significant U S export restrictions, which limits their ability to access leading-edge chips and participate in Western-led standards bodies. Open-sourcing an interconnect spec is a way to drive adoption and influence the industry without needing to go through those channels. If developers and vendors — particularly in China and in markets less affected by export controls — start building around UB-Mesh, Huawei gains ecosystem leverage that's harder for policy to constrain than hardware sales. Skyler: Now, the competitive context here is important. This lands right in the middle of a fight that already includes InfiniBand, U E C, UALink — which is the accelerator-side fabric spec from the UALink consortium — and various proprietary hyperscaler fabrics. Everyone is trying to define what the interconnect stack looks like for the next generation of A I infrastructure. Huawei entering that fight with an open-source play, especially one with this kind of scope, is a move worth watching. Skyler: The skeptic in me notes that "we're going to replace PCIe and TCP/IP" has been said before in various forms and hasn't happened yet, because both of those are embedded extraordinarily deeply in the ecosystem. But the broader push toward unified, A I-optimized interconnect — whether it's U E C, UALink, UB-Mesh, or something we haven't seen yet — is clearly where the industry is moving. Huawei just made a loud, public bet on their version of that future. We'll see if the ecosystem shows up. Skyler: Alright, that's the show for today. Ultra Ethernet gaining enterprise champions, Meta giving us a rare look inside gigawatt-scale cluster networking, a European municipality drawing a line on datacenter growth, and Huawei swinging for the fences with a unified interconnect pitch. A lot moving at once, which honestly is just the state of this industry right now. Line Rate, brought to you by Ada and Zen Semiconductor — Ada helps companies integrate A I into their real-world workflows, and Zen is an umbrella company with a broad portfolio across the silicon industry. If you're finding the show useful, please subscribe and share it with the engineers and architects in your network who should be paying attention to this stuff. Show notes with links to all the sources are available wherever you're listening. I'll catch you next time.