When One Datacenter Is No Longer Enough

Skyler: Welcome to Line Rate — the show about datacenter fabric, AI interconnect, and the hardware moving data at the speed this industry demands. I'm Skyler Monroe, and if you care about what's happening at the physical layer of A I infrastructure, you're in the right place. Quick thanks to our sponsors: Ada, who helps businesses actually integrate A I into their workflows in ways that stick, and Zen Semiconductor, an umbrella company with a portfolio of companies across the silicon industry — including Ada for A I deployment. Alright, we've got a packed show today — multi-site training headaches, a 112 billion dollar optical market projection, three simultaneous physical walls hitting A I datacenters at once, Marvell's optics story, and a little palate cleanser at the end. Let's get into it. Skyler: So the first story comes from The Next Platform, and I'll be upfront — it's a sponsored post. But the topic it raises is genuinely worth talking about because it's a real problem that a lot of teams are quietly wrestling with right now. The premise is simple: a single datacenter is no longer enough for frontier A I training. And that's not marketing fluff — that's a structural shift in how this industry operates. Skyler: Here's the thing. When you're training models at the scale of GPT-4 class or beyond, you're talking about tens of thousands of accelerators running in tight coordination. And for a long time, the assumption was that you'd rack all of those in one facility, connected over something like InfiniBand or Ethernet with very low latency, and call it a day. The fabric was the problem to solve, and you solved it inside the four walls of one building. Skyler: But that assumption is breaking down, and it's breaking down for a few reasons simultaneously. Power availability is one — you simply cannot get 100, 200, 300 megawatts of clean, reliable power at a single site in most markets right now. The grid doesn't support it, permitting takes years, and even if you get the power, cooling at that density is its own beast. So hyperscalers and large A I shops are being pushed to spread workloads geographically whether they want to or not. Skyler: And that creates a networking problem that is genuinely hard. Inside a single datacenter, your fabric is operating at microsecond latencies. You're running something like 400 gig or 800 gig ports, tight collectives, all-reduce operations that depend on every node responding in lockstep. The moment you introduce a wide-area link — even a high-quality dark fiber run between two campuses — you've introduced latency that can be an order of magnitude higher. Milliseconds instead of microseconds. Skyler: For inference, that's manageable. You can shard across sites with some engineering creativity. But for synchronous distributed training? That latency kills you. Gradient synchronization doesn't tolerate sloppy timing. The collective operations that underpin model training — all-reduce, all-gather, reduce-scatter — these are designed around the assumption of a low-latency, high-bandwidth, lossless fabric. Stretch that fabric across a WAN link and the math changes dramatically. Skyler: What makes this an interconnect story specifically is that the solutions being explored aren't just "throw more bandwidth at it." Teams are looking at asynchronous training approaches, pipeline parallelism across sites instead of tensor parallelism, and custom transport layers that can absorb some of the latency variance without stalling the whole job. There's also real interest in what optical transport can do here — coherent DWDM between campus sites with very low added latency. But none of this is a solved problem, and the standards bodies haven't really caught up yet. This is the frontier, and it's a messy one. Skyler: Story two. ET Datacenters is reporting on a market analysis projecting that A I is shifting the primary bottleneck in datacenter infrastructure from compute to communication — and that the optical market is on track to hit 112 billion dollars. That's a big number, and I want to contextualize it. Skyler: For years, the story in A I infrastructure was GPUs. Who has the most H100s, who gets allocation, how do you cool them. The compute was the constraint. But there's a growing recognition — and the numbers are starting to back this up — that as accelerator density increases, the interconnect becomes the binding constraint. You can have a rack full of the fastest GPUs on the planet, and if your fabric can't feed them data fast enough, they sit idle. And GPU idle time at these prices is an extremely expensive problem. Skyler: The 112 billion dollar figure encompasses a pretty broad optical market — transceivers, coherent optics, co-packaged optics, active optical cables, the whole stack. But the interesting thing is how the growth is being driven specifically by A I workloads rather than traditional networking. Traditional datacenter optical spend was growing steadily. A I is bending the curve. Skyler: And the reason is density. A single A I training pod — think an NVLink domain or a fat-tree InfiniBand cluster — consumes an extraordinary amount of optical bandwidth compared to an equivalent amount of traditional compute. Every GPU needs to talk to every other GPU at high bandwidth, low latency. The all-to-all communication patterns of transformer training are brutal on fabric. You're not doing point-to-point flows — you're doing collective operations that touch every node simultaneously. Skyler: So optical is scaling with A I in a way it never scaled with general-purpose compute, because the communication-to-compute ratio for A I workloads is just fundamentally higher. A training cluster doesn't just need a fast network — it needs a fast network that's also extremely consistent. Jitter kills you. Packet drops kill you. The optical layer has to be rock solid. Skyler: Co-packaged optics is the most interesting piece of this puzzle to me. The idea of integrating the optical engine directly with the switching silicon — eliminating the pluggable transceiver and the electrical trace between them — that's a power and bandwidth density play that makes a lot of sense for A I fabric specifically. You get lower power per bit, higher port density on the switch, and you reduce the reach of the high-speed electrical signal. The challenge is manufacturability, ecosystem lock-in concerns, and the fact that the supply chain for pluggables is mature and co-packaged is still being stood up. But the direction is clear. Skyler: The 112 billion dollar projection is a big bet on optical continuing to win as the medium of choice for A I fabric, both inside the rack and between racks and pods. Given the physics, I think that bet is pretty well placed. Skyler: Okay, story three, and this one is a Next Platform piece that I think is genuinely important framing for anyone building A I infrastructure right now. The title says three physical walls are hitting A I datacenter infrastructure simultaneously, and the compound effects are worse than any one of them alone. Let me walk through what those walls are. Skyler: Wall one is power density. The power consumption per rack has been escalating dramatically. You've gone from 10 to 20 kilowatts per rack in traditional datacenter design to 30, 60, 100 kilowatts per rack for A I. The NVIDIABlackwell systems are pushing toward 120 kilowatts per rack in some configurations. Conventional air cooling tops out well below that. So you're being forced into liquid cooling — direct liquid cooling to the chip, rear-door heat exchangers, immersion in some cases — and that requires facility modifications that are expensive, slow, and not always possible in existing builds. Skyler: Wall two is bandwidth density at the switch. As you pack more accelerators into a rack, the aggregate bandwidth demand going up to the spine layer grows proportionally. And you can only put so many optical ports on a top-of-rack switch before you hit limits — physical space, power draw of the optical engines, the routing complexity. The switch silicon itself is a constraint. You can't just add ports infinitely. The bandwidth-per-unit-area of switching is hitting a ceiling, which is part of why co-packaged optics and larger radix switches are so interesting — they're direct responses to this specific wall. Skyler: Wall three is the signaling reach problem. As you increase data rates — going from 100 gig to 400 gig to 800 gig per lane in some cutting-edge work — the distance that signal can travel over copper shrinks dramatically. At very high data rates, copper gives you centimeters, not meters. So you're forced to go optical much sooner in the signal path. That means more optical components, more cost, more complexity in the package. And it's driving the interest in things like linear-drive optics, where you skip the DSP in the optical module to reduce latency and power, but then you need a cleaner electrical signal from the host side. Skyler: The compound effect the article is pointing at is that these three walls interact. Solving the power density problem with liquid cooling changes your physical layout, which affects your cabling strategy, which affects your optical reach budget. Solving the bandwidth density problem with higher-radix switches changes your network topology, which changes your latency profile, which affects your A I training efficiency. None of these are independent variables. You pull one lever and three other things change. Skyler: For fabric architects specifically, this is the environment you're working in. There's no clean solution that optimizes everything simultaneously. You're making tradeoffs, and the tradeoffs you make at the physical layer propagate all the way up to how efficiently your A I workload runs. That connection between physical infrastructure decisions and A I training performance is closer than most people outside this community realize. Skyler: Story four — Marvell. The Next Platform is reporting that optics is still driving more of Marvell's A I business than their custom silicon work, and that's a really interesting data point about where the value is being captured in this ecosystem right now. Skyler: Marvell has been executing a two-track A I strategy. Track one is custom A I accelerators — ASICs designed for specific customers, the kind of work they're doing with hyperscalers who want to build their own silicon rather than buy from NVIDIA. Track two is optical connectivity — their PAM4 DSPs, their electro-optic integration work, the components that go inside pluggable transceivers and eventually co-packaged optics modules. And what this reporting suggests is that track two is actually generating more A I revenue right now than track one. Skyler: That might seem counterintuitive if you've been following the custom silicon narrative, which has gotten a lot of attention. But it makes sense when you think about adoption curves. Custom A I silicon for a hyperscaler involves a long design cycle, massive NRE, risk on both sides, and then a ramp that's gated on software and deployment readiness. It's a big swing with a long payoff timeline. Optical components go into every transceiver being shipped right now, at scale, into a market that's already booming. Skyler: Marvell's DSP business for optical is essentially a picks-and-shovels play on the A I buildout. Every time a hyperscaler deploys more A I fabric — whether it's InfiniBand, Ethernet, whatever — they need transceivers, and a significant fraction of those transceivers have a Marvell DSP inside. The revenue is immediate and it scales with deployment. Skyler: The custom chip story is still important for Marvell's long-term positioning. If they can establish themselves as the go-to ASIC partner for hyperscale A I — and they've had some wins that suggest they're on track — that's a much larger revenue opportunity per customer. But it's a longer game. What the optics revenue does in the meantime is fund the R&D and demonstrate that Marvell is deeply embedded in the A I infrastructure stack. Skyler: There's also a strategic angle here around co-packaged optics. Marvell is one of the companies most actively working on integrating optical engines with switching and compute silicon. If that transition happens — and I think it will, the question is timing — having deep expertise in the optical DSP layer is a significant advantage. You're not just a silicon company buying your way into optics. You're an optics company who also does silicon. That's a different and arguably stronger position. Skyler: The competitive landscape here is intense. Broadcom has optical exposure too. There are pure-play optical DSP companies. And the transceiver vendors themselves are doing more integration. But Marvell's data suggests they're holding a strong position in a market that's growing fast, and the optics revenue is real, near-term proof of that. Skyler: Alright, story five. I promise I didn't forget about you. ServeTheHome has a review of the Qotom Q51551UT — a ten-port mini PC with four 10 GbE ports and six 2.5 GbE ports, built around an Intel N-series processor. And look, this is not an A I fabric play. This is not a hyperscale story. But I include it because I think it's a useful grounding moment. Skyler: This little box exists because there is a real and growing market for high-port-count networking at the edge — home labs, small office environments, industrial networking, network appliance use cases. The fact that you can now buy a consumer-grade mini PC with ten Ethernet ports, some of them running at 10 gig, for a reasonable price, is a direct reflection of how much the economics of networking silicon have shifted. Skyler: Five years ago, 10 GbE was still aspirational for a lot of smaller deployments. Now it's showing up in a commodity mini PC alongside 2.5 GbE ports. The silicon cost has come down dramatically. The ecosystem for multi-gig Ethernet has matured. And the demand signal from edge computing, from smart home infrastructure, from small-scale virtualization — all of that has driven volume that pushes prices down. Skyler: For the networking engineers in the audience, this is also a reminder that the same SerDes and PHY technology that's pushing 800 gig in the datacenter is also filtering down through the stack over time. The physics and the manufacturing processes developed for hyperscale eventually commoditize. What costs thousands of dollars per port at the leading edge costs pennies per port a decade later. Skyler: The Qotom box itself — ServeTheHome found it to be a decent performer for its class. The 10 GbE ports are handled by Intel's I226-V controllers, which are solid for this segment. Throughput was in line with expectations. It's not going to route 800 gig spine traffic, but it's a competent piece of gear for what it's designed to do. And honestly, it's fun to see the breadth of the market this stuff reaches. Skyler: That's a wrap on today's stories. We covered multi-site A I training and why WAN latency is now a fabric problem, a 112 billion dollar optical market on the back of A I communication bottlenecks, three compounding physical constraints hitting A I datacenters simultaneously, Marvell's optics-first A I revenue story, and a ten-port mini PC that is very much not running a spine-leaf topology. As always, this is a fast-moving space and we'll be back with more as it develops. Line Rate, brought to you by Ada and Zen Semiconductor — Ada helps businesses integrate A I into their workflows, and Zen Semiconductor is an umbrella company with a portfolio across the silicon industry. Subscribe wherever you get your podcasts, share it with someone who lives in the fabric stack, and check the show notes for links to everything we covered today. See you next time.