Improve AI Cluster Utilization by Measuring Communication Bottlenecks
Key takeaways
Better use of existing GPU capacity: When communication limits scaling, improving data movement can help an existing cluster achieve better performance.
Better time to solution: Distributed workloads that use AllReduce and AlltoAll collectives stall when communication cannot overlap with computation. Reducing that stall time shortens iteration time and time to result.
Lower evaluation risk: Collective duration, tail latency, and application throughput reveal whether the bottleneck is compute or communication. That measurement guides the next hardware decision.
Why the network affects GPU productivity
AI infrastructure teams often respond to slow workloads by adding more GPUs. That can help when computation is the bottleneck. It is less effective when GPUs spend significant time waiting for data movement, synchronization, routing, or collective communication.
For the technical buyer, the goal is not simply to install more accelerators, it is to complete training runs and serve inference requests faster, with predictable cost and power. A communication-aware network can help unlock compute capacity when distributed workloads are limited by the communication path.
The key question is not only how many GPUs a cluster contains, but how much useful work each GPU completes, and how quickly.
Why utilization matters at cluster scale
A GPU waiting on communication still consumes power and occupies rack space. This means it contributes to the cost of the job without contributing to the result. And that’s before factoring in initial purchase costs. Improving GPU utilization can help an organization complete the same workload in less time and handle more requests with the same hardware.
Better GPU utilization can give an organization more flexibility:
Complete training sooner: Reduce time to solution when communication contributes materially to iteration time.
Serve more inference requests: Increase useful throughput without immediately expanding the GPU hardware.
Improve procurement decisions: Determine whether the next performance target requires more GPUs or a better-balanced system.
Support distributed deployments: Improve the economies of smaller clusters that operate across locations or facilities.
How distributed AI becomes communication-bound
Adding GPUs can increase throughput when compute is the limiting factor. Distributed scaling can also expose communication and synchronization limits.
In many distributed training implementations, accelerators exchange gradients through collective operations such as AllReduce. The next dependent phase cannot proceed until the required results are available. Communication may overlap with computation, but when that overlap is incomplete, a delayed participant becomes a straggler. A single straggler can significantly slow down performance.
As the cluster grows, the communication pattern can become more demanding:
More GPUs generate more messages.
More endpoints increase coordination and synchronization complexity.
Small-message traffic can become more important.
Congestion can increase the chance of stragglers.
A slow or overloaded link can affect the duration of a collective operation.
Inference creates a different set of communication challenges. Mixture-of-Experts (MoE) models may route tokens to expert GPUs through all-to-all traffic. Some distributed inference architectures also move or share Key-Value (KV) cache data, creating additional bandwidth and latency-sensitive traffic.
These operations can be bursty and latency sensitive. The result is that GPUs can be available but not productive. For distributed workloads, the inter-node network becomes part of the compute pipeline.
The network is part of the compute pipeline
AI cluster performance depends on the interaction between several components:
GPU computation: The amount of useful work each accelerator can complete
Host memory and PCIe transfers: The path between the accelerator, host, and network adapter
Network adapters and switches: The devices that move traffic between endpoints
Communication libraries: The communication middleware and libraries responsible for collective and point-to-point communication
Congestion management: The mechanisms that control traffic when multiple flows compete for shared paths
Scheduling and workload placement: The way processes and workloads are assigned to hosts and accelerators
Application-level routing: The way training, inference, and expert traffic is distributed across the system
Peak bandwidth is important, but it is only one part of the performance equation. A network can advertise a high line rate and still produce poor application performance if it has:
Low effective bandwidth growth for small messages
High or variable tail latency
Congestion during collective operations
Inefficient routing for the workload
Packet loss, inefficient error handling, and expensive retries and recovery
Endpoint processing bottlenecks
Insufficient software optimization
Any AI/ML network performance evaluation should use workload-level metrics, not only port speed. For training, the relevant question may be how long one complete iteration takes. For inference, it may be how many tokens per second the system delivers at the required latency and quality. For the business, it may be the cost of each completed training run or generated token.
The Cornelis approach to useful work per accelerator
Cornelis approaches AI networking as a utilization problem, not only as a bandwidth problem. The objective is to help more accelerators remain productive by improving how data moves through the fabric and by reducing avoidable pressure on the endpoints.
Start with efficient data movement
CN5000 Omni-Path® is designed to support communication-intensive workloads through the following capabilities:
RDMA-based data movement: Move data between endpoints while reducing unnecessary host involvement.
Credit-based flow control: Manage traffic injection according to available resources in the network path.
Dynamic adaptive routing: Adjust traffic paths to respond to network conditions.
Congestion-management mechanisms: Help keep traffic moving when multiple flows compete for shared network resources.
Optimized host software: Expose the network capabilities through the software stack used by the application.
High message-rate performance: Accelerate communication-intensive workloads that send many small messages.
What the benchmark results show
A public Cornelis point-to-point demonstration compares CN5000 with an NDR 400G configuration using latency, bandwidth, and message-rate tests. The demonstration reports approximately 979 nanoseconds of CN5000 latency compared with approximately 1.45 microseconds for the tested NDR 400G configuration. It also reports approximately 14 million messages per second for CN5000 compared with approximately 3.9 million messages per second for the tested NDR 400G configuration.
These results describe communication performance under the tested point-to-point conditions. They do not mean that every training or inference workload will run 3.75 times faster. Application benefit depends on the workload’s communication pattern, the amount of communication that can overlap with computation, topology, process placement, software, and system configuration.
A workload evaluation should therefore measure both communication behavior and application results. Useful measures may include collective duration, iteration time, tokens per second, application throughput, time to result, power per completed workload, and cost per result.
How to get started
Start with a baseline of the current AI infrastructure.
Measure representative jobs: Record GPU utilization, collective duration, network throughput, tail latency, tokens per second, cost per token, time to result, and power per completed workload.
Identify the limiting path: Determine whether the primary constraint is GPU compute, memory capacity, host processing, network bandwidth, message rate, congestion, process placement, or collective communication.
Reproduce the traffic pattern: Test the message sizes, directions, collectives, all-to-all traffic, and concurrency levels that occur in the target workload.
Compare complete configurations: Evaluate the adapter, switch, topology, software stack, firmware, communication library, and application together.
Make the procurement decision from measured results: Determine whether the result supports more throughput, shorter job times, deferred expansion, lower energy per result, or another approved business outcome.
Communication measurements become useful only when they lead to an application or procurement decision.
Deployment evaluation should include server compatibility, software integration, fabric configuration, and operational support. Cornelis has publicly announced CN5000 qualification and integration across Lenovo ThinkSystem V3 and V4 servers and Lenovo’s EveryScale solution, providing an example of validated AI and HPC infrastructure integration.
Cornelis can help organizations assess whether the limiting factor is GPU compute, host processing, network bandwidth, message rate, congestion, or collective communication. The goal is not to recommend more hardware by default. The goal is to identify where the system is losing useful capacity and determine how much of that capacity can be recovered.
The fastest AI system is not necessarily the one with the most GPUs. It is the one that keeps the GPUs it already owns doing useful work.
To request a workload assessment or product evaluation, contact your Cornelis representative. You can also reach out to the Cornelis Sales Team.