The Network Traffic Map in NVIDIA’s Unified Fabric Manager (UFM) provides a visual representation of the network topology and traffic flows, which is particularly useful for identifying congestion points, verifying network configurations, and optimizing workload distribution.
However, when troubleshooting a single node’s hardware failure, the Network Traffic Map is less effective, as it focuses on network-level issues rather than individual hardware components.

To check the status and link layer of InfiniBand interfaces, the ibstat command is used. For example:
ibstat -d mlx5_0
This command provides detailed information about the InfiniBand device, including its state (e.g., Active), physical state (e.g., LinkUp), and link layer (e.g., InfiniBand).
Reference: NVIDIA DGX BasePOD Deployment Guide – Network Operator Section

Spectrum-X achieves network isolation in multi-tenant environments by implementing Layer 3 Virtual Network Identifiers (L3VNIs) per Virtual Routing and Forwarding (VRF) instance. This approach allows each tenant to have a separate routing table and network segment, ensuring that traffic is isolated and secure between tenants.
Reference Extracts from NVIDIA Documentation:
* “Spectrum-X enhances multi-tenancy with performance isolation to ensure tenants’ AI workloads perform optimally and consistently.”

Accessing theBlueField DPU Operating System (OS)is possible throughrshim, either over PCIe or USB, and viaSSH through the OOB interfacewhen in DPU mode.
From theNVIDIA BlueField Software Documentation:
“You can access the BlueField OS through the rshim interface. The rshim module enables host-to-DPU communication either via PCIe (default) or USB.”
* B. rshim over PCIe: Default when BlueField is installed in a host.
* D. rshim over USB: Useful for provisioning or systems without PCIe drivers.
Incorrect Options:
* A (NIC mode): BlueField acts as a transparent NIC; OS access is not available to the host.
* C (Redfish): Redfish is for out-of-band management, not direct OS-level access.
Reference: Accessing BlueField OS – rshim via PCIe and USB Methods

NVIDIA AIR (AI-Ready Infrastructure) is a cloud-based simulation platform designed to model and validate data center network deployments, including Spectrum-X Ethernet networks, using realistic topologies and configurations. When creating a custom topology in NVIDIA AIR, users can define network links between devices (e.g., spine and leaf switches) using a DOT file format, which is based on the Graphviz graph visualization software. The question asks for the correct syntax to define a link between port 1 on a spine switch (spine-01) and port 41 on a leaf switch (gpu-leaf-01) in a NVIDIA AIR simulation.
According to NVIDIA’s official NVIDIA AIR documentation, the DOT file format is used to specify network topologies, including nodes (devices) and links (connections between ports). The syntax for defining a link in a DOT file uses a double dash (–) to indicate a connection between two ports, with each port specified in the format “<node>”:”<port>”. For Spectrum-X networks, which typically use Cumulus Linux or SONiC on NVIDIA Spectrum switches, ports are commonly labeled as swpX (switch port X) rather than ethX (Ethernet interface), especially for switch-to-switch connections in a leaf-spine topology. The correct syntax for the link between port 1 on spine-01 and port 41 on gpu-leaf-01 is:
“spine-01″:”swp01” — “gpu-leaf-01″:”swp41”
This syntax uses swp01 and swp41 to denote switch ports, consistent with Cumulus Linux conventions, and the double dash (–) to indicate the link, as required by the DOT file format.
Exact Extract from NVIDIA Documentation:
“You can create custom topologies in Air using a DOT file, which is the file type used with the open-source graph visualization software, Graphviz. DOT files define nodes, attributes, and connections for generating a topology for a network. The following is an example of a link definition in a DOT file:
“leaf01″:”swp31” — “spine01″:”swp1”
This specifies a connection between port swp31 on leaf01 and port swp1 on spine01. Port names typically follow the switch port naming convention (e.g., swpX) for Cumulus Linux-based switches.”
-NVIDIA Air Custom Topology Guide
This extract confirms that option A is the correct answer, as it uses the proper DOT file syntax with swp01 and swp41 for port names and the double dash (–) for the link, aligning with NVIDIA AIR’s topology definition process for Spectrum-X simulations.
Analysis of Other Options:
* B. “spine-01″:”swp1” to “gpu-leaf-01″:”swp41”: This option uses the correct port naming convention (swp1 and swp41) but incorrectly uses the word to as the connector instead of the double dash (–). The DOT file format requires — to define links, making this syntax invalid for NVIDIA AIR.
* C. “spine-01″:”eth1” to “gpu-leaf-01″:”eth41”: This option uses ethX port names, which are typically used for host interfaces (e.g., servers) rather than switch ports in Cumulus Linux or SONiC environments. Switch ports in Spectrum-X topologies are labeled swpX. Additionally, the use of to instead of — is incorrect for DOT file syntax, making this option invalid.
* D. “spine-01″:”eth1” – “gpu-leaf-01″:”eth41”: This option uses a single dash (-) instead of the required double dash (–) and incorrectly uses ethX port names instead of swpX. The ethX naming is not standard for switch ports in Spectrum-X, and the single dash is not valid DOT file syntax, making this option incorrect.
Why “spine-01″:”swp01” — “gpu-leaf-01″:”swp41” is the Correct answer:
Option A correctly adheres to the DOT file syntax used in NVIDIA AIR for defining network links:
* Node and Port Naming: The nodes spine-01 and gpu-leaf-01 are specified with their respective ports swp01 and swp41, following the swpX convention for switch ports in Cumulus Linux-based Spectrum- X switches.
* Link Syntax: The double dash (–) is the standard connector in DOT files to indicate a link between two ports, as required by Graphviz and NVIDIA AIR.
* Spectrum-X Context: In a Spectrum-X leaf-spine topology, connections between spine and leaf switches (e.g., Spectrum-4 switches) use switch ports labeled swpX, making swp01 and swp41 appropriate for this simulation.
This syntax ensures that the NVIDIA AIR simulation accurately models the physical connection between spine-01 port 1 and gpu-leaf-01 port 41, enabling validation of the Spectrum-X network topology. The DOT file can be uploaded to NVIDIA AIR to generate the topology, as described in the documentation.

The correct utility isibnetdiscover.
From the official NVIDIA InfiniBand Utilities Guide:
“ibnetdiscover scans the fabric and returns a topology of all switches and end nodes, including their GUIDs, LIDs, port numbers, and link speeds.” It generates a fabric map with node-to-port relationships and shows:
* GUIDs
* LIDs (Local IDs)
* Link speeds and widths
* Switch-to-host connections
This is essential for network topology validation and mapping physical port additions.
Incorrect Options:
* ib_check_routes- for routing table diagnostics.
* ibhosts- shows host information but not switch-level port mapping.
* ibswitches- shows switch info, but lacks port-level LID/link speed mapping.
Reference: NVIDIA InfiniBand Tools – ibnetdiscover Utility

The ib_write_bw tool is part of the Perftest package and is specifically designed to measure the bandwidth of RDMA write operations between two InfiniBand endpoints. It provides accurate assessments of RDMA throughput, which is crucial for verifying the performance of InfiniBand networks in high-performance computing and AI training environments.
Reference:ib_write_bw – NVIDIA Enterprise Support Portal

MLNX_OFED (Mellanox OpenFabrics Enterprise Distribution) is an NVIDIA-tested and packaged version of the OpenFabrics Enterprise Distribution (OFED) for Linux. It provides the necessary drivers and tools to support InfiniBand and Ethernet interconnects using the same RDMA (Remote Direct Memory Access) and kernel bypass APIs. MLNX_OFED enables high-performance networking capabilities essential for AI clusters, including support for up to 400Gb/s InfiniBand and RoCE (RDMA over Converged Ethernet).
Reference Extracts from NVIDIA Documentation:
* “MLNX_OFED is an NVIDIA tested and packaged version of OFED that supports two interconnect types using the same RDMA (remote DMA) and kernel bypass APIs called OFED verbs – InfiniBand and Ethernet.”
* “Up to 400Gb/s InfiniBand and RoCE (based on the RDMA over Converged Ethernet standard) over 10
/25/40/50/100/200/400GbE are supported.”

NVIDIA NetQ is a highly scalable network operations toolset that provides visibility, troubleshooting, and validation of networks in real-time. It delivers actionable insights and operational intelligence about the health of data center networks-from the container or host all the way to the switch and port-enabling a NetDevOps approach.
NetQ can be used as the functional test platform for the network CI/CD in conjunction with NVIDIA Air.
Customers benefit from testing the new configuration with NetQ in the NVIDIA Air environment (“digital twin”) and fix errors before deploying to their production.

The NVIDIA Spectrum-X Reference Architecture (RA) 1.0.1 is designed for Ethernet AI cloud deployments and includes the SN5600 Spectrum-4 switches and BlueField-3 SuperNICs. This architecture supports adaptive routing and DOCA programmable congestion control (PCC) for lossless RoCE traffic, optimizing performance for AI workloads.
The SN5600 switch offers 64 ports of 800GbE in a dense 2U form factor, providing high throughput and low latency essential for AI applications.

TheMINHOProuting protocol, while efficient in finding minimal paths, does not inherently prevent credit loops. This can lead to deadlocks in the network. In contrast, routing protocols like UPDOWN and FAT TREE are designed to avoid such loops, ensuring more reliable network operation.
Reference:Optimized Routing for Large-Scale InfiniBand Networks

* UFM Telemetry: Provides real-time monitoring and analysis of network performance, collecting data such as port counters and cable information to assess the health and efficiency of the network.
* UFM Enterprise: Focuses on comprehensive network management and optimization, enabling administrators to monitor, operate, and optimize InfiniBand scale-out computing environments effectively.
* UFM Cyber AI: Detects and mitigates network security threats by analyzing telemetry data to identify anomalies and potential security issues within the network infrastructure.
Reference Extracts from NVIDIA Documentation:
* “UFM Telemetry provides real-time monitoring and analysis of network performance.”
* “UFM Enterprise is a powerful platform for managing InfiniBand scale-out computing environments.”
* “UFM Cyber-AI enhances the benefits of UFM Telemetry and UFM Enterprise services by detecting and mitigating network security threats.”

To enable SHARP (Scalable Hierarchical Aggregation and Reduction Protocol) aggregation using theNCCL- SHARP plugin, the following two environment variables are required:
* NCCL_COLLNET_ENABLE=1
Enables NCCL’s support for CollNet (Collective Network) operations, including SHARP.
* NCCL_SHARP_AUTOINIT=1
Automatically initializes the SHARP plugin when available, activating SHARP-based collectives.
From theNVIDIA NCCL User Guide – SHARP Plugin Section:
“NCCL_COLLNET_ENABLE must be set to enable collective network acceleration features.”
“NCCL_SHARP_AUTOINIT enables automatic SHARP plugin integration at NCCL runtime.” Incorrect Options:
* B. NCCL_ALGO=CollNet- This variable controls the algorithm used for collectives but does not enable SHARP.
* C. NCCLSPECTRUM_ENABLE- This is not a documented NCCL variable.
Reference: NCCL SHARP Plugin Guide & NCCL User Guide – Environment Variables Section

The SN5600 smart-leaf/spine/super-spine switch offers 64 ports of 800GbE in a dense 2U form factor. The SN5600 offers diverse connectivity in combinations of 1 to 800GbE and boasts an industry-leading total throughput of 51.2Tb/s.
Reference:NVIDIA Spectrum SN5600 Ethernet Switch – Bluum

NVIDIA NetQ is a comprehensive network operations tool designed to provide real-time visibility into the health and performance of NVIDIA networking environments, including Spectrum-X. It offers detailed telemetry and analytics, allowing administrators to monitor adaptive routing behaviors, detect congestion, and analyze traffic patterns. By leveraging NetQ, you can ensure that adaptive routing is functioning as intended and that the network is optimized for AI workloads.
Reference Extracts from NVIDIA Documentation:
* “The NVIDIA NetQ network validation and ASIC monitoring tool set provide visibility into the network health and behavior. The NetQ flow telemetry analysis shows the paths that data flows take as they traverse the network, providing network latency and performance insights.”
* “By leveraging telemetry from Spectrum Ethernet switches and BlueField-3 SuperNICs, NVIDIA NetQ can detect network issues proactively and troubleshoot network issues faster for optimal use of network capacity.”

Direct Memory Access (DMA) in InfiniBand networks allows data to be transferred directly between the memory of two devices without involving the CPU. This capability significantly reduces CPU overhead, lowers latency, and increases throughput, making it ideal for AI workloads that demand efficient data transfers.

AnInfiniBand (IB) Routerconnects two or moreInfiniBand subnets, making it possible for nodes in different subnets to communicate throughroute-managed communication.
From the officialNVIDIA InfiniBand Routers Documentation:
“An InfiniBand router provides connectivity between two or more InfiniBand subnets, enabling communication between hosts that are not on the same subnet while preserving isolation and scalability.”
* Ensures fabric scalability by allowing subnet segmentation.
* Uses LID routing across subnet managers (SMs).
* Essential in large clusters with thousands of nodes.
Incorrect Options:
* AandBare incorrect: InfiniBand does not connect directly to Ethernet or SANs without a gateway.
* Cis unrelated: NVLink is a GPU interconnect, not tied to InfiniBand routers.
Reference: NVIDIA InfiniBand Routers Guide

The correct answer isPXN (Peer eXchange Network).
From theNVIDIA NCCL Documentation:
“PXN enables communication between GPUs connected via NVLink and NICs by treating the GPUs as a distributed switch. This architecture improves bandwidth utilization by enabling any GPU to communicate with the NIC via the shortest path available, even if it’s not directly connected to the NIC.” This enhances GPU-to-NIC and NIC-to-GPU transfers, leveraging the NVLink topology. It significantly boosts performance in multi-GPU setups where not every GPU is directly connected to the NIC.
Other options:
* Adaptive Routingis a fabric-level feature for dynamic path rerouting.
* Graph Search Optimizationis used internally for topology modeling in NCCL.
* SHARP v2is a switch-based collective acceleration method, unrelated to PXN.
Reference: NVIDIA NCCL User Guide – PXN Feature Section

AI workloads, particularly in large-scale training scenarios, are characterized by a small number of high- bandwidth, long-lived flows known as “elephant flows.” These flows can dominate network traffic and are prone to causing congestion if not managed effectively.
Traditional flow-based load balancing mechanisms, such as Equal-Cost Multipath (ECMP), distribute traffic based on flow hashes. However, in AI workloads with lowentropy (i.e., limited variability in flow characteristics), ECMP can lead to uneven traffic distribution and congestion on certain paths.
Adaptive routing techniques, which dynamically adjust paths based on real-time network conditions, are more effective in managing AI traffic patterns and mitigating congestion risks.
Reference:Powering Next-Generation AI Networking with NVIDIA SuperNICs