Changing Networking Mode on Mellanox Cards

Document Scope

This article provides a comprehensive guide on how to transition Mellanox VPI (Virtual Protocol Interconnect) ports from InfiniBand mode to Ethernet mode using the Mellanox Firmware Tools software package. In environments where Mellanox VPI adapters are deployed, there might be a need to switch between different network protocols, such as transitioning from InfiniBand to Ethernet for various reasons like changing network requirements, application compatibility, or infrastructure upgrades.

Installation

Mellanox Firmware Tools(MFT) - Download

• Mellanox Firmware Tools Download

Mellanox Firmware Tools - Installation

• Getting Started With MFT

Usage

The following is an example of using MST to change the VPI ports from InfiniBand to ethernet. 

**** We can use mst status to find the device name of any Mellanox NICs installed on the system.

[root@c117245 ~]# mst status

MST modules:
------------
MST PCI module is not loaded
MST PCI configuration module is not loaded

PCI Devices:
------------

c1:00.0

**** You must start MST before using mst status. Once MST is started, the details of the Mellanox NIC will be listed by mst status.

[root@c117245 ~]# mst start
Starting MST (Mellanox Software Tools) driver set
Loading MST PCI module - Success
Loading MST PCI configuration module - Success
Create devices
Unloading MST PCI module (unused) - Success
[root@c117245 ~]# mst status
MST modules:
------------
MST PCI module is not loaded
MST PCI configuration module loaded

MST devices:
------------
/dev/mst/mt4119_pciconf0 - PCI configuration cycles access.
domain:bus:dev.fn=0000:c1:00.0 addr.reg=88 data.reg=92 cr_bar.gw_offset=-1
Chip revision is: 00

**** We can then use mlxconfig to set the VPI port's link type. Here, 1 is for InfiniBand and 2 is for Ethernet.

[root@c117245 ~]# mlxconfig -d /dev/mst/mt4119_pciconf0 set LINK_TYPE_P1=2 LINK_TYPE_P2=2

Device #1:
----------

Device type: ConnectX5
Name: MCX556A-ECA_Ax
Description: ConnectX-5 VPI adapter card; EDR IB (100Gb/s) and 100GbE; dual-port QSFP28; PCIe3.0 x16; tall bracket; ROHS R6
Device: /dev/mst/mt4119_pciconf0

Configurations: Next Boot New
LINK_TYPE_P1 IB(1) ETH(2)
LINK_TYPE_P2 IB(1) ETH(2)

Apply new Configuration? (y/n) [n] : y
Applying... Done!

**** Please reboot the system to apply the configuration.

The Mellanox NIC's VPI ports has now been set to ethernet. Alternatively, you can set the link type to 1 to set the VPI ports to InfiniBand.

Checking Your Work

You can use ibstat to confirm that the changes you made worked. 

[root@c117245 ~]# ibstat
CA 'mlx5_0'
CA type: MT4119
Number of ports: 1
Firmware version: 16.32.1010
Hardware version: 0
Node GUID: 0x08c0eb03007c72e6
System image GUID: 0x08c0eb03007c72e6
Port 1:
State: Active
Physical state: LinkUp
Rate: 100
Base lid: 0
LMC: 0
SM lid: 0
Capability mask: 0x00010000
Port GUID: 0x0ac0ebfffe7c72e6
Link layer: Ethernet

CA 'mlx5_1'
CA type: MT4119
Number of ports: 1
Firmware version: 16.32.1010
Hardware version: 0
Node GUID: 0x08c0eb03007c72e7
System image GUID: 0x08c0eb03007c72e6
Port 1:
State: Down
Physical state: Disabled
Rate: 40
Base lid: 0
LMC: 0
SM lid: 0
Capability mask: 0x00010000
Port GUID: 0x0ac0ebfffe7c72e7
Link layer: Ethernet