Search

Intel X710 Firmware Updater - Step-by-step Guide

Arjay Reyes
Arjay Reyes
  • Updated

Overview   

If your MSI server is filling its event log with PCIe errors whenever the network is busy, the cause is most likely the firmware on its onboard Intel X710 network controller, not a hardware fault. 

Intel's NVM version 9.54 addresses this and Exxact will provide the update package after identifying the fault. This article will walk you through:
  • Checking your system status
  • Updating that firmware to 9.54 resolves these repeat errors.
    Note: Please use only the firmware package provided by Exxact Support. It includes a configuration validated with the system manufacturer for this platform, so a newer public Intel release may not behave the same way.
  • Confirmation and monitoring that update is pushed into the system
Some Linux knowledge is recommended but the steps outlined will walk you through the commands to run and how to push the firmware update. Please note that the firmware update is a sensitive operation, so please take your time and follow each step in order. 

Symptoms Indicating a Firmware Update is Needed

  • The BMC event log (SEL) fills with "Critical Interrupt – PCI PERR" entries.
  • The operating system logs a steady stream of corrected PCIe errors (Replay Timer Timeout) for the X710 ports, often about one per second while the network is busy.
  • Errors appear under heavy network traffic, such as multi-node jobs or large data transfers.
  • In more severe cases, system can shut down unexpectedly after a long run of these errors.
To check your logs, you can run:
sudo ipmitool sel elist | grep -i perr
sudo journalctl -k | grep -iE "aer|replay timer"
 

Why does this happen?

The network controller has its own small firmware (called NVM), separate from the system BIOS and the operating system. On NVM 7.30, one of the controller's PCIe timing settings (the replay timer) is too short for this board's PCIe connection. This means under heavy traffic, the controller will keep timing out and re-sending data, and every retry will be logged as an error. Most of these errors are corrected automatically, but the volume can flood the logs and, in the worst case, destabilize the system. Changing BIOS PCIe or power settings, replacing the server board, or reinstalling the operating system doesn't fix it, because the problem lives in the controller's own firmware. Updating that firmware to 9.54 (or latest) should help resolve these repeat errors.

Prerequisites

  • Local console access, such as a monitor and keyboard or the BMC remote console (KVM). The network link can drop during the update, so please don't run it over SSH, especially not over the X710 ports. 
  • Root (sudo) access to the system
  • The X710 firmware package, which you can receive from Exxact Support. Please open a ticket with our support team to assist and diagnose the issue. Feel free to reference this article if you are experiencing the symptoms listed above.
    https://support.exxactcorp.com/hc/en-us/requests/new?ticket_form_id=13800290848663 
This update should take approximately 30 minutes.

Important Note: A firmware update that's interrupted or done incorrectly can leave the network controller, or the system, unusable, and it may need a hardware repair. Please update one system at a time. If anything looks different from what's described here, stop and contact Exxact Systems Support before proceeding.

 

Step-by-step Instructions

Step 1: Check your System's Network Devices

Step 1A: Confirm the system has ONLY onboard Intel Ethernet Controller X710 ports

lspci | grep -iE "intel.*ethernet"

You should see only the onboard Intel Ethernet Controller X710 entries. 
If any other Intel network cards appear (for example, an add-in Intel Ethernet card) please power off the system and remove them before updating. The update tool scans every Intel network device in the system and could try to flash them too.

Note: Non-Intel cards, such as NVIDIA/Mellanox adapters, are not affected by the update tool and can stay installed.

Step 1B: Confirm the Intel network card's driver is loaded

lsmod | grep i40e

If this returns nothing, please contact Exxact Support before continuing.

Step 1C: Record current firmware version

ethtool -i <interface> | grep firmware-version

Replace `<interface>` with the name of one of the X710 ports. You can list interface names with `ip link`.

 

Step 2: Extract Firmware Package

Download x710_fw.tar.gz to the system. Our support team will provide it to you upon request after diagnosing the symptoms align with the issue you are seeing with your Intel X710.

Then extract it:
tar -xzf x710_fw.tar.gz

 

Step 3: Change into firmware tool's folder

Note: The update tool must be run from inside its own folder. If you run it from anywhere else, it can't find its configuration and firmware files, and the update will fail. 
 
cd ./700Series/Linux_x64/

 

Step 4: Run the update

sudo ./nvmupdate64e
The tool scans the system and lists the Intel network devices it found, along with whether an update is available.

When the tool prompts you:
- Which devices to update: type A (All) and press Enter. This updates every X710 port on the system.
- Back up the NVM images: type Y (Yes) and press Enter. This saves a copy of the current firmware before the update.

The update then starts. It typically takes 5 to 15 minutes. While it runs:
  • Do not close the window, end the session, or press Ctrl+C.
  • Do not power off, reboot, or unplug the system.
  • Network ports may lose their link during the update, this is expected.

Important Note: If the update is still running after 20 minutes, or shows an error, do not reboot or power off the system. Leave it as it is and contact Exxact Support right away.

Once the update completes you should see the following output:

 

Step 5: Reboot

When the tool reports that the update is complete, reboot the system:
sudo reboot

If the X710 ports don't get a network link after the reboot, do a full power cycle: shut the system down completely, wait 30 seconds, then power it back on.

 

Step 6: Verify

Check the firmware version again:
ethtool -i <interface> | grep firmware-version
It should now show firmware version 9.54.
 

Step 7: Confirm the errors are gone

Run your normal network-heavy workload (or a large file transfer across the X710 ports) for a while, then check the logs again:

sudo ipmitool sel elist | grep -i perr
sudo journalctl -k -b | grep -iE "aer|replay timer"

You shouldn't see any new PCI PERR or Replay Timer Timeout entries since the reboot. 

Note: PCIe errors logged before the update may still appear in the SEL.

Check the timestamps: only entires dated after the reboot in Step 6 matter. 
To make the check easier, please save a copy of the SEL and then clear it before the reboot. That way, anything that shows up afterward is new.

sudo ipmitool sel elist > ~/sel_before_update.txt
sudo ipmitool sel clear

Please send Exxact Support the before and after firmware versions from Step 1c and Step 6, along with this log check, and reach out if anything doesn't look right.

 

Prevention Tips, Maintenance, or Best Practice

  • Update one system at a time: if something unexpected happens, only one system is affected and it's easier to troubleshoot.
  • Use a console, not the network you're updating: running the update from a local or BMC console means a link drop can't interrupt it.
  • Don't interrupt a firmware update: if the system seems stuck, leave the system powered on and contact us; an interrupted flash is what turns a fixable issue into a hardware repair.
  • Keep your before and after version output: it's a quick record of what firmware each system is running.
  • Keep an eye on PCIe errors: a sudden rise in corrected PCIe errors is worth reporting before it turns into downtime.
  • Need further help? Contact Exxact Support at https://support.exxactcorp.com/hc/en-us at any step. We're happy to walk through the update with you.


 

Was this article helpful?

0 out of 0 found this helpful

Have more questions? Submit a request

Comments

0 comments

Article is closed for comments.