Analysis of Lenovo XCC WebGUI Not Displaying Disk Details

Symptom

1. The XCC WebGUI page does not display detailed information for disks connected to the 430 HBA / 730 RAID card.
2. Below is the disk information displayed on the XCC WebGUI page for a server configured with a 430 HBA card:

   

3. Below is the disk information displayed on the XCC WebGUI page for a server configured with a 730 RAID card:

   

Affected Device Information

Model, firmware version running when the issue was reported, and the latest firmware version currently available on the official website:

Reproduction and Principle

Reproduction:

1. Lenovo Laboratory has reproduced the same situation reported by your company. The following two situations cause disk detailed information to be undisplayed:
   • XCC is not assigned an EID
   • 430 HBA / 730 RAID card is not assigned an EID
2. The situation of not being assigned an EID can only occur during the server POST process.
3. If assigned an EID during the POST process, this situation will not occur during normal operation after entering the operating system.

Important principles involved in subsequent analysis:

1. Communication between XCC and the 430 HBA / 730 RAID card is based on the MCTP protocol (Management Component Transport Protocol).
2. Each node communicating based on the MCTP protocol must be assigned an Endpoint ID (EID).
3. The MCTP master (Intel ME) performs Discovery during the server POST process and assigns an EID to each device.

Changes related to Intel ME Code and Intel ME regarding Device Discovery

1. Fixes related to Intel ME Code and Intel ME regarding Device Discovery:
   • Device Discovery is led by Intel ME as the MCTP master. In unofficial intermediate versions of Intel ME Code, bugs related to Discovery were found; Intel has fixed them. The uEFI 3.11 released in Q3 2021 includes all corresponding fixes made by Intel ME code.
2. Supplementary Information:
   • Faulty machine uEFI version: 2.3.
   • uEFI Change History:https://download.lenovo.com/servers/mig/2021/11/08/54899/lnvgy_fw_uefi_ive174g-3.11_anyos_32-64.chg

Cause Analysis and Corresponding Fix/Enhancement Solutions for Cases Where XCC Does Not Receive an EID

1. For cases where XCC does not receive an EID, XCC includes one fix and one enhancement.
   • During laboratory reproduction, it was found that the residual MCTP status and EID information from the previous session of XCC affect XCC's ability to successfully obtain EID information during the next boot. Therefore, XCC includes a fix that clears XCC's EID information and completely resets the MCTP status during a DC Cycle, enabling XCC to successfully obtain EID information during Intel ME's Discovery process.
   • Considering the possibility that XCC may still not be discovered and fail to obtain EID information during the Discovery process, XCC includes an enhancement where XCC actively sends a command to Intel ME to trigger a re-Discovery by Intel ME.
2. Note:
   • XCC version running on the faulty machine: 2.7.
   • Resetting only XCC and Intel ME can only resolve the issue of XCC failing to obtain EID information.
   • After resetting XCC and Intel ME, a DC cycle is required for both XCC and the card to obtain EID information.
   • XCC change history:https://download.lenovo.com/servers/mig/2021/11/04/54890/lnvgy_fw_xcc_cdi382o-7.22_anyos_noarch.chg

Cause analysis for cards not receiving EID and corresponding fixes/enhancements

1. For the issue where cards do not receive EID, both XCC and card firmware have added corresponding fixes and enhancements.
   • Due to observations that cards take longer to obtain EID in scenarios such as Option ROM Disabled, the card may fail to acquire EID within the limited time because the Discovery timeout provided is insufficient. Therefore, XCC has added an enhancement: after Intel ME Discovery, a waiting period is extended to allow the card sufficient time to obtain EID information.
   • Card firmware identified bugs causing cards to fail receiving EID. The Q3 2021 version of card firmware includes corresponding fixes and enhancements.
2. Note:
   • Firmware version of the faulty machine's 430-16i is 09.00.07.00; firmware version of the 730 RAID card is 24.21.0-0097.
   • 730 card firmware change history:https://download.lenovo.com/servers/mig/2021/06/10/54022/lnvgy_fw_raid_mr3.0.730-24.21.0-0151-4_windows_x86-64.chg
   • 430-16i firmware change history:https://download.lenovo.com/servers/mig/2021/11/18/54976/lnvgy_fw_storehba_mpt3.5.430-18.04.00.00-0_windows_x86-64.chg

Conclusion and Solution

1. Conclusion:The uEFI Intel ME Code, XCC, and the 430 HBA / 730 RAID card firmware were not robust enough in handling EID assignment, leading to the issue.
   • The uEFI, XCC, and 430 HBA / 730 RAID card firmware released in Q3 2021 include fixes and enhancements related to the identified issues.
   • Servers in the lab where the issue could be reproduced did not show the problem of XCC or 430 HBA / 730 RAID cards failing to receive EID after updating to the firmware combination released in Q3 2021.
2. Solution:
   • The solution is to upgrade affected machines to the firmware combination released in Q3 2021.
     • The uEFI, XCC, and 430 HBA / 730 RAID card firmware on machines experiencing the issue differ significantly from the versions released in Q3 2021 (see "Affected Device Information" above).