Fan monitoring on Hyperscale X (HPE Apollo 420) (CC version 11.26.107)

The discrepancy between the HPE iLO fan status and the Commvault HyperScale Health Report is primarily due to how Commvault collects and reports hardware health information. The HyperScale Health Report does not continuously monitor hardware in real time. Instead, it performs periodic hardware health checks typically every 30 to 60 minutes, and in some cases once every hour to gather hardware status from each node. As a result, if a fan failure is detected in HPE iLO, the Health Report may continue to show all fans as healthy until the next scheduled hardware check is completed.

Another factor is the way Commvault interprets fan status. The monitoring logic is designed to report an actual fan failure rather than simply identifying that a fan is missing. If a fan is physically removed or a fan slot is unpopulated, the system may assume that no fan is expected in that location and therefore not generate a failure alert. Additionally, Commvault relies on system-level utilities such as ipmitool and smartctl to retrieve hardware information. If these utilities do not report the issue because of firmware behavior, hardware-specific differences, or other integration limitations, the HyperScale Health Report will continue to display the component as healthy.

The HyperScale Health Report gathers hardware health information by executing these system-level commands on each node and sending the collected data to the Command Center, where it is consolidated into a single view of the appliance's health. This includes the status of components such as disks, power supplies, fans, and network interfaces. Since this information is based on periodic polling rather than continuous monitoring, the report reflects the latest successfully collected hardware status rather than the current real-time state.

Although the HyperScale Health Report is generally reliable for day-to-day monitoring, its purpose is to provide a consolidated operational view of the environment rather than serve as a real-time hardware management tool. The accuracy of the report depends on the polling interval, the information returned by the underlying hardware monitoring commands, and the hardware vendor's implementation. For this reason, it is possible for vendor-specific management interfaces, such as HPE iLO, to detect and display hardware issues before they appear in the HyperScale Health Report.

In summary, the HyperScale Health Report is well suited for routine monitoring, historical tracking, and centralized alerting, but it should not be considered the authoritative source for immediate hardware failures. For critical or time-sensitive hardware events, the vendor's management interface, such as HPE iLO, should always be used to verify the current hardware status and assist with troubleshooting.

​@Bronco, sorry to say this, but your response goes into considerable detail about how Hyperscale monitors the health of the underlying hardware, yet ultimately concludes with: "Sorry, but we are not the authoritative source. You should implement additional monitoring."

Given the information provided, it seems reasonable to assume that this condition existed for quite some time or at least well beyond the scheduler interval used by Hyperscale. To be honest, I am still surprised that Commvault relies on monitoring intervals with this level of delay. Why not perform these health checks every five minutes? If something breaks, I want to know about it as quickly as possible, not hours later. Imagine the temperature is rising because of airflow issues. I would like to know this sooner than later.

More importantly, when a customer purchases an appliance based on a validated reference architecture, and the software vendor is already collecting and reporting on the health status of the platform, I believe it is reasonable to expect that monitoring to be comprehensive enough to cover these types of failures.

From a customer perspective, being told to deploy additional monitoring after the fact feels like a gap in the solution rather than a customer responsibility. If the platform can detect and report hardware health information, I would expect critical issues such as this to be surfaced proactively and in a timely manner.

 @M. Milewski assuming you can access the system yourself I would use impitool yourself and see what it reports back from the CLI and raise a ticket at Commvault to push for improvements.