Thanks for details.
Right now since the browse request, just an extract of what I have from the server, live :Â
>root@MyLinuxMAanonymized Log_Files]#
Message from syslogd@MyLinuxMAanonymized at Apr 13 14:22:14 ...
 kernel:watchdog: BUG: soft lockup - CPU#1 stuck for 23s! !CVODS:2921670]
Apr 13 14:22:14 MyLinuxMAanonymized kernel: CPU: 1 PID: 2921670 Comm: CVODS Kdump: loaded Tainted: P Â Â Â Â W Â OEL Â --------- - Â - 4.18.0-372.32.1.el8_6.x86_64 #1
Message from syslogd@MyLinuxMAanonymized at Apr 13 14:22:46 ...
 kernel:watchdog: BUG: soft lockup - CPU#0 stuck for 23s! CVODS:2921670]
Apr 13 14:22:46 MyLinuxMAanonymized kernel: CPU: 0 PID: 2921670 Comm: CVODS Kdump: loaded Tainted: P Â Â Â Â W Â OEL Â --------- - Â - 4.18.0-372.32.1.el8_6.x86_64 #1
Message from syslogd@MyLinuxMAanonymized at Apr 13 14:23:03 ...
 kernel:watchdog: BUG: soft lockup - CPU#50 stuck for 22s! 5CVODS:2921662]
Apr 13 14:23:03 MyLinuxMAanonymized kernel: CPU: 50 PID: 2921662 Comm: CVODS Kdump: loaded Tainted: P Â Â Â Â W Â OEL Â --------- - Â - 4.18.0-372.32.1.el8_6.x86_64 #1
Message from syslogd@MyLinuxMAanonymized at Apr 13 14:23:11 ...
 kernel:watchdog: BUG: soft lockup - CPU#22 stuck for 23s! pCVODS:2921670]
Apr 13 14:23:11 MyLinuxMAanonymized kernel: CPU: 22 PID: 2921670 Comm: CVODS Kdump: loaded Tainted: P Â Â Â Â W Â OEL Â --------- - Â - 4.18.0-372.32.1.el8_6.x86_64 #1
Message from syslogd@MyLinuxMAanonymized at Apr 13 14:23:27 ...
 kernel:watchdog: BUG: soft lockup - CPU#26 stuck for 23s! fCVODS:2921662]
Apr 13 14:23:27 MyLinuxMAanonymized kernel: CPU: 26 PID: 2921662 Comm: CVODS Kdump: loaded Tainted: P Â Â Â Â W Â OEL Â --------- - Â - 4.18.0-372.32.1.el8_6.x86_64 #1
Message from syslogd@MyLinuxMAanonymized at Apr 13 14:23:42 ...
 kernel:watchdog: BUG: soft lockup - CPU#4 stuck for 22s! :CVODS:2921670]
Apr 13 14:23:42 MyLinuxMAanonymized kernel: CPU: 4 PID: 2921670 Comm: CVODS Kdump: loaded Tainted: P Â Â Â Â W Â OEL Â --------- - Â - 4.18.0-372.32.1.el8_6.x86_64 #1
Message from syslogd@MyLinuxMAanonymized at Apr 13 14:23:51 ...
 kernel:watchdog: BUG: soft lockup - CPU#58 stuck for 22s! aCVODS:2921662]
Apr 13 14:23:51 MyLinuxMAanonymized kernel: CPU: 58 PID: 2921662 Comm: CVODS Kdump: loaded Tainted: P Â Â Â Â W Â OEL Â --------- - Â - 4.18.0-372.32.1.el8_6.x86_64 #1
Message from syslogd@MyLinuxMAanonymized at Apr 13 14:24:11 ...
 kernel:watchdog: BUG: soft lockup - CPU#36 stuck for 23s! kCVODS:2921670]
Apr 13 14:24:11 MyLinuxMAanonymized kernel: CPU: 36 PID: 2921670 Comm: CVODS Kdump: loaded Tainted: P Â Â Â Â W Â OEL Â --------- - Â - 4.18.0-372.32.1.el8_6.x86_64 #1
Message from syslogd@MyLinuxMAanonymized at Apr 13 14:24:18 ...
 kernel:watchdog: BUG: soft lockup - CPU#2 stuck for 22s! .CVODS:2921662]
Apr 13 14:24:18 MyLinuxMAanonymized kernel: CPU: 2 PID: 2921662 Comm: CVODS Kdump: loaded Tainted: P Â Â Â Â W Â OEL Â --------- - Â - 4.18.0-372.32.1.el8_6.x86_64 #1
i root@MyLinuxMAanonymized Log_Files]# uptime
 14:24:26 up 5 days, 21:08,  1 user,  load average: 38.35, 22.22, 11.21
:root@MyLinuxMAanonymized Log_Files]#
Â
(of course MA hostname is different in reality/case).
@Laurent ,
Development confirmed that the Diag is in 54 and up.
From UpdateInfo.log you see we remove all the Diags, then we install the MR.
So when I checked your logs I could see we removed the Diag.
1629126 140672227698496 04/05 13:12:54 5371325 All updates to remove = 'linux-x8664_11.0.0B80-SP28_PreRelease-4206-BIN:1101'], sucessfulUpdates = 'linux-x8664_11.0.0B80-SP28_PreRelease-4206-BIN'], list lengths = 1 1
The job id is from the Persistent Recovery job, in Jobmanager.log you would see:
>---- RESTORE 3RD PARTY REQUEST ----]
However from the FREL, the /var/log/messages have rolled over as the Browse was done on 07/04.
We will check the logs you sent then.
Thanks for the followup, Sebastien. I was off this wednesday.
I just uploaded the logs of the MA, as, well, there’s no job reference itself for the simple ‘browse’ session, mentionning it’s related to Incident 230411-419.
Â
I am not 100% sure how to interpret what you wrote.
Were you talking about my case/logs about Diag removed and MR56Â installed?Â
Or just that Diag is not available anymore as MR56 includes it in any way ? (Sorry for my blurry mind )
@Laurent ,
The logs have rolled over, can you please reproduce the issue and send us the logs.
I can only see that the Diag has been removed and MR56 installed.
Meanwhile I have sent an email to Development.
I will get back to you.
Best Regards,
Sebastien
@Laurent ,
It should be in MR54 already, so MR56 should also have it.
https://documentation.commvault.com/2022e/expert/assets/service_pack/updates/11_28_56.htm
Soft CPU lockups in _raw_spin_lock on RHEL 8.7 with 4.18.0-425.10.1 kernel Adding Linux driver support for Debian 10 kernels up to 4.19.0-23 | 4090 |
Â
But sure, please log a new case and we will check.
Best Regards,
Sebastien
Â
Hi guys !Â
Â
Adding to this thread as it was quite useful to solve the issue.
Last wednesday I applied the latest MR, 11.28.56 onto my 11.28.52 + DIAG that had the issue fixed.
I have been asked to perform a file-level restore from a windows VM backup, using my same RHEL8.7 linux MA. And it failed to succeed. I also had new occurences of ‘kernel:watchdog : BUG : soft lockup - cpu stuck for …’
I think I will open a new case, as I don’t think I can apply the same DIAG after this MR56.
Wasn’t MR54 supposed to embed this fix ?
Â
Regards,
Laurent.
Â
@Laurent ,
It looks like it is RHEL 8.7 latest kernel on which cvblk driver support not present.
My colleague has your case and will send it to you.
Bonne journéeÂ
Seb
Merci Sebastien, it worked after I applied the diagÂ
@Laurent ,
It looks like it is RHEL 8.7 latest kernel on which cvblk driver support not present.
My colleague has your case and will send it to you.
Bonne journéeÂ
Seb
Thanks @Sebastien Merluzzi , to accelerate and not go into usual troubleshooting/error reproducing/send logs, is there something I can mention to get the immediate download link to this hotfix ?Â
@Laurent @Egor Skepko ,
The Hotfix is in MR54Â which will be available on Tuesday 4th April (EST).
So you can log a case with us and we will provide the Diag.
@Laurent There wil be new hotfix next week that should solve your issue aswell. Otherwise you can create case at commvault or as them to send you update.
Hi I’m interested on this as it looks like I experience the same issue
@Egor Skepko That’s correct. Please mark this question as solved.
@Sebastien Merluzzi We have installed the DIAG on one of the MA (11.28.48) to resolve the drivers on OS version 8.7 and after the installation i did restore and its work fine. This DIAG wil be availble next week at 11.28.54 SP
@Egor Skepko ,
Please work on the case you have with my colleague, as I can see he is scheduling a session with you.
@Sebastien Merluzzi Sorry for delay, we are running on 4.18.0-425.13.1.el8_7.x86_64 kernel, and we we are not going to downgrade to 8.6. So we juist wait for the fix in april?Â
Sure, my colleague will contact you.
@Sebastien Merluzzi Hello, yeah in the case i said that we gonna use Windows MA but didnt said to close the case.Â
Â
But yes pls can you provide me fix now so i can test it out.
Looks like issue is Red Hat Enterprise Linux 8.7 crashing due to latest kernel on which cvblk driver support not present.
Can you confirm the version you have please and if so, can you downgrade to RHEL 8.6?
If that’s the issue, we have a fix SP28-HotFix-4090 which will be available in the next Maintenance Release in April.
Otherwise we can provide the Diag in your case and see if that fixes your issue.
Â
Hi @Egor Skepko ,
From the case you said that you will use a Windows MA which is working as expected.
I checked with the Engineer and requested to re-open the case and escalate to Development with the crash dump..
Best Regards,
Sebastien