Skip to main content
Solved

Primary On-Prem MA - Windows based and crashing processes on MA

  • February 26, 2025
  • 6 replies
  • 75 views

Forum|alt.badge.img+1

Does anyone have in their Windows Application Event Log of your Media Agent Server any event id’s 1001 that look like this:

Fault bucket 2119584666510853566, type 5
Event Name: RADAR_PRE_LEAK_64
Response: Not available
Cab Id: 0

Problem signature:
P1: CVODS.exe
P2: 11.280.1000.1507
P3: 10.0.17763.2.0.0

On mine, the event name and the exe changes but does repeat. Sometimes it’s VSBK.exe or a random Windows exe.I used to have a ton of these.

Present day it’s much less. Last year the Application Event log was littered with these and the Media Agent Server became unusable.  Support was at a loss. It might be easy to say there was a hardware issue. This server was in service from 2019 - 2024 and operated 24x7 without issue.

Since that terrible week last year the MA was migrated to a different server to get backups going again. These app crashes in the Application Event log still occur on the new server, but they are few and isn’t effecting backups anymore. When I say migrated, I moved the DDB, moved my Fiber Channel disk paths, and Index cache to a completely different server and powered down the original server.
  

At this point and on this new server I’m just trying to get insight as to what is causing these processes to crash. I’ve looked at memory dump files and seems to be memory leak problem. Because this is now different hardware, it seems more specific to Windows Server. I’ve disabled pagefile as I have enormous amounts of ram. Windows Defender has all Commvault processes and paths excluded from real time protection. Including mount paths and DDB.
Anything else I should check or ideas? is this normal and I somehow missed it till the old server locked up last year, 2024?
I left out a lot of detail. I thought this was a decent starting point. If it helps to know more, please ask. Backups are stable and running better than ever, I just want to try and solve for these windows events if I can.

Thank you,

Best answer by Pradeep

Hi ​@De-Duped 

The reported error is primarily related to a memory leak in a running application. However, to perform a deeper analysis, we require additional details, including Windows Application and System Logs.

In parallel, would request to refer below document and enable windows error reporting under Commvault process manager to capture the dumps during the crash events.

This will help us understand the root cause and provide next step to fix the issue you may also log support case to investigate the details after collecting the required information.

https://documentation.commvault.com/11.20/enabling_windows_error_reporting.html 

 

View original
Did this answer your question?

6 replies

Forum|alt.badge.img+11
  • Vaulter
  • 248 replies
  • Answer
  • March 1, 2025

Hi ​@De-Duped 

The reported error is primarily related to a memory leak in a running application. However, to perform a deeper analysis, we require additional details, including Windows Application and System Logs.

In parallel, would request to refer below document and enable windows error reporting under Commvault process manager to capture the dumps during the crash events.

This will help us understand the root cause and provide next step to fix the issue you may also log support case to investigate the details after collecting the required information.

https://documentation.commvault.com/11.20/enabling_windows_error_reporting.html 

 


dude
Byte
Forum|alt.badge.img+15
  • Byte
  • 329 replies
  • March 1, 2025

I had a very similar case recently. Do you happen to have the additional key on your MAs?

sPipelineMode (value=SR:P;R:P)


Onno van den Berg
Commvault Certified Expert
Forum|alt.badge.img+19

@dude Do you happen to have information what this setting actually changes and why you have it in place. I was wondering about it as it came forward in another post some years ago in where I also asked for details but I never received an answer. It also doesn't popup in the updated additional settings database lookup page.


dude
Byte
Forum|alt.badge.img+15
  • Byte
  • 329 replies
  • March 2, 2025

@Onno van den Berg I initially had it in place due to performance issues related to Tape Copies which never really solved the problem and we end up finding other things that were causing the issues.

This setting more recently was actually crashing CVD/CVODS on our MAs, there is the reason I was asking.

From Tier 2 support regarding sPipeLineMode;

 "The sPipelineMode Mode key overrides the SDT transfer method which will prevent quit flags from being properly called at the end of a stream or when a stream fails. This results in a hung process which leads to the affected process (in this case CVD and CVODS) to fail out and crash, leaving behind the zombie thread which never quit. By disabling sPipelineMode key and reverting to SDT pipelines, this condition will be avoided"

 

 


Onno van den Berg
Commvault Certified Expert
Forum|alt.badge.img+19

Ahh ok. So, this setting was still in place while it should have been removed as it didn't make any difference in terms of performance. 


Forum|alt.badge.img+1
  • Author
  • Byte
  • 2 replies
  • March 5, 2025

Thank you all for your reply. 

@Pradeep Thank you for the reminder about the error reporting. Support had me configure this back in October 2024, but nothing was populated at that time. With your reminder, I checked again and found that a DMP file was finally generated in January 2025 for the vsbk process. I reviewed the DMP file, and it appears to be benign and unrelated to the random exe crashes on the MA.

@dude I checked my MA properties Additional Settings and i do not have that entry. However I do have:
nClientGrpMaxCPUPercent iDataAgent 50      

nClusterMount   Virtual Server 1

nIscsiEnable iDataAgent 1

nRescanAllHBA Virtual Server BOOLEAN true

sSNAP_IsISCSI iDataAgent STRING Y

 

Again, thank you for all of your insights. Maybe things will be better on new hardware and after the environment is upgraded out of 11.28. The server is strong and performing better than ever. I was just looking to tidy up.


Reply


Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings