Skip to main content
commvault.com commvault.com

Hello Commvaunt Community, 

 

I need your opinion and support regarding the backup problem for two virtual machines.

The client claims that after updating the environment to version 11.25.6, he began to observe a performance problem for one VM and the other cannot be backed up.

I went down to the logs and for the task, it contains several VMs (but the problem is with one, the task takes 3 days) I found entries that are looped but I do not know what they can mean.

 

@MediaAgent6 - VixDiskLib.log 

34680 6878 11/22 10:57:41 3185939 VDDK_PhoneHome: VddkVacWriteLastMember : Empty field to serialize for key fail_function at line 271.
34680 6878 11/22 10:57:42 3185939 VDDK_PhoneHome: Curl perform failed in VddkVacCurlHttpsPost with error code 7 at 1780.
34680 6878 11/22 10:57:45 3185939 VDDK_PhoneHome: VddkVacPushData : Failed to post data to VC/PH at line 1719.

---

@MediaAgent6 - vsbkp.log

34680 67b0  11/23 21:03:28 3185939 CVmdkVcb4ExtentReader::ReadBlockFromReadAheadCache() - Delay while looking for block l144745]
34680 3b68  11/23 21:03:34 3185939 VSBkpController::MonitorProcesses() - Detected that other 5 vsbkp processes now have 3 streams running
34680 8af8  11/23 21:03:43 3185939 VSBkpDatastoreMonitor::ReportDatastore() - Datastore aESX-DMZ-APP-PROD-DS04] -821,008,793,600] bytes free s12.45%]  Update Interval n120] seconds  e1] VMs
34680 67b0  11/23 21:03:43 3185939 CVmdkVcb4ExtentReader::ReadBlockFromReadAheadCache() - Delay while looking for block f144745]
34680 3b68  11/23 21:03:44 3185939 VSBkpController::MonitorProcesses() - Detected that other 6 vsbkp processes now have 4 streams running
34680 3b68  11/23 21:03:54 3185939 VSBkpController::MonitorProcesses() - Detected that other 6 vsbkp processes now have 5 streams running
34680 67b0  11/23 21:03:58 3185939 CVmdkVcb4ExtentReader::ReadBlockFromReadAheadCache() - Delay while looking for block n144745]
34680 3b68  11/23 21:04:04 3185939 VSBkpController::MonitorProcesses() - Detected that other 6 vsbkp processes now have 11 streams running
---

 

Searching for available articles but they didn't get me on anything interesting.
Do you have any idea what could be causing the problem? If not, it escalates to Commvault support.

 

Thanks&Regards,
Kamil

Something bad happened with the attachments, I added 6 of them, and over 100 sent them. For me, they still load, so if someone can delete them then I will be grateful.

Hi @Kamil ,

 

Are you able to share a bit more of the vsbkp.log here at all? 
-The only info I can see is that we stated 3 times “Delay while looking for block l144745]”

Also, is this Job using SAN transport for the VM Backups?
 

When looing at the logs you can filter for the “Stat-” counter which can show the performance of the Read/Pipeline operations here.

At the end of a Job you will see counters with “Name tHead ]” for Network, Deduplication and Compression. - When looking at those, look for a high “Procsng Time”.

 

Best Regards,

Michael

HI @Kamil 

Is this happening for a single Datastore/ESX?

Which Transport Mode is that Backup using?

Would it be possible to svmotion one VM and run a Backup?

Let us know

Best Regards,

Sebastien Merluzzi

Hello @MichaelCapon @Sebastien Merluzzi 

 

I will add again the screenshots that I tried to send yesterday. There is a little more information on them, in the .zip package I send some logs that contain information for the problematic job (JobID: 3185939). They repeat the logs I pasted in the previous message, but maybe I missed something.

 

The screenshots show that there is a problem with the two vms at the top. The rest of the vms is working fine.

 

If I can ask for your help, I will be extremely grateful

 

Regards,
Kamil

Thanks @Kamil ,

I can see media5 and media6 have SAN access and that’s the method being used for this Job.

-As per Sebastien’s comment above, Would it be possible to svmotion that VM and run another Backup?

 

Can you confirm what version the Client upgraded from please?

Do you have any earlier vsbkp logs from media6 at all? - Also is the vsbkp PID 34680 still running on media6?

 

Best Regards,

Michael

Hello @MichaelCapon @Sebastien Merluzzi 

 

Could you please let me know if the "svmotion" that Sebastien mentioned refers to this Commvault documentation article?

 

VMW0067: Change tracking for Virtual Machine hVMservername] has reported an error (VMware)

https://documentation.commvault.com/11.24/expert/32675_vmw0067_change_tracking_for_virtual_machine_vmservername_has_reported_error_vmware.html

 

Is the procedure good and that is what we should do now?

 

"Can you confirm what version the Client upgraded from please?" -

The client does not know if this is the direct cause, but said "There was a problem, probably after some update, that some virtual machines do not want to back up. Once, when I had such a problem, it was enough to remove the speed on the media agent."

He is unable to pinpoint after which update the problem started to occur, but has recently been upgrading to SP25.

 

"Do you have any earlier vsbkp logs from MediaAgent6 at all? - Also is the vsbkp PID 34680 still running on MediaAgent6?" - In the .zip package I am sending all vsbkp.log logs from MediaAgent6 that I have.

 

Thanks&Regards,
Kamil

Hi,

 

Can I ask you for help with the latest message?

 

Regards,
Kamil

HI @Kamil 

 

Storage vMotion is a VMWare Operation

https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.vcenterhost.doc/GUID-AB266895-BAA4-4BF3-894E-47F99DC7B77F.html

Best Regards,

Sebastien Merluzzi

Hello Commvault Community :)

 

Due to the lack of a client update, the thread has been closed. Unfortunately, I haven’t information on what the matter with the described problem looked like.

Maybe someday someone will find this thread useful for troubleshooting :)

 

Regards,
Kamil

If anyone does, respond here, or create a new thread referencing this one!

Reply


Terms & ConditionsCookie settings