Skip to main content

Hi,

has anyone implemented vm backups in rhev-m virtualized environment ? 

Thanks

we have many customers using that..do you have any specific questions or concerns?


Good,

i have 2 question/problems.
i have two environments rhev-m one 4.1 and one 4.3.
In the first environment commvault send command to rhev-m to delete snapshot, but the rhev-m fails and in some case force restart node. This envirnment is in dismission (backup suspended). In the other env. in some case have the same problem whtout  reboot node, but is not possible delete the snapshot in any way (opened case to redhat). The vm backup whit this snapshot not work.

Thanks Andrea


Good,

i have 2 question/problems.
i have two environments rhev-m one 4.1 and one 4.3.
In the first environment commvault send command to rhev-m to delete snapshot, but the rhev-m fails and in some case force restart node. This envirnment is in dismission (backup suspended). In the other env. in some case have the same problem whtout  reboot node, but is not possible delete the snapshot in any way (opened case to redhat). The vm backup whit this snapshot not work.

Thanks Andrea

Hey @ITStorageEWL, sorry to hear that you are having issues.

One quick thing to check - Are you using linux proxies? If so, is lvmetad disabled?

 


yes, using linux proxies and lvmetad is disabled.


yes, using linux proxies and lvmetad is disabled.

Got it - thank you.

When no backups are running, check the proxy to see if it has any additional disks attached from other VMs - that could cause snapshots to fail to be removed. If so, remove the disks (detach, not delete) and attempt the snapshot deletion again. Beyond that, I know there were some specific issues in early RHV releases that caused this problem, but I am not aware of issues with 4.1+ It will be interesting to hear what Red Hat find?

The vsbkp.log on the proxy should have more info about the snapshot delete requests.

 


when the backup starts there are no snapshots hanging on the proxy, from the log you can see that the backup starts and wants to delete this snapshot on the rhev-m that you cannot delete in any way.
We opened a case to red-hat, unfortunately they are not as fast as cv support.

 

3687718 384559 03/08 00:10:52 94675 RhevSDKWrapper::CVRhevOperations::pollForRhevJobCompletion() - job 98218c82-241a-4ad2-8d8c-eb584bbb05ce]::Removing Snapshot _GX_BACKUP_xxxipa4.it.xxxxxxxx.local_92072_1868119_vsarhevprinfra of VM xxxipa4.it.xxxxxxxx.local] - current state started] - waiting for FINISHED]...
3687718 384559 03/08 00:11:02 94675 RhevSDKWrapper::CVRhevOperations::pollForRhevJobCompletion() - job 98218c82-241a-4ad2-8d8c-eb584bbb05ce]::Removing Snapshot _GX_BACKUP_xxxipa4.it.xxxxxxxx.local_92072_1868119_vsarhevprinfra of VM xxxipa4.it.xxxxxxxx.local] - current state started] - waiting for FINISHED]...
3687718 384559 03/08 00:11:12 94675 RhevSDKWrapper::CVRhevOperations::pollForRhevJobCompletion() - job 98218c82-241a-4ad2-8d8c-eb584bbb05ce]::Removing Snapshot _GX_BACKUP_xxxipa4.it.xxxxxxxx.local_92072_1868119_vsarhevprinfra of VM xxxipa4.it.xxxxxxxx.local] - current state failed] - waiting for FINISHED]...
 


when the backup starts there are no snapshots hanging on the proxy, from the log you can see that the backup starts and wants to delete this snapshot on the rhev-m that you cannot delete in any way.
We opened a case to red-hat, unfortunately they are not as fast as cv support.

 


 

Hey @ITStorageEWL - We’re the vendor to beat on support response ;)

Did you get anywhere with Red Hat? from the logs it does look like a failure to remove on their side, so I think you did the right thing reaching out to them.


@ITStorageEWL , following up on @Damian Andre’s note.  Any update from Red Hat support?


For this moment only steps to manual remove snapshot, we are waiting for root cause.


Appreciate it.  I’ll check back with you occassionally to see (barring you beating me to the punch :sweat_smile: )


@ITStorageEWL , gentle follow up to see if Red Hat support were able to give you a solution yet.


Red hat support only indicated how to force the removal of snapshots from the system (VM guest). 
From our analysis the problem seems to have been caused by backup RHEV-M (on VMware ) with snapshot. this backups scheduled during VM guest backups with VSA. We have changed the type of backup foe RHEV-M (with agent File System) and the scheduling.


Understood, and thanks for the reply!  Are the backups now completing?


Yes, backup working without errors.


Understood, and thanks for the reply!  Are the backups now completing?

Hi Mike, we had experienced this issue as well in my environment. Unfortunately for us we cannot do file system agent backup because it’s quite a lot of VMs(about 500) in RHEV. Has commvault found any solution to this? When we ran RHEV VM backups, the snapshots didn’t get deleted causing the VMs to freeze and quite a messy service interruption. Manual deletion of snapshots is quite a task for all the VMs. Is there any known issue on this.


Hi @Shirley , what version of rhev-m do you have installed?


@Shirley With vm backups with VSA on rhev-m 4.1 old infrastructure and rhev-m 4.3 new, we had to change the vm type from thin to thick. This is because of how red hat handles snapshots.


@Shirley With vm backups with VSA on rhev-m 4.1 old infrastructure and rhev-m 4.3 new, we had to change the vm type from thin to thick. This is because of how red hat handles snapshots.

Hi @ITStorageEWL We are running 4.3.10. So we have to do this for all the VMs? We have about 500 of them. Are the RHEV VM backups successful now?


@Shirley With vm backups with VSA on rhev-m 4.1 old infrastructure and rhev-m 4.3 new, we had to change the vm type from thin to thick. This is because of how red hat handles snapshots.

Also which Commvault Service Pack are you on? Previously support had told us it’s a service pack support problem but I don’t think this was the cause. I’m currently on 20.53, I last attempted the backup while on SP17.


@Shirley With vm backups with VSA on rhev-m 4.1 old infrastructure and rhev-m 4.3 new, we had to change the vm type from thin to thick. This is because of how red hat handles snapshots.

Also which Commvault Service Pack are you on? Previously support had told us it’s a service pack support problem but I don’t think this was the cause. I’m currently on 20.53, I last attempted the backup while on SP17.

11.20.32


@Shirley With vm backups with VSA on rhev-m 4.1 old infrastructure and rhev-m 4.3 new, we had to change the vm type from thin to thick. This is because of how red hat handles snapshots.

Hi @ITStorageEWL We are running 4.3.10. So we have to do this for all the VMs? We have about 500 of them. Are the RHEV VM backups successful now?

we save 250 or so VMs with VSA on rhev, now we have no problems. We certainly allocated a lot more space this way. Make some tests.


Reply