commvault.com commvault.com
Solved

Backup on Red Hat Enterprise Virtualization Management (RHEV-M)

I
Rank
Userlevel 1
Badge +6

Hi,

has anyone implemented vm backups in rhev-m virtualized environment ? 

Thanks

icon

Best answer by ITStorageEWL 23 March 2021, 16:40

View original

21 replies

A
Rank
Userlevel 2
Badge +3

we have many customers using that..do you have any specific questions or concerns?

I
Rank
Userlevel 1
Badge +6

Good,

i have 2 question/problems.
i have two environments rhev-m one 4.1 and one 4.3.
In the first environment commvault send command to rhev-m to delete snapshot, but the rhev-m fails and in some case force restart node. This envirnment is in dismission (backup suspended). In the other env. in some case have the same problem whtout  reboot node, but is not possible delete the snapshot in any way (opened case to redhat). The vm backup whit this snapshot not work.

Thanks Andrea

Damian Andre
Rank
Userlevel 7
Badge +23

Good,

i have 2 question/problems.
i have two environments rhev-m one 4.1 and one 4.3.
In the first environment commvault send command to rhev-m to delete snapshot, but the rhev-m fails and in some case force restart node. This envirnment is in dismission (backup suspended). In the other env. in some case have the same problem whtout  reboot node, but is not possible delete the snapshot in any way (opened case to redhat). The vm backup whit this snapshot not work.

Thanks Andrea

Hey @ITStorageEWL, sorry to hear that you are having issues.

One quick thing to check - Are you using linux proxies? If so, is lvmetad disabled?

 

I
Rank
Userlevel 1
Badge +6

yes, using linux proxies and lvmetad is disabled.

Damian Andre
Rank
Userlevel 7
Badge +23

yes, using linux proxies and lvmetad is disabled.

Got it - thank you.

When no backups are running, check the proxy to see if it has any additional disks attached from other VMs - that could cause snapshots to fail to be removed. If so, remove the disks (detach, not delete) and attempt the snapshot deletion again. Beyond that, I know there were some specific issues in early RHV releases that caused this problem, but I am not aware of issues with 4.1+ It will be interesting to hear what Red Hat find?

The vsbkp.log on the proxy should have more info about the snapshot delete requests.

 

I
Rank
Userlevel 1
Badge +6

when the backup starts there are no snapshots hanging on the proxy, from the log you can see that the backup starts and wants to delete this snapshot on the rhev-m that you cannot delete in any way.
We opened a case to red-hat, unfortunately they are not as fast as cv support.

 

3687718 384559 03/08 00:10:52 94675 RhevSDKWrapper::CVRhevOperations::pollForRhevJobCompletion() - job [98218c82-241a-4ad2-8d8c-eb584bbb05ce]:[Removing Snapshot _GX_BACKUP_xxxipa4.it.xxxxxxxx.local_92072_1868119_vsarhevprinfra of VM xxxipa4.it.xxxxxxxx.local] - current state [started] - waiting for [FINISHED]...
3687718 384559 03/08 00:11:02 94675 RhevSDKWrapper::CVRhevOperations::pollForRhevJobCompletion() - job [98218c82-241a-4ad2-8d8c-eb584bbb05ce]:[Removing Snapshot _GX_BACKUP_xxxipa4.it.xxxxxxxx.local_92072_1868119_vsarhevprinfra of VM xxxipa4.it.xxxxxxxx.local] - current state [started] - waiting for [FINISHED]...
3687718 384559 03/08 00:11:12 94675 RhevSDKWrapper::CVRhevOperations::pollForRhevJobCompletion() - job [98218c82-241a-4ad2-8d8c-eb584bbb05ce]:[Removing Snapshot _GX_BACKUP_xxxipa4.it.xxxxxxxx.local_92072_1868119_vsarhevprinfra of VM xxxipa4.it.xxxxxxxx.local] - current state [failed] - waiting for [FINISHED]...
 

Damian Andre
Rank
Userlevel 7
Badge +23

when the backup starts there are no snapshots hanging on the proxy, from the log you can see that the backup starts and wants to delete this snapshot on the rhev-m that you cannot delete in any way.
We opened a case to red-hat, unfortunately they are not as fast as cv support.

 


 

Hey @ITStorageEWL - We’re the vendor to beat on support response ;)

Did you get anywhere with Red Hat? from the logs it does look like a failure to remove on their side, so I think you did the right thing reaching out to them.

Mike Struening RETIRED
Rank
Userlevel 7
Badge +23

@ITStorageEWL , following up on @Damian Andre’s note.  Any update from Red Hat support?

https://www.linkedin.com/in/michael-struening
I
Rank
Userlevel 1
Badge +6

For this moment only steps to manual remove snapshot, we are waiting for root cause.

Mike Struening RETIRED
Rank
Userlevel 7
Badge +23

Appreciate it.  I’ll check back with you occassionally to see (barring you beating me to the punch :sweat_smile: )

https://www.linkedin.com/in/michael-struening
Mike Struening RETIRED
Rank
Userlevel 7
Badge +23

@ITStorageEWL , gentle follow up to see if Red Hat support were able to give you a solution yet.

https://www.linkedin.com/in/michael-struening
I
Rank
Userlevel 1
Badge +6

Red hat support only indicated how to force the removal of snapshots from the system (VM guest). 
From our analysis the problem seems to have been caused by backup RHEV-M (on VMware ) with snapshot. this backups scheduled during VM guest backups with VSA. We have changed the type of backup foe RHEV-M (with agent File System) and the scheduling.

Mike Struening RETIRED
Rank
Userlevel 7
Badge +23

Understood, and thanks for the reply!  Are the backups now completing?

https://www.linkedin.com/in/michael-struening
I
Rank
Userlevel 1
Badge +6

Yes, backup working without errors.

S
Badge

Understood, and thanks for the reply!  Are the backups now completing?

Hi Mike, we had experienced this issue as well in my environment. Unfortunately for us we cannot do file system agent backup because it’s quite a lot of VMs(about 500) in RHEV. Has commvault found any solution to this? When we ran RHEV VM backups, the snapshots didn’t get deleted causing the VMs to freeze and quite a messy service interruption. Manual deletion of snapshots is quite a task for all the VMs. Is there any known issue on this.

I
Rank
Userlevel 1
Badge +6

Hi @Shirley , what version of rhev-m do you have installed?

I
Rank
Userlevel 1
Badge +6

@Shirley With vm backups with VSA on rhev-m 4.1 old infrastructure and rhev-m 4.3 new, we had to change the vm type from thin to thick. This is because of how red hat handles snapshots.

S
Badge

@Shirley With vm backups with VSA on rhev-m 4.1 old infrastructure and rhev-m 4.3 new, we had to change the vm type from thin to thick. This is because of how red hat handles snapshots.

Hi @ITStorageEWL We are running 4.3.10. So we have to do this for all the VMs? We have about 500 of them. Are the RHEV VM backups successful now?

S
Badge

@Shirley With vm backups with VSA on rhev-m 4.1 old infrastructure and rhev-m 4.3 new, we had to change the vm type from thin to thick. This is because of how red hat handles snapshots.

Also which Commvault Service Pack are you on? Previously support had told us it’s a service pack support problem but I don’t think this was the cause. I’m currently on 20.53, I last attempted the backup while on SP17.

I
Rank
Userlevel 1
Badge +6

@Shirley With vm backups with VSA on rhev-m 4.1 old infrastructure and rhev-m 4.3 new, we had to change the vm type from thin to thick. This is because of how red hat handles snapshots.

Also which Commvault Service Pack are you on? Previously support had told us it’s a service pack support problem but I don’t think this was the cause. I’m currently on 20.53, I last attempted the backup while on SP17.

11.20.32

I
Rank
Userlevel 1
Badge +6

@Shirley With vm backups with VSA on rhev-m 4.1 old infrastructure and rhev-m 4.3 new, we had to change the vm type from thin to thick. This is because of how red hat handles snapshots.

Hi @ITStorageEWL We are running 4.3.10. So we have to do this for all the VMs? We have about 500 of them. Are the RHEV VM backups successful now?

we save 250 or so VMs with VSA on rhev, now we have no problems. We certainly allocated a lot more space this way. Make some tests.

Reply


Terms & ConditionsCookie settings