Hi Team,
My DDB Backup operations are failing with the following error message: Snap creation failed on volumes holding DDB paths.
A quick review of the job logs points to insufficient space (0 extents). What could this mean?
Regards
Winston
Best answer by Jon Vengust
View original
+5
Identifying this issue may be somewhat hard on face value as the error appears generic when reported to our Job Manager service (the error you see within the Commcell Console).
This error is occurring at the MediaAgent level and as such, we'll need access for further review.
Evidence/Logs
When troubleshooting this particular issue, we need to review the clBackupParent.log on the MA to pinpoint root cause. Most likely, we'll observe log entries as per below upon snapshot creation:
/var/log/commvault/Log_Files/clBackupParent.log (default log location unless specified otherwise)
126243 1ed23 11/26 18:00:25 323114 DDBBackupManager::SnapVolumes(336) - SnapAndMount of [/dev/mapper/data-lvol0] failed. Error [Failed to create snapshot of device <DDB_DEVICE_PATH> with name DDBSnap_<SNAPSHOT_ID> and size 4096MB: Volume group "<VOLUME_GROUP>" has insufficient free space (0 extents): 64 required.
We're concerned with the extents provisioned for the volume group being insufficient in accommodating DDB snapshot growth. This can be confirmed by reviewing the Total PE, Alloc PE and Free PE values outputted in the following command: vgdisplay <VOLUME_GROUP>
Sample output
# vgdisplay <VOLUME_GROUP>
--- Volume group ---
VG Name <VOLUME_GROUP>
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 16937
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 1
Open LV 1
Max PV 0
Cur PV 1
Act PV 1
VG Size <500.00 GiB
PE Size 4.00 MiB
Total PE 127999
Alloc PE / Size 127999 / <500.00 GiB
Free PE / Size 0 / 0
VG UUID <UUID_STRING>
Guide
Total PE = Total number of physical extents for this volume group
Alloc PE = Allocated number of physical extents for this volume group
Free PE = Free number of physical extents for this volume group
As stated in the clBackupParent.log entries, we require enough extents available (as per the Required value) to take the snapshot.
Resolution
Until this problem is resolved, we’ll be unable to take a traditional DDB Backup with the current configuration.
In the above sample output, we can see that 127999 of 127999 extents have been provisioned and we require more. Your Linux team should be contacted hereafter in order to extend this volume group in accordance with how many extents are required in the reported log output (clBackupParent.log). Alternatively, there are several guides online to assist with this matter (ie. HowtoForge). With this performed, re-run the DDB Backup operation and report your findings.
If the issue does persist further please seek Commvault Support and raise a ticket for a deeper look. Please let me know if you have any further queries/concerns with the above information. Otherwise, if you found this guide helpful, don’t forget to upvote this post for visibility!
Hi Jon
Thank you very much for the detailed answer ! Great to explanation on how DDB structure works for Linux MA
Regards
WW
Identifying this issue may be somewhat hard on face value as the error appears generic when reported to our Job Manager service (the error you see within the Commcell Console).
This error is occurring at the MediaAgent level and as such, we'll need access for further review.
Evidence/Logs
When troubleshooting this particular issue, we need to review the clBackupParent.log on the MA to pinpoint root cause. Most likely, we'll observe log entries as per below upon snapshot creation:
/var/log/commvault/Log_Files/clBackupParent.log (default log location unless specified otherwise)
126243 1ed23 11/26 18:00:25 323114 DDBBackupManager::SnapVolumes(336) - SnapAndMount of [/dev/mapper/data-lvol0] failed. Error [Failed to create snapshot of device <DDB_DEVICE_PATH> with name DDBSnap_<SNAPSHOT_ID> and size 4096MB: Volume group "<VOLUME_GROUP>" has insufficient free space (0 extents): 64 required.
We're concerned with the extents provisioned for the volume group being insufficient in accommodating DDB snapshot growth. This can be confirmed by reviewing the Total PE, Alloc PE and Free PE values outputted in the following command: vgdisplay <VOLUME_GROUP>
Sample output
# vgdisplay <VOLUME_GROUP>
--- Volume group ---
VG Name <VOLUME_GROUP>
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 16937
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 1
Open LV 1
Max PV 0
Cur PV 1
Act PV 1
VG Size <500.00 GiB
PE Size 4.00 MiB
Total PE 127999
Alloc PE / Size 127999 / <500.00 GiB
Free PE / Size 0 / 0
VG UUID <UUID_STRING>
Guide
Total PE = Total number of physical extents for this volume group
Alloc PE = Allocated number of physical extents for this volume group
Free PE = Free number of physical extents for this volume group
As stated in the clBackupParent.log entries, we require enough extents available (as per the Required value) to take the snapshot.
Resolution
Until this problem is resolved, we’ll be unable to take a traditional DDB Backup with the current configuration.
In the above sample output, we can see that 127999 of 127999 extents have been provisioned and we require more. Your Linux team should be contacted hereafter in order to extend this volume group in accordance with how many extents are required in the reported log output (clBackupParent.log). Alternatively, there are several guides online to assist with this matter (ie. HowtoForge). With this performed, re-run the DDB Backup operation and report your findings.
If the issue does persist further please seek Commvault Support and raise a ticket for a deeper look. Please let me know if you have any further queries/concerns with the above information. Otherwise, if you found this guide helpful, don’t forget to upvote this post for visibility!
It is a great explanation and especially you provided with live which provide clarity when I matching my setup
Enter your username or e-mail address. We'll send you an e-mail with instructions to reset your password.
