Solved

Failing DDB Backup on a Linux-based MA due to extent exhaustion

  • 8 January 2021
  • 3 replies
  • 2708 views

Userlevel 3
Badge +4

Hi Team, 

My DDB Backup operations are failing with the following error message: Snap creation failed on volumes holding DDB paths. 

A quick review of the job logs points to insufficient space (0 extents). What could this mean?  

Regards

Winston

icon

Best answer by Jon Vengust 8 January 2021, 02:30

View original

3 replies

Userlevel 3
Badge +5

Identifying this issue may be somewhat hard on face value as the error appears generic when reported to our Job Manager service (the error you see within the Commcell Console).

 

This error is occurring at the MediaAgent level and as such, we'll need access for further review.

 

Evidence/Logs

 

When troubleshooting this particular issue, we need to review the clBackupParent.log on the MA to pinpoint root cause. Most likely, we'll observe log entries as per below upon snapshot creation:

 

/var/log/commvault/Log_Files/clBackupParent.log (default log location unless specified otherwise)

 

126243 1ed23 11/26 18:00:25 323114 DDBBackupManager::SnapVolumes(336) - SnapAndMount of [/dev/mapper/data-lvol0] failed. Error [Failed to create snapshot of device <DDB_DEVICE_PATH> with name DDBSnap_<SNAPSHOT_ID> and size 4096MB:   Volume group "<VOLUME_GROUP>" has insufficient free space (0 extents): 64 required.

 

We're concerned with the extents provisioned for the volume group being insufficient in accommodating DDB snapshot growth. This can be confirmed by reviewing the Total PEAlloc PE and Free PE values outputted in the following command: vgdisplay <VOLUME_GROUP>

 

Sample output

 

# vgdisplay <VOLUME_GROUP>

 

--- Volume group ---

VG Name              <VOLUME_GROUP>

System ID

Format                lvm2

Metadata Areas        1

Metadata Sequence No  16937

VG Access             read/write

VG Status             resizable

MAX LV                0

Cur LV                1

Open LV               1

Max PV                0

Cur PV                1

Act PV                1

VG Size <500.00 GiB

PE Size 4.00 MiB

Total PE 127999

Alloc PE / Size 127999 / <500.00 GiB

Free PE / Size 0 / 0

VG UUID <UUID_STRING>

 

Guide

 

Total PE = Total number of physical extents for this volume group

Alloc PE = Allocated number of physical extents for this volume group

Free PE = Free number of physical extents for this volume group

 

As stated in the clBackupParent.log entries, we require enough extents available (as per the Required value) to take the snapshot.

 

Resolution

 

Until this problem is resolved, we’ll be unable to take a traditional DDB Backup with the current configuration.

 

In the above sample output, we can see that 127999 of 127999 extents have been provisioned and we require more. Your Linux team should be contacted hereafter in order to extend this volume group in accordance with how many extents are required in the reported log output (clBackupParent.log). Alternatively, there are several guides online to assist with this matter (ie. HowtoForge). With this performed, re-run the DDB Backup operation and report your findings.

 

If the issue does persist further please seek Commvault Support and raise a ticket for a deeper look. Please let me know if you have any further queries/concerns with the above information. Otherwise, if you found this guide helpful, don’t forget to upvote this post for visibility!

Userlevel 1
Badge +7

Identifying this issue may be somewhat hard on face value as the error appears generic when reported to our Job Manager service (the error you see within the Commcell Console).

 

This error is occurring at the MediaAgent level and as such, we'll need access for further review.

 

Evidence/Logs

 

When troubleshooting this particular issue, we need to review the clBackupParent.log on the MA to pinpoint root cause. Most likely, we'll observe log entries as per below upon snapshot creation:

 

/var/log/commvault/Log_Files/clBackupParent.log (default log location unless specified otherwise)

 

126243 1ed23 11/26 18:00:25 323114 DDBBackupManager::SnapVolumes(336) - SnapAndMount of [/dev/mapper/data-lvol0] failed. Error [Failed to create snapshot of device <DDB_DEVICE_PATH> with name DDBSnap_<SNAPSHOT_ID> and size 4096MB:   Volume group "<VOLUME_GROUP>" has insufficient free space (0 extents): 64 required.

 

We're concerned with the extents provisioned for the volume group being insufficient in accommodating DDB snapshot growth. This can be confirmed by reviewing the Total PEAlloc PE and Free PE values outputted in the following command: vgdisplay <VOLUME_GROUP>

 

Sample output

 

# vgdisplay <VOLUME_GROUP>

 

--- Volume group ---

VG Name              <VOLUME_GROUP>

System ID

Format                lvm2

Metadata Areas        1

Metadata Sequence No  16937

VG Access             read/write

VG Status             resizable

MAX LV                0

Cur LV                1

Open LV               1

Max PV                0

Cur PV                1

Act PV                1

VG Size <500.00 GiB

PE Size 4.00 MiB

Total PE 127999

Alloc PE / Size 127999 / <500.00 GiB

Free PE / Size 0 / 0

VG UUID <UUID_STRING>

 

Guide

 

Total PE = Total number of physical extents for this volume group

Alloc PE = Allocated number of physical extents for this volume group

Free PE = Free number of physical extents for this volume group

 

As stated in the clBackupParent.log entries, we require enough extents available (as per the Required value) to take the snapshot.

 

Resolution

 

Until this problem is resolved, we’ll be unable to take a traditional DDB Backup with the current configuration.

 

In the above sample output, we can see that 127999 of 127999 extents have been provisioned and we require more. Your Linux team should be contacted hereafter in order to extend this volume group in accordance with how many extents are required in the reported log output (clBackupParent.log). Alternatively, there are several guides online to assist with this matter (ie. HowtoForge). With this performed, re-run the DDB Backup operation and report your findings.

 

If the issue does persist further please seek Commvault Support and raise a ticket for a deeper look. Please let me know if you have any further queries/concerns with the above information. Otherwise, if you found this guide helpful, don’t forget to upvote this post for visibility!

It is a great explanation and especially you provided with live which provide clarity when I matching my setup

Userlevel 3
Badge +4

Hi Jon 

Thank you very much for the detailed answer ! Great to explanation on how DDB structure works for Linux MA

Regards

WW

Reply