Skip to main content
Solved

Mount Path does not have enough space/Disk Library running out of space

  • 25 October 2021
  • 10 replies
  • 3022 views

Hi all,

after some time we are facing another serious issue. There is no available space on the disk library. Ayayay.

We have tried to find if there are any unprunable jobs. There were some of them and therefore we have set option to ignore cycle retention for disable subclients. Unfortunately, only small amount of GBs have been aged.

 

Now, there is a question what to do next. I have no idea how to find what can be deleted in order to make more free space for the backups. Moreover, there will be quit a big deduplication ration, so even manual deletion of some jobs do not have to be useful. Maybe, one useful information can be that during the last month there were increase in data cca 10TB, which is 10 percent increase. Is there possibility to figure out what data did this increase?

 

Is there any generule rule or useful tool within the Commvault to fight with this issue?

10 replies

Mike Struening RETIRED
Rank
Userlevel 7
Badge +23

There is!

you can run my favorite report called the Data Retention Forecast and Compliance report on the library low on space.  This will show you ALL of the unaged jobs, when they will prune (and why) and even give some handy stats on the deduplication engines.

As you mentioned, if your dedupe ratio is really high, you likely won’t get much back (though conversely, new backups don’t need much).

Run that and share what you see (there’s generally a theme across the older jobs).  If space has crept up, this will likely solve the issue.

Now, if you noticed a large jump in space usage, it’s possible you sealed a store and have started a new baseline though you’ll see multiple stores for the same Policy at the bottom of the report.

Finally, it’s possible that data is not physically aging (and logical retention is not the issue).  We would want to see what the report shows, though it’s also possible to look at the SIDBPhysicalDeletes.log file via GxTail.exe to see if pruning is happening or not.

All Aging issues start with the above report, so let me know what you see :nerd:

https://www.linkedin.com/in/michael-struening
D
Rank
Userlevel 4
Badge +14

There is!

you can run my favorite report called the Data Retention Forecast and Compliance report on the library low on space.  This will show you ALL of the unaged jobs, when they will prune (and why) and even give some handy stats on the deduplication engines.

As you mentioned, if your dedupe ratio is really high, you likely won’t get much back (though conversely, new backups don’t need much).

Run that and share what you see (there’s generally a theme across the older jobs).  If space has crept up, this will likely solve the issue.

Now, if you noticed a large jump in space usage, it’s possible you sealed a store and have started a new baseline though you’ll see multiple stores for the same Policy at the bottom of the report.

Finally, it’s possible that data is not physically aging (and logical retention is not the issue).  We would want to see what the report shows, though it’s also possible to look at the SIDBPhysicalDeletes.log file via GxTail.exe to see if pruning is happening or not.

All Aging issues start with the above report, so let me know what you see :nerd:

 

Hi @Mike Struening , thanks for the answer! As you advised me I have run the Data Retention Forecast and Compliance report. Unfortunately, there seems to be no obvisious problem with unaged jobs. Almost all jobs are waiting for aging according to their retention rules.

Furthermore, it is possible that the data increase is caused by moving some policy copies from one library to the another one. However, it is hard to verify this assumption.

 

As a workaround, it was decided to find an old storage device with a little more free storage space, but I am pretty sure it will be time ticking bomb.

 

Mike Struening RETIRED
Rank
Userlevel 7
Badge +23

Valid concern, especially if it is older….

if the jobs all appear to be unaged based on your retention, it’s possible you have issues with the Dedupe stores, which should show up at the bottom of the report.

I don’t want to dismiss the possibility of either problematic stores/aging issues, or additional baselines.

Did you see anything in sidbprune, sidbphysicaldeletes, or sidbengine.log?

https://www.linkedin.com/in/michael-struening
Damian Andre
Rank
Userlevel 7
Badge +23

What type of storage do you have @drPhil - i.e type and brand (NAS/Isilon)

Mike Struening RETIRED
Rank
Userlevel 7
Badge +23

Hi @drPhil , hope all is well!

How are things looking in regards to space?  Have you had a chance to review the sidbprune, sidbphysicaldeletes, or sidbengine.log files, or look at the storage type?

Possible that there’s a drilling holes issue if your library doesn’t support it (where we delete partial chunk files to save space, if your library doesn’t support that feature, then you won’t get back space until the whole chunk is prunable).

https://www.linkedin.com/in/michael-struening
Scott Moseman
Rank
Userlevel 6
Badge +18

Maybe, one useful information can be that during the last month there were increase in data cca 10TB, which is 10 percent increase. Is there possibility to figure out what data did this increase?


Subclient Growth
https://cloud.commvault.com/webconsole/softwarestore/#!/135/665/11301

This report is for App Size, not Data Written, but I have used this to help find Subclients have grown the most over a certain period of time.  Maybe it can help find where the growth originated.

Thanks,
Scott
 

Commvault Enterprise Success
D
Rank
Userlevel 4
Badge +14

Hi @Mike Struening , @Scott Moseman@Scott Moseman !

 

Thanks for interesting ideas!

 

In sidbprune, sidbphysicaldeletes, or sidbengine.log files I have found no error strings or something similar. Everything looks as usual there.

 

However, in Subclient Growth report I have found that one database client (Oracle) is growing very fast. This report displays last 30 days

 

 

Do you think that it is possible that this grow could be caused by that sublient?

Scott Moseman
Rank
Userlevel 6
Badge +18

 

 

Do you think that it is possible that this grow could be caused by that sublient?


I cannot say conclusively, but that is quite a bit of growth for 30 days.

Thanks,
Scott

Commvault Enterprise Success
D
Rank
Userlevel 4
Badge +14

Hi @Mike Struening , @Scott Moseman@Damian Andre 

 

The solution was neither pretty nor creative… We have increased disk space capacity with another disk array device. And after some time, suddenly, there was a huge data aging and now the disk space is no issue.

 

Thanks to all for your support!

Mike Struening RETIRED
Rank
Userlevel 7
Badge +23

Sometimes that’s the answer!  Years ago I had a checklist for engineers to refer to for ‘out of disk space’ issues, and one of the items was ‘add more space’.  The last option, with heavy warning was to delete space.

It’s easy to remove things, nearly impossible to put them back!

https://www.linkedin.com/in/michael-struening

Reply


Terms & ConditionsCookie settings