Skip to main content
Solved

Low disk space on client - Information on what/how to make space


Forum|alt.badge.img+3

Hello,

 

One of the Media Agents keeps running out of space and I am trying to determine why this is happening and what else stored on this particular drive can be deleted.

 

 

I believe that this is caused by new content being written to our DR_Backup folder as it is over 400GB (drive is only 500GB).

 

 

 

I would like to know what they files are for and to what extent they can be reduced / deleted.

 

Additionally there is another folder on the same drive that I believe can be deleted but I wanted to double-check. The folder “CV_Install” looks like it contains the installer/setup from when our backup server was implemented (before my time) but pressing delete on folders without being sure is a good way to have a bad day.

 

 

This might be related to another issue I need to chase up where Disaster jobs are failing but I can’t determine why.

 

 

I inherited responsibility of our Commvault system late late year without any formal training so please excuse any noob questions.

 

Thanks

 

Best answer by Kegan Levett

So the issue has been resolved thanks to help from Commvault Support.

 

A network change prevented our Azure Storage (Storage Resources>Libraries>Azure Storage) from communicating down/up to our server. In Azure itself adding the IP range to the Firewall settings to allow access from the internet was all that was needed. The SET files removed themselves as well, so plenty of space on the drive.

 

Thanks for the help.

View original
Did this answer your question?
If you have a question or comment, please create a topic

14 replies

Chris Hollis
Vaulter
Forum|alt.badge.img+15

Hi @Kegan Levett 
 

As per your DR backup settings, you should only be retaining a maximum of 7 DR backup sets in that directory.


By default you run 1 x DR backup per day, so in theory, you should only see the last weeks worth of CS DB backups in that directory.


Based on your screenshot, you have 624 items… are all these items SET folders? if you sort by date modified, how old is the oldest SET folder?

I suspect what is happening here is that as a result of your DR backups ‘completing with errors’ we are retaining every backup until one completes successfully (this is based on this article: https://documentation.commvault.com/11.24/expert/11965_impact_of_jobs_that_completed_with_errors.html - note ‘SQL Server’ does not count a ‘CWE’ job as a FULL for next cycle during data aging).

We need to determine why these jobs are completing with errors.. so if you could please view admin job history as per the procedure here: https://documentation.commvault.com/11.24/expert/6444_viewing_admin_job_history.html

Select Disaster Recovery Backup and the view the results > please share some screenshots on the job details / attempts / status tabs. 

Feel free to right click the job > view logs and share (attach) as a txt file for analysis (please cross check/remove any sensitive data). 

Additionally, with reference to this directory: 


As per the time stamps, they are all extremely old. I see no issue cleaning these up right now to get you back a little breathing space.

Additional articles for reference:
https://documentation.commvault.com/11.24/expert/43516_commserve_recovery_using_disaster_recovery_dr_backups.html

https://documentation.commvault.com/11.24/expert/43517_configuration_of_disaster_recovery_dr_backups.html

Regards,

Chris.


Forum|alt.badge.img+3

Hi @Chris Hollis,

 

Firstly, yes they are all SET folders, going back to the 18th of November. It looks like five or six are generated per day judging by the “date modified” field, which makes sense as the DR tape is written to every four hours

 

These are the results of the DR admin job history:

 

I have uploaded two log files as 2365111 actually displays the error code [72:53]

 

Thanks for your help so far.


Chris Hollis
Vaulter
Forum|alt.badge.img+15

@Kegan Levett 

Going to need you to get me logs from the most recent job that has run. 

Whe viewing the logs from the GUI, make sure you hit the ‘select all’ button in the log pane so that we get everything. 


Forum|alt.badge.img+3

@Chris Hollis 

 

No worries, let’s try again.


Chris Hollis
Vaulter
Forum|alt.badge.img+15

@Kegan Levett 

Looks like we’re having issues uploading the DB to cloud.
 

File    : CloudActivity-CommServeDR.log
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
14512 9270  03/03 08:01:47 2368176 CVRESTBaseRemoteFile::SendRequestRetry() - Error:  Message: This request is not authorized to perform this operation.
RequestId:83fb2dbb-401e-0025-7d78-2e6557000000
Time:2022-03-02T21:01:47.2829591Z
14512 91b4  03/03 08:01:47 2368176 CVRESTBaseRemoteFile::SendRequestRetry() - Error:  Message: This request is not authorized to perform this operation.
RequestId:1dba4c04-401e-001a-3b78-2eadf4000000
Time:2022-03-02T21:01:47.2825794Z
14512 9270  03/03 08:01:47 2368176 SendSinglePart() - failed to send the remote file - DR/<commserve>/SET_8416/DM2_<commserve>_2022_03_03_08_00_FULL.dmp
14512 91b4  03/03 08:01:47 2368176 SendSinglePart() - failed to send the remote file - DR/<commserve>/SET_8416/AppStudioDB_<commserve>_2022_03_03_08_00_FULL.dmp
14512 9270  03/03 08:01:47 2368176 HandleSendFileThread() - failed for DR/<commserve>/SET_8416/DM2_<commserve>_2022_03_03_08_00_FULL.dmp
14512 91b4  03/03 08:01:47 2368176 HandleSendFileThread() - failed for DR/<commserve>/SET_8416/AppStudioDB_<commserve>_2022_03_03_08_00_FULL.dmp
14512 8b6c  03/03 08:01:47 ####### Unknown error :AuthorizationFailure
14512 8b6c  03/03 08:01:47 2368176 CVRESTBaseRemoteFile::SendRequestRetry() - Error:  Message: This request is not authorized to perform this operation.
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@


File    : CloudFileAccess.log
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
14512 73e0  03/03 08:01:47 2368176 CloseFile() - commvault/DR/<commserve>/SET_8416/AppStudioDB_<commserve>_2022_03_03_08_00_FULL.dmp, error =  Message: This request is not authorized to perform this operation.
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
 

File    : commserveDR.log
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

14512 5300  03/03 08:01:47 2368176 CommservDR::uploadFileToCloud() - Error: Failed to upload file [C:\Program Files\Commvault\ContentStore\CommserveDR\DR_2368176\DM2_<commserve>_2022_03_03_08_00_FULL.dmp] on cloud [commvault/DR/<commserve>/SET_8416/DM2_<commserve>_2022_03_03_08_00_FULL.dmp]. Error Message:[44103 :  Message: This request is not authorized to perform this operation.
 

File    : JobManager.log
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
7496  8a88  03/03 08:08:57 2368176             JobSvr Obj Copy to Staging location completed.
7496  8a88  03/03 08:08:57 2368176             Scheduler   -> COMPLETED WITH ERRORS/WARNINGS <-  Completed JobType[Disaster Recovery Backup] CompletionStatus[With Errors]


All other backup phases indicate job success.

What feature realease/maintenance release do you have installed in your environment at the moment? (right click commserve > properties > version tab).

If you disable the cloud log upload feature, can you monitor jobs and see if they succeed?
 

Instead of select, deselect this option. 

If jobs succeed, either there is a prerequisite that is missed from this article: 
https://documentation.commvault.com/11.24/expert/115116_uploading_dr_backups_to_commvault_cloud_services_portal.html or there may be a fix you need to install to rectify.

I’ll know more once you can confirm the above for me.

 

Thanks

Chris.
 


Forum|alt.badge.img+3

@Chris Hollis 

 

Version is 11.20.9

 

I have unselected the “Upload backup metadata” tickbox. The next scheduled run is at 4pm (currently 1:20pm). What is the best way to run a backup immediately?

 

Thanks again


Chris Hollis
Vaulter
Forum|alt.badge.img+15

@Kegan Levett 

Instructions are here on performing a backup: https://documentation.commvault.com/commvault/v11_sp20/article?p=43530.htm

No need to select any other options, just hit OK once done.

Additionally, we will need to look at getting your environment updated... your present maintenance release is very far behind.. at 11.20.9.. we are 11.20.94!

i’d recommend downloading and installing the latest available which can be found here:
https://cloud.commvault.com/webconsole/downloadcenter/packageDetails.do?packageId=22468&status=0&type=details

Installation instructions: https://documentation.commvault.com/commvault/v11_sp20/article?p=2671.htm

Maintenance release info: https://documentation.commvault.com/commvault/v11_sp20/article?p=124866.htm


Let me know how the DR backup goes without cloud enabled.

 

Regards,

Chris


Forum|alt.badge.img+3

@Chris Hollis 

 

Still have issues with the DR backup. Haven’t done the update yet, I’ll have to read through the procedure first.

Also attached the log file for the job.

 

 


Forum|alt.badge.img+3

Hi again,

 

Just bumping for a response. The upgrade on our Commserve / Media agents haven’t happened yet (need to sort out a time) however clearing out the old install files has bought some time for this issue to be resolved.

 

Cheers


Mike Struening
Vaulter
Forum|alt.badge.img+23

@Kegan Levett , it could be a permissions issue getting your Commserve access to the cloud.

I would suggest upgrading ASAP as you are barely patched at all and it’s the first thing support will ask :nerd:

If that doesn’t resolve the issue, call in a support case .  I’m seeing various different solutions that are contextual to each case.

Can you share the case number if you do?


Forum|alt.badge.img+3

Thanks @Mike Struening 

 

I’m going through the documentation to do the upgrade after hours, and I’m wondering if there is a rollback procedure available as a precaution? My intention is to do a snapshot of the server and media agents, just checking if there is anything else I should do.

Additionally it appears that the upgrade has attempted to run previously but failed for some reason. Is a restart required normally? My in-house doco states that Commvault caches a copy on the latest Service Pack on the Commcell server at this file location.

 

 


Mike Struening
Vaulter
Forum|alt.badge.img+23

At worst, if the whole installation binary folder is problematic, you can always wipe out the install and restore the database.

The only thing you want to ensure you save on the Commserve is a copy of the Commserve database (on another drive or location altogether).


Forum|alt.badge.img+3
  • Author
  • Bit
  • 9 replies
  • Answer
  • March 11, 2022

So the issue has been resolved thanks to help from Commvault Support.

 

A network change prevented our Azure Storage (Storage Resources>Libraries>Azure Storage) from communicating down/up to our server. In Azure itself adding the IP range to the Firewall settings to allow access from the internet was all that was needed. The SET files removed themselves as well, so plenty of space on the drive.

 

Thanks for the help.


Mike Struening
Vaulter
Forum|alt.badge.img+23

Thanks for sharing!


Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings