Low disk space on client - Information on what/how to make space

Hi @Kegan Levett 
 

As per your DR backup settings, you should only be retaining a maximum of 7 DR backup sets in that directory.

By default you run 1 x DR backup per day, so in theory, you should only see the last weeks worth of CS DB backups in that directory.

Based on your screenshot, you have 624 items… are all these items SET folders? if you sort by date modified, how old is the oldest SET folder?

I suspect what is happening here is that as a result of your DR backups ‘completing with errors’ we are retaining every backup until one completes successfully (this is based on this article: https://documentation.commvault.com/11.24/expert/11965_impact_of_jobs_that_completed_with_errors.html - note ‘SQL Server’ does not count a ‘CWE’ job as a FULL for next cycle during data aging).

We need to determine why these jobs are completing with errors.. so if you could please view admin job history as per the procedure here: https://documentation.commvault.com/11.24/expert/6444_viewing_admin_job_history.html

Select Disaster Recovery Backup and the view the results > please share some screenshots on the job details / attempts / status tabs. 

Feel free to right click the job > view logs and share (attach) as a txt file for analysis (please cross check/remove any sensitive data). 

Additionally, with reference to this directory: 

As per the time stamps, they are all extremely old. I see no issue cleaning these up right now to get you back a little breathing space.

Additional articles for reference:
https://documentation.commvault.com/11.24/expert/43516_commserve_recovery_using_disaster_recovery_dr_backups.html

https://documentation.commvault.com/11.24/expert/43517_configuration_of_disaster_recovery_dr_backups.html

Regards,

Chris.

@Kegan Levett 

Going to need you to get me logs from the most recent job that has run. 

Whe viewing the logs from the GUI, make sure you hit the ‘select all’ button in the log pane so that we get everything. 

@Kegan Levett 

Looks like we’re having issues uploading the DB to cloud.
 

File    : CloudActivity-CommServeDR.log
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
14512 9270  03/03 08:01:47 2368176 CVRESTBaseRemoteFile::SendRequestRetry() - Error:  Message: This request is not authorized to perform this operation.
RequestId:83fb2dbb-401e-0025-7d78-2e6557000000
Time:2022-03-02T21:01:47.2829591Z
14512 91b4  03/03 08:01:47 2368176 CVRESTBaseRemoteFile::SendRequestRetry() - Error:  Message: This request is not authorized to perform this operation.
RequestId:1dba4c04-401e-001a-3b78-2eadf4000000
Time:2022-03-02T21:01:47.2825794Z
14512 9270  03/03 08:01:47 2368176 SendSinglePart() - failed to send the remote file - DR/<commserve>/SET_8416/DM2_<commserve>_2022_03_03_08_00_FULL.dmp
14512 91b4  03/03 08:01:47 2368176 SendSinglePart() - failed to send the remote file - DR/<commserve>/SET_8416/AppStudioDB_<commserve>_2022_03_03_08_00_FULL.dmp
14512 9270  03/03 08:01:47 2368176 HandleSendFileThread() - failed for DR/<commserve>/SET_8416/DM2_<commserve>_2022_03_03_08_00_FULL.dmp
14512 91b4  03/03 08:01:47 2368176 HandleSendFileThread() - failed for DR/<commserve>/SET_8416/AppStudioDB_<commserve>_2022_03_03_08_00_FULL.dmp
14512 8b6c  03/03 08:01:47 ####### Unknown error :AuthorizationFailure
14512 8b6c  03/03 08:01:47 2368176 CVRESTBaseRemoteFile::SendRequestRetry() - Error:  Message: This request is not authorized to perform this operation.
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

File    : CloudFileAccess.log
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
14512 73e0  03/03 08:01:47 2368176 CloseFile() - commvault/DR/<commserve>/SET_8416/AppStudioDB_<commserve>_2022_03_03_08_00_FULL.dmp, error =  Message: This request is not authorized to perform this operation.
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
 

File    : commserveDR.log
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

14512 5300  03/03 08:01:47 2368176 CommservDR::uploadFileToCloud() - Error: Failed to upload file [C:\Program Files\Commvault\ContentStore\CommserveDR\DR_2368176\DM2_<commserve>_2022_03_03_08_00_FULL.dmp] on cloud [commvault/DR/<commserve>/SET_8416/DM2_<commserve>_2022_03_03_08_00_FULL.dmp]. Error Message:[44103 :  Message: This request is not authorized to perform this operation.
 

File    : JobManager.log
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
7496  8a88  03/03 08:08:57 2368176             JobSvr Obj Copy to Staging location completed.
7496  8a88  03/03 08:08:57 2368176             Scheduler   -> COMPLETED WITH ERRORS/WARNINGS <-  Completed JobType[Disaster Recovery Backup] CompletionStatus[With Errors]

All other backup phases indicate job success.

What feature realease/maintenance release do you have installed in your environment at the moment? (right click commserve > properties > version tab).

If you disable the cloud log upload feature, can you monitor jobs and see if they succeed?
 

Instead of select, deselect this option. 

If jobs succeed, either there is a prerequisite that is missed from this article: 
https://documentation.commvault.com/11.24/expert/115116_uploading_dr_backups_to_commvault_cloud_services_portal.html or there may be a fix you need to install to rectify.

I’ll know more once you can confirm the above for me.

Thanks

Chris.
 

@Kegan Levett 

Instructions are here on performing a backup: https://documentation.commvault.com/commvault/v11_sp20/article?p=43530.htm

No need to select any other options, just hit OK once done.

Additionally, we will need to look at getting your environment updated... your present maintenance release is very far behind.. at 11.20.9.. we are 11.20.94!

i’d recommend downloading and installing the latest available which can be found here:
https://cloud.commvault.com/webconsole/downloadcenter/packageDetails.do?packageId=22468&status=0&type=details

Installation instructions: https://documentation.commvault.com/commvault/v11_sp20/article?p=2671.htm

Maintenance release info: https://documentation.commvault.com/commvault/v11_sp20/article?p=124866.htm

Let me know how the DR backup goes without cloud enabled.

Regards,

Chris

Hi again,

Just bumping for a response. The upgrade on our Commserve / Media agents haven’t happened yet (need to sort out a time) however clearing out the old install files has bought some time for this issue to be resolved.

Cheers

Thanks @Mike Struening 

I’m going through the documentation to do the upgrade after hours, and I’m wondering if there is a rollback procedure available as a precaution? My intention is to do a snapshot of the server and media agents, just checking if there is anything else I should do.

Additionally it appears that the upgrade has attempted to run previously but failed for some reason. Is a restart required normally? My in-house doco states that Commvault caches a copy on the latest Service Pack on the Commcell server at this file location.