commvault.com commvault.com
Solved

Disaster recovery backup - Completed with errors

  • 4 June 2021
  • 7 replies
  • 3602 views
Z
Rank
Badge +1

For weeks now, the DR backup has been ending with errors.

We have reviewed the log and events. We don't see any errors.

 

How can I find more information to know exactly the problems that the copy is having

 

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Machine : b2s-backup01
File    : ResourceManager.log
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

9904  2d2c  06/03 10:01:09 43461 Resource   Reserved Resource: NoOfReservations [1], StoragePolicy [CommserveDR], MediaAgent[b2s-backup01], StartNewMedia = false, InterruptedJobList []
9904  24cc  06/03 10:02:06 43461 Resource   Releasing Stream(s) for Job [43461]

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Machine : b2s-backup01
File    : UpdateIndex.log
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

4416  46f8  06/03 10:01:16 43461 TUpdateIndexHandler:: Initializing TupdateIndex object for client [b2s-backup01],job [43461],bType [1],commCellId [2],appNumber [1],appType [43]
4416  46f8  06/03 10:01:16 ##### UPDATEINDEX JobToken received: '43461:8:1:0'
4416  46f8  06/03 10:01:16 ##### Init() - Initializing job control [token=43461:8:1:0,cn=b2s-backup01] in single threaded mode, serverName [B2S-BACKUP01.b2s.loc], ControlFlag [7], Job Id [43461]
4416  46f8  06/03 10:01:16 43461 TMonitorJob: Registry key MediaAgent\heartBeatPeriod is not present, defaulting to 5 minutes
4416  46f8  06/03 10:01:16 43461 TMonitorJob: Registry key MediaAgent\JMQuitCheckPeriod is not present, defaulting to 1 minute
4416  46f8  06/03 10:01:16 43461 UPDATEINDEX index checkpointing remains disabled based on task options
4416  46f8  06/03 10:01:16 43461 UPDATEINDEX Effective failover flags are: UseICS=FALSE, UseTL=FALSE, Checkpointing=FALSE (scheduler flags=[TRUE, FALSE, FALSE])
4416  46f8  06/03 10:01:47 43461 UPDATEINDEX updateRestartInfo: offset indicates archive file (2,19,80909) successfully closed
4416  46f8  06/03 10:01:51 43461 UPDATEINDEX received backup completion message
4416  46f8  06/03 10:01:51 43461 INDEXCACHEDIR: in IndexCacheDir::backup()
4416  46f8  06/03 10:01:51 43461 UPDATEINDEX terminating due to backup completion
4416  46f8  06/03 10:01:51 43461 UPDATEINDEX Attempt statistics: Streams=1, Items=290, Chunks=1, aFiles=1, processing time=0.070664 s, total time=34.914619 s, average processing time=0.000243 s, average wait time=0.119739 s
 

icon

Best answer by Christian Negron 7 June 2021, 18:35

View original

7 replies

S
Rank
Userlevel 7
Badge +15

Hi @zori92 

Thank you for the question and welcome to the Community!

From the screenshots, it looks like the first phase of DR backup is getting failed - backup to disk.

The last 3 phases, which is backup to CommserveDR storage policy copy actually look to be successful.

The log file we need to check is commserveDR.log.

There are a few common reasons why the first backup to disk phase might fail:

  • Disk space constraints on the local disk - check the installation volume for disk space. DR backups are staged, typically in C:\Program Files\Commvault\ContentStore\CommserveDR before being copied to the network location.
  • Network location configured for DR backups is full, unavailable or cannot be written to, potentially credentials configured may have changed/expired
  • Cloud upload destination is failing

Either way, the commserveDR.log should point towards the failure.

Thanks,

Stuart

Customer Support
Z
Rank
Badge +1

Thanks for the welcome! I am excited to learn and grow with the community.


I will check the steps you have told me about.

 

 

Sergio.

Mike Struening RETIRED
Rank
Userlevel 7
Badge +23

Based on your error, I’m going to assume you are writing somewhere with specified credentials that no longer have rights or the password changed.

@zori92 , I’m going to add in a link to a blog we did a few weeks back about Ransomware.  When you check your Commserve DR configuration, make sure you are a) writing to a network share and b) utilizing our Cloud DR save feature.

Writing to a network share ensures that a physical loss of the Commserve doesn’t lose the DR backup as well.  Sending a DR to our cloud service saves you time and energy should you get hit by a Ransomware disaster.

 

https://www.linkedin.com/in/michael-struening
Z
Rank
Badge +1

Hi Mike,

Thanks for you time :)

 

The points a and b that you tell me are verified, I have been able to verify that in the cloud the last copy uploaded to the cloud was on May 11.

I've been trying all day to figure out the problem.

 

L
Rank
Userlevel 6
Badge +15

@zori92 Have you reviewed the settings of the DR-backup from the Control Panel ?

Commcell Console / Control Panel, then select DR-Backup :

 

Then check the Export and Backup settings : 

The ‘upload backup metadate to Commvault Cloud’ is the option that’s reported to have last succeeded may 11. Maybe this is the issue, and most probably for firewall/security reasons on your side…

 

 But check also your Pre/Post process tab, in case you have scripts that are to be executed :

I do have a script to make another copy of the DR backups to some other locations for extra security purposes. Failure to execute this script would result in the ‘Complete with Warnings’ DR backup job status…

 

Also maybe check the CVCloudService.log at the time of your DR-backup schedule for traces of the data copy to Cloud.

C
Rank
Userlevel 2
Badge +4

Hello @zori92!

 

I see a few points have been brought up to check. Below I will include some additional information; 

 

  1. Were you able to confirm the backup is writing successfully to \Program Files\Commvault\ContentStore\CommserveDR? 

 

  1. Going to the metadata location, is the data from the CommserveDR folder copying to this location? Are the folders also delete per the “Number of retained metadata backups” value? I have seen where older folders remained and the system was unable to delete them, giving a line item error that led to the Complete w/errors status and with all stages completing. 

     

  2. I see you noted the last valid Cloud backup  is from the 11th, this shows us the cloud backups are not completing fully. Things to check;
  1. Cloud URLs are whitelisted within firewall rules; https://documentation.commvault.com/11.21/expert/4787_external_urls_for_commvault_features.html

Upload Disaster Recovery (DR) Backups to Commvault Cloud

  • cloud.commvault.com

  • cloudupload.commvault.com

  • cvdrbackup1.blob.core.windows.net

  

Upload and access DR Backups from the Cloud Services website.
  1. Has the documentation been followed for configuring the Cloud backup? Overview: https://documentation.commvault.com/11.21/expert/92700_configuring_automatic_uploads_of_disaster_recovery_dr_backups_to_commvault_cloud_services_portal.html

Specifically the portion where you register the product: https://documentation.commvault.com/11.21/expert/40674_register_your_product.html

 

  1. The logs are not showing much in the screenshot other than the SQL backup completing successfully. Would you be able to attach the entire Commserve.log and CVCloudService.log files in your reply? I would be happy to check. 

 

(SQL log files may be helpful as well, this can be exported via SQL Management Studio)

(Commvault log file location: \Program Files\Commvault\ContentStore\Log Files)

 

Mike Struening RETIRED
Rank
Userlevel 7
Badge +23

@zori92 , wanted to follow up and see if you had a chance to review the information from @Christian - Support .

Thanks!

https://www.linkedin.com/in/michael-struening

Reply


Terms & ConditionsCookie settings