Disaster Recovery Backups have started failing

Hi @Ken_H 

I would first manually open a connection to the UNC path from <server_57> and authenticate with the user configured in the DR backup location config.

If that works, re-enter credentials in the DR backup location config and test the DR backup.

If the manual test doesn't work, check if the firewall is active on <server_216> and allows incoming smb traffic, secondly check if the user configured in the DR backup location config has rights on the share.

These DR backups have been running longer than I’ve been working with CommVault and I’m not sure where they are configured.  The CommVault online documentation talks about using the Command Center to make these changes (as shown here) but when I try to follow those directions I get “Something went wrong. Server may be under maintenance. Trace ID: cvoDRcf1NY” . 

Can you tell me where to find this configuration in the Java GUI?

Ken

In the top section you can open the control panel, here you will find a button DR backup

Thanks for the help @Jos Meijer.

I have a CommServe Hardening PDF from CommVault about hardening the security to keep backups safe in the event of a cyber attack.  One of the recommendations is:

• Disable NETBIOS on the CommServe host. To do this, open the Network and Sharing Center, select Change Adapter Settings, right-click the network connection, and select Properties. Select Internet Protocol Version 4 (TCP/IPv4) and click the Advanced button in the displayed dialog. Select the WINS tab and select the Disable NetBIOS over TCP/IP.

Oops.  Turns out disabling NetBIOS breaks the DR Backup.  I’ve enabled NetBIOS, rebooted, and the DR backups are back to normal.

Ken

Good to hear you have found the cause👍
Not so good to hear that this limits hardening 😐

Hi @Ken_H 

Just to let you know, I do Commserve hardening too in multiple deployments, I also do disable NETBIOS on Commserve host and don’t have any issues with DR backup on network share. Maybe it’s something environmental in your case but I definitely wouldn’t blame disabling NETBIOS alone.

Disabling NetBIOS broke DR backups and enabling it allowed backups to resume.  Any suggestions on how to get around this problem would be gratefully received.

Ken

@Ken_H , I unchecked the Best Answer.  Unless @Jos Meijer has some more ideas, it might be best to open a support case.

As NetBios brakes the UNC access you mostlikely are using a short name / single label name ( \\servername\D$ ) for the target server and not an FQDN address ( \\servername.dns.suffix\D$ ), is that correct?

If so, do you have a DNS server configured?

If so please try an FQDN address.

Let us know the outcome 🙂

Also as @Jos Meijer already mentioned check communication. Try to telnet <server_216> from server <server_57> on ports 139 and 445 and see if you could open these ports.

Sigh.  It turns out that CommVault will not just let you change the server name to the fully qualified domain name and gives the message:

Failed to set [Backup Metadata Folder] for [DR Backup] with error [Failed to save Disaster Recovery Destination in CommServe database because folder [\\<server_216.domain.suffix>\D$\DR_Dump_Prod] is already being used for Disaster Recovery.]

So I changed the configuration from:

   \\<server_216>\D$\DR_Dump_Prod

to 

   \\<server_216.domain.suffix>\D$\DR_Dump_PRD

Just because I’m paranoid, I’m going to let this run for the weekend before attempting to disable NetBIOS again.

My apologies, I should have forseen this.
As the folder is already in use and the location has been configured before the database will prevent such change.

Your action by assinging a different location is the only way.
Keep in mind that even after changing the DR location, aging on old sets (on unc path with short name) will be performed according to the retention settings. Thus when disabling NetBios will result in periodic errors that pruning on the old data path cannot be performed as the path will become inaccessible as experienced before.
This will eventually stop though when all sets, existing on the old location, have been attemted and failed to be removed. Then only aging/pruning will be continued on the new path.
You could lower the amount of retained DR sets before disabling NetBios to prune the sets on the old location and “force” an ealier transition this way. If this path is choosen then just monitor the old path, when fully emptied then disable NetBios and set the amount of DR sets back to the original setting.

Update:  I updated the DR backup to use a fully qualified domain name last Friday and left it to confirm that DR backups run normally.  After days of uneventful operation, yesterday (Wednesday) I disabled NetBIOS on the CommServer host and DR backups immediately began to fail.  I’ll enable NetBIOS now while investigating why this is so problematic.

Thanks for letting us know the solution @Ken_H 

Glad to hear it's fixed now 🙂