Disk library and physical Media Agent change as disaster recovery.


Userlevel 1
Badge +3

Hi all!

My company, using six MA to create and store backups. One MA with a separated storage for long term retention outside, and an another one for create local backups on a branch office site. On the main site there are four MA in two node grids. MA1 & MA2 is a grid and MA3 & MA4 is an another. They are sharing their libraries and DDBs.
From the branch office, local backups are copied to the main site and main backups are copied to long term site as DR backups. MAs are physical on main site and virtual on others, and disk storages are used on all sites.

Currently, we are planing to change our disk storages and physical MAs on main site. And of course, it is a good chance to upgrade OS on MAs from Win2012R2 to Win2019. During the process, library content should be moved from the old disk storage to the new one, and DDBs from old MA to new. One MA stores 40 - 60 TB backup data, and of course, I would like to do it with minimum downtime. 
I have found descriptions about library move and DDB move processes between MAs. But after these steps, Subclients and Storage Policy copies should be configured for new MA. Therefore, I'm planing to change MAs as MA disaster recovery. It means, following steps:

- Install OS on new physical server with new name and IP.
- Configure disks (library & DDB) as it is on the old MA. (library disks from new storage, shares also created)
- Copy the content of libraries, DDB and Index Cache from old MA to new disks on new MA. It mainly can be done during operation by RoboCopy tool. RoboCopy can be started several times to refresh changes during CommServe operation. 
- Disable All Job Activity and Scheduler on CommServe. Finish all running jobs.
- Stop Commvault services on old MA.
- Start RoboCopy to refresh last changes on new MA. Check RoboCopy logs to be sure, every modification transferred to new MA.
- Shutdown old MA
- Rename new MA to the same name with old and change IP.
- After new MA restarted, install MA components remotely from Commcell Console.
- After installation, check readiness and DDB status.
- Enable All Job Activity and Scheduler on CommServe

I have installed a test environment with two MA in grid and third MA for aux copies. Some backups and copies was made for a week, and steps above was done. New MA operates without any issue. Only Index Cache had to be reconfigured, because that was default after MA installation. Therefore I think, the described process above can work, but I would like to avoid hidden traps.
If something goes wrong before activity enabled on CommServe, hopefully old MA can be restarted, but I like to be as prepared as possible.

What do you think about this idea?


8 replies

Userlevel 7
Badge +23

Looks solid to me.  Since you are copying and not moving the data (and keeping the same names and IP addresses), you can always fall back to brining the current MA back online if anything fails, but your process looks sound to me!

Badge +5

Hello,

I have a question about this.

If no Robocopy would have been used to copy DDB and Index cache and Disk Library would be a SAN (FC) storage.

Would just dettaching the disk library from the old MediaAgent and attaching it to the new MediaAgent work?

Thanks

Userlevel 7
Badge +16

Hi @VBalazs 

I would install the new media agents via local install and not remote install, reason for this is that you just power off the old media agent thus according to the commserve database there already is an installation present on the media agent and only a repair and add package option is available.

Another thought is, why not prepare the media agents as a new grid (now remote push install is possible), create a new DDB setup, create additional storage policy copies, perform an aux copy, then disable the current primary copy and promote the new copy to primary? This way you have one tool to manage, you will have a clean new copy with latest DDB standards instead (assuming the current ma config originated from an earlier DDB version). You can still keep using the current grid till all transfers finished and can easily fall back to the old setup.

Userlevel 7
Badge +16

Hi @Sergio V 

Yes SAN storage can be remapped, works fine 🙂

Userlevel 1
Badge +3

Hi @Jos Meijer 

I can use remote install from Commcell Console, because when I select the Add/Remove Software menu on the Tools tab, and choose Install Software, then necessary clients and SW components can be selected.
If new media agents prepared, then all copies, off site copies and subclients have to be reconfigured. It is like a new backup system implementation. That was what I tried to avoid with steps above.

Userlevel 7
Badge +16

You are right, didn’t expect that route to ignore the already installed packages but it does 👍

Regarding my suggestion, indeed there are some more steps, but doesn’t consume much time and provides more flexibility. But you should do it as you see fit 😋
The subclient adjustment though is not needed, the storage policy is configured on subclient level and this will automatically adopt the storage policy copy setup.

Userlevel 1
Badge +3

Sorry for my late answer. On reflection, your proposal really has some benefits. I will consider it.
Thank you.

Userlevel 1
Badge +3

Due to the long delivery times, I have only just started replacements on prod system. I'm started with smalest MA and trying with RoboCopy solution as I describe at the begining.
New MA installed, Index Cache, DDB copyed successfully, but Disk Library partitions has an issue. The old content does not fit on the new disk library.
When I check partition property, then I can see, there are more data on the partition as the size of it. You can see in the picture. Block size 64K on that partition and Windows level compression not enabled. That can be enabled only with 4K block size or below.
Commvault Hardware compression not enabled on storage path, I think, that can be anabled only for Tape and not for Disk library. Software compression enabled on subclients, but that can be seen on Windows OS level? What @Mike Struening think?
At this point, RoboCopy extracts the contents of the old storage path and copies it uncompressed. This will take up much more space on the new storage. There isn't that much space.
Is there any solution to copy Disk Library content on Windows level? Or it should be copyed on LUN level?
Or should I prepare a new MA, DDB, Storage Policy Copies and use Aux Copy as @Jos Meijer suggested?

 

Reply