Solved

What to do after losing backup storage

  • 17 December 2021
  • 15 replies
  • 306 views

Userlevel 4
Badge +13

Hello everyone,

My secondary site has a Windows server connected to HPE MSA disk shelves.  Earlier this year we added another disk shelf but my storage admin now tells me it’s improperly configured and he wants to correct it which will result in the lose of all the files on that individual shelf.  The storage appears to CommVault as J: (51TB), K: (51TB), L: (51TB), and M: (18TB).  It is just the M: allocation that needs to be corrected.

All backups have two copies with one at my primary and one at my secondary site so I *should* be able to re-copy the affected backups over to the secondary site once the storage is fixed.  Does anyone know if there’s a procedure or web site that shows how to do this?  It would be the equivalent of recovering from an disk failure.

Any help would be appreciated.

Ken

icon

Best answer by YYZ 17 December 2021, 19:34

View original

15 replies

Userlevel 7
Badge +23

@Ken_H , is there any parity on the drives so you can replace and rebuild it at that level, or is this not RAID, just a drive presented?

You could Aux Copy all of the data off, ensure the jobs are there, replace the disk, then Aux Copy it back (and promote the disk copy back to Primary).

That’s likely your easiest and cleanest method.

Userlevel 4
Badge +13

@Mike Struening, This is my secondary storage and all backups are configured to have a copy on both the primary and secondary media agents.  It sounds like all I’d need to do is re-run an aux copy from the primary media agent.  How would I ensure that all backups are fully replicated after the change?

Ken

Userlevel 7
Badge +23

Oh, even better!

Beyond normal restore testing after completion, you could (afterwards) set the retention on the current copy to Spool Copy.  This will only allow jobs to prune once they are fully copied over to all dependent copies (there’s no days or cycles count).

If the source copy clears out, then you’re good.  If any jobs remain, then the Aux didn’t complete (or there are other issues).

Of course, worth repeating that it pays to do some quick browsing/restore testing JUST to be sure!

Userlevel 4
Badge +9

Do you have all 4 Drive Letters providing Mount Paths to a Single Library?

If you have free space on the  3 lots of 51TB (enough so the capacity of the 18TB mount paths can fit) you could move the mount paths

https://documentation.commvault.com/11.24/expert/9302_moving_mount_path.html

As per the documentation - allow 24 hours and the ‘source’ mount paths on the 18TB should clear down to zero

 

 

 

Userlevel 4
Badge +13

@YYZ, the documentation for Move MountPath says “Do not move a mount path to another active mount path” so I’m not sure this will work for me.

Userlevel 4
Badge +9

That's correct. You cannot drop an existing mount path within another mount path. 

You would have to create new/empty folders

Userlevel 4
Badge +13

I tried a little experiment to move everything off M:.

  1. Update M: to “Disable mount path for new data”
  2. Went to Storage Policies > Email > Copy 2 > View > Jobs > Job 388543 > right-click > Delete job > erase and reuse media > OK
  3. Right-click on the Commcell > All tasks > Data aging > immediate > OK
  4. Back to Policies > Email > right-click > Aux Copy > OK > waited for job to finish
  5. Client > Exchange Mailbox > right-click > View > Archive history > OK > highlight job 388543 > right-click > View Media > only shows the copy at my primary site.

So … the aux copy did not replace the backup I manually deleted on to a different drive.  That’s unfortunate.

Userlevel 4
Badge +13

@Mike Struening  I don’t see anything on the properties of Copy 1 in the storage policy for a “Spool Copy”.  I’m not sure where you’re getting that.

Userlevel 7
Badge +23

@Ken_H , I should have been more specific.  Spool Copy is a setting on the retention tab for any copy that has dependent Aux Copies.

This should help:

https://documentation.commvault.com/11.24/expert/119069_spool_copies.html

Let me know if you have more questions!

Userlevel 7
Badge +23

I tried a little experiment to move everything off M:.

  1. Update M: to “Disable mount path for new data”
  2. Went to Storage Policies > Email > Copy 2 > View > Jobs > Job 388543 > right-click > Delete job > erase and reuse media > OK
  3. Right-click on the Commcell > All tasks > Data aging > immediate > OK
  4. Back to Policies > Email > right-click > Aux Copy > OK > waited for job to finish
  5. Client > Exchange Mailbox > right-click > View > Archive history > OK > highlight job 388543 > right-click > View Media > only shows the copy at my primary site.

So … the aux copy did not replace the backup I manually deleted on to a different drive.  That’s unfortunate.

I may be missing something, though for Spool copy you need to let the job Aux copy over before it gets deleted.

Userlevel 4
Badge +13

The Enabling Spool Copy page says:

  1. ensure that the storage policy copy is assigned as a source to another active synchronous copy.

  2. From the CommCell Browser, go to Policies > Storage Policies ><Storage Policy>.

  3. Right-click the storage policy copy that was assigned as a source, and click Properties.

  4. In the Copy Properties dialog box, click the Retention tab, click Spool Copy (No Retention), and then click OK.

But when I go to Copy 1 > Properties > Retention tab, there is no Spool Copy option.  Sigh.  If I had a dollar for every time the user interface didn’t match the documentation, I would be a very rich man.

Userlevel 4
Badge +13

Thanks everyone for your help.  It seems clear that this is so far outside the normal bounds of what can be done with CommVault that I’ve opened ticket 211217-687 to get some assistance.

Ken

Userlevel 7
Badge +23

If it’s not there, then maybe there’s no Synchronous Aux Copy yet?  I’m darn curious now!

Once you have an answer, let us know.  The main issue is of course your concern, though the not-appearing-spool-option is curious.

Userlevel 4
Badge +13

Ignoring the two most recent aux copy jobs that I ran manually today, you can see that aux copies run every 8 hours starting at 2:00 AM.

 

Userlevel 4
Badge +13

So … I misunderstood the instructions for Moving a Mount Path which say:

  • The move mount path operation overwrites the destination mount path. Therefore, ensure that the destination mount path is an empty folder.

I mistakenly thought this meant I couldn’t move to one of the existing drives but it turns out that all I needed was to create a new folder.  I created L:\MP4 and was able to move everything off M: to that new folder so I’m in a good position to change the configuration on the M: drive.  Thanks everyone.  @YYZ was correct and I’ll mark their reply as the solution.

Ken

 

Reply