Commserve Upgrade - Best Practices

  • 25 January 2021
  • 17 replies
  • 7883 views

Userlevel 7
Badge +23

With all of the cool features coming in our Feature Releases, I thought it would be handy to create a list of Best Practices for planning and Performing a Commserve update to a higher Feature Release.

There’s definitely a lot of Proactive work that can go into your process that will save you from potential headaches or even nightmare scenarios!

Have any of your own?  Feel free to share below!


17 replies

Badge

Hello Mike,

Thank you for your post. Very helpful. I'm a newbie when it comes to commvault. I am being asked to create an update/upgrade procedure.

We have 2x Media Agents(Physical) and one virtual Commserve Server on HA (VMWare).

My understanding is the following.

  1. When I upgrade the Comserve Server I need to also upgrade the two media agents for backups to work.

  2. At that stage when everything has had a new feature or hotfix installed; then it would be the case for regression testing. (testing that everything works as it did beforehand)

  3. if something is wrong I would need to manually re-install the older version on the media agents. The commserv server could be rolled back as a snapshot (due to being virtual).

In regards to reverting back the Commcell/Commvault versions of the media agents on the 2x physicals. I have thought of the following as an option for the physical media agents.

If I add another partition; and clone the existing primary partition to that disk before performing the upgrade. If there is a problem; I should be able to do the following?

  1. Boot from the cloned disk (taken just before the upgrade on MA)

  2. Restore the commserv server from snapshot (as VM)

  3. Everything works!

Badge

Further to this. Is CommServe LiveSync what I'm looking for. EG: 1. Failover to standby 2. Perform updates. 3. Failback 4. Test 5. Then failover again if regression testing fails. Otherwise I perform the same on the failover?

Userlevel 1
Badge +1

Hello,

 

Another good advice during the installation is to open the GXTail.exe and check the “Install.log”, “DatabaseUpgrade.log” and “UpdateInfo.log”. The installation window do not show much information, so when our department upgrade to CV11SP18 it took a long time and this logfiles showed us that the process still running and going without problems. 

 

Regards

Userlevel 7
Badge +23

My understanding is the following.

When I upgrade the Comserve Server I need to also upgrade the two media agents for backups to work

Welcome to Commvault - glad to be able to help you out!

Yes, it is always best practice to have your Media Agents in-sync with the update level of the CommServe to avoid any issues. Sometimes it's not possible (if you have a huge environment and it takes time to update), in the majority of cases things will function as expected, but Media Agents should be updated as soon as possible.

 

At that stage when everything has had a new feature or hotfix installed; then it would be the case for regression testing. (testing that everything works as it did beforehand)

 

Sounds good

if something is wrong I would need to manually re-install the older version on the media agents. The commserv server could be rolled back as a snapshot (due to being virtual).

 

I would say this is the worst-case scenario for sure. We always want to try to resolve the issue going forward rather than revert back. You have to be super careful with rolling back snapshots - the database is closely tied to the data and indexes stored on the media agent. If you start rolling back snapshots then those things all become out of sync - for example the DDB open time is stored in the commserve database, if the time does not match what is expected, the DDB store will be marked as invalid and has to be validated/reconstructed again. Likewise, if data was written to the library and you rollback the snapshot on the Commserve, we lose track of that data and won't know to delete it. These days, there are mechanisms to try to recover from this scenario but you don't really want to rely on them if you don’t have to. In an unlikely event where things are not working after an update, open a support case (critical if you need), and we should work to resolve it with the updated code and evaluate the options to roll back if needed - but that is very rare.

 

In regards to reverting back the Commcell/Commvault versions of the media agents on the 2x physicals. I have thought of the following as an option for the physical media agents.

 

Theoretically, this could work, but again, one misstep and things will be out of sync. Reinstalling the software on a media agent is usually pretty quick. I’d opt to go that route rather than keeping clones/snapshots around just in case. 

Userlevel 7
Badge +23

Hello Mike,

Thank you for your post. Very helpful. I'm a newbie when it comes to commvault. I am being asked to create an update/upgrade procedure.

We have 2x Media Agents(Physical) and one virtual Commserve Server on HA (VMWare).

My understanding is the following.

  1. When I upgrade the Comserve Server I need to also upgrade the two media agents for backups to work.

  2. At that stage when everything has had a new feature or hotfix installed; then it would be the case for regression testing. (testing that everything works as it did beforehand)

  3. if something is wrong I would need to manually re-install the older version on the media agents. The commserv server could be rolled back as a snapshot (due to being virtual).

In regards to reverting back the Commcell/Commvault versions of the media agents on the 2x physicals. I have thought of the following as an option for the physical media agents.

If I add another partition; and clone the existing primary partition to that disk before performing the upgrade. If there is a problem; I should be able to do the following?

  1. Boot from the cloned disk (taken just before the upgrade on MA)

  2. Restore the commserv server from snapshot (as VM)

  3. Everything works!

Glad to be of help!  This community just launched yesterday (for everyone) and the feedback is already so amazing!

I’ll answer both of your questions separately:

In the first section, your roll back method should work, though if you have issues post update that aren’t an emergency, start a thread on the forums and see if the community can help (before rolling back).

@Damian Andre answered this part, though I wanted to share some more info with you.  Here’s some documentation about testing a Commserve DR with detailed steps/links.  Definitely wise to practice to a) make the process part of your mental muscle memory AND to b) confirm everything is in place to work in a real emergency!

For your second reply, it definitely sounds like you should look into CS LiveSync.  Makes the entire process MUCH easier.

 

Userlevel 7
Badge +23

Hello,

 

Another good advice during the installation is to open the GXTail.exe and check the “Install.log”, “DatabaseUpgrade.log” and “UpdateInfo.log”. The installation window do not show much information, so when our department upgrade to CV11SP18 it took a long time and this logfiles showed us that the process still running and going without problems. 

 

Regards

That’s a good one!  It’s also quite hypnotic to watch the logs scroll down the screen :nerd:

Userlevel 6
Badge +13

Plan the Timing out in advance - Make sure any key stakeholders know the CS will be down during this planned upgrade

 

Any advise on how to improve that? With different databases which require the availability of backups , getting downtimes of the backup environment is very difficult.
For example: hana backups use a constant stream (thank god, remember the time individual jobs were started), and backup of the logs are running 24h/24h.

That makes it difficult to arrange maintenance windows, as each customer want a different maintenance window of course :) As this is quite common, I wonder what the advise of commvault or other customers is regarding this? 

 

Userlevel 7
Badge +23

That’s a very good question, and the difficulty increases exponentially as you scale upwards.

I’d say that this is a good time to talk about setting up a failover CS so there’s essentially no downtime.

For any upgrades to clients, or client groups (or even Media Agents and the clients that use them), you’ll have more flexibility in speaking to the direct server owners/users/stakeholders since you can address those upgrades at more individualized times.

Userlevel 4
Badge +11

This is all very helpful- Appreciate it

Userlevel 5
Badge +10

Good tips, although there can be problems with the CommServe upgrade that are not CommServe related.  The most common one I have seen is that the MongoDB fails to upgrade. So in case there is a very tight change window it definitely can help to have a VM snapshot of the CommServe.

Userlevel 4
Badge +13

@Smnhs and @Mike Struening did something change here?

I don’t think the backups stop working once commcell is updated with a new FR and media agents isn’t.


When I do the updates i usaually go by updating commcell and two media agents for test. Then move on to the production ones when all looks good.

But during the time period all backups/restores etc. do work. Of course no new features can be used though, until media agent is updated.

 

//Henke

 

Userlevel 7
Badge +23

@Smnhsand @Mike Struening did something change here?

I don’t think the backups stop working once commcell is updated with a new FR and media agents isn’t.


When I do the updates i usaually go by updating commcell and two media agents for test. Then move on to the production ones when all looks good.

But during the time period all backups/restores etc. do work. Of course no new features can be used though, until media agent is updated.

 

//Henke

 

@Henke you are correct.  You don’t have to upgrade everything in one shot, though it’s not a bad idea to keep the MAs and CS at the same level if possible.

 

Certain obstacles (features, OS levels) will dictate what you can and can’t do.

Userlevel 7
Badge +23

Hello,

after a successful Feature Release upgrade of all CommCell and clients, is it possible to purge the contents of the software cache only related to the old version?

I recently upgraded from SP17 to SP21, and I still have:

  • CV_SoftwareCache\CVMedia\11.0.0\SP17_2206210_R682
  • CV_SoftwareCache\CVMedia\11.0.0\SP21_2957659_R830

 

Thanks 

Best Regards

Good question!  I’m going to split this off to a new topic :-)

Userlevel 6
Badge +13

regarding the upgrade process: as mentioned before, timing is always important. One of the actions what commvault is doing with a upgrade is dong an SQL backup.

That sql backup can take some time.
With the standard DR backup, you can select ‘differential’, isn’t that also possible during the upgrade process, so that downtime becomes a tad smaller?

Badge

I did an upgrade a few months back and only upgraded the CC and MA’s at the time, but not Exchange clients.  Backups ran without any errors for a few days, however couldn't restore a mailbox DB until the client was upgraded to the same level as the CC and MA’s  Just my 2 cents….:o)

Userlevel 3
Badge +9

i would like to add on the following actions

  1. Leverage VM snapshot for quick revert without reconfigure whenever possible for CS-MA
  2. Reboot the CS-MA if there are pending reboot after windows patches installations
Userlevel 1
Badge +13

@Mike Struening RETIRED   

 

hello make , i have a situation , our DDB got corrupted , and now running full ddb recon job,

we dont have much space in disk library 

and we are very old version 11.14, 

all our backup are stopped and ddb recon job is running since 7 days ,

can i suspend the current ddb job and do a commserv upgrade and resume the job, because it took 7 days to reach around 75% so i dont want it be killed and rerun again

Reply