Commserve Upgrade - Best Practices

  • 25 January 2021
  • 11 replies
  • 257 views

Userlevel 4
Badge +11

With all of the cool features coming in our Feature Releases, I thought it would be handy to create a list of Best Practices for planning and Performing a Commserve update to a higher Feature Release.

There’s definitely a lot of Proactive work that can go into your process that will save you from potential headaches or even nightmare scenarios!

Have any of your own?  Feel free to share below!


11 replies

Badge

Hello Mike,

Thank you for your post. Very helpful. I'm a newbie when it comes to commvault. I am being asked to create an update/upgrade procedure.

We have 2x Media Agents(Physical) and one virtual Commserve Server on HA (VMWare).

My understanding is the following.

  1. When I upgrade the Comserve Server I need to also upgrade the two media agents for backups to work.

  2. At that stage when everything has had a new feature or hotfix installed; then it would be the case for regression testing. (testing that everything works as it did beforehand)

  3. if something is wrong I would need to manually re-install the older version on the media agents. The commserv server could be rolled back as a snapshot (due to being virtual).

In regards to reverting back the Commcell/Commvault versions of the media agents on the 2x physicals. I have thought of the following as an option for the physical media agents.

If I add another partition; and clone the existing primary partition to that disk before performing the upgrade. If there is a problem; I should be able to do the following?

  1. Boot from the cloned disk (taken just before the upgrade on MA)

  2. Restore the commserv server from snapshot (as VM)

  3. Everything works!

Badge

Further to this. Is CommServe LiveSync what I'm looking for. EG: 1. Failover to standby 2. Perform updates. 3. Failback 4. Test 5. Then failover again if regression testing fails. Otherwise I perform the same on the failover?

Badge +1

Hello,

 

Another good advice during the installation is to open the GXTail.exe and check the “Install.log”, “DatabaseUpgrade.log” and “UpdateInfo.log”. The installation window do not show much information, so when our department upgrade to CV11SP18 it took a long time and this logfiles showed us that the process still running and going without problems. 

 

Regards

Userlevel 5
Badge +10

My understanding is the following.

When I upgrade the Comserve Server I need to also upgrade the two media agents for backups to work

Welcome to Commvault - glad to be able to help you out!

Yes, it is always best practice to have your Media Agents in-sync with the update level of the CommServe to avoid any issues. Sometimes it's not possible (if you have a huge environment and it takes time to update), in the majority of cases things will function as expected, but Media Agents should be updated as soon as possible.

 

At that stage when everything has had a new feature or hotfix installed; then it would be the case for regression testing. (testing that everything works as it did beforehand)

 

Sounds good

if something is wrong I would need to manually re-install the older version on the media agents. The commserv server could be rolled back as a snapshot (due to being virtual).

 

I would say this is the worst-case scenario for sure. We always want to try to resolve the issue going forward rather than revert back. You have to be super careful with rolling back snapshots - the database is closely tied to the data and indexes stored on the media agent. If you start rolling back snapshots then those things all become out of sync - for example the DDB open time is stored in the commserve database, if the time does not match what is expected, the DDB store will be marked as invalid and has to be validated/reconstructed again. Likewise, if data was written to the library and you rollback the snapshot on the Commserve, we lose track of that data and won't know to delete it. These days, there are mechanisms to try to recover from this scenario but you don't really want to rely on them if you don’t have to. In an unlikely event where things are not working after an update, open a support case (critical if you need), and we should work to resolve it with the updated code and evaluate the options to roll back if needed - but that is very rare.

 

In regards to reverting back the Commcell/Commvault versions of the media agents on the 2x physicals. I have thought of the following as an option for the physical media agents.

 

Theoretically, this could work, but again, one misstep and things will be out of sync. Reinstalling the software on a media agent is usually pretty quick. I’d opt to go that route rather than keeping clones/snapshots around just in case. 

Userlevel 4
Badge +11

Hello Mike,

Thank you for your post. Very helpful. I'm a newbie when it comes to commvault. I am being asked to create an update/upgrade procedure.

We have 2x Media Agents(Physical) and one virtual Commserve Server on HA (VMWare).

My understanding is the following.

  1. When I upgrade the Comserve Server I need to also upgrade the two media agents for backups to work.

  2. At that stage when everything has had a new feature or hotfix installed; then it would be the case for regression testing. (testing that everything works as it did beforehand)

  3. if something is wrong I would need to manually re-install the older version on the media agents. The commserv server could be rolled back as a snapshot (due to being virtual).

In regards to reverting back the Commcell/Commvault versions of the media agents on the 2x physicals. I have thought of the following as an option for the physical media agents.

If I add another partition; and clone the existing primary partition to that disk before performing the upgrade. If there is a problem; I should be able to do the following?

  1. Boot from the cloned disk (taken just before the upgrade on MA)

  2. Restore the commserv server from snapshot (as VM)

  3. Everything works!

Glad to be of help!  This community just launched yesterday (for everyone) and the feedback is already so amazing!

I’ll answer both of your questions separately:

In the first section, your roll back method should work, though if you have issues post update that aren’t an emergency, start a thread on the forums and see if the community can help (before rolling back).

@Damian Andre answered this part, though I wanted to share some more info with you.  Here’s some documentation about testing a Commserve DR with detailed steps/links.  Definitely wise to practice to a) make the process part of your mental muscle memory AND to b) confirm everything is in place to work in a real emergency!

For your second reply, it definitely sounds like you should look into CS LiveSync.  Makes the entire process MUCH easier.

 

Userlevel 4
Badge +11

Hello,

 

Another good advice during the installation is to open the GXTail.exe and check the “Install.log”, “DatabaseUpgrade.log” and “UpdateInfo.log”. The installation window do not show much information, so when our department upgrade to CV11SP18 it took a long time and this logfiles showed us that the process still running and going without problems. 

 

Regards

That’s a good one!  It’s also quite hypnotic to watch the logs scroll down the screen :nerd:

Userlevel 3
Badge +7

Plan the Timing out in advance - Make sure any key stakeholders know the CS will be down during this planned upgrade

 

Any advise on how to improve that? With different databases which require the availability of backups , getting downtimes of the backup environment is very difficult.
For example: hana backups use a constant stream (thank god, remember the time individual jobs were started), and backup of the logs are running 24h/24h.

That makes it difficult to arrange maintenance windows, as each customer want a different maintenance window of course :) As this is quite common, I wonder what the advise of commvault or other customers is regarding this? 

 

Userlevel 4
Badge +11

That’s a very good question, and the difficulty increases exponentially as you scale upwards.

I’d say that this is a good time to talk about setting up a failover CS so there’s essentially no downtime.

For any upgrades to clients, or client groups (or even Media Agents and the clients that use them), you’ll have more flexibility in speaking to the direct server owners/users/stakeholders since you can address those upgrades at more individualized times.

Userlevel 3
Badge +5

This is all very helpful- Appreciate it

Userlevel 3
Badge +6

Good tips, although there can be problems with the CommServe upgrade that are not CommServe related.  The most common one I have seen is that the MongoDB fails to upgrade. So in case there is a very tight change window it definitely can help to have a VM snapshot of the CommServe.

Badge

Hello,

after a successful Feature Release upgrade of all CommCell and clients, is it possible to purge the contents of the software cache only related to the old version?

I recently upgraded from SP17 to SP21, and I still have:

  • CV_SoftwareCache\CVMedia\11.0.0\SP17_2206210_R682
  • CV_SoftwareCache\CVMedia\11.0.0\SP21_2957659_R830

 

Thanks 

Best Regards

Reply