commvault.com commvault.com
Solved

Do not update to version 11.30 or 11.32 (drama inside)

  • 3 July 2023
  • 8 replies
  • 1725 views
J
Rank
Userlevel 1
Badge +2

I want to explain a problem we have:

Hardware WORM was active on all of our disk libraries.

 

We are not aware of having activated this functionality and as support indicated, there is no trace that we have executed a workflow to activate it.

Despite that, the WORM functionality was not active in any of our Storage Policies, only at the disk library level. We could change the retention of our copies, delete backups, micro-prunning working, and our databases were NEVER sealed.

 

We updated version 11.28 to 11.30 and suddenly the “WORM Storage” feature is activated in ALL storage policies at storage pool level.

As per the below document, this changes are applied on SP30 onwards, if hardware WORM is enabled:

https://documentation.commvault.com/2023/essential/149133_changes_in_commvault_platform_release_2023.html

 

Now we cannot delete backups, nor change the retention of our copies and ALL databases will be sealed every X days, micro-prunning disable. In addition, we have not been able to plan the increase of space in the libraries (run out of space on one of them), nor have we been able to adjust the retention of copies before this functionality was activated.

Total disaster.

Why was the "WORM Storage" functionality activated at the storage policy level if we did not have it activated at that level?

Our copies are a mix between incremental copies and synthetic copies.

Now when the database is sealed and a new one is created, there is no baseline and the first copy that is executed is a full copy. Full copy on ALL infraestructure.

Also a problem if you have agents installed all over the world with low bandwidth and now you need to run a full copy. All lines saturated, having to pause the copies and given the size of the copy, it is possible that when the copy is finished the database will be sealed again...the snowball effect that backup administrators love so much

The same thing happnes if you have aux copies and the destination database is sealed. More drama.

We ask support to deactivate the "WORM Storage" functionality of all storage policies. The issue has been at level 2 for more than two weeks. Asked managers to escalate the issue to engineering/development and they continue to ignore us.

I have been working in IT for many years, and the support we are receiving from Commvault is not what we are used to from other vendors. This is enough to consider another product backup like Veeam and not looking back.

Imagine working with this size of data and getting no help

 

Incident number #230620-137

icon

Best answer by Jbrown 6 July 2023, 08:25

View original

8 replies

Jbrown
Rank
Userlevel 1
Badge

Hi, firstly on behalf of Commvault Support we’d like to apologise for the issues you have experienced but can assure you that steps are being taken to rectify this for you as soon as possible. You would have received contact from one of our Support Managers this morning, outlining the current status of your Support Incident and our current action plan. Our management team shall continue to have oversight of this incident to ensure progress is made and frequent communication with you is upheld. 

Should you have any further questions or concerns I would request you reach out to us via the email thread we have with you as this has 24x7 oversight from our management layer.

In regards to the technical issue you have outlined, this is of course very unfortunate and we are doing all we can to resolve this for you, but it is not something affecting all Commvault customers as the vast majority have upgraded successfully without reporting any such issues. Rest assured, we will work hard to resolve the situation you are in, and to avoid other customers potentially being impacted.

Onno van den Berg
Rank
Userlevel 7
Badge +19

Curios to learn what happend to your environment @jla and why it happend, because we, as many other customers, upgraded to FR30 without this side-effect and we manage multiple environments. 

K
Rank
Userlevel 1
Badge +5

Did you manage to get this resolved @jla ? Interested to see if this was Commvault that enabled WORM or something else?

J
Rank
Userlevel 1
Badge +2

Did you manage to get this resolved @jla ? Interested to see if this was Commvault that enabled WORM or something else?

Totally NO. Give me a few days and explain what happened.

Jbrown
Rank
Userlevel 1
Badge

To answer your questions @Onno van den Berg and @KevinDodd on how this situation arose; the reason why WORM was enabled in 11.30 from 11.28 was because Hardware WORM had been manually marked as enabled by someone at the customer site on the Commvault library properties. This option was previously named as “Mark Archive File As Read-Only”. As this was enabled, all storage pools were marked as WORM pools. This corrects the incorrect configurations; if WORM was never supported or intended to be used it should not have been enabled in Commvault on the storage level.

Doc reference: Changes in Commvault Platform Release 2023

Commvault Platform Release 2023

Disk Library

The WORM storage lock option is added (at the storage pool level) to the Storage Policy Copy Properties dialog box in CommCell Console.

When WORM storage lock is enabled on a deduplication enabled storage pool, periodic sealing is configured on the storage pool and the jobs are converted to full on the new deduplication database store.

 

We continue to work closely with Mr @jla to assist in recovering from the issues caused by this unfortunate misconfiguration.

K
Rank
Userlevel 1
Badge +5

Thanks for the explanation @Jbrown - very reassuring.

Where can I find this option? I’ve checked the properties of the copy jobs under the storage policy but can’t see it? Just want to make sure it is DISABLED in our environment

S
Rank
Userlevel 6
Badge +14

@KevinDodd

It’s the properties of the Disk Library and Storage Policy.

There are a couple of screenshots on the top of this Post.

This is the old option: “Mark Archive File As Read-Only”

Best Regards,

Sebastien

J
Rank
Userlevel 1
Badge +2

As @Jbrown has commented,

The reason why WORM was enabled in 11.30 from 11.28 was because Hardware WORM was enabled (We are currently investigating how it got activated in all disk libraries)
Previously, this option "Hardware WORM" was named "Mark Archive File As Read-Only".

With the help of support we disabled WORM, uncheck the DDB sealing frecuency and enabled micro-prunning again. Also, deleted old backups jobs to free space.

We are still in the process of moving large amounts of data but we are in a better position compared to a few weeks ago.

I would like to thanks to the Commvault support team; This is an example of how Commvault, as a company, is committed to helping customers, and being a leader.

I would also like to extend my thanks to the following individuals (in no particular order):

- Technical Specialists/Engineers/Support: Dheeraj, Arun, Selvan, Josh, Melissa, Edward, JBrown, Kiran and Vaidotas.
- Support Managers: Anish, Daniel, Blaine

Also, to the rest of the teams who have directly or indirectly assisted us.

J

 

Reply


Terms & ConditionsCookie settings