Skip to main content
commvault.com commvault.com
Solved

Do not update to version 11.30 or 11.32 (drama inside)

J
Byte
Forum|alt.badge.img+4
  • jla
  • Byte
  • 13 replies

I want to explain a problem we have:

Hardware WORM was active on all of our disk libraries.

 

We are not aware of having activated this functionality and as support indicated, there is no trace that we have executed a workflow to activate it.

Despite that, the WORM functionality was not active in any of our Storage Policies, only at the disk library level. We could change the retention of our copies, delete backups, micro-prunning working, and our databases were NEVER sealed.

 

We updated version 11.28 to 11.30 and suddenly the “WORM Storage” feature is activated in ALL storage policies at storage pool level.

As per the below document, this changes are applied on SP30 onwards, if hardware WORM is enabled:

https://documentation.commvault.com/2023/essential/149133_changes_in_commvault_platform_release_2023.html

 

Now we cannot delete backups, nor change the retention of our copies and ALL databases will be sealed every X days, micro-prunning disable. In addition, we have not been able to plan the increase of space in the libraries (run out of space on one of them), nor have we been able to adjust the retention of copies before this functionality was activated.

Total disaster.

Why was the "WORM Storage" functionality activated at the storage policy level if we did not have it activated at that level?

Our copies are a mix between incremental copies and synthetic copies.

Now when the database is sealed and a new one is created, there is no baseline and the first copy that is executed is a full copy. Full copy on ALL infraestructure.

Also a problem if you have agents installed all over the world with low bandwidth and now you need to run a full copy. All lines saturated, having to pause the copies and given the size of the copy, it is possible that when the copy is finished the database will be sealed again...the snowball effect that backup administrators love so much

The same thing happnes if you have aux copies and the destination database is sealed. More drama.

We ask support to deactivate the "WORM Storage" functionality of all storage policies. The issue has been at level 2 for more than two weeks. Asked managers to escalate the issue to engineering/development and they continue to ignore us.

I have been working in IT for many years, and the support we are receiving from Commvault is not what we are used to from other vendors. This is enough to consider another product backup like Veeam and not looking back.

Imagine working with this size of data and getting no help

 

Incident number #230620-137

Best answer by Jbrown

To answer your questions @Onno van den Berg and @KevinDodd on how this situation arose; the reason why WORM was enabled in 11.30 from 11.28 was because Hardware WORM had been manually marked as enabled by someone at the customer site on the Commvault library properties. This option was previously named as “Mark Archive File As Read-Only”. As this was enabled, all storage pools were marked as WORM pools. This corrects the incorrect configurations; if WORM was never supported or intended to be used it should not have been enabled in Commvault on the storage level.

Doc reference: Changes in Commvault Platform Release 2023

Commvault Platform Release 2023

Disk Library

The WORM storage lock option is added (at the storage pool level) to the Storage Policy Copy Properties dialog box in CommCell Console.

When WORM storage lock is enabled on a deduplication enabled storage pool, periodic sealing is configured on the storage pool and the jobs are converted to full on the new deduplication database store.

 

We continue to work closely with Mr @jla to assist in recovering from the issues caused by this unfortunate misconfiguration.

View original
Did this answer your question?

8 replies

Jbrown
Vaulter
Forum|alt.badge.img
  • JbrownVaulter
  • Vaulter
  • 2 replies
  • July 3, 2023

Hi, firstly on behalf of Commvault Support we’d like to apologise for the issues you have experienced but can assure you that steps are being taken to rectify this for you as soon as possible. You would have received contact from one of our Support Managers this morning, outlining the current status of your Support Incident and our current action plan. Our management team shall continue to have oversight of this incident to ensure progress is made and frequent communication with you is upheld. 

Should you have any further questions or concerns I would request you reach out to us via the email thread we have with you as this has 24x7 oversight from our management layer.

In regards to the technical issue you have outlined, this is of course very unfortunate and we are doing all we can to resolve this for you, but it is not something affecting all Commvault customers as the vast majority have upgraded successfully without reporting any such issues. Rest assured, we will work hard to resolve the situation you are in, and to avoid other customers potentially being impacted.

Scott H
Jon Vengust
2 people like this
  • Scott H
    Scott H
  • Jon Vengust
    Jon Vengust
Onno van den Berg
Commvault Certified Expert
Forum|alt.badge.img+19

Curios to learn what happend to your environment @jla and why it happend, because we, as many other customers, upgraded to FR30 without this side-effect and we manage multiple environments. 

Scott H
Jon Vengust
2 people like this
  • Scott H
    Scott H
  • Jon Vengust
    Jon Vengust
K
Byte
Forum|alt.badge.img+5

Did you manage to get this resolved @jla ? Interested to see if this was Commvault that enabled WORM or something else?

J
Byte
Forum|alt.badge.img+4
  • jla
  • Author
  • Byte
  • 13 replies
  • July 6, 2023
KevinDodd wrote:

Did you manage to get this resolved @jla ? Interested to see if this was Commvault that enabled WORM or something else?

Totally NO. Give me a few days and explain what happened.

Jbrown
Vaulter
Forum|alt.badge.img
  • JbrownVaulter
  • Vaulter
  • 2 replies
  • Answer
  • July 6, 2023

To answer your questions @Onno van den Berg and @KevinDodd on how this situation arose; the reason why WORM was enabled in 11.30 from 11.28 was because Hardware WORM had been manually marked as enabled by someone at the customer site on the Commvault library properties. This option was previously named as “Mark Archive File As Read-Only”. As this was enabled, all storage pools were marked as WORM pools. This corrects the incorrect configurations; if WORM was never supported or intended to be used it should not have been enabled in Commvault on the storage level.

Doc reference: Changes in Commvault Platform Release 2023

Commvault Platform Release 2023

Disk Library

The WORM storage lock option is added (at the storage pool level) to the Storage Policy Copy Properties dialog box in CommCell Console.

When WORM storage lock is enabled on a deduplication enabled storage pool, periodic sealing is configured on the storage pool and the jobs are converted to full on the new deduplication database store.

 

We continue to work closely with Mr @jla to assist in recovering from the issues caused by this unfortunate misconfiguration.

Scott Moseman
S
M
3 people like this
  • Scott Moseman
    Scott Moseman
  • S
    Sebastien Merluzzi
  • M
    Maithreyi
K
Byte
Forum|alt.badge.img+5

Thanks for the explanation @Jbrown - very reassuring.

Where can I find this option? I’ve checked the properties of the copy jobs under the storage policy but can’t see it? Just want to make sure it is DISABLED in our environment

S
Vaulter
Forum|alt.badge.img+14

@KevinDodd

It’s the properties of the Disk Library and Storage Policy.

There are a couple of screenshots on the top of this Post.

This is the old option: “Mark Archive File As Read-Only”

Best Regards,

Sebastien

J
Byte
Forum|alt.badge.img+4
  • jla
  • Author
  • Byte
  • 13 replies
  • July 11, 2023

As @Jbrown has commented,

The reason why WORM was enabled in 11.30 from 11.28 was because Hardware WORM was enabled (We are currently investigating how it got activated in all disk libraries)
Previously, this option "Hardware WORM" was named "Mark Archive File As Read-Only".

With the help of support we disabled WORM, uncheck the DDB sealing frecuency and enabled micro-prunning again. Also, deleted old backups jobs to free space.

We are still in the process of moving large amounts of data but we are in a better position compared to a few weeks ago.

I would like to thanks to the Commvault support team; This is an example of how Commvault, as a company, is committed to helping customers, and being a leader.

I would also like to extend my thanks to the following individuals (in no particular order):

- Technical Specialists/Engineers/Support: Dheeraj, Arun, Selvan, Josh, Melissa, Edward, JBrown, Kiran and Vaidotas.
- Support Managers: Anish, Daniel, Blaine

Also, to the rest of the teams who have directly or indirectly assisted us.

J

 

Damian Andre
Jbrown
M
6 people like this
  • Damian Andre
    Damian Andre
  • Jbrown
    Jbrown
  • M
    Maithreyi
  • Ralph
    Ralph
  • M
    mparise-cvlt
  • M
    mihaib200

Reply


Most helpful members this week

Terms & ConditionsCookie settings

Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings