Skip to main content
commvault.com commvault.com

Storage and Deduplication

Discuss any topic related to storage or deduplication best practices

  • 1,139 Topics
  • 4,859 Replies
1139 Topics
Patrick Dijkgraaf
Commvault Certified Expert
Patrick DijkgraafCommvault Certified Expert
asked in Storage and Deduplication
Amazon S3 Standard-IA/Deep Archive WORM questions

Hi all,one of our customers (running CV11.24.43) is interested in configuring a Selective Copy to Amazon S3 Standard-IA/Deep Archive (Combined Storage Tier). So far, no problem.We intend to copy Monthly Fulls. Basic retention will be 365 days and Yearly Fulls will get extended retention set to 10 years.Also, they want to lock the objects to prevent them from being deleted before the retention is met. The “Enable WORM Storage” workflow should take care of that.But it does raise a few questions:Would you recommend using Deduplication in this scenario, or not? If we use Dedupe, I suppose a DDB seal will take place automatically ever 365 days, right? In this combined storage tier, metadata is written to Standard-IA, and actual backup data is stored in the Deep Archive tier, right? Do we set the object-level retention on both tiers? Retention of Index V2 backups does not follow the Storage Policy settings, and might be pruned earlier than the configured retention on the Storage Policy. What

6503
O
Vaulter
Melissa Adams
Commvault Certified Expert
Melissa AdamsCommvault Certified Expert
asked in Storage and Deduplication
AWS Object Lock issue

Hey all -I have created a bucket with OL enabled, created a new library, storage pool, secondary policies and ran the Enable WORM Storage workflow.  Everything went great but now as I’m starting my Aux Copy, I’m getting this error:12504 1938  05/12 12:07:09 4148  [DEVICE_IO] CVRFAMZS3::SendRequest() - Error:  Message: Content-MD5 OR x-amz-checksum- HTTP header is required for Put Object requests with Object Lock parameters12504 1938  05/12 12:07:09 4148  [DEVICE_IO]  Message: Content-MD5 OR x-amz-checksum- HTTP header is required for Put Object requests with Object Lock parametersI’m not sure how to fix this issue.  Anyone have ideas?

6726
M
Vaulter
K
Bit
Kegan LevettBit
asked in Storage and Deduplication
Projecting / Estimating new tape requirements

Hello, Is there a feature in Commvault that estimates / projects what amount of new tapes would be required for a specific future time range? Just looking for something more concrete than observations from new tapes being uses / purchase history.

951
Mike Struening
Vaulter
J
Byte
JoergByte
asked in Storage and Deduplication
How many drives can we configure to one SAN-Attached tape Library.

V11.25 SAN-Attached Tape LibraryHow many drives are supported at one library? We use a big VTL and have no problem to configure 500 and more to one tape lib. Now we have unclear failures and I want to check in general if we have reached a hard coded limit. We work with 600 and more drives.I looked up documentation.commvault.com but cannot find anything about limits.Can you help? 

1984
Mike Struening
Vaulter
T
Byte
Tom EvansByte
asked in Storage and Deduplication
Network Bandwidth Throttling

Hi,Im having issues with throttling network utilisation for aux copies. I have about 10 Storage Policies all with Secondary copies.I’ve configured “Throttle Network Bandwidth (MB/HR)” to 25000 for the secondary copy for all storage policy. If my maths is correct that would be approx 50Mbps per aux copy. Even with all 10 running at the same times utilisation shouild only be approx 500Mbps.However through network montioring i can see when these aux copies are running they are using well over the 50Mbps configured (and saturating the network).Is my maths wrong is or the throttling configuration not  working how i expect it to work?Thanks in advance for any responses.

24486
Mike Struening
Vaulter
T
Byte
Thos GieskesByte
asked in Storage and Deduplication
what are the password complexity rules for a mount paths service account?

hi, we had a services engineer create a new mount path using special caracters for the services account.how ever it gave an error when trying to login, that there where insufficient permissions.they only changed was the password to be less complex (just basic caracters) so there are expecting they used a caracter that is not supported.i checked the docs and i could not find articles stating this for the a Mount path. i was able to find ones for Commserve livesync and other components.do the same rules apply to a mount path service account? Kind Regards, Thos Gieskes.

2471
M
Vaulter
Nikos.Kyrm
Byte
Nikos.KyrmByte
posted in Storage and Deduplication
Auxiliary copy VS Dash copy

Hello to all!In case of jobs replication (secondary copy), I see that these 2 options are available: Auxiliary copy & Dash copy.Whats the pros and cons of each?Dash copy is for deduplicated data (with Commvault dedup engine) and the Auxiliary copy for the rest “transitional data” / jobs?Lets share your thoughts.Nikos

64423
Jos Meijer
Commvault Certified Expert
M
Bit
Marty at ARABit
asked in Storage and Deduplication
VTL tape size

Recently created 2 Tb VTL tapes on DD9400. However CV is detecting the size as 399.99 Gb. 

1365
M
Vaulter
E
Byte
Eduardo BragaByte
asked in Storage and Deduplication
DDB Verification and Aux Copy Jobs impact on DDB disk

Could the DDB Verification and the Auxiliary Copy jobs impact on DDB disk in terms of Query & Insert time (the calculation)? We have some Database's Average Q&I Times across Partitions in the last 3 and 14 days with very high values. Backups jobs are running fine. No Performance degradation during backups with a good Deduplication ratio. There are no significant events. Most of the time we’re doing Database Log backups (So, no dedup here) and running Admin Jobs to copy data onto tapes or to create secondary copies. At night, we're doing incremental backups (VMware and FS agents). We have an issue with Synthetic Full backups (https://community.commvault.com/commvault-q-a-2/synthetic-full-is-not-running-based-on-automatic-schedule-settings-2744) and I'm wondering about how this issue also can impact on those times. Some info:VMware VM Media Agents are running updated Linux. 8 CPU cores, 32 GB RAM. DDB Flash disk. Kernel Parameter Configuration done. DDB Verification: Incremental

7306
christopherlecky
Byte
L
Byte
LiorRNByte
asked in Storage and Deduplication
Can I change the ddb network interface

Hi,Can I change the ddb network interface? and if I can, how?

3877
christopherlecky
Byte
Michal128
Byte
Michal128Byte
asked in Storage and Deduplication
Archive backup from local Commserve on tapes to AZURE.

Hello, One of our client has Archive backup on tapes which is configured in one location and that type of infra is managed by local Commserve server. There is a question about the possibility of migration that Archive backup to AZURE. In the end client would like to get all archive backup in AZURE cloud with possibility of manage the backup from Commserve in AZURE cloud which is a different machine (Global solution for organization). And my question what will be the best approach for that migration and how to calculate the storage usage in AZURE for that migration process? Regards, Michal 

3577
christopherlecky
Byte
Stefano Castelli
Stefano Castelli
asked in Storage and Deduplication
Low deduplication ratio on Oracle RAC DBs: why?

Hello everyone and thanks in advance for any hint.I’m running a Commvault V11Sp20 installation of the software and recently had a 6TB Oracle Database migrated from a stand-alone Oracle 12 DB to a three-node Oracle RAC 19 installation. Backups work properly and, “randomly” the three nodes are involved in backup jobs.Problem is: dedupe ratio is low.While the stand-alone data reached 95/97% of dedupe ratio, these backups for the Oracle RAC instances reach 40/65% of dedupe ratio. That is, a single Full backup takes something like 3TB of disk library (the very same disk library hosting the GDP previously used for storing the stand-alone Oracle DBs backups). Any hints about why the dedupe ratio is so low? Thanks in advance for your kind opinion. Regards

60910
F
Bit
L
Byte
LucyByte
asked in Storage and Deduplication
Aux copy behaviour - scalable resource allocation

Hello all, I have a question about my aux copy behaviour.We use S3 type storage on site for our backups and aux these to an private cloud provider. I’ve noticed that in the aux copies certain clients jobs will consistently be skipped. Initially I thought the issue was bandwidth as the aux copy never completed. We have improved the bandwidth and throughput is much better now, however the aux copies are still not completing. If I go to the secondary copy and show jobs (unticking time range and excluding “available”) I will have a number of jobs going back weeks, with none of the jobs from that client showing partially available (implying it is part way through copying). Interestingly, today I have started the aux copy with “Use scalable resource allocation” UNTICKED, and those old jobs have immediately been picked up and started copying.Anyone have any ideas why this would be? I’m curious what impact this will have on my environment. I just don’t get why most jobs were copying and it was

170411
Mike Struening
Vaulter
Melissa Adams
Commvault Certified Expert
Melissa AdamsCommvault Certified Expert
asked in Storage and Deduplication
Egress Charges on AWS Aux copy

Hey all - can someone explain this statement in the Cloud Architecture Guide for AWS (page 29)?Do not utilize an on-premises MediaAgent to perform Auxiliary copy as Cloud egress costs (S3 Pricing >) will be incurred.We have customers doing this very thing and one is having an issue with high egress charges.  We are not sure that the charges are from Commvault but want to make sure.  The customer turned off data verification but that did not make a difference.  They are writing Oracle logs directly to the cloud library (instead of primary onsite) and are very tight on hard drive space and have a lot of index restores happening but I don’t see either of those things causing large Egress.  The index restores would come from the primary onsite copy.Can someone point me in the right direction on if there are other things with this aux copy that could cause egress?Thanks!Melissa

2385
Mike Struening
Vaulter
S
Byte
SkarraByte
asked in Storage and Deduplication
Oracle channels and effects on deduplication efficiency

Is there any downside to using multiple channels (data streams) for Oracle RMAN backups when client-side deduplication is configured?  i.e. is deduplication negatively impacted vs using a single channel.Thank you.

3292
S
Byte
T
Byte
tmyooByte
asked in Storage and Deduplication
PTL mount/unmount error during Oracle DB recovery

HelloAt the request of the client company, I proceeded with the recovery of Oracle DB backed up in PTL Media as follows.1. Backed up original DB to PTL Media. (SAN)2. The DB backed up with PTL was restored to the recovery server. (Network, using Backup Server MA)Different drives were used for backup and recovery.No issues occurred during the backup.The following message is output with a Failed unmount error during the recovery process:The path is being accessed by another application.Advice : please make sure that no other device explorer application like SAN explorer are running on the machine.PTL devices are used by Commvault only.I would appreciate it if you could advise me on what to check to resolve the issue.

761
Mike Struening
Vaulter
K
Bit
Kegan LevettBit
asked in Storage and Deduplication
Drive stuck at the "Media is being Unmounted" status

Hello,I have a tape that appears to be stuck in one of the library drives. The tape itself  reports that it is stuck in the drive. I have tried the Clean / Reload / Unload Drive options to do effect. I also restarted the Quantum tape library to see if that helps, again no luck.     Cheers  

25246
K
Bit
T
Byte
tmyooByte
asked in Storage and Deduplication
Backup data from the disk library is not Aging normally.

HelloCommvault Version uses the V11 SP16.The OS of the Media Server with DISK LIBRARY where Aging is not working is AIX.When checking the contents of DISK LIBRARY in Commvault, the data shows 2.21TB written.When checking the CV_MAGNETIC path usage capacity of DISK LIBRARY PATH on the media server, it shows that you are using approximately 11TB.If you have any experience in solving this phenomenon, please share your solution.  

1903
Mike Struening
Vaulter
X
Byte
Xb-WolfByte
asked in Storage and Deduplication
selective copy media not copied: how do I back up the missed jobs?

Hi, I have a selective copy that backup a week's data on a tape. This tape i export every friday offsite.This week i I did not have more scratch and if i check media not copied on a selective copy i see more data.Now have scratch tape, and if i check jobs faile for storage policy there are more on status: not selected.how do i relaunch those jobs? if i search on job history same jobs id, there are on status completed because not saved only selective copy

6402
X
Byte
M
Byte
marcelo bernardes fernandesByte
asked in Storage and Deduplication
error 10:64

Has anyone seen the error below? I just created a new Storage Policy , associated it to the client and did the test and it is returning the following error:  

23468
Mike Struening
Vaulter
Pasqual Döhring
Byte
Pasqual DöhringByte
posted in Storage and Deduplication
Understanding DDB Statistics

Hello there. In the last couple of months I was asked by a few customers where all the data growth on their disk libraries came from. And by digging deeper into that topic I tried to understand the data and statistics from the DDBs. I had to realize that some of those statistics seem to be very odd and that I don’t understand them. So maybe someone can enlighten me. Let’s take a look at a screenshot from statistics of a “normal” DDB: So what do we got here:Deduplication Engine Name: self-explanatoryCreation Time: self-explanatoryVersion: I just take this as it is but is there anything out there but Version 11? Does it refer to Commvault 11? I thought a recent DDB should be Version 5 (or 4.2 if you prefer) and a somewhat older DDB should be version 4. So what does “11” mean”?Total Application Size: AFAIK this is the total data that got backed up into this DDB. All fulls, incrementals and so on before compression and dedupe.Total Data Size on Disk: Seems to be self-explanatoryDeduplicati

15806
Pasqual Döhring
Byte
Mohit Chordia
Byte
Mohit ChordiaByte
asked in Storage and Deduplication
Isilon Smart lock as WORM Enabled Library

Hi Team,Iam testing Isilon Smart lock feature for storage level immutability with Commvault  DASH Aux Copies.My Dedupe is hosted on local drive ( non-worm) on windows media agent .Library is created from smart lock directory from Isilon .My question is that how does CV writes to WORM enabled storage ? As per the current settings on Isilon Smart lock there is autocommit time period of 5 mins  ... files that have been in a SmartLock directory for a 5 mins of time without being modified will  automatically committed to a WORM state.Can Commvault handle this because i guess there will be chunks which will not modify for 5 mins and need to update later during Aux copy process? Will the Aux copy fail or create new chunks/folders ?What about DDB ? Keeping DDB on non-worm and data on worm is recommended way ?Regards,Mohit

7929
SoundTheDreadAlarm
Byte
Melissa Adams
Commvault Certified Expert
Melissa AdamsCommvault Certified Expert
asked in Storage and Deduplication
Video data backups/secondary copies with Dedupe

Good morning I have a customer that is backing up in car videos (local sheriff’s dept) and he is deduping this data.  We have created a new stand alone dedupe database for this data and are using a selective copy to copy weekly fulls to another target.  We have set up another stand alone dedupe database for this copy.  The customer is having issues getting this data copied over - it’s taking multiple days.  The full backup is about 40 TB in size.  We have recommended not using dedupe for this copy, but the customer does not have enough capacity for non-deduped copies.  Are there any optimizations that can be made to speed up this aux copy?  They currently have only 1 subclient backing up all the data - if they split this out, would that help in using multiple streams for the aux copy?  Any other suggestions? Thanks

71623
Mike Struening
Vaulter
L
Byte
lborekByte
asked in Storage and Deduplication
DDB and disk library limits

Hi! Some time ago we set up Commcell dedicated to Hadoop environment. One media agent, one DDB (two partitions, hosted on local SSDs), deduplication block size increased to 512KB, disk library from few external SAN attached storage arrays. And it keep growing :) Few incremental jobs a day, writing ~10TB, synthetic full each 10 days. Without any performance issues so far. DDB Q&I ~ 500ns. Both backup and restore time acceptable. It’s kind of archive, not typical backup solution. Last month I was asked to disable retention, because ongoing migration activities. And it keep growing faster. I assume until 2PB on disk library by the end of year.  So wondering when I’d hit the ceiling for this setup? I know it breaks media agent and deduplication sizing, but until Q&I time are fine everything is acceptable? Or i would hit other limit? It need to survive until end of this year! :)  Some numbers below : Thanks!   

10411
Mike Struening
Vaulter
C
Byte
christopheByte
asked in Storage and Deduplication
On premise S3 Cloud and DDB data verification

Hi,i have a question about DDB data verification on a Scality ring S3 cloud (on premise).We have around 800TB of stored data on S3.Today the ddb verification move on average 1000 TB every month (1.5X as the volume of data stored on S3 scality).We have set the DDB data verification option to “Quick Verification of Deduplication Database”it is possible to reduce the number of GET per month by changing the way DDB verification run?regards,  

1995
O
Vaulter
description of image

Badge winners

  • Topic Starter 1
    rahul_nairhas earned the badge Topic Starter 1
  • Contributor 5
    Kalaivananhas earned the badge Contributor 5
  • Topic Starter 1
    leanhas earned the badge Topic Starter 1
  • Topic Starter 25
    Sergio Vhas earned the badge Topic Starter 25
  • Contributor 5
    LukasBChas earned the badge Contributor 5
Show all badges
Terms & ConditionsCookie settings