Solved

Aux Copy Schedule

  • 23 August 2022
  • 8 replies
  • 655 views

Badge +2

Do aux copies start after the backup copy time retention time was reached. Or do Aux copies start at the same start as the backup copy but just to a different location? I am aware Selective only does Full  and Synchronize does Incremental and Full

icon

Best answer by Jos Meijer 24 August 2022, 00:59

View original

8 replies

Userlevel 7
Badge +23

Hi @DBA , thanks for the post!

When you say ‘start’, are you referring to the scheduling of the Aux Copy operation, or the times of the jobs being copied?

The jobs themselves are identical copies in all ways.

The scheduled operation itself will start when the time of the schedule arrives.

Badge +2

I was referring to when the job starts getting copied, it has a daily schedule setup but we are a bit behind in our Aux copies, they just keep adding up, and have been the the last few months.

Userlevel 7
Badge +16

Backup Copies are extracting data from snapshots to a disklibrary, at least based on your description this is my assumption. The Auxiliary Copy will then copy the data to for example a secondary disklibrary or a tape library. An Auxiliary Copy can be started parallel or during a backup copy, only when data at least is available partially the Auxiliary Copy can begin copying. Looking at your last reply you have the setting where Auxiliary processes are re-scanning for new data, when you have to many new daily data or the transfer speed (probably due to lack of either streams/readers, compute or network bandwidth capacity) this effect can occur where Auxiliary copies keep running.

Some questions:

  • What type of data is being Backup Copied / Auxiliary Copied?
  • What is the daily new amount of data coming from the Backup Copy?
  • What is the transfer speed of the Auxiliary Copy?
  • Is this deduplicated data?
  • Do you have a fixed amount readers configured in the Auxliary Copy schedule or do you have it set to Max?
  • Do you use scalable resource allocation in the Auxliary Copy schedule?
    It probably is as your Auxiliary Copies keep getting new data, this option does that if the option “Pick new data for Scalable Resource Auxiliary Copy job” (within Media Management → Auxiliary Copy) is set to value 1.
    If configured what is the Batch Size configured in the Advanced Settings within the schedule → Scalable Resource Allocation Tab?
  • Is the receiving media agent in the Auxiliary Copy being used as a primary target for any backups? If not is the media agent configured with Optimize for concurrent LAN backups (located on Control tab)?
  • Is the Storage Policy configured with the option: “Distribute data evenly among multiple streams for offline read operations”
  • Is the Storage Policy configured with the option “Enable stream randomization for backup/archiver jobs”

 

Badge +2

Hope these answers help out. I was not certain how to verify the info on the third from last question from the bottom. Do you think anything should be changed.  Appreciate the help

  • What type of data is being Backup Copied / Auxiliary Copied? 95% Window servers and the rest are Linux servers 
  • What is the daily new amount of data coming from the Backup Copy? Its slightly different every night but ranges from 60-70TB per night
  • What is the transfer speed of the Auxiliary Copy? Estimate of 700 GB/h
  • Is this deduplicated data? Its deduplicated data on the primary but not the Aux side
  • Do you have a fixed amount readers configured in the Auxliary Copy schedule or do you have it set to Max? It is set to Max
  • Do you use scalable resource allocation in the Auxliary Copy schedule? Yes
    It probably is as your Auxiliary Copies keep getting new data, this option does that if the option “Pick new data for Scalable Resource Auxiliary Copy job” (within Media Management → Auxiliary Copy) is set to value 1. Yes that is set to 1
    If configured what is the Batch Size configured in the Advanced Settings within the schedule → Scalable Resource Allocation Tab? “The max jobs process is se to 1,000” under the scalable resource allocation.
  • Is the receiving media agent in the Auxiliary Copy being used as a primary target for any backups? If not is the media agent configured with Optimize for concurrent LAN backups (located on Control tab)?  One media agents is being used a a proxy for IntelliSnap for one server 18TB’s. Not sure if that answers your question. ??
  • Is the Storage Policy configured with the option: “Distribute data evenly among multiple streams for offline read operations” Yes
  • Is the Storage Policy configured with the option “Enable stream randomization for backup/archiver jobs” Yes

Also there are Aux copies scheduled to start every day at 11am. 

Userlevel 7
Badge +16

The basic issue here seems to be that you have max 70 TB per day with 700 GB/hr auxiliary copy transfer speed. This will take you about 102 hours to complete.

At first glance it seems this is due to either a re-hydration or network performance issue.

  • What storage device and disk type are you using on the source location?
  • Is the target location disk or tape?
    If disk, what storage device and disk type are you using on the target location?
    Also what is the reason for not using deduplication on this disk target?
    If tape, what tape device and tape type are you using on the target location?
    Also how many drives are simultaneously available for the auxiliary copy?
  • How are the source and target storage device connected?
  • Are source and target managed by the same media agent(s) or separate media agent(s).
    If separate, what is the network bandwidth available looking at NIC and network capabilities?
Badge +2

@Jos Meijer These are  Aux copies - DDB is beside the point 

Userlevel 6
Badge +17

Simple question:  Is your network connection pegged during the Aux Copy jobs, or does there appear to be room to spare?  If your network connection is saturated, obviously you cannot go any faster.  If there’s plenty of more bandwidth, the bottleneck is likely either read speeds from the source or write speeds to the destination.  Or maybe you need more streams.

https://documentation.commvault.com/2022e/expert/109223_performance_analysis_tool.html

I might suggest trying the Performance Analysis Tool to see if it can identify any bottleneck.

Thanks,
Scott

Badge +2

The speed time or thoughput for for Aux copies is pretty good 700 GB/h or so. We did do a performance test and what is confusing is the read time is slower than then  write time by a lot. I would have expected the opposite.

Reply