Solved

AIX Job results folder full

  • 29 September 2021
  • 7 replies
  • 1012 views

Badge +1

HI 
I am backing up an AIX (7.2) sever with an Oracle data base (19C), data base size is ± 20TB. Problem I am having is the job results folder keeps filling up, till now my colleagues have just been extending the volume to make the problem go away, however we are now sitting with jobs result folder that is 50Gb in size.

I have arranged root access to the server in question and found that the problem is actually in the CV_CLDB folder where I have a CacheTable.dat file that is currently around 35Gb in size and a CacheTable.idx that is currently around 14Gb in size, so my understanding is that these should shrink when pruning takes place which set for 3 days but it would appear that this is not happening, I have tried changing the pruning frequency to check if it makes a difference but nothing, not sure how to reduce the size of these files at this point, and don't want to keep throughing space at the problem.

Commvault version installed is 11.20.36

icon

Best answer by Ledoesp 29 September 2021, 16:14

View original

7 replies

Userlevel 5
Badge +13

I think the CV_CLDB folder is created when you have enabled source side dedupe with disk cache, please disable if you confirm it is enabled or decrease its size to a lower value

https://documentation.commvault.com/11.24/expert/93891_online_help.html#b59047_deduplication_advanced_client_properties

https://documentation.commvault.com/11.24/expert/12451_modifying_deduplication_properties_for_client.html

 

Limit the Max Cache size to <n> MB

Use this option to set the maximum size of the source-side disk cache (CV_CLDB). The default value is 4096 MB (4 GB). 

 

Normally you enable this setting when the client and Media Agent are WAN connected and not when they are in the same datacenter.

Badge +1

Thanks made the change and managed to get a backup going again, having said that the job results folder on the AIX server still reports as full, I take it this will clear out after some time ?

Userlevel 5
Badge +13

I would remove the source-side disk cache (CV_CLDB) manually if you have access to the operating system and you have disabled the source side disk cache option for this AIX client. 

Badge +1

Seems I was mistaken the job (file system backup) is not actually running it was stuck in the scan phase for some time and has finally come back with the same error indicating to no space on the job results folder.

So I have tried running it with client side dedup and disk cache limited to 4096MB, client side dedup disabled, as well as storage policy setting and all come back with the same alert.

Description: File scan failed to write to dirchange file [/opt/commvault/iDataAgent/jobResults/2/1368/DCTmp.cvf]. Possible reasons include nonexistent path/file, hardware problems or insufficient disk space.
Source: <ClientServer>, Process: FileScan

How do I shrink this CV_CLDB ?

 

Userlevel 5
Badge +13

Interesting, you are not running an Oracle backup but a File system backup, depending of the amount of objects the Job Results folder might need to have enough space to accomodate our collect files.

 

Can you expand or point the Jobs Results folder to another file system within the machine? Also please make sure you are filtering the Oracle dbf files from the FS backup.

You can use this link to help you calculate the space needed for the Job Results folder (provided you are filtering the Oracle Database from FS backup)

https://documentation.commvault.com/11.25/expert/6599_job_results_directory_disk_space_calculation.html

 

Also I mentioned before that use of the disk cache is for limited bandwith between client and MA, I would not expect to use this setting if you are trying to protect a +-20 TB Oracle database as the MA should be as close in the network to this type of client. In short, do not use the disk cache setting and keep source side dedupe.

Badge +1

The job was originally setup to run with storage policy settings which is client side dedup, and that is for both oracle and File systems backups, all of which was working fine till a few weeks ago when only this server (from multiple with the same config) started running out of space on the job results folder, and to my knowledge nothing has changed in that time. We have several similar sized systems running the same OS and same backup config with no issues all there CL_CVDB's are under 20GB, so this one perplexes me some what.

Any way will log a call with the OS team to resize the volume and continue testing from there.

Userlevel 5
Badge +13

Below the default settings for the deduplication at the primary copy level of the storage policy and the client deduplication options:

Question is what value was defined for the disk cache? You haven’t mentioned the value before. Job results pruning does not clean the entries for the client side disk cache, once it has reached the maximum size defined it starts to write from the beginning (in a circular way) so planning the disk space is needed. You’ll have a disk cache per subclient, you mentioned a 20 TB Oracle database, it makes sense that it has reached the maximum size defined provided Oracle agent is used for protecting the database, no sure if other similar sized systems you mentioned have also a 20 TB Oracle database or not or the Max Cache size defined is different to this system.

As a guess, imagine that the database has encryption enabled, this will trigger a lot of hashes that will fill very quickly the dedupe cache on the client.

 

If you disable this setting, you won’t have to worry about resizing the file system for the Job Results path. 

 

 

Reply