Solved

Timeout during backup on Netapp CIFS GRID

  • 8 November 2021
  • 9 replies
  • 97 views

Userlevel 1
Badge +4

Hello,

We are facing the following error:

During the backup job, there is a timeout then disconnection from the CIFS share (backup repository). The backup does not end but continues on the next share.

Configuration:

Commvault 11.22.27

Commserve hostname: lumgt300

Media Agent: 3 MA (lumgt310, lumgt311, lumgt312)

Backup repository : configured in GRID CIFS Share on Ontap 9.7

 

Log error on CVD.log, on 08/11 at 01:28:

4736 1138 11/08 01:28:08 565467 1479749-1175120 [MEDIAFS] SiFS :: WrtToFileInt: Cannot write to metadata file. Input Size [1048028], Bytes written [0], SysErr = 64, Err = 0xECCC0016: {CQiFile :: Write (225) /ErrNo.22. (Invalid argument)} 4736 1138 08/11 01:28:08 565467 1479749-1175120 [MEDIAFS] The operation failed. errorno [1205] 4736 1138 11/08 01:28:08 565467 1479749-1175120 [MEDIAFS] SiFS :: WrtToFile: [ERROR] WriteToCache failed. Ret = [- 1] 4736 1138 11/08 01:28:08 565467 1479749-1175120 [DM_BASE] ProcessCompletedBlocks [9206]: Unable to write non dedup data to meta data file. iRet [-1] 4736 1138 11/08 01:28:08 565467 1479749-1175120 [DM_BASE] WaitAndProcessHeadBlock [9316]: Unable to process completed blocks. iRet [-1] 4736 1138 11/08 01:28:08 565467 1479749-1175120 [DM_BASE] WaitAndProcessAllBlocks [9337]: WaitHeadBlock failed. iRet [-1], Pending [0] 4736 1138 11/08 01:28:08 565467 1479749-1175120 [DM_BASE] OnPlFsCLDedupSignature [6018]: Unable to wait for all the blocks. iRet [-1] 4736 1138 11/08 01:28:08 565467 1479749-1175120 [DM_BASE] WriteATagHdrDataPairForCVSIDevice: 1706 [ERROR] Cannot process the CLDEDUP-signature buffer. 4736 1138 11/08 01:28:08 565467 1479749-1175120 [DM_RECEIVER] DataReceiver :: WritePipeLineBuffer: Error: DataWriter Write Failed 4736 1138 11/08 01:28:08 565467 1479749-1175120 [DSBACKUP] Failed to process the pipeline buffer 4736 1138 11/08 01:28:08 565467 1479749-1175120 [DSBACKUP] telling Index that backup is done, backupState = 0

A case to Netapp has been opened, waiting on reply…

Event details:

 

Thx in advance,

Gilles

icon

Best answer by Gilles SCHMIDT 16 November 2021, 10:45

View original

9 replies

Userlevel 1
Badge +4

Hi again,

After parsing VixDiskLib.log, I’m thinking issue will be due to “transport Mode” for VSA backup. Regarding the conf, the only mode compatible is NBD but subclient is probably configured as Auto (I have to ask to customer), first try during VM backup is SAN but :

→ SupportSanTransport: A disk is on a datastore that is incompatible with SAN mode.

Time on log is exactly the time as the issue occured.

Could it be the reason why ?

 

Gilles

Userlevel 7
Badge +23

Looking through our incident history, “The parameter used for the current operation is not supported by the Operating System” generally indicates hardware issues on the library side.

The main errors in the logs are dedupe errors, so also likely relevant to the hardware.

Do you see anything on the hardware side in the OS logs?

Userlevel 1
Badge +4

Hi Mike,

Hardware library is a Netapp CIFS GRID (4 cifs shares used by 3 MediaAgents). Deduplication is not configured on Netapp side. A case has been opened to Netapp, I’m waiting a reply from them.

I will keep you in touch.

Userlevel 7
Badge +23

Thanks, @Gilles SCHMIDT !

Userlevel 7
Badge +23

Hi @Gilles SCHMIDT , hope all is going well!

Any word from NetApp?

Userlevel 1
Badge +4

Hi Mike, hope you’re doing well !

The error “error occured in disk media… The parameter used for the current operation is not supported by the Operating System”  is a warning probably due to the dedup configuration.  This will be examined later...

I’m focusing on the real error  « failed to mount the disk media in library…. » which appears at 03:51:10 (CVMA.log). I’ve found the following technical note from Netapp :

As recommended by Commvault, I’ve increased the SMB credits to 256, I’m waiting on customer’s reply.

Gilles

 

Userlevel 7
Badge +23

Thanks!  Keep us posted :nerd:

Userlevel 1
Badge +4

Issue related to disconnection from the backup repository CIFS GRID is solved after increasing the smb credits on Ontap SVM. Important to know when we use a cifs grid as backup repository

 

Userlevel 7
Badge +23

Thanks for sharing!

Reply