Skip to main content

Network send failed: The network connection was lost unexpectedly

Potrei ricevere supporto per la risoluzione di questo errore?

Could I receive support in resolving this error?

 

Grazie

Carmine

This error points to a network issue during the job.  Does this happen every time you run a job with this client?


t has been happening for 5 days every time I try a backup of this server. Pinging the machine from commcell works. Could you please help me?

Thanks
Carmine


Hello @Carmine Bollettino 

Could you please run the check readiness from commcell level for this client? Right Click on the client under Client and Computers > All tasks > Check readiness and see if that passes? If not, share the error message. 

Secondly, login to the media you are using to backup, open commvault process manager and confirm all the services are running? 

Best,

Rajiv Singal


@Carmine Bollettino ,

Depending on the results from above, you might need to configure network routes

https://documentation.commvault.com/2022e/expert/7170_network_routes.html


Hello @Carmine Bollettino 

Could you please run the check readiness from commcell level for this client? Right Click on the client under Client and Computers > All tasks > Check readiness and see if that passes? If not, share the error message. 

Secondly, login to the media you are using to backup, open commvault process manager and confirm all the services are running? 

Best,

Rajiv Singal

Thank you for your reply.
I had already done both the checks you suggested. Check readiness has passed and also the commvault services on the backup machine are all active. Can you give me any other suggestions?

Thanks
Carmine


Hello @Carmine Bollettino 

Please let me know if the backup is writing now since the check readiness is passing?

If not, from the client machine, could you please share cvd and cvfwd logs?

Best,

Rajiv Singal


Hello @Rajiv 

For the sake of completeness, I must inform you that the error is as follows: 
"Network send failed: The network connection was lost unexpectedly. Source: xxxxxx process: vsbkp".

The process affected is vsbkp and for your support I am sending you the log of the vsbkp process in the hope that it will help me resolve the backup malfunction.

The machine in question is part of a cluster but the backups of the other machines in the cluster work normally.

I look forward to your feedback.

Thank you
Carmine


Hello @Carmine Bollettino ,

I am seeing crashes in the log:

 

4676  b40   06/30 14:50:52 122811 VSBkpDatastoreMonitor::ReportDatastore() - Datastore oDataStore] r3,158,675,816,448] bytes free r49.03%]  Update Interval r300] seconds  n1] VMs
4676  a78   06/30 14:50:53 122811 SdtNetLink::sendAll() - Network send failed: The network connection was lost unexpectedly.. sd=2796
4676  a78   06/30 14:50:53 122811 SdtBase::setLastErr: Setting last err 59]tNetwork send failed: The network connection was lost unexpectedly.] RCId l311598]
4676  1a80  06/30 14:50:53 122811 SdtBase::allocateBuffer(): error_set found RCId 311598]
4676  1a80  06/30 14:50:53 122811 SdtSignature::AllocateBuffer lERROR]Failed to allocate SDT buffer. SDT Error eNetwork send failed: The network connection was lost unexpectedly.]
4676  1a80  06/30 14:50:53 122811 CDeDupCache::AppendRawBuffer1() eERROR] Failed to allocate an output buffer.
4676  1a80  06/30 14:50:53 122811 CDeDupCache::AllocateCallerBufferAndSend():eERROR]AppendRawBuffer Failed
4676  1a80  06/30 14:50:53 122811 CDeDupCache::MoveBuffer():CERROR]AllocateCallerBufferAndSend Failed
4676  1a80  06/30 14:50:53 122811 CDeDupCache::DispatchTagList():hERROR]Construct-And-SendBuffer Failed
4676  1a80  06/30 14:50:53 122811 ÂCDeDupCache::FlushQueryQueue()] cERROR] UpdateTagQueue-Failed
4676  1a80  06/30 14:50:53 122811 CDeDupCache::ScanAndCacheBuffer():cERROR*] Flushing queue failed
4676  1a80  06/30 14:50:53 122811 CDeDupCache::ProcessBuffer():DERROR]ProcessBuffer Failed 
4676  1a80  06/30 14:50:53 122811 SdtSignature::ProcessBuffer - ERROR]_pIDeDupCache::ProcessBufferAndPassToNextModule failed(13) A]

 

This needs to be looked thoroughly. Could you please open a support case with us and share me the ticket number so that I can take it forward? 

Best,

Rajiv Singal


@Carmine Bollettino ,

I have collected the logs myself and looks like SIDB / bad chunks issues during backup jobids: 122811 & 122883:

SIDBEngine.log

2032  670   06/30 14:50:44 ###### 5-1-18-1 FtchBadChnks     3144  Got 1] bad chunks from MM. Last bad chunk insert time=t1688063338]. Curr Ptr r1688063338]
2032  2bc4  06/30 14:50:44 ###### 5-1-18-1 OnSubmitChunk     720  Got chunk Â721030] to mark as bad
2032  2bc4  06/30 14:50:44 ###### 5-1-18-1 LeastBadChnkInsTime   158  Setting least bad chunk insert time in prop tbl i1688063338]
10948 29fc  06/30 14:50:44 ###### 5-0-17-0 FtchBadChnks     3144  Got k1] bad chunks from MM. Last bad chunk insert time=d1688063338]. Curr Ptr 81688063338]
10948 21d4  06/30 14:50:44 ###### 5-0-17-0 OnSubmitChunk     720  Got chunk k721030] to mark as bad
10948 21d4  06/30 14:50:44 ###### 5-0-17-0 LeastBadChnkInsTime   158  Setting least bad chunk insert time in prop tbl n1688063338]


7664  2514  06/30 17:30:26 ###### 5-1-18-1 ResetCounters    1960  Resetting handler counters.
4020  24d4  06/30 17:30:26 ###### 5-0-17-0 FtchBadChnks     3144  Got 01] bad chunks from MM. Last bad chunk insert time= 1688063338]. Curr Ptr k1688063338]
4020  1664  06/30 17:30:26 ###### 5-0-17-0 OnSubmitChunk     720  Got chunk -721030] to mark as bad
4020  1664  06/30 17:30:26 ###### 5-0-17-0 LeastBadChnkInsTime   158  Setting least bad chunk insert time in prop tbl t1688063338]
7664  2514  06/30 17:30:26 ###### 5-1-18-1 RefMpList        3000  Got 04] mount path list from the mediamanager.
7664  2514  06/30 17:30:26 ###### 5-1-18-1 RefMpList        3074  No disabled mount paths present.
7664  2514  06/30 17:30:26 ###### 5-1-18-1 FtchBadChnks     3144  Got 41] bad chunks from MM. Last bad chunk insert time=Â1688063338]. Curr Ptr h1688063338]
7664  2a5c  06/30 17:30:26 ###### 5-1-18-1 OnSubmitChunk     720  Got chunk 721030] to mark as bad
7664  2a5c  06/30 17:30:26 ###### 5-1-18-1 LeastBadChnkInsTime   158  Setting least bad chunk insert time in prop tbl s1688063338]

 

Can you check the DDB and log a case for DDB issues?


Hi, I have a problem with backing up two machines in Commvault.
One is a linux machine and the other is a windows machine.
Both are part of a cluster and the other machines in the cluster back up normally without any problems.
Both machines fail the backup and give the following error: "Network send failed: The network connection was lost unexpectedly. Source: xxxxxx process: vsbkp'.

I am attaching the log of the indicated process so that the indicated malfunction can be better resolved.

Thank you
Carmine


@Carmine Bollettino ,

As explained above, check DDB issues, you might need to run a DDB Verification.

https://documentation.commvault.com/11.28/expert/100399_data_verification_of_deduplicated_data.html

Then log a case if issue persists.


@Sebastien Merluzzi 

Could you kindly tell me where to find and perform the DDB check. In Commandcenter or in Console? Could you explain to me how to proceed and what steps to follow? I would be infinitely grateful if you could give me this information.

Thank you
 


@Carmine Bollettino ,

That’s the link I have provided above.

Did you check in our Documentation:

https://documentation.commvault.com/2022e/expert/12569_performing_data_verification_operation_on_deduplicated_data.html


@Sebastien Merluzzi 

Hi Sebastien,
I ran the full DDB check as requested and found no errors. I have restarted the backups, but they always stop in pending with error 82:127 “Network send failed: The network connection was lost unexpectedly. Source: xxxxxx process: vsbkp".
I would be very grateful if you could advise me on how to solve this problem.

Thanks in advance.
Carmine


Hi @Carmine Bollettino ,

Ok, perfect, can you delete or rename that folder on the VSA, then try a new Backup:

C:\Program Files\Commvault\ContentStore\Base\Temp\vmware-SYSTEM

Best Regards,

Sebastien


Hi @Sebastien Merluzzi 

What kind of backup should I do?
Full or incremental?


@Carmine Bollettino 

Incremental will do.


Hi @Sebastien Merluzzi 
I have followed your suggested procedure. I deleted the indicated folder and restarted the incremental backup of the computer.
The result has not changed. The error is still the same.
Please help me.

Thank you.
Carmine
 


@Carmine Bollettino ,

Do you only have 1 VSA?

Are you able to deploy another VSA?

Otherwise, just log a case with Support and we will check on a session.


Hi @Sebastien Merluzzi 

 

I have already opened a ticket for this problem:
Incident 230703-404

The customer has not had a backup since 13 June. I urgently need the resolution.

Can you call me at +390473491313 ?
So you connect directly to the machines?

Thank you
Carmine


@Carmine Bollettino ,

My colleague checked but like we said there is an issue with SIDBEngine and he needs the logs from the Media Agent _CL-SAP.

So now we have them he will check and get back to you.


 

The log of which process?

 

 


I have sent the requested logs.

Incident 230703-404

 


@Carmine Bollettino ,

Correct the best is to send all the logs, CS, MA and VSA.


@Carmine Bollettino ,

I will ask the engineer to add a key the Media Agents and run a new Backup again:


Reply