Network send failed: The network connection was lost unexpectedly
Potrei ricevere supporto per la risoluzione di questo errore?
Could I receive support in resolving this error?
Â
Grazie
Carmine
Network send failed: The network connection was lost unexpectedly
Potrei ricevere supporto per la risoluzione di questo errore?
Could I receive support in resolving this error?
Â
Grazie
Carmine
This error points to a network issue during the job. Does this happen every time you run a job with this client?
t has been happening for 5 days every time I try a backup of this server. Pinging the machine from commcell works. Could you please help me?
Thanks
Carmine
Hello
Could you please run the check readiness from commcell level for this client? Right Click on the client under Client and Computers > All tasks > Check readiness and see if that passes? If not, share the error message.Â
Secondly, login to the media you are using to backup, open commvault process manager and confirm all the services are running?Â
Best,
Rajiv Singal
Depending on the results from above, you might need to configure network routes
https://documentation.commvault.com/2022e/expert/7170_network_routes.html
HelloÂ
Could you please run the check readiness from commcell level for this client? Right Click on the client under Client and Computers > All tasks > Check readiness and see if that passes? If not, share the error message.Â
Secondly, login to the media you are using to backup, open commvault process manager and confirm all the services are running?Â
Best,
Rajiv Singal
Thank you for your reply.
I had already done both the checks you suggested. Check readiness has passed and also the commvault services on the backup machine are all active. Can you give me any other suggestions?
Thanks
Carmine
Hello
Please let me know if the backup is writing now since the check readiness is passing?
If not, from the client machine, could you please share cvd and cvfwd logs?
Best,
Rajiv Singal
Hello
For the sake of completeness, I must inform you that the error is as follows:Â
"Network send failed: The network connection was lost unexpectedly. Source: xxxxxx process: vsbkp".
The process affected is vsbkp and for your support I am sending you the log of the vsbkp process in the hope that it will help me resolve the backup malfunction.
The machine in question is part of a cluster but the backups of the other machines in the cluster work normally.
I look forward to your feedback.
Thank you
Carmine
Hello
I am seeing crashes in the log:
Â
4676  b40  06/30 14:50:52 122811 VSBkpDatastoreMonitor::ReportDatastore() - Datastore oDataStore] r3,158,675,816,448] bytes free r49.03%]  Update Interval r300] seconds  n1] VMs
4676 Â a78 Â 06/30 14:50:53 122811 SdtNetLink::sendAll() - Network send failed: The network connection was lost unexpectedly.. sd=2796
4676 Â a78 Â 06/30 14:50:53 122811 SdtBase::setLastErr: Setting last err 59]tNetwork send failed: The network connection was lost unexpectedly.] RCId l311598]
4676 Â 1a80 Â 06/30 14:50:53 122811 SdtBase::allocateBuffer(): error_set found RCId 311598]
4676 Â 1a80 Â 06/30 14:50:53 122811 SdtSignature::AllocateBuffer lERROR]Failed to allocate SDT buffer. SDT Error eNetwork send failed: The network connection was lost unexpectedly.]
4676 Â 1a80 Â 06/30 14:50:53 122811 CDeDupCache::AppendRawBuffer1() eERROR] Failed to allocate an output buffer.
4676 Â 1a80 Â 06/30 14:50:53 122811 CDeDupCache::AllocateCallerBufferAndSend():eERROR]AppendRawBuffer Failed
4676 Â 1a80 Â 06/30 14:50:53 122811 CDeDupCache::MoveBuffer():CERROR]AllocateCallerBufferAndSend Failed
4676 Â 1a80 Â 06/30 14:50:53 122811 CDeDupCache::DispatchTagList():hERROR]Construct-And-SendBuffer Failed
4676 Â 1a80 Â 06/30 14:50:53 122811 ÂCDeDupCache::FlushQueryQueue()] cERROR] UpdateTagQueue-Failed
4676 Â 1a80 Â 06/30 14:50:53 122811 CDeDupCache::ScanAndCacheBuffer():cERROR*] Flushing queue failed
4676 Â 1a80 Â 06/30 14:50:53 122811 CDeDupCache::ProcessBuffer():DERROR]ProcessBuffer FailedÂ
4676 Â 1a80 Â 06/30 14:50:53 122811 SdtSignature::ProcessBuffer - ERROR]_pIDeDupCache::ProcessBufferAndPassToNextModule failed(13) A]
Â
This needs to be looked thoroughly. Could you please open a support case with us and share me the ticket number so that I can take it forward?Â
Best,
Rajiv Singal
I have collected the logs myself and looks like SIDB / bad chunks issues during backup jobids:Â 122811 & 122883:
SIDBEngine.log
2032  670  06/30 14:50:44 ###### 5-1-18-1 FtchBadChnks   3144  Got 1] bad chunks from MM. Last bad chunk insert time=t1688063338]. Curr Ptr r1688063338]
2032  2bc4  06/30 14:50:44 ###### 5-1-18-1 OnSubmitChunk   720  Got chunk Â721030] to mark as bad
2032  2bc4  06/30 14:50:44 ###### 5-1-18-1 LeastBadChnkInsTime  158  Setting least bad chunk insert time in prop tbl i1688063338]
10948 29fc  06/30 14:50:44 ###### 5-0-17-0 FtchBadChnks   3144  Got k1] bad chunks from MM. Last bad chunk insert time=d1688063338]. Curr Ptr 81688063338]
10948 21d4  06/30 14:50:44 ###### 5-0-17-0 OnSubmitChunk   720  Got chunk k721030] to mark as bad
10948 21d4  06/30 14:50:44 ###### 5-0-17-0 LeastBadChnkInsTime  158  Setting least bad chunk insert time in prop tbl n1688063338]
7664  2514  06/30 17:30:26 ###### 5-1-18-1 ResetCounters   1960  Resetting handler counters.
4020  24d4  06/30 17:30:26 ###### 5-0-17-0 FtchBadChnks   3144  Got 01] bad chunks from MM. Last bad chunk insert time= 1688063338]. Curr Ptr k1688063338]
4020  1664  06/30 17:30:26 ###### 5-0-17-0 OnSubmitChunk   720  Got chunk -721030] to mark as bad
4020  1664  06/30 17:30:26 ###### 5-0-17-0 LeastBadChnkInsTime  158  Setting least bad chunk insert time in prop tbl t1688063338]
7664  2514  06/30 17:30:26 ###### 5-1-18-1 RefMpList     3000  Got 04] mount path list from the mediamanager.
7664  2514  06/30 17:30:26 ###### 5-1-18-1 RefMpList     3074  No disabled mount paths present.
7664  2514  06/30 17:30:26 ###### 5-1-18-1 FtchBadChnks   3144  Got 41] bad chunks from MM. Last bad chunk insert time=Â1688063338]. Curr Ptr h1688063338]
7664  2a5c  06/30 17:30:26 ###### 5-1-18-1 OnSubmitChunk   720  Got chunk 721030] to mark as bad
7664  2a5c  06/30 17:30:26 ###### 5-1-18-1 LeastBadChnkInsTime  158  Setting least bad chunk insert time in prop tbl s1688063338]
Â
Can you check the DDB and log a case for DDB issues?
Hi, I have a problem with backing up two machines in Commvault.
One is a linux machine and the other is a windows machine.
Both are part of a cluster and the other machines in the cluster back up normally without any problems.
Both machines fail the backup and give the following error: "Network send failed: The network connection was lost unexpectedly. Source: xxxxxx process: vsbkp'.
I am attaching the log of the indicated process so that the indicated malfunction can be better resolved.
Thank you
Carmine
As explained above, check DDB issues, you might need to run a DDB Verification.
https://documentation.commvault.com/11.28/expert/100399_data_verification_of_deduplicated_data.html
Then log a case if issue persists.
Could you kindly tell me where to find and perform the DDB check. In Commandcenter or in Console? Could you explain to me how to proceed and what steps to follow? I would be infinitely grateful if you could give me this information.
Thank you
Â
That’s the link I have provided above.
Did you check in our Documentation:
Hi Sebastien,
I ran the full DDB check as requested and found no errors. I have restarted the backups, but they always stop in pending with error 82:127 “Network send failed: The network connection was lost unexpectedly. Source: xxxxxx process: vsbkp".
I would be very grateful if you could advise me on how to solve this problem.
Thanks in advance.
Carmine
Hi
Ok, perfect, can you delete or rename that folder on the VSA, then try a new Backup:
C:\Program Files\Commvault\ContentStore\Base\Temp\vmware-SYSTEM
Best Regards,
Sebastien
Hi
What kind of backup should I do?
Full or incremental?
Incremental will do.
Hi
I have followed your suggested procedure. I deleted the indicated folder and restarted the incremental backup of the computer.
The result has not changed. The error is still the same.
Please help me.
Thank you.
Carmine
Â
Do you only have 1 VSA?
Are you able to deploy another VSA?
Otherwise, just log a case with Support and we will check on a session.
I have already opened a ticket for this problem:
Incident 230703-404
The customer has not had a backup since 13 June. I urgently need the resolution.
Can you call me at +390473491313 ?
So you connect directly to the machines?
Thank you
Carmine
My colleague checked but like we said there is an issue with SIDBEngine and he needs the logs from the Media Agent _CL-SAP.
So now we have them he will check and get back to you.
Â
The log of which process?
Â
Â
I have sent the requested logs.
Incident 230703-404
Â
Correct the best is to send all the logs, CS, MA and VSA.
I will ask the engineer to add a key the Media Agents and run a new Backup again:
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.