commvault.com commvault.com
Question

Erroce code 82:127

Carmine Bollettino
Rank
Badge +4

Network send failed: The network connection was lost unexpectedly

Potrei ricevere supporto per la risoluzione di questo errore?

Could I receive support in resolving this error?

 

Grazie

Carmine

27 replies

O
Rank
Userlevel 6
Badge +15

This error points to a network issue during the job.  Does this happen every time you run a job with this client?

Carmine Bollettino
Rank
Badge +4

t has been happening for 5 days every time I try a backup of this server. Pinging the machine from commcell works. Could you please help me?

Thanks
Carmine

Rajiv
Rank
Userlevel 4
Badge +11

Hello @Carmine Bollettino 

Could you please run the check readiness from commcell level for this client? Right Click on the client under Client and Computers > All tasks > Check readiness and see if that passes? If not, share the error message. 

Secondly, login to the media you are using to backup, open commvault process manager and confirm all the services are running? 

Best,

Rajiv Singal

S
Rank
Userlevel 6
Badge +14

@Carmine Bollettino ,

Depending on the results from above, you might need to configure network routes

https://documentation.commvault.com/2022e/expert/7170_network_routes.html

Carmine Bollettino
Rank
Badge +4

Hello @Carmine Bollettino 

Could you please run the check readiness from commcell level for this client? Right Click on the client under Client and Computers > All tasks > Check readiness and see if that passes? If not, share the error message. 

Secondly, login to the media you are using to backup, open commvault process manager and confirm all the services are running? 

Best,

Rajiv Singal

Thank you for your reply.
I had already done both the checks you suggested. Check readiness has passed and also the commvault services on the backup machine are all active. Can you give me any other suggestions?

Thanks
Carmine

Rajiv
Rank
Userlevel 4
Badge +11

Hello @Carmine Bollettino 

Please let me know if the backup is writing now since the check readiness is passing?

If not, from the client machine, could you please share cvd and cvfwd logs?

Best,

Rajiv Singal

Carmine Bollettino
Rank
Badge +4

Hello @Rajiv 

For the sake of completeness, I must inform you that the error is as follows: 
"Network send failed: The network connection was lost unexpectedly. Source: xxxxxx process: vsbkp".

The process affected is vsbkp and for your support I am sending you the log of the vsbkp process in the hope that it will help me resolve the backup malfunction.

The machine in question is part of a cluster but the backups of the other machines in the cluster work normally.

I look forward to your feedback.

Thank you
Carmine

1 Attachment
Rajiv
Rank
Userlevel 4
Badge +11

Hello @Carmine Bollettino ,

I am seeing crashes in the log:

 

4676  b40   06/30 14:50:52 122811 VSBkpDatastoreMonitor::ReportDatastore() - Datastore [DataStore] [3,158,675,816,448] bytes free [49.03%]  Update Interval [300] seconds  [1] VMs
4676  a78   06/30 14:50:53 122811 SdtNetLink::sendAll() - Network send failed: The network connection was lost unexpectedly.. sd=2796
4676  a78   06/30 14:50:53 122811 SdtBase::setLastErr: Setting last err [59][Network send failed: The network connection was lost unexpectedly.] RCId [311598]
4676  1a80  06/30 14:50:53 122811 SdtBase::allocateBuffer(): error_set found RCId [311598]
4676  1a80  06/30 14:50:53 122811 SdtSignature::AllocateBuffer [ERROR]Failed to allocate SDT buffer. SDT Error [Network send failed: The network connection was lost unexpectedly.]
4676  1a80  06/30 14:50:53 122811 CDeDupCache::AppendRawBuffer1() [ERROR] Failed to allocate an output buffer.
4676  1a80  06/30 14:50:53 122811 CDeDupCache::AllocateCallerBufferAndSend():[ERROR]AppendRawBuffer Failed
4676  1a80  06/30 14:50:53 122811 CDeDupCache::MoveBuffer():[ERROR]AllocateCallerBufferAndSend Failed
4676  1a80  06/30 14:50:53 122811 CDeDupCache::DispatchTagList():[ERROR]Construct-And-SendBuffer Failed
4676  1a80  06/30 14:50:53 122811 [CDeDupCache::FlushQueryQueue()] [ERROR] UpdateTagQueue-Failed
4676  1a80  06/30 14:50:53 122811 CDeDupCache::ScanAndCacheBuffer():[ERROR*] Flushing queue failed
4676  1a80  06/30 14:50:53 122811 CDeDupCache::ProcessBuffer():[ERROR]ProcessBuffer Failed 
4676  1a80  06/30 14:50:53 122811 SdtSignature::ProcessBuffer - [ERROR]_pIDeDupCache::ProcessBufferAndPassToNextModule failed(13) []

 

This needs to be looked thoroughly. Could you please open a support case with us and share me the ticket number so that I can take it forward? 

Best,

Rajiv Singal

S
Rank
Userlevel 6
Badge +14

@Carmine Bollettino ,

I have collected the logs myself and looks like SIDB / bad chunks issues during backup jobids: 122811 & 122883:

SIDBEngine.log

2032  670   06/30 14:50:44 ###### 5-1-18-1 FtchBadChnks     3144  Got [1] bad chunks from MM. Last bad chunk insert time=[1688063338]. Curr Ptr [1688063338]
2032  2bc4  06/30 14:50:44 ###### 5-1-18-1 OnSubmitChunk     720  Got chunk [721030] to mark as bad
2032  2bc4  06/30 14:50:44 ###### 5-1-18-1 LeastBadChnkInsTime   158  Setting least bad chunk insert time in prop tbl [1688063338]
10948 29fc  06/30 14:50:44 ###### 5-0-17-0 FtchBadChnks     3144  Got [1] bad chunks from MM. Last bad chunk insert time=[1688063338]. Curr Ptr [1688063338]
10948 21d4  06/30 14:50:44 ###### 5-0-17-0 OnSubmitChunk     720  Got chunk [721030] to mark as bad
10948 21d4  06/30 14:50:44 ###### 5-0-17-0 LeastBadChnkInsTime   158  Setting least bad chunk insert time in prop tbl [1688063338]


7664  2514  06/30 17:30:26 ###### 5-1-18-1 ResetCounters    1960  Resetting handler counters.
4020  24d4  06/30 17:30:26 ###### 5-0-17-0 FtchBadChnks     3144  Got [1] bad chunks from MM. Last bad chunk insert time=[1688063338]. Curr Ptr [1688063338]
4020  1664  06/30 17:30:26 ###### 5-0-17-0 OnSubmitChunk     720  Got chunk [721030] to mark as bad
4020  1664  06/30 17:30:26 ###### 5-0-17-0 LeastBadChnkInsTime   158  Setting least bad chunk insert time in prop tbl [1688063338]
7664  2514  06/30 17:30:26 ###### 5-1-18-1 RefMpList        3000  Got [4] mount path list from the mediamanager.
7664  2514  06/30 17:30:26 ###### 5-1-18-1 RefMpList        3074  No disabled mount paths present.
7664  2514  06/30 17:30:26 ###### 5-1-18-1 FtchBadChnks     3144  Got [1] bad chunks from MM. Last bad chunk insert time=[1688063338]. Curr Ptr [1688063338]
7664  2a5c  06/30 17:30:26 ###### 5-1-18-1 OnSubmitChunk     720  Got chunk [721030] to mark as bad
7664  2a5c  06/30 17:30:26 ###### 5-1-18-1 LeastBadChnkInsTime   158  Setting least bad chunk insert time in prop tbl [1688063338]

 

Can you check the DDB and log a case for DDB issues?

Carmine Bollettino
Rank
Badge +4

Hi, I have a problem with backing up two machines in Commvault.
One is a linux machine and the other is a windows machine.
Both are part of a cluster and the other machines in the cluster back up normally without any problems.
Both machines fail the backup and give the following error: "Network send failed: The network connection was lost unexpectedly. Source: xxxxxx process: vsbkp'.

I am attaching the log of the indicated process so that the indicated malfunction can be better resolved.

Thank you
Carmine

S
Rank
Userlevel 6
Badge +14

@Carmine Bollettino ,

As explained above, check DDB issues, you might need to run a DDB Verification.

https://documentation.commvault.com/11.28/expert/100399_data_verification_of_deduplicated_data.html

Then log a case if issue persists.

Carmine Bollettino
Rank
Badge +4

@Sebastien Merluzzi 

Could you kindly tell me where to find and perform the DDB check. In Commandcenter or in Console? Could you explain to me how to proceed and what steps to follow? I would be infinitely grateful if you could give me this information.

Thank you
 

S
Rank
Userlevel 6
Badge +14

@Carmine Bollettino ,

That’s the link I have provided above.

Did you check in our Documentation:

https://documentation.commvault.com/2022e/expert/12569_performing_data_verification_operation_on_deduplicated_data.html

Carmine Bollettino
Rank
Badge +4

@Sebastien Merluzzi 

Hi Sebastien,
I ran the full DDB check as requested and found no errors. I have restarted the backups, but they always stop in pending with error 82:127 “Network send failed: The network connection was lost unexpectedly. Source: xxxxxx process: vsbkp".
I would be very grateful if you could advise me on how to solve this problem.

Thanks in advance.
Carmine

S
Rank
Userlevel 6
Badge +14

Hi @Carmine Bollettino ,

Ok, perfect, can you delete or rename that folder on the VSA, then try a new Backup:

C:\Program Files\Commvault\ContentStore\Base\Temp\vmware-SYSTEM

Best Regards,

Sebastien

Carmine Bollettino
Rank
Badge +4

Hi @Sebastien Merluzzi 

What kind of backup should I do?
Full or incremental?

S
Rank
Userlevel 6
Badge +14

@Carmine Bollettino 

Incremental will do.

Carmine Bollettino
Rank
Badge +4

Hi @Sebastien Merluzzi 
I have followed your suggested procedure. I deleted the indicated folder and restarted the incremental backup of the computer.
The result has not changed. The error is still the same.
Please help me.

Thank you.
Carmine
 

S
Rank
Userlevel 6
Badge +14

@Carmine Bollettino ,

Do you only have 1 VSA?

Are you able to deploy another VSA?

Otherwise, just log a case with Support and we will check on a session.

Carmine Bollettino
Rank
Badge +4

Hi @Sebastien Merluzzi 

 

I have already opened a ticket for this problem:
Incident 230703-404

The customer has not had a backup since 13 June. I urgently need the resolution.

Can you call me at +390473491313 ?
So you connect directly to the machines?

Thank you
Carmine

S
Rank
Userlevel 6
Badge +14

@Carmine Bollettino ,

My colleague checked but like we said there is an issue with SIDBEngine and he needs the logs from the Media Agent _CL-SAP.

So now we have them he will check and get back to you.

Carmine Bollettino
Rank
Badge +4

 

The log of which process?

 

 

Carmine Bollettino
Rank
Badge +4

I have sent the requested logs.

Incident 230703-404

 

S
Rank
Userlevel 6
Badge +14

@Carmine Bollettino ,

Correct the best is to send all the logs, CS, MA and VSA.

S
Rank
Userlevel 6
Badge +14

@Carmine Bollettino ,

I will ask the engineer to add a key the Media Agents and run a new Backup again:

Reply


Terms & ConditionsCookie settings