Question

Kubernetes Backup Failing

  • 27 January 2021
  • 10 replies
  • 195 views

Added a Kubernetes Cluster. The pods are discovered fine however during the backup I see the following on the vsbkp.log.

Job goes pending and after a while it fails.

4868  3940  01/27 13:37:18 JOB ID CKubsInfo::OpenVmdk() - Listing volume [qbc] failed with error [0xFFFFFFFF:{KubsApp::ListDir(1401)} + {KubsVol::ListDir(643)} + {KubsWS::ListDir(193)/ErrNo.-1.(Unknown error)-Exec failed with unhandled exception: set_fail_handler: 20: Invalid HTTP status.}]

4868  3940  01/27 13:37:18 JOBID CKubsFSVolume::Close() - Closing Disk: [qbc] of VM [test-clst`StatefulSet`qbc`b5bf0bb6-a0fc-4123-ac68-9de3c3800807]

Documentation is very shallow and not enough KBs around kubernetes. 

Ideas?


10 replies

Userlevel 7
Badge +15

Curious as to the version you are trying to protect @dude ?

Curious as to the version you are trying to protect @dude ?

CV SP20.32 Kubernetes 1.16.8

Userlevel 1
Badge +1

@dude 

If possible can you share the vsbkp.log and yaml of the application you are trying to protect?
It will help us better to understand the issue being faced.

 

-Manoranjan

@dude 

If possible can you share the vsbkp.log and yaml of the application you are trying to protect?
It will help us better to understand the issue being faced.

 

-Manoranjan

I have opened a ticket with commvault to discuss further.

Badge

@Dude, 

if you found a solution to this problem, can you share the same as we are also facing the same issue?

Userlevel 7
Badge +19

That’s on me, @raj5725 !  Not sure how this one slipped through as ‘solved’!

Here is the last action for the case:

Here is a summary of the issues seen during troubleshooting on Tuesday.

Also it is advised to update to the latest available hotfixes as there are some fixes for Kubernetes backup.


On the call we added a new Kubernetes instance using  API IP: port number (previously the url for Rancher was used)

1.    Tested one Application
-    This failed due to a failed mount attempt on the volume.  This error was seen on the Kubernetes side 
-    please check with storage team on this 

 

2.    Failed backup attempt also left pods in a stuck creating or terminating state
-    Solution for this was to force delete the pod  (k delete pod “podname” –force –grace-period=0

I did notice the case was closed, though @dude responded afterwards (though the case was closed so there was no further response).

@dude , do you recall what was the main fix for this issue?

ps I unmarked this as answered.

I dont think there was a proper fix to it. There was a lot of errors around disk mounts and unmounts that were unfamiliar. We decided not to pursue CV and Kubernetes. Documentation had very little info about configuring and troubleshooting at the time. Support Ticket did not help nor provided the confidence/results we expected.

Sorry not much of a help in this area. 

 

Userlevel 7
Badge +19

No apology need, @dude !

@raj5725 , can you create a support incident and share the case number with me?

Badge

@Mike Struening 

we are yet to install CV in production mode. Before going into production we were testing the CV +K8 integration in a test environment and we got these errors. we wanted to confirm before going into production. Let me add the CV licenses if possible to log a support incident 

Userlevel 7
Badge +19

Sounds like a solid plan.  Keep me posted!

Reply