Solved

Kubernetes User Requirements

  • 10 February 2021
  • 9 replies
  • 852 views

Userlevel 2
Badge +6

Hi All

Does anybody know the exact requirements (roles, privilleges, namespace, whatever) for a Kubernetes Service Account? Documentation skips over that part and Support so far was unable to answer this.

Thanks,

Stefan

icon

Best answer by Manoranjan Reddy 10 February 2021, 19:52

View original

9 replies

Userlevel 7
Badge +23

@Stefan Vollrath, looking to get someone to answer this for you.

 

Also tagging in @Seema Ghai for the documentation piece.

Userlevel 1
Badge +1

@Stefan Vollrath

Roles : Should be cluster-admin role #2
Namespace: Can be anything, default is used in below example. Use suffix -n NS001 to kubectl commands to use service account in a different namespace, say NS001

Use below commands for creating service account with required clusterrolebindings to use for backup and restore with  Commvault.

1)    #Create service account named cvbackup
        kubectl create serviceaccount cvbackup

2)    #Create clusterrolebinding
        kubectl create clusterrolebinding default-sa-crb --clusterrole=cluster-admin --serviceaccount=default:cvbackup

        TOKENNAME=`kubectl get sa/cvbackup  -o jsonpath='{.secrets[0].name}'`
        TOKEN=`kubectl  get secret $TOKENNAME -o jsonpath='{.data.token}'| base64 --decode`

3)    #Prints token
        echo $TOKEN

   

Sample usage of above CLI commands

In this case cvbackup is the Service Account and output from #3 is token to be used at time of adding cluster in Command Center

 

Screenshot to add Kubernetes Cluster in Command Center


We’ll update documentation to improve missing pieces.

Thanks
Manoranjan Reddy

Userlevel 2
Badge +6

Looks a lot like the guide I found on Oracles KB, even the tickmarks are placed identical :-)

Ended up using belows Variant for use in Powershell, using Namespace default did not work for me, so settled for kube-system.
kubectl -n kube-system create serviceaccount backup-service-account

kubectl create clusterrolebinding backup-cluster-admin --clusterrole=cluster-admin --serviceaccount=kube-system:backup-service-account

$TOKENNAME=kubectl -n kube-system get serviceaccount/backup-service-account -o jsonpath='{.secrets[0].name}'

$TOKEN = kubectl -n kube-system get secret $TOKENNAME -o jsonpath='{.data.token}'

$DECODEDTOKEN=[Text.Encoding]::Utf8.GetString([Convert]::FromBase64String($TOKEN))

kubectl config set-credentials backup-service-account --token=$DECODEDTOKEN

kubectl config set-context --current --user=backup-service-account

That worked out fine for a Day, after that the Token apparently expired. So there must be some Value I missed to tweak. Could redo the procedure above and re-add the Token, but CommandCenter does not have a Change/Modify Page for these Instance Properties yet, at least none I could find.

Userlevel 2
Badge +6

Scratch my previous Comment, at least partially.

Found the two reasons why it stopped working for me.

  1. CV uses the global http Proxy even if the selected VSA Proxy could directly contact the Kubernetes Cluster. our Proxy currently has issues and that’s why this connection was dropped. Option to specify Web-Proxy on case-to-case base would be better, especially for multi-tenant Envs or segregated Networks.
  2. CommServe Database lost the Secret Key; vdiscovery.log stated “Credentials incomplete for x”. and Attribute “K8s Secret Key” in App_InstanceProp Table was empty all the sudden. Sidenote: the key is saved in plain-text in this field, shouldn’t that be at least encoded or encrypted like all other passwords?
Userlevel 7
Badge +23

 

  1. CV uses the global http Proxy even if the selected VSA Proxy could directly contact the Kubernetes Cluster. our Proxy currently has issues and that’s why this connection was dropped. Option to specify Web-Proxy on case-to-case base would be better, especially for multi-tenant Envs or segregated Networks.

Do you mean we are inheriting the HTTP proxy configured in the system settings (through IE or group policy)? i.e

netsh winhttp show proxy

 

  1. CommServe Database lost the Secret Key; vdiscovery.log stated “Credentials incomplete for x”. and Attribute “K8s Secret Key” in App_InstanceProp Table was empty all the sudden. Sidenote: the key is saved in plain-text in this field, shouldn’t that be at least encoded or encrypted like all other passwords?

 

Will flag with a PM

Userlevel 2
Badge +6

Only place that has Proxy configured and should use it is the CommServe itself (Control Panel > Internet Options > HTTP Proxy) to upload data to Commvault Cloud.

VSA Nodes neither have Proxies set up in Windows nor in IE, nor should they use any.

Before disabling Proxy in ControlPanel:

5880 a998  02/11 09:44:22 13463177 CreateK8sInfoObject() - Creating K8sInfo object
35880 a998 02/11 09:44:22 13463177 CVMInfo::CVMInfo() - Start >>>
35880 a998 02/11 09:44:22 13463177 CVMInfo::Init() - Start >>>
35880 a998 02/11 09:44:22 13463177 CK8sInfo::Connect() - Connecting using HTTP proxy [outappl.pnet.ch:3128], user name []
35880 a998 02/11 09:44:22 13463177 CK8sInfo::Connect() - Connecting to server:https://k8s-zf-t-backup-01.pnet.ch:8443
35880 a998 02/11 09:44:22 13463177 RestCall(48) - Exception: Failed to connect to any resolved proxy endpoint
35880 a998 02/11 09:44:22 13463177 CK8sInfo::Connect() - Connection failed with error 0xFFFFFFFF:{K8sCluster::Connect(243)} + {K8sVol::Init(21)/ErrNo.-1.(Unknown error)-Api failed}
35880 a998 02/11 09:44:22 13463177 VSDisc::PrepareVMInfo() - Failed to connect to server=[https://k8s-zf-t-backup-01.pnet.ch:8443] user=[ServiceAccount|backup-service-account]

And after:

33688 8350  02/11 09:47:59 13463183 CreateK8sInfoObject() - Creating K8sInfo object
33688 8350 02/11 09:47:59 13463183 CVMInfo::CVMInfo() - Start >>>
33688 8350 02/11 09:47:59 13463183 CVMInfo::Init() - Start >>>
33688 8350 02/11 09:47:59 13463183 CK8sInfo::Connect() - Credentials incomplete for [https://k8s-zf-t-backup-01.pnet.ch:8443:ServiceAccount|backup-service-account]
33688 8350 02/11 09:47:59 13463183 VSDisc::PrepareVMInfo() - Failed to connect to server=[https://k8s-zf-t-backup-01.pnet.ch:8443] user=[ServiceAccount|backup-service-account]

 

Userlevel 7
Badge +23

Only place that has Proxy configured and should use it is the CommServe itself (Control Panel > Internet Options > HTTP Proxy) to upload data to Commvault Cloud.

VSA Nodes neither have Proxies set up in Windows nor in IE, nor should they use any.

 

I understand now. This is a common issue I saw for VSA proxy in cloud. I’m not sure exactly if this works for Kubernetes, but I have a hunch that it may:

https://documentation.commvault.com/additionalsetting/details?name=%22httpProxyName%22&id=6356

You can configure this and it (hopefully) will ignore the control panel HTTP Proxy setting

Userlevel 2
Badge +6

@Damian Andre thanks for the hint, will give it a shot. I do think this is kinda design-issue. VSA simply using the CS setting is wrong simple as that. But above that, the http-Proxy Setting would need to be something Instance or Subclient Level dependent. Example from Realworld: Our MA/VSAs are intended for multi-use (shit was expensive), talking to local vCenter (and maybe later Hyper-V), now managing Kubernetes onPrem and later probably expanding to Kubernetes in different Clouds with different out Proxies, or none if VPN is established. Some of that cannot be accessed via WebProxy, some might but then horribly slow, some requires WebProxy. Can’t manage that with one Additional Setting. Maybe something for Dev to sleep over.

 

Userlevel 7
Badge +23

@Damian Andrethanks for the hint, will give it a shot. I do think this is kinda design-issue. VSA simply using the CS setting is wrong simple as that. But above that, the http-Proxy Setting would need to be something Instance or Subclient Level dependent. Example from Realworld: Our MA/VSAs are intended for multi-use (shit was expensive), talking to local vCenter (and maybe later Hyper-V), now managing Kubernetes onPrem and later probably expanding to Kubernetes in different Clouds with different out Proxies, or none if VPN is established. Some of that cannot be accessed via WebProxy, some might but then horribly slow, some requires WebProxy. Can’t manage that with one Additional Setting. Maybe something for Dev to sleep over.

 

Right on - I have forwarded it for review internally. There is definitely a better way to handle this I think.

Reply