commvault.com commvault.com
Solved

Error 91:465 - Unable to connect to vCenter server

  • 10 February 2023
  • 4 replies
  • 4424 views
T
Rank
Userlevel 3
Badge +6

We have a primary commserve and vcenter. We have multiple sites each with a physical media agent, esxi servers and netapp as storage. The media agents talk back to the primary vcenter to back up the local vm’s at each site. For some reason jobs started failing at one site, it was working before but now I am seeing the following error: 

Error Code: [91:465]
Description: Unable to connect to vCenter server [VCENTER] as user [CVSERVICEACCOUNT] from access node [REMOTEMEDIAAGENT]. [Unable to connect to the remote server]
Source: REMOTEMEDIAAGENT, Process: vsdiscovery

Now from this remote media agent, I am literally able to log into the vcenter using the service account. So this does not make any sense why every job is failing with the above error message. Does anyone know what I can check or do to help resolve this? 
 

icon

Best answer by Damian Andre 23 February 2023, 00:06

View original

4 replies

S
Rank
Userlevel 6
Badge +14

Hi @tsmtmiller ,

Can you check VSDISCOVERY.log and share parts of it hiding sensitive data please?

This login failure is usually between VMware and Active Directory since the service account is an AD account. This can happen due to any time out or authentication issue between vCenter and DC.

Use vSphere local account (eg, administrator@vsphere.local) so it doesn't need to authenticate with an AD server.

If you are using a custom account make sure it has all the permissions required based on the documentation below:
https://documentation.commvault.com/11.24/expert/32514_permissions_for_custom_user_accounts_vmware.html

Also, check below:

  • Ensure the user account is not locked out (account might getting locked out but unlocks itself after a short period of time).
  • Ensure the vCenter host being reported in the error can log into the VMware GUI using Internet Explorer with the same username\password that is failing
  • Ensure the username reported as being unable to connect has the correct permissions in vCenter – this is particularly important when upgrading from vCenter 5.1 to 5.5.
  • For vCenter 5.5 and the newer VDDK 5.5 – permissions are required at the root level of vCenter  and there are some additional requirements as per KB2063054
  • Ensure SSO on vCenter is working

Port Requirements

In an environment with firewalls, the vCenter, ESX servers, and Virtual Server Agent must be able to communicate with each other. To ensure that all components can communicate through the firewall, ensure that the ports for web services (default: 443) and TCP/IP (default: 902) are opened for bidirectional communication on each of these machines.

Additional port requirements can apply for the MediaAgent when you use specific features such as Live Browse, Live File Recovery, and Live Mount. For more information, see Entering Required Firewall Settings.

Prior to performing any backup or restore operations, ensure that the following port requirements are met. 

Note: If a non-default port is used with VDDK 5.5, backup or restore operations can fail. See KB Article VMW0013.

vCenter: Port for web service (default: 443) must be opened. If vCenter is configured to use non-default ports, the non-default ports must also be opened.
ESX Server: Ports for web service (default: 443) and TCP/IP (default: 902) must be opened for the vStorage APIs for Data Protection.
vCloud Director: A port for the vCloud REST API (default: 443) must be opened.

Additional information can be found here

Best Regards,

Sebastien

T
Rank
Userlevel 3
Badge +6

Hello thanks for responding. Oddly, we got one successful backup on the 19th. After that, its back to failures. One thing I am suspicious about is that in the VSdiscovery log, the IP at the top of the log belongs to the “NFS” IP assigned to server. This IP does not have access to the CVCS or VCENTER as far as I know. I will ask our network team to allow it.   

 

 

 

1 Attachment
Damian Andre
Rank
Userlevel 7
Badge +23

You could try changing the binding order on the server so that the NFS network is not the default

I just pulled a random article from google:

https://infosight.hpe.com/InfoSight/media/cms/active/public/tmg_HPE_Nimble_Storage_Deployment_Considerations_for_Standalone_Servers_and_Failover_Clusters_on_Windows_Server_2016_doc_version_family.whz/vzk1473886140334.html

 

Restart the commvault services after the change and see if that helps.

T
Rank
Userlevel 3
Badge +6

You could try changing the binding order on the server so that the NFS network is not the default

I just pulled a random article from google:

https://infosight.hpe.com/InfoSight/media/cms/active/public/tmg_HPE_Nimble_Storage_Deployment_Considerations_for_Standalone_Servers_and_Failover_Clusters_on_Windows_Server_2016_doc_version_family.whz/vzk1473886140334.html

 

Restart the commvault services after the change and see if that helps.

Thanks, actually Sebastien Merluzzi and you helped. Looks like whatever this NFS adaptor was, I disabled it and now I see backups going through. Also for whatever reason, some of the VSS services on the CVMA were stopped. 

Reply


Terms & ConditionsCookie settings