Backup Failure occurring failing to Request HTTP Sessions vCenter

Issue :Commvault backup jobs were failing .Found from the logs Backup jobs are trying to establish an HTTP request to the vCenter and failing to establish the communication with the vCenter.

Log snippets from commvault


vsbkp.log from the proxy server

15:57:34 2357789 _Connect() - Exception - System.Net.WebException: The request failed with HTTP status 503: Service Unavailable.at System.Web.Services.Protocols.SoapHttpClientProtocol.ReadResponse(SoapClientMessage message, WebResponse response, Stream responseStream, Boolean asyncCall)    at System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke(String methodName, Object[] parameters)at Vim25Api.VimService.RetrieveServiceContent(ManagedObjectReference _this)    at VISDKWrapper.SvcConnection._Connect(String url, String username, String password)
 VSBkpCoordinator::OnMsgVMBackupGetNextVM
1 CVMWareInfo::Connect() - VISDKCppBridge::Connect failed [The request failed with HTTP status 503: Service Unavailable.

vpxd logs from vCenter

 error vpxd[7F99DECD9700] [Originator@6876 sub=HTTP session map] Out of HTTP sessions: Limited to 2000
 error vpxd[7F99D7468700] [Originator@6876 sub=HTTP session map] Out of HTTP sessions: Limited to 2000



Cause : This issue occurs due to active SOAP sessions exceeds 2000 on the VCSA exhausting the http sessions. By default the VCSA can handle only 2000 sessions. SOAP connections can be generated byVMware vSphere Client sessions, Third-party API connections, vCenter Server plugins

Resolution :  To work around this issue, increase <maxSessionCount> in vpxd.cfg file.

• Took a backup of the vpxd.cfg from /etc/vmware-vpx/
• Stopped the vpxd service
• Added the highlighted entries in the below order

<config>
   ...
   <vmacore>
      ...
      <soap>
      <maxSessionCount>6000</maxSessionCount>
      </soap>
      ...
   </vmacore>
   ...
</config>


• After starting the vpxd services service-control --start vmware-vpx
• Initiated the backup and it was successful

Please note the highlighted lines won’t be available to edit and we have to manually include them in the vpxd.cfg file.


Below are some of the articles for your reference

https://kb.vmware.com/kb/50114010 

https://forum.commvault.com/forums/53839/ShowThread.aspx

https://documentation.commvault.com/commvault/v11/article?p=32618.htm