VAPI Endpoint service failure

VAPI Endpoint service failure 

ERROR | state-manager1            | StsBuilder                     | Failed to acquire token for the solution user

2018-11-15T21:50:42.647Z | ERROR | state-manager1            | DefaultStateManager            | Could not initialize endpoint runtime state.

com.vmware.vapi.endpoint.config.ConfigurationException: com.vmware.vim.sso.client.exception.AuthenticationFailedException: Provided credentials are not valid.

        at com.vmware.vapi.endpoint.cis.StsBuilder.createToken(StsBuilder.java:182)

        at com.vmware.vapi.endpoint.cis.StsBuilder.rebuild(StsBuilder.java:77)

        at com.vmware.vapi.endpoint.cis.StsBuilder.buildInitial(StsBuilder.java:54)

        at com.vmware.vapi.state.impl.DefaultStateManager.build(DefaultStateManager.java:354)

        at com.vmware.vapi.state.impl.DefaultStateManager$1.doInitialConfig(DefaultStateManager.java:168)

        at com.vmware.vapi.state.impl.DefaultStateManager$1.run(DefaultStateManager.java:151)

        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

        at java.util.concurrent.FutureTask.run(FutureTask.java:266)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

Caused by: com.vmware.vim.sso.client.exception.AuthenticationFailedException: Provided credentials are not valid.

        at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor.handleFaultCondition(SecurityTokenServiceImpl.java:852)

        at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor.sendRequest(SecurityTokenServiceImpl.java:788)

        at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor.executeRoundtrip(SecurityTokenServiceImpl.java:714)

        at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl.acquireTokenByCertificate(SecurityTokenServiceImpl.java:473)

        at com.vmware.vapi.endpoint.cis.StsBuilder.createToken(StsBuilder.java:179)

Please follow the steps below

/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store vpxd --text | less

MACHINE_SSL_CERT

TRUSTED_ROOTS

TRUSTED_ROOT_CRLS

machine

vpxd

vpxd-extension

vsphere-webclient

SMS

/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store <stored name> --alias <alias name> --output /certificate/<certificate usage name>.crt

/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store machine --alias machine --output /certificate/machine.crt

/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store vpxd-extension --alias vpxd-extension --output /certificate/vpxd-extension.crt

/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store vpxd --alias vpxd --output /certificate/vpxd.crt

/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store vsphere-webclient --alias vsphere-webclient --output /certificate/vsphere-webclient.crt

dir-cli service update --name vpxd-extension-d74bb41a-f337-4667-9294-9df39322b428- --cert /cert/vpxd.crt --login administrator@vsphere.local

root@VCENTER51 [ /certificate ]# less vsphere-webclient.crt

root@VCENTER51 [ /certificate ]# /usr/lib/vmware-vmafd/bin/dir-cli service list

Enter password for administrator@vsphere.local:

1. WebClient_2016.11.23_145904

2. machine-2253fa42-d1ee-11e7-ba38-0050569c439f

3. vsphere-webclient-2253fa42-d1ee-11e7-ba38-0050569c439f

4. machine-d74bb41a-f337-4667-9294-9df39322b428

5. vsphere-webclient-d74bb41a-f337-4667-9294-9df39322b428

6. vpxd-d74bb41a-f337-4667-9294-9df39322b428

7. vpxd-extension-d74bb41a-f337-4667-9294-9df39322b428

/usr/lib/vmware-vmafd/bin/dir-cli service update --name machine-d74bb41a-f337-4667-9294-9df39322b428 --cert /certificate/machine.crt --login administrator@vsphere.local

/usr/lib/vmware-vmafd/bin/dir-cli service update --name vsphere-webclient-d74bb41a-f337-4667-9294-9df39322b428 --cert /certificate/vsphere-webclient.crt --login administrator@vsphere.local

/usr/lib/vmware-vmafd/bin/dir-cli service update --name vpxd-d74bb41a-f337-4667-9294-9df39322b428 --cert /certificate/vpxd.crt --login administrator@vsphere.local

/usr/lib/vmware-vmafd/bin/dir-cli service update --name vpxd-extension-d74bb41a-f337-4667-9294-9df39322b428 --cert /certificate/vpxd-extension.crt --login administrator@vsphere.local

Manually Replacing the solution user certificate on VCSA/PSC 6.x

Manually Replacing the solution user certificate on VCSA/PSC 6.x

Note :

-- Take a snapshot of the VC or PSC before proceeding.

-- PSC has two solution users : machine and vsphere-webclient.

-- VC has four solution users : machine, vpxd, vpxd-extension and vsphere-webclient.

-- VC with embedded PSC has four solution users : machine, vpxd, vpxd-extension and vsphere-webclient.

-- SSO admin username and password would vary depending on the configuration.

-- Windows Install directory may vary depending on your installation. 

-- Run the below command to get the machine ID of the node :

Appliance : /usr/lib/vmware-vmafd/bin/vmafd-cli get-machine-id --server-name localhost

Windows : "%VMWARE_CIS_HOME%"\vmafdd\vmafd-cli get-machine-id --server-name localhost

0. Take a backup of the old solution user certificates and its private keys.

Appliance :

/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store machine --alias machine --output /root/certificate/machine.crt

/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store vpxd --alias vpxd --output /root/certificate/vpxd.crt

/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store vpxd-extension --alias vpxd-extension --output /root/certificate/vpxd-extension.crt

/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store vsphere-webclient --alias vsphere-webclient --output /root/certificate/vsphere-webclient.crt

/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store machine --alias machine --output /root/certificate/machine.key

/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store vpxd --alias vpxd --output /root/certificate/vpxd.key

/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store vpxd-extension --alias vpxd-extension --output /root/certificate/vpxd-extension.key

/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store vsphere-webclient --alias vsphere-webclient --output /root/certificate/vsphere-webclient.key

Windows :

"%VMWARE_CIS_HOME%"\vmafdd\vecs-cli entry getcert --store machine --alias machine --output c:\certificate\machine.crt

"%VMWARE_CIS_HOME%"\vmafdd\vecs-cli entry getcert --store vpxd --alias vpxd --output c:\certificate\vpxd.crt

"%VMWARE_CIS_HOME%"\vmafdd\vecs-cli entry getcert --store vpxd-extension --alias vpxd-extension --output c:\certificate\vpxd-extension.crt

"%VMWARE_CIS_HOME%"\vmafdd\vecs-cli entry getcert --store vsphere-webclient --alias vsphere-webclient --output c:\certificate\vsphere-webclient.crt

"%VMWARE_CIS_HOME%"\vmafdd\vecs-cli entry getcert --store machine --alias machine --output c:\certificate\machine.key

"%VMWARE_CIS_HOME%"\vmafdd\vecs-cli entry getcert --store vpxd --alias vpxd --output c:\certificate\vpxd.key

"%VMWARE_CIS_HOME%"\vmafdd\vecs-cli entry getcert --store vpxd-extension --alias vpxd-extension --output c:\certificate\vpxd-extension.key

"%VMWARE_CIS_HOME%"\vmafdd\vecs-cli entry getcert --store vsphere-webclient --alias vsphere-webclient --output c:\certificate\vsphere-webclient.key

1. Create the configuration file for all the solution users :

a) machine.cfg :

[ req ]

distinguished_name = req_distinguished_name

encrypt_key = no

prompt = no

string_mask = nombstr

req_extensions = v3_req

[ v3_req ]

basicConstraints = CA:false

keyUsage = nonRepudiation, digitalSignature, keyEncipherment

subjectAltName = DNS:vcenter-nl-1.filmlogistics.local     # FQDN of VC or PSC

[ req_distinguished_name ]

countryName = NL

stateOrProvinceName = Utrecht

localityName = Zeist

0.organizationName = VMware

organizationalUnitName = mID-74453a6d-4f01-4191-a95c-d3a905c1a516     # machine ID of the VC/PSC

commonName = machine

b) vpxd.cfg :

[ req ]

distinguished_name = req_distinguished_name

encrypt_key = no

prompt = no

string_mask = nombstr

req_extensions = v3_req

[ v3_req ]

basicConstraints = CA:false

keyUsage = nonRepudiation, digitalSignature, keyEncipherment

subjectAltName = DNS:vcenter-nl-1.filmlogistics.local     # FQDN of VC or PSC

[ req_distinguished_name ]

countryName = NL

stateOrProvinceName = Utrecht

localityName = Zeist

0.organizationName = VMware

organizationalUnitName = mID-74453a6d-4f01-4191-a95c-d3a905c1a516     # machine ID of the VC/PSC

commonName = vpxd

c) vpxd-extension.cfg :

[ req ]

distinguished_name = req_distinguished_name

encrypt_key = no

prompt = no

string_mask = nombstr

req_extensions = v3_req

[ v3_req ]

basicConstraints = CA:false

keyUsage = nonRepudiation, digitalSignature, keyEncipherment

subjectAltName = DNS:vcenter-nl-1.filmlogistics.local     # FQDN of VC or PSC

[ req_distinguished_name ]

countryName = NL

stateOrProvinceName = Utrecht

localityName = Zeist

0.organizationName = VMware

organizationalUnitName = mID-74453a6d-4f01-4191-a95c-d3a905c1a516     # machine ID of the VC/PSC

commonName = vpxd-extension

d) vsphere-webclient.cfg :

[ req ]

distinguished_name = req_distinguished_name

encrypt_key = no

prompt = no

string_mask = nombstr

req_extensions = v3_req

[ v3_req ]

basicConstraints = CA:false

keyUsage = nonRepudiation, digitalSignature, keyEncipherment

subjectAltName = DNS:vcenter-nl-1.filmlogistics.local     # FQDN of VC or PSC

[ req_distinguished_name ]

countryName = NL

stateOrProvinceName = Utrecht

localityName = Zeist

0.organizationName = VMware

organizationalUnitName = mID-74453a6d-4f01-4191-a95c-d3a905c1a516     # machine ID of the VC/PSC

commonName = vsphere-webclient

2. Create the certificate signing request and generate a new certificate for all the solution certs :

Appliance :

openssl req -new -nodes -out machine.csr -newkey rsa:2048 -keyout machine.key -config machine.cfg

openssl x509 -req -days 3650 -in machine.csr -out machine.crt -CA /var/lib/vmware/vmca/root.cer -CAkey /var/lib/vmware/vmca/privatekey.pem -extensions v3_req -CAcreateserial -extfile machine.cfg

openssl req -new -nodes -out vpxd.csr -newkey rsa:2048 -keyout vpxd.key -config vpxd.cfg

openssl x509 -req -days 3650 -in vpxd.csr -out vpxd.crt -CA /var/lib/vmware/vmca/root.cer -CAkey /var/lib/vmware/vmca/privatekey.pem -extensions v3_req -CAcreateserial -extfile vpxd.cfg

openssl req -new -nodes -out vpxd-extension.csr -newkey rsa:2048 -keyout vpxd-extension.key -config vpxd-extension.cfg

openssl x509 -req -days 3650 -in vpxd-extension.csr -out vpxd-extension.crt -CA /var/lib/vmware/vmca/root.cer -CAkey /var/lib/vmware/vmca/privatekey.pem -extensions v3_req -CAcreateserial -extfile vpxd-extension.cfg

openssl req -new -nodes -out vsphere-webclient.csr -newkey rsa:2048 -keyout vsphere-webclient.key -config vsphere-webclient.cfg

openssl x509 -req -days 3650 -in vsphere-webclient.csr -out vsphere-webclient.crt -CA /var/lib/vmware/vmca/root.cer -CAkey /var/lib/vmware/vmca/privatekey.pem -extensions v3_req -CAcreateserial -extfile vsphere-webclient.cfg

Windows :

"%VMWARE_OPENSSL_BIN%" req -new -nodes -out machine.csr -newkey rsa:2048 -keyout machine.key -config machine.cfg

"%VMWARE_OPENSSL_BIN%" x509 -req -days 3650 -in machine.csr -out machine.crt -CA /var/lib/vmware/vmca/root.cer -CAkey /var/lib/vmware/vmca/privatekey.pem -extensions v3_req -CAcreateserial -extfile machine.cfg

"%VMWARE_OPENSSL_BIN%" req -new -nodes -out vpxd.csr -newkey rsa:2048 -keyout vpxd.key -config vpxd.cfg

"%VMWARE_OPENSSL_BIN%" x509 -req -days 3650 -in vpxd.csr -out vpxd.crt -CA /var/lib/vmware/vmca/root.cer -CAkey /var/lib/vmware/vmca/privatekey.pem -extensions v3_req -CAcreateserial -extfile vpxd.cfg

"%VMWARE_OPENSSL_BIN%" req -new -nodes -out vpxd-extension.csr -newkey rsa:2048 -keyout vpxd-extension.key -config vpxd-extension.cfg

"%VMWARE_OPENSSL_BIN%" x509 -req -days 3650 -in vpxd-extension.csr -out vpxd-extension.crt -CA /var/lib/vmware/vmca/root.cer -CAkey /var/lib/vmware/vmca/privatekey.pem -extensions v3_req -CAcreateserial -extfile vpxd-extension.cfg

"%VMWARE_OPENSSL_BIN%" req -new -nodes -out vsphere-webclient.csr -newkey rsa:2048 -keyout vsphere-webclient.key -config vsphere-webclient.cfg

"%VMWARE_OPENSSL_BIN%" x509 -req -days 3650 -in vsphere-webclient.csr -out vsphere-webclient.crt -CA /var/lib/vmware/vmca/root.cer -CAkey /var/lib/vmware/vmca/privatekey.pem -extensions v3_req -CAcreateserial -extfile vsphere-webclient.cfg

3. Delete the old certificate entries from VECS store :

Appliance :

/usr/lib/vmware-vmafd/bin/vecs-cli entry delete --store machine --alias machine -y

/usr/lib/vmware-vmafd/bin/vecs-cli entry delete --store vpxd --alias vpxd -y

/usr/lib/vmware-vmafd/bin/vecs-cli entry delete --store vpxd-extension --alias vpxd-extension -y

/usr/lib/vmware-vmafd/bin/vecs-cli entry delete --store vsphere-webclient --alias vsphere-webclient -y

Windows :

"%VMWARE_CIS_HOME%"\vmafdd\vecs-cli entry delete --store machine --alias machine --cert machine.crt --key machine.key

"%VMWARE_CIS_HOME%"\vmafdd\vecs-cli entry delete --store vpxd --alias vpxd --cert vpxd.crt --key vpxd.key

"%VMWARE_CIS_HOME%"\vmafdd\vecs-cli entry delete --store vpxd-extension --alias vpxd-extension --cert vpxd-extension.crt --key vpxd-extension.key

"%VMWARE_CIS_HOME%"\vmafdd\vecs-cli entry delete --store vsphere-webclient --alias vsphere-webclient --cert vsphere-webclient.crt --key vsphere-webclient.key

4. Update the  new certificates in the VECS store :

Appliance :

/usr/lib/vmware-vmafd/bin/vecs-cli entry create --store machine --alias machine --cert machine.crt --key machine.key

/usr/lib/vmware-vmafd/bin/vecs-cli entry create --store vpxd --alias vpxd --cert vpxd.crt --key vpxd.key

/usr/lib/vmware-vmafd/bin/vecs-cli entry create --store vpxd-extension --alias vpxd-extension --cert vpxd-extension.crt --key vpxd-extension.key

/usr/lib/vmware-vmafd/bin/vecs-cli entry create --store vsphere-webclient --alias vsphere-webclient --cert vsphere-webclient.crt --key vsphere-webclient.key

Windows :

"%VMWARE_CIS_HOME%"\vmafdd\vecs-cli entry create --store machine --alias machine --cert machine.crt --key machine.key

"%VMWARE_CIS_HOME%"\vmafdd\vecs-cli entry create --store vpxd --alias vpxd --cert vpxd.crt --key vpxd.key

"%VMWARE_CIS_HOME%"\vmafdd\vecs-cli entry create --store vpxd-extension --alias vpxd-extension --cert vpxd-extension.crt --key vpxd-extension.key

"%VMWARE_CIS_HOME%"\vmafdd\vecs-cli entry create --store vsphere-webclient --alias vsphere-webclient --cert vsphere-webclient.crt --key vsphere-webclient.key

5. List all the solution users : 

/usr/lib/vmware-vmafd/bin/dir-cli service list --login administrator@vsphere.local --password 'YA,Bkc9ID8.)o_Sxr5t6'

1. machine-74453a6d-4f01-4191-a95c-d3a905c1a516

2. vsphere-webclient-74453a6d-4f01-4191-a95c-d3a905c1a516

3. vpxd-74453a6d-4f01-4191-a95c-d3a905c1a516

4. vpxd-extension-74453a6d-4f01-4191-a95c-d3a905c1a516 

6. Update all the solution users with the new certificate :

Appliance :

/usr/lib/vmware-vmafd/bin/dir-cli service update --name machine-74453a6d-4f01-4191-a95c-d3a905c1a516 --cert machine.crt --login administrator@vsphere.local --password 'YA,Bkc9ID8.)o_Sxr5t6'

/usr/lib/vmware-vmafd/bin/dir-cli service update --name vsphere-webclient-74453a6d-4f01-4191-a95c-d3a905c1a516 --cert vsphere-webclient.crt --login administrator@vsphere.local --password 'YA,Bkc9ID8.)o_Sxr5t6'

/usr/lib/vmware-vmafd/bin/dir-cli service update --name vpxd-74453a6d-4f01-4191-a95c-d3a905c1a516 --cert vpxd.crt --login administrator@vsphere.local --password 'YA,Bkc9ID8.)o_Sxr5t6'

/usr/lib/vmware-vmafd/bin/dir-cli service update --name vpxd-extension-74453a6d-4f01-4191-a95c-d3a905c1a516 --cert vpxd-extension.crt --login administrator@vsphere.local --password 'YA,Bkc9ID8.)o_Sxr5t6'

Windows :

"%VMWARE_CIS_HOME%"\vmafdd\dir-cli service update --name machine-74453a6d-4f01-4191-a95c-d3a905c1a516 --cert machine.crt --login administrator@vsphere.local --password 'YA,Bkc9ID8.)o_Sxr5t6'

"%VMWARE_CIS_HOME%"\vmafdd\dir-cli service update --name vsphere-webclient-74453a6d-4f01-4191-a95c-d3a905c1a516 --cert vsphere-webclient.crt --login administrator@vsphere.local --password 'YA,Bkc9ID8.)o_Sxr5t6'

"%VMWARE_CIS_HOME%"\vmafdd\dir-cli service update --name vpxd-74453a6d-4f01-4191-a95c-d3a905c1a516 --cert vpxd.crt --login administrator@vsphere.local --password 'YA,Bkc9ID8.)o_Sxr5t6'

"%VMWARE_CIS_HOME%"\vmafdd\dir-cli service update --name vpxd-extension-74453a6d-4f01-4191-a95c-d3a905c1a516 --cert vpxd-extension.crt --login administrator@vsphere.local --password 'YA,Bkc9ID8.)o_Sxr5t6'

7 . Update the auto-deploy, imagebuilder and eam service with vpxd-extension certificate.

Appliance :

python /usr/lib/vmware-vpx/scripts/updateExtensionCertInVC.py -e com.vmware.vim.eam -c /certificate/vpxd-extension.crt -k /certificate/vpxd-extension.key -s <VC_FQDN> -u <SSO_admin>

python /usr/lib/vmware-vpx/scripts/updateExtensionCertInVC.py -e com.vmware.rbd -c /certificate/vpxd-extension.crt -k /certificate/vpxd-extension.key -s <VC_FQDN> -u <SSO_admin>

python /usr/lib/vmware-vpx/scripts/updateExtensionCertInVC.py -e com.vmware.imagebuilder  -c /certificate/vpxd-extension.crt -k /certificate/vpxd-extension.key -s <VC_FQDN> -u <SSO_admin>

Windows :

"%VMWARE_PYTHON_BIN%" C:\Program Files\VMware\vCenter Server\vpxd\scripts\updateExtensionCertInVC.py -e com.vmware.vim.eam -c C:\Certificates\vpxd-extension.crt -k C:\Certificates\vpxd-extension.key -s <vcenter_FQDN> -u <SSO_admin>

"%VMWARE_PYTHON_BIN%" C:\Program Files\VMware\vCenter Server\vpxd\scripts\updateExtensionCertInVC.py -e com.vmware.rbd -c C:\Certificates\vpxd-extension.crt -k C:\Certificates\vpxd-extension.key -s <vcenter_FQDN> -u <SSO_admin>

"%VMWARE_PYTHON_BIN%" C:\Program Files\VMware\vCenter Server\vpxd\scripts\updateExtensionCertInVC.py -e com.vmware.imagebuilder -c C:\Certificates\vpxd-extension.crt -k C:\Certificates\vpxd-extension.key -s <vcenter_FQDN> -u <SSO_admin>

8. Restart all the services.

Reference: 

https://docs.vmware.com/en/VMware-vSphere/6.0/com.vmware.vsphere.security.doc/GUID-BD70615E-BCAA-4906-8E13-67D0DBF715E4.html

vCenter Web Client crash

web client service crashes java.lang.OutOfMemoryError: PermGen space and java.lang.OutOfMemoryError
vSphere web client refused to start with memory errors
log location:
Windows: c:\programdata\VMware\vCenter\Logs\vsphere-client
Appliance: /var/log/vmware/vsphere-client


wrapper.log
INFO | jvm 1 | 2018/04/03 15:34:25 | Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread “org.springframework.scheduling.timer.TimerFactoryBean#0”
INFO | jvm 1 | 2018/04/03 15:34:33 |
INFO | jvm 1 | 2018/04/03 15:34:33 | Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread “http-bio-9443-exec-10”
INFO | jvm 1 | 2018/04/03 15:35:12 |
INFO | jvm 1 | 2018/04/03 15:35:12 | Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread “http-bio-9443-exec-6”
vsphere_client_virgo.log
[2018-04-03T15:28:45.855-04:00] [ERROR] http-bio-9090-exec-2 com.vmware.vise.util.concurrent.WorkerThread http-bio-9090-exec-2 terminated with exception: java.lang.OutOfMemoryError: PermGen space
[2018-04-03T15:28:46.773-04:00] [ERROR] http-bio-9090-exec-5 com.vmware.vise.util.concurrent.WorkerThread http-bio-9090-exec-5 terminated with exception: java.lang.OutOfMemoryError: PermGen space


Cause: insufficient  Web-client heap size/Insufficient PermGen space?configuration change


Scenario 1:  Heap-size
Resolution:
• Ensure there is sufficient free memory on the vCenter
free -m

• Review and Increase (double) the heap size of the web client
Appliance: cloudvm-ram-size -l vsphere-client
windows: C:\Program Files\VMware\vCenter Server\visl-integration\usr\sbin\cloudvm-ram-size.bat -l
cloudvm-ram-size.bat -C XXX 
• start vsphere-client and observe if this still crashes.
Scenario 2: PermGen
• Take a copy of the file service-layout.mfx as service-layout.mfx.bak
  Appliance path: /etc/vmware/
  Windows Path: C:\ProgramData\VMware\vCenterServer\cfg\
• Edit service-layout.mfx with a text editor
•  Change MaxPermMB size from 256 to 512 for the row vspherewebclientsvc. (increase accordingly depending on the number of plugins configured with vCenter_webclient)
• start vsphere-client service
Scenario 3: Problem persists even after increasing/maxing out scenario 1 and scenario 2.
• backup configuration file before you proceed
  cp /usr/lib/vmware-vsphere-client/server/wrapper/bin/vsphere-client /usr/lib/vmware-vsphere-client/server/wrapper/bin/vsphereclient.bak
• Edit the file using a text editor
vi /usr/lib/vmware-vsphere-client/server/wrapper/bin/vsphere-client
• Look for the line  “RUN_AS_USER=vsphere-client” and hash this
  vi vsphere-client