Saturday, January 19, 2019

VAPI Endpoint service failure

VAPI Endpoint service failure 


ERROR | state-manager1            | StsBuilder                     | Failed to acquire token for the solution user
2018-11-15T21:50:42.647Z | ERROR | state-manager1            | DefaultStateManager            | Could not initialize endpoint runtime state.
com.vmware.vapi.endpoint.config.ConfigurationException: com.vmware.vim.sso.client.exception.AuthenticationFailedException: Provided credentials are not valid.
        at com.vmware.vapi.endpoint.cis.StsBuilder.createToken(StsBuilder.java:182)
        at com.vmware.vapi.endpoint.cis.StsBuilder.rebuild(StsBuilder.java:77)
        at com.vmware.vapi.endpoint.cis.StsBuilder.buildInitial(StsBuilder.java:54)
        at com.vmware.vapi.state.impl.DefaultStateManager.build(DefaultStateManager.java:354)
        at com.vmware.vapi.state.impl.DefaultStateManager$1.doInitialConfig(DefaultStateManager.java:168)
        at com.vmware.vapi.state.impl.DefaultStateManager$1.run(DefaultStateManager.java:151)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: com.vmware.vim.sso.client.exception.AuthenticationFailedException: Provided credentials are not valid.
        at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor.handleFaultCondition(SecurityTokenServiceImpl.java:852)
        at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor.sendRequest(SecurityTokenServiceImpl.java:788)
        at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor.executeRoundtrip(SecurityTokenServiceImpl.java:714)
        at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl.acquireTokenByCertificate(SecurityTokenServiceImpl.java:473)
        at com.vmware.vapi.endpoint.cis.StsBuilder.createToken(StsBuilder.java:179)


Please follow the steps below


/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store vpxd --text | less

MACHINE_SSL_CERT
TRUSTED_ROOTS
TRUSTED_ROOT_CRLS
machine
vpxd
vpxd-extension
vsphere-webclient
SMS


/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store <stored name> --alias <alias name> --output /certificate/<certificate usage name>.crt
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store machine --alias machine --output /certificate/machine.crt
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store vpxd-extension --alias vpxd-extension --output /certificate/vpxd-extension.crt
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store vpxd --alias vpxd --output /certificate/vpxd.crt
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store vsphere-webclient --alias vsphere-webclient --output /certificate/vsphere-webclient.crt


dir-cli service update --name vpxd-extension-d74bb41a-f337-4667-9294-9df39322b428- --cert /cert/vpxd.crt --login administrator@vsphere.local

root@VCENTER51 [ /certificate ]# less vsphere-webclient.crt
root@VCENTER51 [ /certificate ]# /usr/lib/vmware-vmafd/bin/dir-cli service list
Enter password for administrator@vsphere.local:
1. WebClient_2016.11.23_145904
2. machine-2253fa42-d1ee-11e7-ba38-0050569c439f
3. vsphere-webclient-2253fa42-d1ee-11e7-ba38-0050569c439f
4. machine-d74bb41a-f337-4667-9294-9df39322b428
5. vsphere-webclient-d74bb41a-f337-4667-9294-9df39322b428
6. vpxd-d74bb41a-f337-4667-9294-9df39322b428
7. vpxd-extension-d74bb41a-f337-4667-9294-9df39322b428


/usr/lib/vmware-vmafd/bin/dir-cli service update --name machine-d74bb41a-f337-4667-9294-9df39322b428 --cert /certificate/machine.crt --login administrator@vsphere.local
/usr/lib/vmware-vmafd/bin/dir-cli service update --name vsphere-webclient-d74bb41a-f337-4667-9294-9df39322b428 --cert /certificate/vsphere-webclient.crt --login administrator@vsphere.local
/usr/lib/vmware-vmafd/bin/dir-cli service update --name vpxd-d74bb41a-f337-4667-9294-9df39322b428 --cert /certificate/vpxd.crt --login administrator@vsphere.local
/usr/lib/vmware-vmafd/bin/dir-cli service update --name vpxd-extension-d74bb41a-f337-4667-9294-9df39322b428 --cert /certificate/vpxd-extension.crt --login administrator@vsphere.local

Manually Replacing the solution user certificate on VCSA/PSC 6.x

Manually Replacing the solution user certificate on VCSA/PSC 6.x


Note :

-- Take a snapshot of the VC or PSC before proceeding.
-- PSC has two solution users : machine and vsphere-webclient.
-- VC has four solution users : machine, vpxd, vpxd-extension and vsphere-webclient.
-- VC with embedded PSC has four solution users : machine, vpxd, vpxd-extension and vsphere-webclient.
-- SSO admin username and password would vary depending on the configuration.
-- Windows Install directory may vary depending on your installation. 
-- Run the below command to get the machine ID of the node :

Appliance : /usr/lib/vmware-vmafd/bin/vmafd-cli get-machine-id --server-name localhost
Windows : "%VMWARE_CIS_HOME%"\vmafdd\vmafd-cli get-machine-id --server-name localhost

0. Take a backup of the old solution user certificates and its private keys.

Appliance :

/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store machine --alias machine --output /root/certificate/machine.crt
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store vpxd --alias vpxd --output /root/certificate/vpxd.crt
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store vpxd-extension --alias vpxd-extension --output /root/certificate/vpxd-extension.crt
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store vsphere-webclient --alias vsphere-webclient --output /root/certificate/vsphere-webclient.crt

/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store machine --alias machine --output /root/certificate/machine.key
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store vpxd --alias vpxd --output /root/certificate/vpxd.key
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store vpxd-extension --alias vpxd-extension --output /root/certificate/vpxd-extension.key
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store vsphere-webclient --alias vsphere-webclient --output /root/certificate/vsphere-webclient.key

Windows :

"%VMWARE_CIS_HOME%"\vmafdd\vecs-cli entry getcert --store machine --alias machine --output c:\certificate\machine.crt
"%VMWARE_CIS_HOME%"\vmafdd\vecs-cli entry getcert --store vpxd --alias vpxd --output c:\certificate\vpxd.crt
"%VMWARE_CIS_HOME%"\vmafdd\vecs-cli entry getcert --store vpxd-extension --alias vpxd-extension --output c:\certificate\vpxd-extension.crt
"%VMWARE_CIS_HOME%"\vmafdd\vecs-cli entry getcert --store vsphere-webclient --alias vsphere-webclient --output c:\certificate\vsphere-webclient.crt

"%VMWARE_CIS_HOME%"\vmafdd\vecs-cli entry getcert --store machine --alias machine --output c:\certificate\machine.key
"%VMWARE_CIS_HOME%"\vmafdd\vecs-cli entry getcert --store vpxd --alias vpxd --output c:\certificate\vpxd.key
"%VMWARE_CIS_HOME%"\vmafdd\vecs-cli entry getcert --store vpxd-extension --alias vpxd-extension --output c:\certificate\vpxd-extension.key
"%VMWARE_CIS_HOME%"\vmafdd\vecs-cli entry getcert --store vsphere-webclient --alias vsphere-webclient --output c:\certificate\vsphere-webclient.key

1. Create the configuration file for all the solution users :

a) machine.cfg :

[ req ]
distinguished_name = req_distinguished_name
encrypt_key = no
prompt = no
string_mask = nombstr
req_extensions = v3_req
[ v3_req ]
basicConstraints = CA:false
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName = DNS:vcenter-nl-1.filmlogistics.local     # FQDN of VC or PSC
[ req_distinguished_name ]
countryName = NL
stateOrProvinceName = Utrecht
localityName = Zeist
0.organizationName = VMware
organizationalUnitName = mID-74453a6d-4f01-4191-a95c-d3a905c1a516     # machine ID of the VC/PSC
commonName = machine

b) vpxd.cfg :

[ req ]
distinguished_name = req_distinguished_name
encrypt_key = no
prompt = no
string_mask = nombstr
req_extensions = v3_req
[ v3_req ]
basicConstraints = CA:false
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName = DNS:vcenter-nl-1.filmlogistics.local     # FQDN of VC or PSC
[ req_distinguished_name ]
countryName = NL
stateOrProvinceName = Utrecht
localityName = Zeist
0.organizationName = VMware
organizationalUnitName = mID-74453a6d-4f01-4191-a95c-d3a905c1a516     # machine ID of the VC/PSC
commonName = vpxd

c) vpxd-extension.cfg :

[ req ]
distinguished_name = req_distinguished_name
encrypt_key = no
prompt = no
string_mask = nombstr
req_extensions = v3_req
[ v3_req ]
basicConstraints = CA:false
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName = DNS:vcenter-nl-1.filmlogistics.local     # FQDN of VC or PSC
[ req_distinguished_name ]
countryName = NL
stateOrProvinceName = Utrecht
localityName = Zeist
0.organizationName = VMware
organizationalUnitName = mID-74453a6d-4f01-4191-a95c-d3a905c1a516     # machine ID of the VC/PSC
commonName = vpxd-extension

d) vsphere-webclient.cfg :

[ req ]
distinguished_name = req_distinguished_name
encrypt_key = no
prompt = no
string_mask = nombstr
req_extensions = v3_req
[ v3_req ]
basicConstraints = CA:false
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName = DNS:vcenter-nl-1.filmlogistics.local     # FQDN of VC or PSC
[ req_distinguished_name ]
countryName = NL
stateOrProvinceName = Utrecht
localityName = Zeist
0.organizationName = VMware
organizationalUnitName = mID-74453a6d-4f01-4191-a95c-d3a905c1a516     # machine ID of the VC/PSC
commonName = vsphere-webclient

2. Create the certificate signing request and generate a new certificate for all the solution certs :

Appliance :
openssl req -new -nodes -out machine.csr -newkey rsa:2048 -keyout machine.key -config machine.cfg
openssl x509 -req -days 3650 -in machine.csr -out machine.crt -CA /var/lib/vmware/vmca/root.cer -CAkey /var/lib/vmware/vmca/privatekey.pem -extensions v3_req -CAcreateserial -extfile machine.cfg

openssl req -new -nodes -out vpxd.csr -newkey rsa:2048 -keyout vpxd.key -config vpxd.cfg
openssl x509 -req -days 3650 -in vpxd.csr -out vpxd.crt -CA /var/lib/vmware/vmca/root.cer -CAkey /var/lib/vmware/vmca/privatekey.pem -extensions v3_req -CAcreateserial -extfile vpxd.cfg

openssl req -new -nodes -out vpxd-extension.csr -newkey rsa:2048 -keyout vpxd-extension.key -config vpxd-extension.cfg
openssl x509 -req -days 3650 -in vpxd-extension.csr -out vpxd-extension.crt -CA /var/lib/vmware/vmca/root.cer -CAkey /var/lib/vmware/vmca/privatekey.pem -extensions v3_req -CAcreateserial -extfile vpxd-extension.cfg

openssl req -new -nodes -out vsphere-webclient.csr -newkey rsa:2048 -keyout vsphere-webclient.key -config vsphere-webclient.cfg
openssl x509 -req -days 3650 -in vsphere-webclient.csr -out vsphere-webclient.crt -CA /var/lib/vmware/vmca/root.cer -CAkey /var/lib/vmware/vmca/privatekey.pem -extensions v3_req -CAcreateserial -extfile vsphere-webclient.cfg

Windows :
"%VMWARE_OPENSSL_BIN%" req -new -nodes -out machine.csr -newkey rsa:2048 -keyout machine.key -config machine.cfg
"%VMWARE_OPENSSL_BIN%" x509 -req -days 3650 -in machine.csr -out machine.crt -CA /var/lib/vmware/vmca/root.cer -CAkey /var/lib/vmware/vmca/privatekey.pem -extensions v3_req -CAcreateserial -extfile machine.cfg

"%VMWARE_OPENSSL_BIN%" req -new -nodes -out vpxd.csr -newkey rsa:2048 -keyout vpxd.key -config vpxd.cfg
"%VMWARE_OPENSSL_BIN%" x509 -req -days 3650 -in vpxd.csr -out vpxd.crt -CA /var/lib/vmware/vmca/root.cer -CAkey /var/lib/vmware/vmca/privatekey.pem -extensions v3_req -CAcreateserial -extfile vpxd.cfg

"%VMWARE_OPENSSL_BIN%" req -new -nodes -out vpxd-extension.csr -newkey rsa:2048 -keyout vpxd-extension.key -config vpxd-extension.cfg
"%VMWARE_OPENSSL_BIN%" x509 -req -days 3650 -in vpxd-extension.csr -out vpxd-extension.crt -CA /var/lib/vmware/vmca/root.cer -CAkey /var/lib/vmware/vmca/privatekey.pem -extensions v3_req -CAcreateserial -extfile vpxd-extension.cfg

"%VMWARE_OPENSSL_BIN%" req -new -nodes -out vsphere-webclient.csr -newkey rsa:2048 -keyout vsphere-webclient.key -config vsphere-webclient.cfg
"%VMWARE_OPENSSL_BIN%" x509 -req -days 3650 -in vsphere-webclient.csr -out vsphere-webclient.crt -CA /var/lib/vmware/vmca/root.cer -CAkey /var/lib/vmware/vmca/privatekey.pem -extensions v3_req -CAcreateserial -extfile vsphere-webclient.cfg

3. Delete the old certificate entries from VECS store :

Appliance :
/usr/lib/vmware-vmafd/bin/vecs-cli entry delete --store machine --alias machine -y
/usr/lib/vmware-vmafd/bin/vecs-cli entry delete --store vpxd --alias vpxd -y
/usr/lib/vmware-vmafd/bin/vecs-cli entry delete --store vpxd-extension --alias vpxd-extension -y
/usr/lib/vmware-vmafd/bin/vecs-cli entry delete --store vsphere-webclient --alias vsphere-webclient -y

Windows :
"%VMWARE_CIS_HOME%"\vmafdd\vecs-cli entry delete --store machine --alias machine --cert machine.crt --key machine.key
"%VMWARE_CIS_HOME%"\vmafdd\vecs-cli entry delete --store vpxd --alias vpxd --cert vpxd.crt --key vpxd.key
"%VMWARE_CIS_HOME%"\vmafdd\vecs-cli entry delete --store vpxd-extension --alias vpxd-extension --cert vpxd-extension.crt --key vpxd-extension.key
"%VMWARE_CIS_HOME%"\vmafdd\vecs-cli entry delete --store vsphere-webclient --alias vsphere-webclient --cert vsphere-webclient.crt --key vsphere-webclient.key

4. Update the  new certificates in the VECS store :

Appliance :
/usr/lib/vmware-vmafd/bin/vecs-cli entry create --store machine --alias machine --cert machine.crt --key machine.key
/usr/lib/vmware-vmafd/bin/vecs-cli entry create --store vpxd --alias vpxd --cert vpxd.crt --key vpxd.key
/usr/lib/vmware-vmafd/bin/vecs-cli entry create --store vpxd-extension --alias vpxd-extension --cert vpxd-extension.crt --key vpxd-extension.key
/usr/lib/vmware-vmafd/bin/vecs-cli entry create --store vsphere-webclient --alias vsphere-webclient --cert vsphere-webclient.crt --key vsphere-webclient.key

Windows :
"%VMWARE_CIS_HOME%"\vmafdd\vecs-cli entry create --store machine --alias machine --cert machine.crt --key machine.key
"%VMWARE_CIS_HOME%"\vmafdd\vecs-cli entry create --store vpxd --alias vpxd --cert vpxd.crt --key vpxd.key
"%VMWARE_CIS_HOME%"\vmafdd\vecs-cli entry create --store vpxd-extension --alias vpxd-extension --cert vpxd-extension.crt --key vpxd-extension.key
"%VMWARE_CIS_HOME%"\vmafdd\vecs-cli entry create --store vsphere-webclient --alias vsphere-webclient --cert vsphere-webclient.crt --key vsphere-webclient.key

5. List all the solution users : 
/usr/lib/vmware-vmafd/bin/dir-cli service list --login administrator@vsphere.local --password 'YA,Bkc9ID8.)o_Sxr5t6'
1. machine-74453a6d-4f01-4191-a95c-d3a905c1a516
2. vsphere-webclient-74453a6d-4f01-4191-a95c-d3a905c1a516
3. vpxd-74453a6d-4f01-4191-a95c-d3a905c1a516
4. vpxd-extension-74453a6d-4f01-4191-a95c-d3a905c1a516 

6. Update all the solution users with the new certificate :

Appliance :
/usr/lib/vmware-vmafd/bin/dir-cli service update --name machine-74453a6d-4f01-4191-a95c-d3a905c1a516 --cert machine.crt --login administrator@vsphere.local --password 'YA,Bkc9ID8.)o_Sxr5t6'
/usr/lib/vmware-vmafd/bin/dir-cli service update --name vsphere-webclient-74453a6d-4f01-4191-a95c-d3a905c1a516 --cert vsphere-webclient.crt --login administrator@vsphere.local --password 'YA,Bkc9ID8.)o_Sxr5t6'
/usr/lib/vmware-vmafd/bin/dir-cli service update --name vpxd-74453a6d-4f01-4191-a95c-d3a905c1a516 --cert vpxd.crt --login administrator@vsphere.local --password 'YA,Bkc9ID8.)o_Sxr5t6'
/usr/lib/vmware-vmafd/bin/dir-cli service update --name vpxd-extension-74453a6d-4f01-4191-a95c-d3a905c1a516 --cert vpxd-extension.crt --login administrator@vsphere.local --password 'YA,Bkc9ID8.)o_Sxr5t6'

Windows :
"%VMWARE_CIS_HOME%"\vmafdd\dir-cli service update --name machine-74453a6d-4f01-4191-a95c-d3a905c1a516 --cert machine.crt --login administrator@vsphere.local --password 'YA,Bkc9ID8.)o_Sxr5t6'
"%VMWARE_CIS_HOME%"\vmafdd\dir-cli service update --name vsphere-webclient-74453a6d-4f01-4191-a95c-d3a905c1a516 --cert vsphere-webclient.crt --login administrator@vsphere.local --password 'YA,Bkc9ID8.)o_Sxr5t6'
"%VMWARE_CIS_HOME%"\vmafdd\dir-cli service update --name vpxd-74453a6d-4f01-4191-a95c-d3a905c1a516 --cert vpxd.crt --login administrator@vsphere.local --password 'YA,Bkc9ID8.)o_Sxr5t6'
"%VMWARE_CIS_HOME%"\vmafdd\dir-cli service update --name vpxd-extension-74453a6d-4f01-4191-a95c-d3a905c1a516 --cert vpxd-extension.crt --login administrator@vsphere.local --password 'YA,Bkc9ID8.)o_Sxr5t6'

7 . Update the auto-deploy, imagebuilder and eam service with vpxd-extension certificate.

Appliance :

python /usr/lib/vmware-vpx/scripts/updateExtensionCertInVC.py -e com.vmware.vim.eam -c /certificate/vpxd-extension.crt -k /certificate/vpxd-extension.key -s <VC_FQDN> -u <SSO_admin>
python /usr/lib/vmware-vpx/scripts/updateExtensionCertInVC.py -e com.vmware.rbd -c /certificate/vpxd-extension.crt -k /certificate/vpxd-extension.key -s <VC_FQDN> -u <SSO_admin>
python /usr/lib/vmware-vpx/scripts/updateExtensionCertInVC.py -e com.vmware.imagebuilder  -c /certificate/vpxd-extension.crt -k /certificate/vpxd-extension.key -s <VC_FQDN> -u <SSO_admin>

Windows :

"%VMWARE_PYTHON_BIN%" C:\Program Files\VMware\vCenter Server\vpxd\scripts\updateExtensionCertInVC.py -e com.vmware.vim.eam -c C:\Certificates\vpxd-extension.crt -k C:\Certificates\vpxd-extension.key -s <vcenter_FQDN> -u <SSO_admin>
"%VMWARE_PYTHON_BIN%" C:\Program Files\VMware\vCenter Server\vpxd\scripts\updateExtensionCertInVC.py -e com.vmware.rbd -c C:\Certificates\vpxd-extension.crt -k C:\Certificates\vpxd-extension.key -s <vcenter_FQDN> -u <SSO_admin>
"%VMWARE_PYTHON_BIN%" C:\Program Files\VMware\vCenter Server\vpxd\scripts\updateExtensionCertInVC.py -e com.vmware.imagebuilder -c C:\Certificates\vpxd-extension.crt -k C:\Certificates\vpxd-extension.key -s <vcenter_FQDN> -u <SSO_admin>

8. Restart all the services.

Reference: 

https://docs.vmware.com/en/VMware-vSphere/6.0/com.vmware.vsphere.security.doc/GUID-BD70615E-BCAA-4906-8E13-67D0DBF715E4.html

vCenter Web Client crash


    web client service crashes java.lang.OutOfMemoryError: PermGen space and java.lang.OutOfMemoryError
    vSphere web client refused to start with memory errors
    log location:
    Windows: c:\programdata\VMware\vCenter\Logs\vsphere-client
    Appliance: /var/log/vmware/vsphere-client

    wrapper.log
    INFO | jvm 1 | 2018/04/03 15:34:25 | Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread “org.springframework.scheduling.timer.TimerFactoryBean#0”
    INFO | jvm 1 | 2018/04/03 15:34:33 |
    INFO | jvm 1 | 2018/04/03 15:34:33 | Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread “http-bio-9443-exec-10”
    INFO | jvm 1 | 2018/04/03 15:35:12 |
    INFO | jvm 1 | 2018/04/03 15:35:12 | Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread “http-bio-9443-exec-6”
    vsphere_client_virgo.log
    [2018-04-03T15:28:45.855-04:00] [ERROR] http-bio-9090-exec-2 com.vmware.vise.util.concurrent.WorkerThread http-bio-9090-exec-2 terminated with exception: java.lang.OutOfMemoryError: PermGen space
    [2018-04-03T15:28:46.773-04:00] [ERROR] http-bio-9090-exec-5 com.vmware.vise.util.concurrent.WorkerThread http-bio-9090-exec-5 terminated with exception: java.lang.OutOfMemoryError: PermGen space

    Cause: insufficient  Web-client heap size/Insufficient PermGen space?configuration change

    Scenario 1:  Heap-size
    Resolution:
    • Ensure there is sufficient free memory on the vCenter
    free -m
    • Review and Increase (double) the heap size of the web client
    Appliance: cloudvm-ram-size -l vsphere-client
    windows: C:\Program Files\VMware\vCenter Server\visl-integration\usr\sbin\cloudvm-ram-size.bat -l
    cloudvm-ram-size.bat -C XXX 
    • start vsphere-client and observe if this still crashes.
    Scenario 2: PermGen
    • Take a copy of the file service-layout.mfx as service-layout.mfx.bak
      Appliance path: /etc/vmware/
      Windows Path: C:\ProgramData\VMware\vCenterServer\cfg\
    • Edit service-layout.mfx with a text editor
    •  Change MaxPermMB size from 256 to 512 for the row vspherewebclientsvc. (increase accordingly depending on the number of plugins configured with vCenter_webclient)
    • start vsphere-client service
    Scenario 3: Problem persists even after increasing/maxing out scenario 1 and scenario 2.
    • backup configuration file before you proceed
      cp /usr/lib/vmware-vsphere-client/server/wrapper/bin/vsphere-client /usr/lib/vmware-vsphere-client/server/wrapper/bin/vsphereclient.bak
    • Edit the file using a text editor
    vi /usr/lib/vmware-vsphere-client/server/wrapper/bin/vsphere-client
    • Look for the line  “RUN_AS_USER=vsphere-client” and hash this
      vi vsphere-client




Unable to add vCenter to Usage Meter 3.6 after replacing the vCenter certificate

 Issue: Unable to add the vCenter endpoint to the usage meter due to certificate error Error: There was a problem checking the certificate v...