Handy Command and Articles to troubleshoot PSC and Certificate related query in vCenter

List of commands:

To find the PNID:

• /usr/lib/vmware-vmafd/bin/vmafd-cli get-pnid --server-name localhost
• "C:\Program Files\VMware\vCenter Server\vmafdd\vmafd-cli.exe" get-pnid --server-name localhost

Which PSC is my VC pointing to:

• /usr/lib/vmware-vmafd/bin/vmafd-cli get-ls-location --server-name localhost
• C:\Program Files\VMware\vCenter Server\vmafdd\vmafd-cli get-ls-location --server-name localhost

Get the site name where my PSC/VC is:

• /usr/lib/vmware-vmafd/bin/vmafd-cli get-site-name --server-name localhost
• C:\Program Files\VMware\vCenter Server\vmafdd\vmafd-cli get-site-name  --server-name localhost

SSO Domain name:

• /etc/vmware/install-defaults/vmdir.domain-name
• C:\ProgramData\VMware\vCenterServer\cfg\install-defaults\vmdir.domain-name

Certificate Manager Location:

• Windows vCenter Server:  C:\Program Files\VMware\vCenter Server\vmcad\certificate-manager
• vCenter Server Appliance:  /usr/lib/vmware-vmca/bin/certificate-manager

Location for the cert store:

• /usr/lib/vmware-vmafd/bin à includes vecs-cli , dir-cli, vmafd-cli

Hard copy of the certificate available in:

• /var/lib/vmware/vmca/root.cer
• ProgramData\VMware\CIS\data\vmca\root.cer

To get the copy of the MACHINE_SSL_CERT and the KEY used :

• /usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store MACHINE_SSL_CERT --alias __MACHINE_CERT --output /var/tmp/Machine_SSL.crt
• /usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --store MACHINE_SSL_CERT --alias __MACHINE_CERT --output /var/tmp/Machine_SSL.key
•  

Location for certificates that we most commonly use :

1. STS certificate : etc/vmware-sso/keys/ssoserver.crt
2. VMDIRD certificate : /usr/lib/vmware-vmdir/share/config/vmdircert.pem

Location for the certs in jexlorer:

     >>Trusted roots ====>ConfigurationàCertificate authority

     >>Lookup service ==>ConfigurationàSitesà”Site-name”àLookupserviceàService Registrations

    >>Solution users =====>Service Principals

    >>STS certs -->Services-->Identity manager-->Tenants-->vsphere.local-->TenanatCredential-1

To view the certificates using openssl command:

• openssl x509 -in certificate.crt -noout -text | less
• openssl s_client -connect “Server FQDN”:443

Eg: To get the machine SSL for vcsa1.gsslabs.org:

      openssl s_client -connect ORLpD1PSC-VIP.catmktg.com:443

Get the lstool output:

• /usr/lib/vmidentity/tools/scripts/lstool.py list --url http://localhost:7080/lookupservice/sdk  > /tmp/psc_services.txt

• "%VMWARE_PYTHON_BIN%" "%VMWARE_CIS_HOME%\VMware Identity Services\lstool\scripts\lstool.py" list --url http://localhost:7080/lookupservice/sdk  > c:\psc_services.txt

Different scenarios lstool is used:

• To output vCenter's service ID only in a particular site:

à/usr/lib/vmidentity/tools/scripts/lstool.py list --url https://PSC.FQDN/lookupservice/sdk --site <site-name> --type vcenterserver --id-only

• To export the inforation as spec file(text file):

         à/usr/lib/vmidentity/tools/scripts/lstool.py get --url https://PSC.FQDN/lookupservice/sdk --id      "vCenter's service-id" --as-spec > /tmp/vcenterserver.txt

• How to edit the spec file and re-register back:

à/usr/lib/vmidentity/tools/scripts/lstool.py reregiter --url https://PSC.FQDN/lookupservice/sdk --id "vCenter's service-id --spec /tmp/vcenterserver.txt --user administrator@vsphere.local --password "VMware!"

• Get the service ID associated with the NODE ID:

              /usr/lib/vmidentity/tools/scripts/lstool.py list --url http://localhost:7080/lookupservice/sdk --node "NODE ID of the PSC/VC" --id-only >/tmp/node.txt

     To get the node ID: /etc/vmware/install-defaults  -à vmdir.ldu-guid

• Determining replication agreements and status with the Platform Services Controller 6.x 

                   https://kb.vmware.com/s/article/2127057

• Manually reviewing certificates in VMware Endpoint Certificate Store

   https://kb.vmware.com/s/article/2111411

• Updating SSL Trust Anchors (When there is a thumbprint mismatch )

                   https://kb.vmware.com/s/article/50112066

• Removing expired certs from trusted roots:

            https://kb.vmware.com/s/article/2146011

• Feature walkthrough for cert replacement:

     https://featurewalkthrough.vmware.com/t/vsphere-6-5/ssl-certificate-replacement-hybrid-mode/19

• Obtaining vSphere certificates from a Microsoft Certificate Authority and creating templates for SSL cert creation

                    https://kb.vmware.com/s/article/2112014

                    https://kb.vmware.com/s/article/2112009

• The steps for the certificate generation/replacement in the Load balancer environment:

      https://kb.vmware.com/s/article/2147627 

• Generating the STS certificates:

   https://docs.vmware.com/en/VMware-vSphere/6.0/com.vmware.vsphere.security.doc/GUID-497233EA-AEF9-464B-A9C3-CCAEEA90C801.html

• Replacing the vmdird certificates :

                  https://docs.vmware.com/en/VMware-vSphere/6.0/com.vmware.vsphere.security.doc/GUID-585CF428-2BBC-47CE-A386-9A39D3DFE0BF.html

• Removing the Service ID :

                https://kb.vmware.com/s/article/2050273 

• Using the cmsso command to unregister vCenter Server from Single Sign-On :

  https://kb.vmware.com/s/article/2106736

• Repoint vCenter Server 6.x between External PSC within a site

                 https://kb.vmware.com/s/article/2113917

Manually Replacing the solution user certificate on VCSA/PSC 6.x

Manually Replacing the solution user certificate on VCSA/PSC 6.x

Note :

-- Take a snapshot of the VC or PSC before proceeding.

-- PSC has two solution users : machine and vsphere-webclient.

-- VC has four solution users : machine, vpxd, vpxd-extension and vsphere-webclient.

-- VC with embedded PSC has four solution users : machine, vpxd, vpxd-extension and vsphere-webclient.

-- SSO admin username and password would vary depending on the configuration.

-- Windows Install directory may vary depending on your installation. 

-- Run the below command to get the machine ID of the node :

Appliance : /usr/lib/vmware-vmafd/bin/vmafd-cli get-machine-id --server-name localhost

Windows : "%VMWARE_CIS_HOME%"\vmafdd\vmafd-cli get-machine-id --server-name localhost

0. Take a backup of the old solution user certificates and its private keys.

Appliance :

/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store machine --alias machine --output /root/certificate/machine.crt

/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store vpxd --alias vpxd --output /root/certificate/vpxd.crt

/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store vpxd-extension --alias vpxd-extension --output /root/certificate/vpxd-extension.crt

/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store vsphere-webclient --alias vsphere-webclient --output /root/certificate/vsphere-webclient.crt

/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store machine --alias machine --output /root/certificate/machine.key

/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store vpxd --alias vpxd --output /root/certificate/vpxd.key

/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store vpxd-extension --alias vpxd-extension --output /root/certificate/vpxd-extension.key

/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store vsphere-webclient --alias vsphere-webclient --output /root/certificate/vsphere-webclient.key

Windows :

"%VMWARE_CIS_HOME%"\vmafdd\vecs-cli entry getcert --store machine --alias machine --output c:\certificate\machine.crt

"%VMWARE_CIS_HOME%"\vmafdd\vecs-cli entry getcert --store vpxd --alias vpxd --output c:\certificate\vpxd.crt

"%VMWARE_CIS_HOME%"\vmafdd\vecs-cli entry getcert --store vpxd-extension --alias vpxd-extension --output c:\certificate\vpxd-extension.crt

"%VMWARE_CIS_HOME%"\vmafdd\vecs-cli entry getcert --store vsphere-webclient --alias vsphere-webclient --output c:\certificate\vsphere-webclient.crt

"%VMWARE_CIS_HOME%"\vmafdd\vecs-cli entry getcert --store machine --alias machine --output c:\certificate\machine.key

"%VMWARE_CIS_HOME%"\vmafdd\vecs-cli entry getcert --store vpxd --alias vpxd --output c:\certificate\vpxd.key

"%VMWARE_CIS_HOME%"\vmafdd\vecs-cli entry getcert --store vpxd-extension --alias vpxd-extension --output c:\certificate\vpxd-extension.key

"%VMWARE_CIS_HOME%"\vmafdd\vecs-cli entry getcert --store vsphere-webclient --alias vsphere-webclient --output c:\certificate\vsphere-webclient.key

1. Create the configuration file for all the solution users :

a) machine.cfg :

[ req ]

distinguished_name = req_distinguished_name

encrypt_key = no

prompt = no

string_mask = nombstr

req_extensions = v3_req

[ v3_req ]

basicConstraints = CA:false

keyUsage = nonRepudiation, digitalSignature, keyEncipherment

subjectAltName = DNS:vcenter-nl-1.filmlogistics.local     # FQDN of VC or PSC

[ req_distinguished_name ]

countryName = NL

stateOrProvinceName = Utrecht

localityName = Zeist

0.organizationName = VMware

organizationalUnitName = mID-74453a6d-4f01-4191-a95c-d3a905c1a516     # machine ID of the VC/PSC

commonName = machine

b) vpxd.cfg :

[ req ]

distinguished_name = req_distinguished_name

encrypt_key = no

prompt = no

string_mask = nombstr

req_extensions = v3_req

[ v3_req ]

basicConstraints = CA:false

keyUsage = nonRepudiation, digitalSignature, keyEncipherment

subjectAltName = DNS:vcenter-nl-1.filmlogistics.local     # FQDN of VC or PSC

[ req_distinguished_name ]

countryName = NL

stateOrProvinceName = Utrecht

localityName = Zeist

0.organizationName = VMware

organizationalUnitName = mID-74453a6d-4f01-4191-a95c-d3a905c1a516     # machine ID of the VC/PSC

commonName = vpxd

c) vpxd-extension.cfg :

[ req ]

distinguished_name = req_distinguished_name

encrypt_key = no

prompt = no

string_mask = nombstr

req_extensions = v3_req

[ v3_req ]

basicConstraints = CA:false

keyUsage = nonRepudiation, digitalSignature, keyEncipherment

subjectAltName = DNS:vcenter-nl-1.filmlogistics.local     # FQDN of VC or PSC

[ req_distinguished_name ]

countryName = NL

stateOrProvinceName = Utrecht

localityName = Zeist

0.organizationName = VMware

organizationalUnitName = mID-74453a6d-4f01-4191-a95c-d3a905c1a516     # machine ID of the VC/PSC

commonName = vpxd-extension

d) vsphere-webclient.cfg :

[ req ]

distinguished_name = req_distinguished_name

encrypt_key = no

prompt = no

string_mask = nombstr

req_extensions = v3_req

[ v3_req ]

basicConstraints = CA:false

keyUsage = nonRepudiation, digitalSignature, keyEncipherment

subjectAltName = DNS:vcenter-nl-1.filmlogistics.local     # FQDN of VC or PSC

[ req_distinguished_name ]

countryName = NL

stateOrProvinceName = Utrecht

localityName = Zeist

0.organizationName = VMware

organizationalUnitName = mID-74453a6d-4f01-4191-a95c-d3a905c1a516     # machine ID of the VC/PSC

commonName = vsphere-webclient

2. Create the certificate signing request and generate a new certificate for all the solution certs :

Appliance :

openssl req -new -nodes -out machine.csr -newkey rsa:2048 -keyout machine.key -config machine.cfg

openssl x509 -req -days 3650 -in machine.csr -out machine.crt -CA /var/lib/vmware/vmca/root.cer -CAkey /var/lib/vmware/vmca/privatekey.pem -extensions v3_req -CAcreateserial -extfile machine.cfg

openssl req -new -nodes -out vpxd.csr -newkey rsa:2048 -keyout vpxd.key -config vpxd.cfg

openssl x509 -req -days 3650 -in vpxd.csr -out vpxd.crt -CA /var/lib/vmware/vmca/root.cer -CAkey /var/lib/vmware/vmca/privatekey.pem -extensions v3_req -CAcreateserial -extfile vpxd.cfg

openssl req -new -nodes -out vpxd-extension.csr -newkey rsa:2048 -keyout vpxd-extension.key -config vpxd-extension.cfg

openssl x509 -req -days 3650 -in vpxd-extension.csr -out vpxd-extension.crt -CA /var/lib/vmware/vmca/root.cer -CAkey /var/lib/vmware/vmca/privatekey.pem -extensions v3_req -CAcreateserial -extfile vpxd-extension.cfg

openssl req -new -nodes -out vsphere-webclient.csr -newkey rsa:2048 -keyout vsphere-webclient.key -config vsphere-webclient.cfg

openssl x509 -req -days 3650 -in vsphere-webclient.csr -out vsphere-webclient.crt -CA /var/lib/vmware/vmca/root.cer -CAkey /var/lib/vmware/vmca/privatekey.pem -extensions v3_req -CAcreateserial -extfile vsphere-webclient.cfg

Windows :

"%VMWARE_OPENSSL_BIN%" req -new -nodes -out machine.csr -newkey rsa:2048 -keyout machine.key -config machine.cfg

"%VMWARE_OPENSSL_BIN%" x509 -req -days 3650 -in machine.csr -out machine.crt -CA /var/lib/vmware/vmca/root.cer -CAkey /var/lib/vmware/vmca/privatekey.pem -extensions v3_req -CAcreateserial -extfile machine.cfg

"%VMWARE_OPENSSL_BIN%" req -new -nodes -out vpxd.csr -newkey rsa:2048 -keyout vpxd.key -config vpxd.cfg

"%VMWARE_OPENSSL_BIN%" x509 -req -days 3650 -in vpxd.csr -out vpxd.crt -CA /var/lib/vmware/vmca/root.cer -CAkey /var/lib/vmware/vmca/privatekey.pem -extensions v3_req -CAcreateserial -extfile vpxd.cfg

"%VMWARE_OPENSSL_BIN%" req -new -nodes -out vpxd-extension.csr -newkey rsa:2048 -keyout vpxd-extension.key -config vpxd-extension.cfg

"%VMWARE_OPENSSL_BIN%" x509 -req -days 3650 -in vpxd-extension.csr -out vpxd-extension.crt -CA /var/lib/vmware/vmca/root.cer -CAkey /var/lib/vmware/vmca/privatekey.pem -extensions v3_req -CAcreateserial -extfile vpxd-extension.cfg

"%VMWARE_OPENSSL_BIN%" req -new -nodes -out vsphere-webclient.csr -newkey rsa:2048 -keyout vsphere-webclient.key -config vsphere-webclient.cfg

"%VMWARE_OPENSSL_BIN%" x509 -req -days 3650 -in vsphere-webclient.csr -out vsphere-webclient.crt -CA /var/lib/vmware/vmca/root.cer -CAkey /var/lib/vmware/vmca/privatekey.pem -extensions v3_req -CAcreateserial -extfile vsphere-webclient.cfg

3. Delete the old certificate entries from VECS store :

Appliance :

/usr/lib/vmware-vmafd/bin/vecs-cli entry delete --store machine --alias machine -y

/usr/lib/vmware-vmafd/bin/vecs-cli entry delete --store vpxd --alias vpxd -y

/usr/lib/vmware-vmafd/bin/vecs-cli entry delete --store vpxd-extension --alias vpxd-extension -y

/usr/lib/vmware-vmafd/bin/vecs-cli entry delete --store vsphere-webclient --alias vsphere-webclient -y

Windows :

"%VMWARE_CIS_HOME%"\vmafdd\vecs-cli entry delete --store machine --alias machine --cert machine.crt --key machine.key

"%VMWARE_CIS_HOME%"\vmafdd\vecs-cli entry delete --store vpxd --alias vpxd --cert vpxd.crt --key vpxd.key

"%VMWARE_CIS_HOME%"\vmafdd\vecs-cli entry delete --store vpxd-extension --alias vpxd-extension --cert vpxd-extension.crt --key vpxd-extension.key

"%VMWARE_CIS_HOME%"\vmafdd\vecs-cli entry delete --store vsphere-webclient --alias vsphere-webclient --cert vsphere-webclient.crt --key vsphere-webclient.key

4. Update the  new certificates in the VECS store :

Appliance :

/usr/lib/vmware-vmafd/bin/vecs-cli entry create --store machine --alias machine --cert machine.crt --key machine.key

/usr/lib/vmware-vmafd/bin/vecs-cli entry create --store vpxd --alias vpxd --cert vpxd.crt --key vpxd.key

/usr/lib/vmware-vmafd/bin/vecs-cli entry create --store vpxd-extension --alias vpxd-extension --cert vpxd-extension.crt --key vpxd-extension.key

/usr/lib/vmware-vmafd/bin/vecs-cli entry create --store vsphere-webclient --alias vsphere-webclient --cert vsphere-webclient.crt --key vsphere-webclient.key

Windows :

"%VMWARE_CIS_HOME%"\vmafdd\vecs-cli entry create --store machine --alias machine --cert machine.crt --key machine.key

"%VMWARE_CIS_HOME%"\vmafdd\vecs-cli entry create --store vpxd --alias vpxd --cert vpxd.crt --key vpxd.key

"%VMWARE_CIS_HOME%"\vmafdd\vecs-cli entry create --store vpxd-extension --alias vpxd-extension --cert vpxd-extension.crt --key vpxd-extension.key

"%VMWARE_CIS_HOME%"\vmafdd\vecs-cli entry create --store vsphere-webclient --alias vsphere-webclient --cert vsphere-webclient.crt --key vsphere-webclient.key

5. List all the solution users : 

/usr/lib/vmware-vmafd/bin/dir-cli service list --login administrator@vsphere.local --password 'YA,Bkc9ID8.)o_Sxr5t6'

1. machine-74453a6d-4f01-4191-a95c-d3a905c1a516

2. vsphere-webclient-74453a6d-4f01-4191-a95c-d3a905c1a516

3. vpxd-74453a6d-4f01-4191-a95c-d3a905c1a516

4. vpxd-extension-74453a6d-4f01-4191-a95c-d3a905c1a516 

6. Update all the solution users with the new certificate :

Appliance :

/usr/lib/vmware-vmafd/bin/dir-cli service update --name machine-74453a6d-4f01-4191-a95c-d3a905c1a516 --cert machine.crt --login administrator@vsphere.local --password 'YA,Bkc9ID8.)o_Sxr5t6'

/usr/lib/vmware-vmafd/bin/dir-cli service update --name vsphere-webclient-74453a6d-4f01-4191-a95c-d3a905c1a516 --cert vsphere-webclient.crt --login administrator@vsphere.local --password 'YA,Bkc9ID8.)o_Sxr5t6'

/usr/lib/vmware-vmafd/bin/dir-cli service update --name vpxd-74453a6d-4f01-4191-a95c-d3a905c1a516 --cert vpxd.crt --login administrator@vsphere.local --password 'YA,Bkc9ID8.)o_Sxr5t6'

/usr/lib/vmware-vmafd/bin/dir-cli service update --name vpxd-extension-74453a6d-4f01-4191-a95c-d3a905c1a516 --cert vpxd-extension.crt --login administrator@vsphere.local --password 'YA,Bkc9ID8.)o_Sxr5t6'

Windows :

"%VMWARE_CIS_HOME%"\vmafdd\dir-cli service update --name machine-74453a6d-4f01-4191-a95c-d3a905c1a516 --cert machine.crt --login administrator@vsphere.local --password 'YA,Bkc9ID8.)o_Sxr5t6'

"%VMWARE_CIS_HOME%"\vmafdd\dir-cli service update --name vsphere-webclient-74453a6d-4f01-4191-a95c-d3a905c1a516 --cert vsphere-webclient.crt --login administrator@vsphere.local --password 'YA,Bkc9ID8.)o_Sxr5t6'

"%VMWARE_CIS_HOME%"\vmafdd\dir-cli service update --name vpxd-74453a6d-4f01-4191-a95c-d3a905c1a516 --cert vpxd.crt --login administrator@vsphere.local --password 'YA,Bkc9ID8.)o_Sxr5t6'

"%VMWARE_CIS_HOME%"\vmafdd\dir-cli service update --name vpxd-extension-74453a6d-4f01-4191-a95c-d3a905c1a516 --cert vpxd-extension.crt --login administrator@vsphere.local --password 'YA,Bkc9ID8.)o_Sxr5t6'

7 . Update the auto-deploy, imagebuilder and eam service with vpxd-extension certificate.

Appliance :

python /usr/lib/vmware-vpx/scripts/updateExtensionCertInVC.py -e com.vmware.vim.eam -c /certificate/vpxd-extension.crt -k /certificate/vpxd-extension.key -s <VC_FQDN> -u <SSO_admin>

python /usr/lib/vmware-vpx/scripts/updateExtensionCertInVC.py -e com.vmware.rbd -c /certificate/vpxd-extension.crt -k /certificate/vpxd-extension.key -s <VC_FQDN> -u <SSO_admin>

python /usr/lib/vmware-vpx/scripts/updateExtensionCertInVC.py -e com.vmware.imagebuilder  -c /certificate/vpxd-extension.crt -k /certificate/vpxd-extension.key -s <VC_FQDN> -u <SSO_admin>

Windows :

"%VMWARE_PYTHON_BIN%" C:\Program Files\VMware\vCenter Server\vpxd\scripts\updateExtensionCertInVC.py -e com.vmware.vim.eam -c C:\Certificates\vpxd-extension.crt -k C:\Certificates\vpxd-extension.key -s <vcenter_FQDN> -u <SSO_admin>

"%VMWARE_PYTHON_BIN%" C:\Program Files\VMware\vCenter Server\vpxd\scripts\updateExtensionCertInVC.py -e com.vmware.rbd -c C:\Certificates\vpxd-extension.crt -k C:\Certificates\vpxd-extension.key -s <vcenter_FQDN> -u <SSO_admin>

"%VMWARE_PYTHON_BIN%" C:\Program Files\VMware\vCenter Server\vpxd\scripts\updateExtensionCertInVC.py -e com.vmware.imagebuilder -c C:\Certificates\vpxd-extension.crt -k C:\Certificates\vpxd-extension.key -s <vcenter_FQDN> -u <SSO_admin>

8. Restart all the services.

Reference: 

https://docs.vmware.com/en/VMware-vSphere/6.0/com.vmware.vsphere.security.doc/GUID-BD70615E-BCAA-4906-8E13-67D0DBF715E4.html