Unable to add permissions or Unable to add any identity source in vCenter 6.x

ssoAdminserver logs

[INFO ][2019-12-19T14:50:25.432Z][k4cu9c27-323-auto-91-h5:70000118] auditlogger - {"user":"Administrator@VSPHERE.LOCAL","client":"","timestamp":"12/19/2019 14:50:25 UTC","description":"Registering the Active Directory as identity source

with domain Name 'LAB.LOCAL'","eventSeverity":"INFO","type":"com.vmware.sso.IdentitySourceManagement"}

[INFO ][2019-12-19T14:50:25.432Z][k4cu9c27-323-auto-91-h5:70000118] IdentitySourceManagementServiceImpl - [User {Name: Administrator, Domain: VSPHERE.LOCAL} with role 'Administrator'] Registering the Active Directory as identity source w

ith domain Name 'LAB.LOCAL'

[INFO ][2019-12-19T14:50:25.488Z][k4cu9c27-323-auto-91-h5:70000118] PooledLdapConnectionFactory - New connection created in pool PooledLdapConnectionIdentity [tenantName=null, username=vcsalab.org@vsphere.local, authType=SRP, us

eGCPort=false, connectionString=ldap://localhost:389]

[WARN ][2019-12-19T14:50:25.551Z][k4cu9c27-323-auto-91-h5:70000118] LdapErrorChecker - Error received by LDAP client: com.vmware.identity.interop.ldap.OpenLdapClientLibrary, error code: 1

[ERROR][2019-12-19T14:50:25.556Z][k4cu9c27-323-auto-91-h5:70000118] IdentityManager - Failed to add identity provider for tenant [vsphere.local]

[ERROR][2019-12-19T14:50:25.556Z][k4cu9c27-323-auto-91-h5:70000118] ServerUtils - Exception 'com.vmware.identity.interop.ldap.OperationsErrorLdapException: Operations error LDAP error [code: 1]'

vmdird syslog

2019-12-18T10:59:17.368888+00:00 err vmdird  t@140252710672128: UpdateServerObject: InternalModifyEntry failed. Error code: 1, Error string: Schema check failed - (9612)(Objectclass (vmwDirServer) is not defined in schema)

2019-12-18T10:59:48.033055+00:00 err vmdird  t@140252710672128: UpdateServerObject: InternalModifyEntry failed. Error code: 1, Error string: Schema check failed - (9612)(Objectclass (vmwDirServer) is not defined in schema)

019-12-19T15:09:08.088685+00:00 err vmdird  t@140036292978432: CoreLogicModifyEntry failed, DN = CN=81FD31A929956E9A1AEC546701B114C6EC48E74A,CN=Certificate-Authorities,cn=Configuration,dc=vsphere,dc=local, (9612)(Schema check failed - (9612)(Objectclass (vmwCertificationAuthority) is not defined in schema))

2019-12-19T15:09:08.089737+00:00 err vmdird  t@140036292978432: VmDirSendLdapResult: Request (Modify), Error (1), Message (Schema check failed - (9612)(Objectclass (vmwCertificationAuthority) is not defined in schema)), (0) socket (127.0.0.1)

(END)

Resolution :

Appliance Based Platform Services Controller:

• Ensure to take a snapshot or Backup of the VCSA and PSC 
• Connect to the Platform Services Controller with an SSH session as root.
• Stop the Platform Services Controller services 
• Run this command to update the VMdir Schema:
• /usr/lib/vmware-vmdir/sbin/vmdird -c -u -f /usr/lib/vmware-vmdir/share/config/vmdirschema.ldif
• Start the Platform Services Controller services 
• Re-add the identity source 

Windows Based Platform Services Controller:

• Log in to the Platform Services Controller machine as an Administrator.
• Open an administrative command prompt.
• Change to the Platform Services Controller installation directory:
• cd C:\Program Files\VMware\vCenter Server\bin
• Note: This is the default installation path. If you have installed the Platform Services controller to another location, modify this command to reflect the correct install location. 
•  stop all services: 
• Run this command to update the VMdir Schema:
• C:\Program Files\VMware\vCenter Server\vmdird\vmdird.exe -c -u -f C:\ProgramData\VMware\vCenterServer\cfg\vmdird\vmdirschema.ldif
• Note: This command uses the default installation path. If you have installed the Platform Services controller to another location, modify this command to reflect the correct install location. 
• start all services and re-add the identity source 

Below is a different KB but you can use it as a reference to update the schema.ldif file 

https://kb.vmware.com/s/article/2144612      -à Deploying or Installing an additional Platform Service Controller 6.0 Update 1b fails during vmafd firstboot (2144612)

missing chunk number 0 for toast value 120710098 in pg_toast_17620; -Postgres corruption

vCenter 6.x with postgres DB corruption making vpxd to crash

vpxd logs

2019-04-20T16:26:10.995-06:00 error vpxd[11244] [Originator@6876 sub=Default] [VdbStatement] SQLError was thrown: "ODBC error: (XX000) - ERROR: missing chunk number 0 for toast value 120710098 in pg_toast_17620;

--> Error while executing the query" is returned when executing SQL statement "SELECT ID, CONFIG_MANAGER FROM VPX_HOST"

2019-04-20T16:26:10.995-06:00 error vpxd[11244] [Originator@6876 sub=DbBulkLoader] [VpxdDbBulkLoader::Load] Failed to load tableDef 19 from database: "ODBC error: (XX000) -

--> Error while executing the query" is returned when executing SQL statement "SELECT ID, CONFIG_MANAGER FROM VPX_HOST"

2019-04-20T16:26:10.995-06:00 error vpxd[11244] [Originator@6876 sub=Default] Win32 invalid_parameter: expression=(null), function=(null), file=(null), line=0

2019-04-20T16:26:11.029-06:00 info vpxd[11244] [Originator@6876 sub=Default] CoreDump: Writing minidump

2019-04-20T16:26:13.940-06:00 panic vpxd[11244] [Originator@6876 sub=Default]

-->

--> Panic: Win32 invalid_parameter error

--> Backtrace:

-->

Postgres logs

MDT 5cbb8054.1bc8 0 VCDB vc ERROR:  missing chunk number 0 for toast value 177603961 in pg_toast_17620

In most cases we end up redeploying the vCenter or restoring the vCenter from a good known backup for the error missing chunk number 0 for toast value 120710098 in pg_toast_17620;

 In my case it was the ESXi host Entity was corrupted . steps.

MDT 5cbb8054.1bc8 0 VCDB vc ERROR:  missing chunk number 0 for toast value 177603961 in pg_toast_17620

The log will indicated which toast value is corrupt

1. Login to the database.

              psql -U vc -d VCDB

2. To identify the table with the corrupt rows.

               VCDB=> select 17620::regclass;

               regclass

               ----------

              vpx_host        In this case its host The corruption is on vpx_host

               (1 row)

3. To find the row do some selects.

             VCDB=> select count(*) from vpx_host;

               count

              -------

               40

4. List down the ESXi host in the DB

              VCDB=>    select id,dns_name from vpx_host;

                 id   |           dns_name

               --------+------------------------------

              26780 | vhusinvctgvm12.managed.local

              141 | vhusinvctgvm08.managed.local

              626 | vhusinvctgvm03.managed.local

              18603 | mnusinvctgvm06.managed.local

             1188 | vhusinvctgvm04.managed.local

              536 | vhusinvctgvm02.managed.local

             2229 | vhusinvctgvm06.managed.local

            240231 | vhusinvctgvm09.managed.local

             277 | vhusinvctgvm10.managed.local

             6554 | mnusinvctgvm10.managed.local

             6833 | mnusinvctgvm12.managed.local

            242255 | vhusinvctgvm16.managed.local

             12140 | gr-vm02.kmhd.local

            148668 | vhusinvsvsvm02.managed.local

            242209 | vhusinvctgvm13.managed.local

             479 | vhusinvctgvm01.managed.local

             242269 | vhusinvctgvm17.managed.local               

            (40 rows)

5.  VCDB=> select * from vpx_host where id<242269;      ---  This is the corrupt host

                ERROR:  missing chunk number 0 for toast value 177603961 in pg_toast_17620

6. Since we have the hostd id tied to multiple entities disconnecting the host will not help us

7. We need to Manually removing an ESX\ESXi host from the vCenter Server database

PNID change issue on 6.7 Update 3

As we know PNID change is supported on 6.7 U3 onwards, however while changing the hostname from VAMI, you may encounter errors “Failed to create replication placeholder” and “Network update failed” :

To fix the issue, we need to replace the pnid_utils.py file which is at /usr/lib/applmgmt/networking/py/vmware/appliance/networking/pnid/ location with attached one, restart Appliance mgmt. service and try changing PNID again.

Quick Esxi commands

 Esxi’s command’s to make your daily troubleshooting 

- Lists all vm's running on hypervisor and provides vmid

   

    vim-cmd vmsvc/getallvms

   

List the inventory ID of the virtual machine with the command:

     vim-cmd vmsvc/getallvms |grep <vm name>

    Note: The first column of the output shows the vmid.

Check the power state of the virtual machine with the command:

      vim-cmd vmsvc/power.getstate <vmid>

Power-on the virtual machine with the command

      vim-cmd vmsvc/power.on <vmid>

Power-off the virtual machine with the command:

      vim-cmd vmsvc/power.off <vmid>

Reboots vmid referenced from getallvms command

   

      vim-cmd vmsvc/power.reboot vmid

    

Power Off (Hard)

get the world ID of the virtual machine

esxcli vm process list

TestComputer

World ID: 1625788

Process ID: 0

VMX Cartel ID: 1625786

UUID: 56 4d 9e d3 8b ce ab 59-9b 22 ac 87 40 6c 48 c3

Display Name: TestComputer

And kill them

esxcli vm process kill -t [soft,hard,force] -w

esxcli vm process kill -t hard -w 1625788

Suspend a vm

vim-cmd vmsvc/power.suspend

Resume a virtual machine

vim-cmd vmsvc/power.suspendResume

Reset a virtual machine

vim-cmd vmsvc/power.reset

Shutdown

vim-cmd vmsvc/power.shutdown

Deletes the vmdk and vmx files from disk

   

      vim-cmd vmsvc/destroy vmid

 

Puts hypervisor into maintenance mode

   

     vim-cmd hostsvc/maintenance_mode_enter

Takes hypervisor out of maintenance mode

     vim-cmd hostsvc/maintenance_mode_exit

Registers vm in hypervisor inventory

     vim-cmd solo/registervm /vmfs/vol/datastore/dir/vm.vmx

Unregisters vm with hypervisor

     vim-cmd vmsvc/unregister vmid

Starts vmware tools installation for VM

     vim-cmd vmsvc/tools.install vmid

Provides information about hypervisor networking

     vim-cmd hostsvc/net/info

Shows daemons running on hypervisor. Can also be used for configuration.

   

     chkconfig -l

Same as linux top for vmware

   

     Esxtop

List of vmkernel errors

   

     vmkerrcode -l

Lists a LOT of information about the esx host

     esxcfg-info

Lists information about NIC's. Can also be used for configuration.

     esxcfg-nics -l

Lists information about virtual switching. Can also be used for configuration.

     esxcfg-vswitch -l

Provides console screen to ssh session

     dcui

Vmware interactive shell

     vsish

Read System Event Log of server

     decodeSel /var/log/ipmi_sel.raw

     esxcfg-vmknic -l

     esxcfg-route -l (for defaultgateway:)

     esxcfg-vswitch -l (for switch)

Restart Management, HA Services

/sbin/services restart

Installing Software. List/Install/Uninstall  VIBs (vSphere Installation bundle)

List vibs

esxcli software vib list

Install a vib

esxcli software vib install -v file:/tmp/[NewVIB].vib

Uninstall a vib (determine the Name of the VIB by the list command)

esxcli software vib remove -n VIBname

Install a patch

esxcli software vib install /tmp/[patchName].zip

Network

firewall state

esxcli network firewall get

Firewall rules

esxcli network firewall ruleset list

Firewall activate a Ruleset, i.e. sshClient

esxcli network firewall ruleset set --ruleset-id=sshClient --enabled=true

Only allow an IP Range for the ssh daemon

esxcli network firewall ruleset allowedip add --ruleset-id sshServer --ip-address 192.168.254.0/24

List Kernel Network Interfaces

esxcli network ip interface list

List routing table

esxcli network ip route ipv4 list

Add a route

esxcli network ip route ipv4 add --gateway 10.1.1.254 --network 10.1.2.0/24

To make the route persistent add the command line to a ESXi startup script. In ESXi 5.1

/etc/rc.local.d/local.sh

in ESXi <=5.0

/etc/rc.local

SNMP

Set Community (Communityname: MGMCOM)

esxcli system snmp set --communities MGMCOM

Set Trap destintion (IP Address Management station: 192.168.254.100)

esxcli system snmp set --targets 192.168.254.100/MGMCOM

Send test trap

esxcli system snmp test

State

esxcli system snmp get

enable IPMI as SNMP source

esxcli system snmp set --hwsrc sensors

enable CIM as SNMP source

esxcli system snmp set --hwsrc indications

Enable snmp

esxcli system snmp set --enable true

Virtual disks

extend a virtual disk, i.e. to 40GB

vmkfstools -X 40G /vmfs/volumes/datastore1/Test/test.vmdk

Convert a Workstation based virtual Disk to a VMFS Disk for use at ESXi. Disk to convert: Computer.vmdk

Rename Disk Computer.vmdk to Computer_WS.vmdk

mv Computer.vmdk Computer_WS.vmdk

Clone the disk

vmkfstools -i Computer_WS.vmdk -d zeroedthick Computer.vmdk

Destination disk format: VMFS thin-provisioned

Cloning disk 'Computer.vmdk'...

Clone: 100% done.

Remove Workstation disk

rm Computer_WS.vmdk

Snapshots

Get VMID

vim-cmd vmsvc/getallvms

Vmid  Name File Guest OS  Version  Annotation

36 .......................................

List all snapshots for a virtual machine

vim-cmd vmsvc/snapshot.get 36

Get Snapshot:

|-ROOT

--Snapshot Name : Installation Complete

--Snapshot Id : 1

--Snapshot Desciption :

--Snapshot Created On : 3/15/2013 13:20:34

--Snapshot State : powered off

--|-CHILD

----Snapshot Name : SNAP1

----Snapshot Id : 2

----Snapshot Desciption :

----Snapshot Created On : 5/14/2013 11:46:19

----Snapshot State : powered on

Create a snapshot,  including the RAM of the Machine

vim-cmd vmsvc/snapshot.create 36 "New Snap" "Snap desc" includeMemory

Delete a snapshot, where 36 is the VMID and 2 the Snapshot ID

vim-cmd vmsvc/snapshot.remove 36 2

ESXi Advanced Kernel Settings/Parameters

Get all

esxcli system settings advanced list

Set one, i.e. disable the shell respectively ssh warnings

esxcli system settings advanced set -o /UserVars/SuppressShellWarning -i 1

ESXi Kernel modules

List loaded kernel modules

vmkload_mod  -l

Get a list of all enabled kernel modules

esxcfg-module -q

Get parameters of a kernel modul, i.e. an Emulex FC HBA module

esxcfg-module -g lpfc820

lpfc820 enabled = 1 options = 'lpfc0_lun_queue_depth=8 lpfc1_lun_queue_depth=8 lpfc2_lun_queue_depth=8 lpfc3_lun_queue_depth=8'

Set kernel module parameters, also i.e. the Emulex FC HBA module

esxcfg-module -s "lpfc0_lun_queue_depth=8 lpfc1_lun_queue_depth=8 lpfc2_lun_queue_depth=8 lpfc3_lun_queue_depth=8" lpfc820

Get info of a module with all possible parameters and a description of each

esxcfg-module -i lpfc820

Storage

Get a list of all storage devices

esxcli storage filesystem list

Rename a datastore

vim-cmd hostsvc/datastore/rename OldName NewName

Get a list of all storage path’s

esxcli storage core path list

Get storage paths for a specific drive

esxcli storage core path list -d naa.600000e00d11000000111a2400000000

Show Storage Array Type and Path selection policies of disk devices

esxcli storage nmp device list

Set Roundrobin(VMW_PSP_RR) path selection policy for a disk device, possible other policies are “Most recently used”:VMW_PSP_MRU or “Fixed”:VMW_PSP_FIXED

esxcli storage nmp device set -P VMW_PSP_RR -d naa.600000e00d11000000111a2400020000

Stat of the visorfs. Here you can find  is a detail description of the visorfs(Hypervisor Filesystem).

vdf

Usage of the visorfs

vdu

Host Services

Enable ESXi Shell

vim-cmd hostsvc/enable_esx_shell

Disable ESXi Shell

vim-cmd hostsvc/disable_esx_shell

Start ESXi Shell

vim-cmd hostsvc/start_esx_shell

Enable the ssh daemon

vim-cmd hostsvc/enable_ssh

Disable ssh daemon

vim-cmd hostsvc/disable_ssh

Start ssh daemon

vim-cmd hostsvc/start_ssh

Enter Maintainance mode

vim-cmd hostsvc/maintenance_mode_enter

vPostgres service fails to start with Fatal error : bogus postmaster.pid

I came across this issue today where vPostgres service was failing on VCSA with error Fatal Error.

# service-control --start vmware-vpostgres

Perform start operation. vmon_profile=None, svc_names=['vmware-vpostgres'], include_coreossvcs=False, include_leafossvcs=False

2019-11-06T21:38:14.283Z   Service vmware-vpostgres state STOPPED

Error executing start on service vmware-vpostgres. Details {

    "resolution": null,

    "detail": [

        {

            "args": [

                "vmware-vpostgres"

            ],

            "id": "install.ciscommon.service.failstart",

            "localized": "An error occurred while starting service 'vmware-vpostgres'",

            "translatable": "An error occurred while starting service '%(0)s'"

        }

    ],

    "componentKey": null,

root@vcenterdr [ /var/log/vmware/vpostgres ]#

/var/log/vmware/vpostgres/serverlog.stderr log content :-

~

~

Starting service process with pid: 61183.

LOG:  skipping missing configuration file "/storage/db/vpostgres/postgresql.conf.repl"

LOG:  skipping missing configuration file "/storage/db/vpostgres/postgresql.conf.repl"

2019-11-06 21:38:14.700 UTC 5dc33d46.eeff 0   FATAL:  bogus data in lock file "postmaster.pid": ""

Location of the postmaster.pid file :-

root@vcenterdr [ /var/log/vmware/vpostgres ]# ls -l /storage/db/vpostgres/postmaster.pid

-rw------- 1 vpostgres users 45 Oct 30 17:20 /storage/db/vpostgres/postmaster.pid

I tried to cat/less the postmaster.pid file and found that it had junk characters.

After moving this pid file, vPosgres service started fine and automatically created fresh pid file.

# mv /storage/db/vpostgres/postmaster.pid /storage/db/vpostgres/postmaster.pid_bkp

Note :- Please do not delete the file, and take proper snapshot/backup before making changes

Unable to run a automated Script to Backup the VCSA .Failed to run a script which makes connection via API calls

Error:  A server error occurred: 'com.vmware.vapi.std.errors.unauthenticated': Unable to authenticate user (Server error id:
'vapi.security.authentication.invalid'). Check $Error[0].Exception.ServerError for more details.

ERROR:vmware.appliance.vapi.auth:Requested SSO authentication but SSO authentication module is not available

vami.log

2019-12-02T18:48:44.336 [50279]INFO:twisted:"127.0.0.1" - - [02/Dec/2019:10:48:44 +0000] "POST /api HTTP/1.1" 200 2783 "-" "vAPI http client"
2019-12-02T18:50:35.336 [50279]ERROR:vmware.appliance.vapi.auth:Could not parse HOK Token
Traceback (most recent call last):
  File "/usr/lib/applmgmt/vapi/py/vmware/appliance/vapi/auth.py", line 183, in authenticate
    token.validate()
  File "/usr/lib/applmgmt/lib/extensions/py/vmware/appliance/extensions/authentication/authentication_sso.py", line 529, in validate
    signing_chain = self.validate_certificate()
  File "/usr/lib/applmgmt/lib/extensions/py/vmware/appliance/extensions/authentication/authentication_sso.py", line 700, in validate_certificate
    'One or more certificates cannot be verified.')AuthenticationError: One or more certificates cannot be verified.
2019-12-02T18:50:35.336 [50279]INFO:twisted:"127.0.0.1" - - [02/Dec/2019:10:50:35 +0000] "POST /api HTTP/1.1" 200 339 "-" "vAPI http client"

Note: PSC had total of 7 STS certificate chain out of which STS of 2 PSC is valid and the rest of STS was stale 

Resolution :

• Ensure to backup the PSC and VCSA 
• Verify if there are any Stale STS certificates of the old PSC are listed 
• Removed the STS certificate chains of the unused PSC 
• Restart the services and PSC and the VCSA

We can refer to the below KB if the issue is not with Certificates 

https://kb.vmware.com/s/article/67483

My certification

vCenter Convergence

vCenter 6.7 U1 Convergence steps 

vCenter with external PSC. 

Mount VCSA 6.7 ISO and copy the converge.json from the following folder to local machine. 

Edit the converge.json with the details. 

Details Below : 

ESXi Host name where the current vCenter is running. 
Root
Password

vCenter FQDN or IP. 
SSO user name
SSO Pwd. 
VCSA root appliance password. 

Domain Name. 
Username to join machine to domain. 
Password for domain user
DNS IP. 

After that open CMD as Admin and browse to D:\vcsa-converge-cli\win32 and run the below command to verify the template. 

 vcsa-util.exe converge --verify-template-only c:\Converge\converge.json

After verifying the template, run  the below command to run the convergence. 


vcsa-util.exe converge --no-ssl-certificate-verification --backup-taken --verbose c:\Converge\converge.json 
                     

Once the converge process is completed, we need to reboot the appliance. 

After reboot we can see vCenter with embedded PSC. 

Backup Failure occurring failing to Request HTTP Sessions vCenter

Issue :Commvault backup jobs were failing .Found from the logs Backup jobs are trying to establish an HTTP request to the vCenter and failing to establish the communication with the vCenter.

Log snippets from commvault


vsbkp.log from the proxy server

15:57:34 2357789 _Connect() - Exception - System.Net.WebException: The request failed with HTTP status 503: Service Unavailable.at System.Web.Services.Protocols.SoapHttpClientProtocol.ReadResponse(SoapClientMessage message, WebResponse response, Stream responseStream, Boolean asyncCall)    at System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke(String methodName, Object[] parameters)at Vim25Api.VimService.RetrieveServiceContent(ManagedObjectReference _this)    at VISDKWrapper.SvcConnection._Connect(String url, String username, String password)
 VSBkpCoordinator::OnMsgVMBackupGetNextVM
1 CVMWareInfo::Connect() - VISDKCppBridge::Connect failed [The request failed with HTTP status 503: Service Unavailable.

vpxd logs from vCenter

 error vpxd[7F99DECD9700] [Originator@6876 sub=HTTP session map] Out of HTTP sessions: Limited to 2000
 error vpxd[7F99D7468700] [Originator@6876 sub=HTTP session map] Out of HTTP sessions: Limited to 2000



Cause : This issue occurs due to active SOAP sessions exceeds 2000 on the VCSA exhausting the http sessions. By default the VCSA can handle only 2000 sessions. SOAP connections can be generated byVMware vSphere Client sessions, Third-party API connections, vCenter Server plugins

Resolution :  To work around this issue, increase <maxSessionCount> in vpxd.cfg file.

• Took a backup of the vpxd.cfg from /etc/vmware-vpx/
• Stopped the vpxd service
• Added the highlighted entries in the below order

<config>
   ...
   <vmacore>
      ...
      <soap>
      <maxSessionCount>6000</maxSessionCount>
      </soap>
      ...
   </vmacore>
   ...
</config>


• After starting the vpxd services service-control --start vmware-vpx
• Initiated the backup and it was successful

Please note the highlighted lines won’t be available to edit and we have to manually include them in the vpxd.cfg file.


Below are some of the articles for your reference

https://kb.vmware.com/kb/50114010 

https://forum.commvault.com/forums/53839/ShowThread.aspx

https://documentation.commvault.com/commvault/v11/article?p=32618.htm

VAPI Endpoint service failure

VAPI Endpoint service failure 

ERROR | state-manager1            | StsBuilder                     | Failed to acquire token for the solution user

2018-11-15T21:50:42.647Z | ERROR | state-manager1            | DefaultStateManager            | Could not initialize endpoint runtime state.

com.vmware.vapi.endpoint.config.ConfigurationException: com.vmware.vim.sso.client.exception.AuthenticationFailedException: Provided credentials are not valid.

        at com.vmware.vapi.endpoint.cis.StsBuilder.createToken(StsBuilder.java:182)

        at com.vmware.vapi.endpoint.cis.StsBuilder.rebuild(StsBuilder.java:77)

        at com.vmware.vapi.endpoint.cis.StsBuilder.buildInitial(StsBuilder.java:54)

        at com.vmware.vapi.state.impl.DefaultStateManager.build(DefaultStateManager.java:354)

        at com.vmware.vapi.state.impl.DefaultStateManager$1.doInitialConfig(DefaultStateManager.java:168)

        at com.vmware.vapi.state.impl.DefaultStateManager$1.run(DefaultStateManager.java:151)

        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

        at java.util.concurrent.FutureTask.run(FutureTask.java:266)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

Caused by: com.vmware.vim.sso.client.exception.AuthenticationFailedException: Provided credentials are not valid.

        at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor.handleFaultCondition(SecurityTokenServiceImpl.java:852)

        at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor.sendRequest(SecurityTokenServiceImpl.java:788)

        at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor.executeRoundtrip(SecurityTokenServiceImpl.java:714)

        at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl.acquireTokenByCertificate(SecurityTokenServiceImpl.java:473)

        at com.vmware.vapi.endpoint.cis.StsBuilder.createToken(StsBuilder.java:179)

Please follow the steps below

/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store vpxd --text | less

MACHINE_SSL_CERT

TRUSTED_ROOTS

TRUSTED_ROOT_CRLS

machine

vpxd

vpxd-extension

vsphere-webclient

SMS

/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store <stored name> --alias <alias name> --output /certificate/<certificate usage name>.crt

/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store machine --alias machine --output /certificate/machine.crt

/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store vpxd-extension --alias vpxd-extension --output /certificate/vpxd-extension.crt

/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store vpxd --alias vpxd --output /certificate/vpxd.crt

/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store vsphere-webclient --alias vsphere-webclient --output /certificate/vsphere-webclient.crt

dir-cli service update --name vpxd-extension-d74bb41a-f337-4667-9294-9df39322b428- --cert /cert/vpxd.crt --login administrator@vsphere.local

root@VCENTER51 [ /certificate ]# less vsphere-webclient.crt

root@VCENTER51 [ /certificate ]# /usr/lib/vmware-vmafd/bin/dir-cli service list

Enter password for administrator@vsphere.local:

1. WebClient_2016.11.23_145904

2. machine-2253fa42-d1ee-11e7-ba38-0050569c439f

3. vsphere-webclient-2253fa42-d1ee-11e7-ba38-0050569c439f

4. machine-d74bb41a-f337-4667-9294-9df39322b428

5. vsphere-webclient-d74bb41a-f337-4667-9294-9df39322b428

6. vpxd-d74bb41a-f337-4667-9294-9df39322b428

7. vpxd-extension-d74bb41a-f337-4667-9294-9df39322b428

/usr/lib/vmware-vmafd/bin/dir-cli service update --name machine-d74bb41a-f337-4667-9294-9df39322b428 --cert /certificate/machine.crt --login administrator@vsphere.local

/usr/lib/vmware-vmafd/bin/dir-cli service update --name vsphere-webclient-d74bb41a-f337-4667-9294-9df39322b428 --cert /certificate/vsphere-webclient.crt --login administrator@vsphere.local

/usr/lib/vmware-vmafd/bin/dir-cli service update --name vpxd-d74bb41a-f337-4667-9294-9df39322b428 --cert /certificate/vpxd.crt --login administrator@vsphere.local

/usr/lib/vmware-vmafd/bin/dir-cli service update --name vpxd-extension-d74bb41a-f337-4667-9294-9df39322b428 --cert /certificate/vpxd-extension.crt --login administrator@vsphere.local

Manually Replacing the solution user certificate on VCSA/PSC 6.x

Manually Replacing the solution user certificate on VCSA/PSC 6.x

Note :

-- Take a snapshot of the VC or PSC before proceeding.

-- PSC has two solution users : machine and vsphere-webclient.

-- VC has four solution users : machine, vpxd, vpxd-extension and vsphere-webclient.

-- VC with embedded PSC has four solution users : machine, vpxd, vpxd-extension and vsphere-webclient.

-- SSO admin username and password would vary depending on the configuration.

-- Windows Install directory may vary depending on your installation. 

-- Run the below command to get the machine ID of the node :

Appliance : /usr/lib/vmware-vmafd/bin/vmafd-cli get-machine-id --server-name localhost

Windows : "%VMWARE_CIS_HOME%"\vmafdd\vmafd-cli get-machine-id --server-name localhost

0. Take a backup of the old solution user certificates and its private keys.

Appliance :

/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store machine --alias machine --output /root/certificate/machine.crt

/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store vpxd --alias vpxd --output /root/certificate/vpxd.crt

/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store vpxd-extension --alias vpxd-extension --output /root/certificate/vpxd-extension.crt

/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store vsphere-webclient --alias vsphere-webclient --output /root/certificate/vsphere-webclient.crt

/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store machine --alias machine --output /root/certificate/machine.key

/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store vpxd --alias vpxd --output /root/certificate/vpxd.key

/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store vpxd-extension --alias vpxd-extension --output /root/certificate/vpxd-extension.key

/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store vsphere-webclient --alias vsphere-webclient --output /root/certificate/vsphere-webclient.key

Windows :

"%VMWARE_CIS_HOME%"\vmafdd\vecs-cli entry getcert --store machine --alias machine --output c:\certificate\machine.crt

"%VMWARE_CIS_HOME%"\vmafdd\vecs-cli entry getcert --store vpxd --alias vpxd --output c:\certificate\vpxd.crt

"%VMWARE_CIS_HOME%"\vmafdd\vecs-cli entry getcert --store vpxd-extension --alias vpxd-extension --output c:\certificate\vpxd-extension.crt

"%VMWARE_CIS_HOME%"\vmafdd\vecs-cli entry getcert --store vsphere-webclient --alias vsphere-webclient --output c:\certificate\vsphere-webclient.crt

"%VMWARE_CIS_HOME%"\vmafdd\vecs-cli entry getcert --store machine --alias machine --output c:\certificate\machine.key

"%VMWARE_CIS_HOME%"\vmafdd\vecs-cli entry getcert --store vpxd --alias vpxd --output c:\certificate\vpxd.key

"%VMWARE_CIS_HOME%"\vmafdd\vecs-cli entry getcert --store vpxd-extension --alias vpxd-extension --output c:\certificate\vpxd-extension.key

"%VMWARE_CIS_HOME%"\vmafdd\vecs-cli entry getcert --store vsphere-webclient --alias vsphere-webclient --output c:\certificate\vsphere-webclient.key

1. Create the configuration file for all the solution users :

a) machine.cfg :

[ req ]

distinguished_name = req_distinguished_name

encrypt_key = no

prompt = no

string_mask = nombstr

req_extensions = v3_req

[ v3_req ]

basicConstraints = CA:false

keyUsage = nonRepudiation, digitalSignature, keyEncipherment

subjectAltName = DNS:vcenter-nl-1.filmlogistics.local     # FQDN of VC or PSC

[ req_distinguished_name ]

countryName = NL

stateOrProvinceName = Utrecht

localityName = Zeist

0.organizationName = VMware

organizationalUnitName = mID-74453a6d-4f01-4191-a95c-d3a905c1a516     # machine ID of the VC/PSC

commonName = machine

b) vpxd.cfg :

[ req ]

distinguished_name = req_distinguished_name

encrypt_key = no

prompt = no

string_mask = nombstr

req_extensions = v3_req

[ v3_req ]

basicConstraints = CA:false

keyUsage = nonRepudiation, digitalSignature, keyEncipherment

subjectAltName = DNS:vcenter-nl-1.filmlogistics.local     # FQDN of VC or PSC

[ req_distinguished_name ]

countryName = NL

stateOrProvinceName = Utrecht

localityName = Zeist

0.organizationName = VMware

organizationalUnitName = mID-74453a6d-4f01-4191-a95c-d3a905c1a516     # machine ID of the VC/PSC

commonName = vpxd

c) vpxd-extension.cfg :

[ req ]

distinguished_name = req_distinguished_name

encrypt_key = no

prompt = no

string_mask = nombstr

req_extensions = v3_req

[ v3_req ]

basicConstraints = CA:false

keyUsage = nonRepudiation, digitalSignature, keyEncipherment

subjectAltName = DNS:vcenter-nl-1.filmlogistics.local     # FQDN of VC or PSC

[ req_distinguished_name ]

countryName = NL

stateOrProvinceName = Utrecht

localityName = Zeist

0.organizationName = VMware

organizationalUnitName = mID-74453a6d-4f01-4191-a95c-d3a905c1a516     # machine ID of the VC/PSC

commonName = vpxd-extension

d) vsphere-webclient.cfg :

[ req ]

distinguished_name = req_distinguished_name

encrypt_key = no

prompt = no

string_mask = nombstr

req_extensions = v3_req

[ v3_req ]

basicConstraints = CA:false

keyUsage = nonRepudiation, digitalSignature, keyEncipherment

subjectAltName = DNS:vcenter-nl-1.filmlogistics.local     # FQDN of VC or PSC

[ req_distinguished_name ]

countryName = NL

stateOrProvinceName = Utrecht

localityName = Zeist

0.organizationName = VMware

organizationalUnitName = mID-74453a6d-4f01-4191-a95c-d3a905c1a516     # machine ID of the VC/PSC

commonName = vsphere-webclient

2. Create the certificate signing request and generate a new certificate for all the solution certs :

Appliance :

openssl req -new -nodes -out machine.csr -newkey rsa:2048 -keyout machine.key -config machine.cfg

openssl x509 -req -days 3650 -in machine.csr -out machine.crt -CA /var/lib/vmware/vmca/root.cer -CAkey /var/lib/vmware/vmca/privatekey.pem -extensions v3_req -CAcreateserial -extfile machine.cfg

openssl req -new -nodes -out vpxd.csr -newkey rsa:2048 -keyout vpxd.key -config vpxd.cfg

openssl x509 -req -days 3650 -in vpxd.csr -out vpxd.crt -CA /var/lib/vmware/vmca/root.cer -CAkey /var/lib/vmware/vmca/privatekey.pem -extensions v3_req -CAcreateserial -extfile vpxd.cfg

openssl req -new -nodes -out vpxd-extension.csr -newkey rsa:2048 -keyout vpxd-extension.key -config vpxd-extension.cfg

openssl x509 -req -days 3650 -in vpxd-extension.csr -out vpxd-extension.crt -CA /var/lib/vmware/vmca/root.cer -CAkey /var/lib/vmware/vmca/privatekey.pem -extensions v3_req -CAcreateserial -extfile vpxd-extension.cfg

openssl req -new -nodes -out vsphere-webclient.csr -newkey rsa:2048 -keyout vsphere-webclient.key -config vsphere-webclient.cfg

openssl x509 -req -days 3650 -in vsphere-webclient.csr -out vsphere-webclient.crt -CA /var/lib/vmware/vmca/root.cer -CAkey /var/lib/vmware/vmca/privatekey.pem -extensions v3_req -CAcreateserial -extfile vsphere-webclient.cfg

Windows :

"%VMWARE_OPENSSL_BIN%" req -new -nodes -out machine.csr -newkey rsa:2048 -keyout machine.key -config machine.cfg

"%VMWARE_OPENSSL_BIN%" x509 -req -days 3650 -in machine.csr -out machine.crt -CA /var/lib/vmware/vmca/root.cer -CAkey /var/lib/vmware/vmca/privatekey.pem -extensions v3_req -CAcreateserial -extfile machine.cfg

"%VMWARE_OPENSSL_BIN%" req -new -nodes -out vpxd.csr -newkey rsa:2048 -keyout vpxd.key -config vpxd.cfg

"%VMWARE_OPENSSL_BIN%" x509 -req -days 3650 -in vpxd.csr -out vpxd.crt -CA /var/lib/vmware/vmca/root.cer -CAkey /var/lib/vmware/vmca/privatekey.pem -extensions v3_req -CAcreateserial -extfile vpxd.cfg

"%VMWARE_OPENSSL_BIN%" req -new -nodes -out vpxd-extension.csr -newkey rsa:2048 -keyout vpxd-extension.key -config vpxd-extension.cfg

"%VMWARE_OPENSSL_BIN%" x509 -req -days 3650 -in vpxd-extension.csr -out vpxd-extension.crt -CA /var/lib/vmware/vmca/root.cer -CAkey /var/lib/vmware/vmca/privatekey.pem -extensions v3_req -CAcreateserial -extfile vpxd-extension.cfg

"%VMWARE_OPENSSL_BIN%" req -new -nodes -out vsphere-webclient.csr -newkey rsa:2048 -keyout vsphere-webclient.key -config vsphere-webclient.cfg

"%VMWARE_OPENSSL_BIN%" x509 -req -days 3650 -in vsphere-webclient.csr -out vsphere-webclient.crt -CA /var/lib/vmware/vmca/root.cer -CAkey /var/lib/vmware/vmca/privatekey.pem -extensions v3_req -CAcreateserial -extfile vsphere-webclient.cfg

3. Delete the old certificate entries from VECS store :

Appliance :

/usr/lib/vmware-vmafd/bin/vecs-cli entry delete --store machine --alias machine -y

/usr/lib/vmware-vmafd/bin/vecs-cli entry delete --store vpxd --alias vpxd -y

/usr/lib/vmware-vmafd/bin/vecs-cli entry delete --store vpxd-extension --alias vpxd-extension -y

/usr/lib/vmware-vmafd/bin/vecs-cli entry delete --store vsphere-webclient --alias vsphere-webclient -y

Windows :

"%VMWARE_CIS_HOME%"\vmafdd\vecs-cli entry delete --store machine --alias machine --cert machine.crt --key machine.key

"%VMWARE_CIS_HOME%"\vmafdd\vecs-cli entry delete --store vpxd --alias vpxd --cert vpxd.crt --key vpxd.key

"%VMWARE_CIS_HOME%"\vmafdd\vecs-cli entry delete --store vpxd-extension --alias vpxd-extension --cert vpxd-extension.crt --key vpxd-extension.key

"%VMWARE_CIS_HOME%"\vmafdd\vecs-cli entry delete --store vsphere-webclient --alias vsphere-webclient --cert vsphere-webclient.crt --key vsphere-webclient.key

4. Update the  new certificates in the VECS store :

Appliance :

/usr/lib/vmware-vmafd/bin/vecs-cli entry create --store machine --alias machine --cert machine.crt --key machine.key

/usr/lib/vmware-vmafd/bin/vecs-cli entry create --store vpxd --alias vpxd --cert vpxd.crt --key vpxd.key

/usr/lib/vmware-vmafd/bin/vecs-cli entry create --store vpxd-extension --alias vpxd-extension --cert vpxd-extension.crt --key vpxd-extension.key

/usr/lib/vmware-vmafd/bin/vecs-cli entry create --store vsphere-webclient --alias vsphere-webclient --cert vsphere-webclient.crt --key vsphere-webclient.key

Windows :

"%VMWARE_CIS_HOME%"\vmafdd\vecs-cli entry create --store machine --alias machine --cert machine.crt --key machine.key

"%VMWARE_CIS_HOME%"\vmafdd\vecs-cli entry create --store vpxd --alias vpxd --cert vpxd.crt --key vpxd.key

"%VMWARE_CIS_HOME%"\vmafdd\vecs-cli entry create --store vpxd-extension --alias vpxd-extension --cert vpxd-extension.crt --key vpxd-extension.key

"%VMWARE_CIS_HOME%"\vmafdd\vecs-cli entry create --store vsphere-webclient --alias vsphere-webclient --cert vsphere-webclient.crt --key vsphere-webclient.key

5. List all the solution users : 

/usr/lib/vmware-vmafd/bin/dir-cli service list --login administrator@vsphere.local --password 'YA,Bkc9ID8.)o_Sxr5t6'

1. machine-74453a6d-4f01-4191-a95c-d3a905c1a516

2. vsphere-webclient-74453a6d-4f01-4191-a95c-d3a905c1a516

3. vpxd-74453a6d-4f01-4191-a95c-d3a905c1a516

4. vpxd-extension-74453a6d-4f01-4191-a95c-d3a905c1a516 

6. Update all the solution users with the new certificate :

Appliance :

/usr/lib/vmware-vmafd/bin/dir-cli service update --name machine-74453a6d-4f01-4191-a95c-d3a905c1a516 --cert machine.crt --login administrator@vsphere.local --password 'YA,Bkc9ID8.)o_Sxr5t6'

/usr/lib/vmware-vmafd/bin/dir-cli service update --name vsphere-webclient-74453a6d-4f01-4191-a95c-d3a905c1a516 --cert vsphere-webclient.crt --login administrator@vsphere.local --password 'YA,Bkc9ID8.)o_Sxr5t6'

/usr/lib/vmware-vmafd/bin/dir-cli service update --name vpxd-74453a6d-4f01-4191-a95c-d3a905c1a516 --cert vpxd.crt --login administrator@vsphere.local --password 'YA,Bkc9ID8.)o_Sxr5t6'

/usr/lib/vmware-vmafd/bin/dir-cli service update --name vpxd-extension-74453a6d-4f01-4191-a95c-d3a905c1a516 --cert vpxd-extension.crt --login administrator@vsphere.local --password 'YA,Bkc9ID8.)o_Sxr5t6'

Windows :

"%VMWARE_CIS_HOME%"\vmafdd\dir-cli service update --name machine-74453a6d-4f01-4191-a95c-d3a905c1a516 --cert machine.crt --login administrator@vsphere.local --password 'YA,Bkc9ID8.)o_Sxr5t6'

"%VMWARE_CIS_HOME%"\vmafdd\dir-cli service update --name vsphere-webclient-74453a6d-4f01-4191-a95c-d3a905c1a516 --cert vsphere-webclient.crt --login administrator@vsphere.local --password 'YA,Bkc9ID8.)o_Sxr5t6'

"%VMWARE_CIS_HOME%"\vmafdd\dir-cli service update --name vpxd-74453a6d-4f01-4191-a95c-d3a905c1a516 --cert vpxd.crt --login administrator@vsphere.local --password 'YA,Bkc9ID8.)o_Sxr5t6'

"%VMWARE_CIS_HOME%"\vmafdd\dir-cli service update --name vpxd-extension-74453a6d-4f01-4191-a95c-d3a905c1a516 --cert vpxd-extension.crt --login administrator@vsphere.local --password 'YA,Bkc9ID8.)o_Sxr5t6'

7 . Update the auto-deploy, imagebuilder and eam service with vpxd-extension certificate.

Appliance :

python /usr/lib/vmware-vpx/scripts/updateExtensionCertInVC.py -e com.vmware.vim.eam -c /certificate/vpxd-extension.crt -k /certificate/vpxd-extension.key -s <VC_FQDN> -u <SSO_admin>

python /usr/lib/vmware-vpx/scripts/updateExtensionCertInVC.py -e com.vmware.rbd -c /certificate/vpxd-extension.crt -k /certificate/vpxd-extension.key -s <VC_FQDN> -u <SSO_admin>

python /usr/lib/vmware-vpx/scripts/updateExtensionCertInVC.py -e com.vmware.imagebuilder  -c /certificate/vpxd-extension.crt -k /certificate/vpxd-extension.key -s <VC_FQDN> -u <SSO_admin>

Windows :

"%VMWARE_PYTHON_BIN%" C:\Program Files\VMware\vCenter Server\vpxd\scripts\updateExtensionCertInVC.py -e com.vmware.vim.eam -c C:\Certificates\vpxd-extension.crt -k C:\Certificates\vpxd-extension.key -s <vcenter_FQDN> -u <SSO_admin>

"%VMWARE_PYTHON_BIN%" C:\Program Files\VMware\vCenter Server\vpxd\scripts\updateExtensionCertInVC.py -e com.vmware.rbd -c C:\Certificates\vpxd-extension.crt -k C:\Certificates\vpxd-extension.key -s <vcenter_FQDN> -u <SSO_admin>

"%VMWARE_PYTHON_BIN%" C:\Program Files\VMware\vCenter Server\vpxd\scripts\updateExtensionCertInVC.py -e com.vmware.imagebuilder -c C:\Certificates\vpxd-extension.crt -k C:\Certificates\vpxd-extension.key -s <vcenter_FQDN> -u <SSO_admin>

8. Restart all the services.

Reference: 

https://docs.vmware.com/en/VMware-vSphere/6.0/com.vmware.vsphere.security.doc/GUID-BD70615E-BCAA-4906-8E13-67D0DBF715E4.html