Replacing vROPS Certificates

Issue:  When using default certificates in vROPS  Due to security requirements it was necessary to replace the default self-signed certificates with a trusted certificate signed by an internal CA.

Unlike vCenter, vROPS uses a common name vc-ops-slice-1 followed by slice-2 for data and other node, where vCenter has SAN names in the default certificate so when you add the root of the local machine trusted store then the vCenter url shows secured.

In order to make the secure SSL we have to generate the CSR and get it signed by any third party SSL vendors or our own Microsoft CA. 

I will be generating a CSR and getting signed by internal Microsoft CA .We can refer this below article from VMware as reference. 

https://kb.vmware.com/s/article/2046591?lang=en_us

Steps: 

1.The first step is to create a new private key from taking a SSH session to master node vROPS.

openssl genrsa -out vrops.key 2048

2. Using the configuration file below make sure to provide details of master node ,data node  of FQDN /IP address and other basic information of certificates like organization name, locality etc. 

sample vrops.cfg file 

*********************************************************************

distinguished_name = req_distinguished_name

encrypt_key = no

prompt = no

string_mask = nombstr

req_extensions = v3_req

[ v3_req ]

basicConstraints = CA:FALSE

keyUsage = digitalSignature, keyEncipherment, dataEncipherment

extendedKeyUsage = serverAuth, clientAuth

subjectAltName = DNS:vrops1.org.com, IP:192.168.0.14,vrops2.org.com,192.168.0.15

[ req_distinguished_name ]

countryName = IN

stateOrProvinceName = VAR

localityName = CAL

0.organizationName = VROPS

organizationalUnitName = VROPSORG

commonName = vrops1.org.com

********************************************************************

openssl req -new -key vrops.key -out vrops.csr -config vrops.cfg

3. Get the certificate signed from your Internal CA using the vrops.csr file for instance we save the certificate obtained as  Server_cert.cer

4. The order of CA's certs in the .PEM file: Cert, Private Key, Intermediate Cert and then Root Cert.

cat server_cert.cer vrops.key cacerts.cer >final.pem.pem

-----BEGIN CERTIFICATE-----

(Your Primary SSL certificate: your_domain_name.crt)

-----END CERTIFICATE-----

-----BEGIN RSA PRIVATE KEY-----

(Your Private Key: your_domain_name.key)

-----END RSA PRIVATE KEY-----

-----BEGIN CERTIFICATE-----

(Your Intermediate certificate: name.crt)

-----END CERTIFICATE-----

-----BEGIN CERTIFICATE-----

(Your Root certificate: TrustedRoot.crt)

-----END CERTIFICATE-----

5.Access the admin page of vRops master node and click on the SSL certificate to install the pem file

6. After the installation will complete in few minutes logout from the admin and refresh the browser and access the vROPS to show as secured SSL

 

vCloud Usage Meter 4.3 DNS settings is not updated after the UM 4.3 ovf Deployment

Issue: I had multiple occasions where the UM 4.3 is deployed using ovf template and DNS settings are provided during the deployment and later checked DNS settings rollbacks to default IP address

Actions Performed :

1.Accesses the /op/vmware/share/vami/vami_config_net and updated the DNS settings but still it rollback to default  DNS IP address

2. Added the DNS domain server in /etc/resolv.conf file but after the reboot the DNS IP address is not persistent 

Cause: We found the resolv.conf is pointing to runtime symbolic which is making not persistent after reboot

Workaround 1: 

• Take an SSH/ console onto the Usage Meter appliance, and run the below commands.
• mv /etc/resolve.conf /etc/resolve.conf.bak
• vi /etc/resolve.conf
• chmod 644 resolve.conf
• Key in the DNS server info in the below format. I have attached a screenshot below.
• nameserver DNSIPAddress1
• nameserver DNSIPAddress1
• Quit out of the file, and perform a reboot.

Workaround 2:

• Navigated to the file: /etc/systemd/network/10-eth0.network 
• vi  /etc/systemd/network/10-eth0.network 
• Added the DNS entries
• Reboot the Appliance or restart the network by running systemctl restart systemd-networkd
• post which the DNS settings is persistent

                                        

Usage Meter 4.3 Fails to connect to vRNI (Network Insight) fails with API ERROR

 Usage Meter v4.3 is not able to connect to vRealize Network Insight (vRNI v5.1.0). Product status in UM = collection error Message Notification = Collection error: APIError 

                                                                                                                                                                     Resolution : In `conf` directory modify `vrnicollector_process.conf` file, add the following lines and restart the vRNI collector service. Ensure the backup the file before making the changes below.

        -httpConnectionTimeout 900

-httpReadTimeout 900

-httpWriteTimeout 900

Restart the vRNI Collector service 

cd /opt/vmware/cloudusagemetering

scripts/stop.sh vrniCol

scripts/start.sh vrniCol

Unable to add vCenter to Usage Meter 3.6 after replacing the vCenter certificate

 Issue: Unable to add the vCenter endpoint to the usage meter due to certificate error

Error: There was a problem checking the certificate vCenter fqdn:443 .Unexpected exception :java.net.UnknownHostException:fqdn of vCenter

Actions performed: 

1.Checked the root certificate on the vCenter and machine SSL ,SAN names etc

2. Running an openssl command  from Usage meter using the IP address of the vCenter returns the certificate 

3. Running and opnessl command from Usage meter using the vCenter FQDN fails to resolve and unable to fetch the vCenter certificate.

Resolution :

1.SSH into the Usage Meter appliance using root

2.Took a backup of /etc/hosts file

3.Added the IP address and the FQDN of the vCenter after which vCenter was successfully added to the Usage Meter.

Top 20 articles for vRealize Operations Manager

Handy KB articles for recent issues in vROPS

1. How to reset the root password in vRealize Operations
2. How to reset the admin password in vRealize Operations Manager
3. Adding additional storage to a node in vRealize Operations
4. vRealize Operations Manager Sizing Guidelines
5. Configure a Certificate For Use With vRealize Operations
6. /storage/log is full on vRealize Operations
7. Replace expired internal certificate in vRealize Operations Manager 6.3 and later
8. Upgrade to vRealize Operations Manager 8.0 fails due to the admin or root account password
9. Upgrade to vRealize Operations Manager 8.0 hangs on step 4 of 9
10. Shutdown and Startup sequence for a vRealize Operations Manager cluster
11. Clearing the Alerts and Alarms Tables in vRealize Operations
12. Upgrade to vRealize Operations Manager 8.x fails due to low space on /dev/sda
13. Upgrade to vRealize Operations Manager 8.0 fails due to low space on /dev/sdc
14. Continuous disk space alerts for /storage/archive Guest File System in vRealize Operations Manager
15. Rebooting nodes in vRealize Operations Manager
16. Minimum Collection User Permissions in vRealize Operations Manager 6.x and later
17. How to take a Snapshot of vRealize Operations
18. Change the IP Address of a vRealize Operations Manager Multi Node Deployment
19. vRealize Operations Data Collection
20. Reload the default certificate in vRealize Operations Manager

Using Custom Certificates in Usage Meter 4.x

As we get to move on with customers and service providers asking to implement to use a secure web browser access to Usage Meter .The old versions of UM has methods to replace the certificates.

Beginning from 3.6.1 versions there is no official or supported methods to use a custom certificates for Usage Meter and we are still allowed with default certificates.

Recently we are receiving request from customer using their own internal or custom signed certificates for Usage Meter 4.x versions. It still in the road map and i believe it will be taken care in the upcoming versions.

I have tested in my lab since the Usage Meter 4.x version uses ngnix web servers so i found a possibility to replace the certificates.

Note: Its official not supported and no supported documentation available at this moment  and you have to repeat ever time you upgrade or patch the Usage Meter appliance make sure to take a snapshot or a valid backup before performing the steps

• Please take a snapshot of the Usage Meter Appliance
• Once you have the certificate and the key file generated from your CA
• Access the Usage Meter appliance via SSH using the root account
• Take a backup of the crt and key files from the location and copy the new crt and key files to     
• ssl_certificate /etc/ssl/certs/nginx-selfsigned.crt
• ssl_certificate_key /etc/ssl/private/nginx-selfsigned.key
• Edit the configuration file 
        vi /opt/vmware/cloudusagemetering/conf/nginx.conf  to match the crt and key location


    

Once the changes have been made make sure to give the usagemeter account full permissions to the directories crt,key,configuration files as shown below

 chown -R usagemeter /opt/vmware/cloudusagemetering/conf/nginx.conf

 chown -R usagemeter /etc/ssl/private/nginx-selfsigned.key

 chown -R usagemeter /etc/ssl/certs/nginx-selfsigned.crt

Once the permission has been provided the usagemeter account should be able to access the file

using the account so login into the UM appliance via SSH using usagemeter account

  chmod 777 /etc/ssl/certs/nginx-selfsigned.crt

  chmod 777 /etc/ssl/private/nginx-selfsigned.key

  chmod 777 /opt/vmware/cloudusagemetering/conf/nginx.conf

Finally reboot the appliance and verify the certificates on the https://IPaddress:8443 browser interface