Mike's Virtualization Blog

My chance to serve!

For as long as I can remember I was what you could call a non-believer or agnostic. As some of you know that changed for me around 6 months ago. It’s been a great change in my life & one I’m very excited & passionate about.

I started attending church at Faithbridge & the people there have been nothing short of awesome! I’ve never felt more welcomed in a place as long as I can remember. One individual in particular has become a mentor and even better someone I can call a friend. So why am I telling my social media followers, colleagues, and friends this?

The truth is I need some help to serve and help some children in Zambia in July. The person I mentioned above, Jason, has asked me to join him & a group of men on a mission trip. I am thrilled to be a part of this particular journey because it allows me to spend time with and share God’s love with the kids there, and help construct an aquaponics system for the orphanage we’ll be serving that will provide an ongoing supply of fish and vegetables for the children and missionaries. More information on Project Samuel is available here.

It would mean a lot to me & the children if you could partner with me on this trip. You can do this in a couple of ways. The first way is through prayer. If you would pray for me and my team, leading up to, and on our journey to Zambia this summer I would be very grateful. Prayer is powerful. It changes things. I would invite you to pray for me personally; for safety and the work God is doing in me to share His love, for God’s preparation of our worksite and the hearts we might affect while there, and for God’s financial provision for us as we raise the money to make this journey possible.

If you can help financially, you can donate directly to me or through our church Faithbridge. If you give directly to the church you will receive documentation at the end of the year documenting your charitable donation. Any financial support is needed by the end of June. I am extremely thankful for any support you can provide, whether it is through prayer or financially.

If you would like to donate directly to me you can do so via PayPal below.




If you would like to donate via Faithbridge & receive documentation of a charitable donation: Faithbridge.org & choose Zambia, then choose my name from the list. This will require you to create an account so that the documentation can be sent to you at the end of the year.

I’ll post updates as the date gets closer & will share the experience & pictures with you all. Thanks for your support!

Mike

Making a Move

For the past 18 months I’ve been a Senior Consultant with the Cloud Infrastructure Management Professional Services Organization here inside of VMware. It’s been a great year and half and I’ve grown both professionally and personally by having the opportunity to work with some great customers, some who pushed me and some who made me pull some hair out.

Back at the end of July I interviewed for a Technical Marketing position, I flew out to Palo Alto for 5 interviews packed into 4.5 hours, it was a great experience and everyone I interviewed with from product managers, the manager of the team, up to the VP of Technical Marketing was awesome. I felt I hit it off with everyone and that the interviews went really well. A few days later I received a call from the recruiter handling the position and was offered a transfer onto the team.

So as of 9/1 I moved onto the Technical Marketing team as a Senior Technical Marketing Manager – vCloud Suite. I’m extremely excited to jump in and get to work in my new role within VMware. While at VMworld last week I got to meet and hang out with the rest of the team and it was really good, I think we have a great group all of which seem like real team players which made me even more excited to start working with them.

The remainder of this week I’ll be transitioning the current Professional Services engagement I’ve been on since March to another consultant, once complete I’ll move full time into my new role.

I’m excited for what the future holds, watch this space and the vSphere Blog for a lot more blogging from me.

vCenter 5.1, Multisite SSO, Linked Mode, Custom SSL certificates, protected by vCenter HeartBeat. Part 3

I’ve been a bit distracted lately and wasn’t able to get this written up as soon as I would have liked, but in any case here is the final part in my three part series of setting vSphere 5.1 with vCenter Heartbeat and custom SSL certificates. Part one can be found here and part two here.

The most important part of this is to remember to install all the components and then go back and replace the certificates. If you install a component, say SSO, then replace its certificate then install another component, like vCenter, the Inventory Service or the Web Client, the install will hang when they try to register with the SSO server.

Like I mentioned in part 2 the KB Article on replacing these certificates is pretty spot on, but is a little cumbersome in areas and those are the areas I’m documenting here, otherewise I’ll just link to the corresponding KB.

So getting to it. We should have at least SSO, the vCenter Inventory Service, vCenter Server, and the Web Client installed.

We need to have our Root and Intermediate certificates for this process. More then likely you have these on your Windows server or workstation already, if not you can get them from your Root CA server. Assuming you do have these already installed:

  • Open mmc.exe
  • Add the certificates snap-in
  • Choose computer account when prompted
  • Navigate to Trusted Root Certificate Authorities and open the Certificates folder under it
  • Find your Root and any Intermediate certificates, one by one right click and choose Export under All Tasks
  • Choose Base-64 encoded X.509 (.cer)
  • Save each one, if you only have a Root with no Intermediates save it as Root64.cer
  • If you have Intermediates, once all are saved you need to combine them into one file, this can be done from the command line by running copy file1.cer+file2.cer Root64.cer
  • Copy the Root64.cer file to the certs folder on the machine you’re going to use openssl on

I would recommend using a Mac or a linux box for this, it is possible to do with Windows but the tools are built into to OS X and Linux and just makes the process a little easier. We’ll start by following this KB on creating some configuration files for each component. I create a certs directory in my OS X home folder and then inside of it create a sub-folder for each service (SSO, Inventory, vCenter, etc.). In each of those folders create the cfg file for that service, all cfg files are the same except for the servers name and the organitionalunitname, the organizationalunitname field specifies the service the cfg file is for, use the following values:

  • SSO = vCenterSSO
  • Inventory Service = vCenterInventoryService
  • vCenter = vCenterServer
  • Web Client = vCenterWebClient
  • Log Browser = vCenterLogBrowser
  • Update Manager = VMwareUpdateManager

Copy and paste the following to make your cfg files, replace the italicized text with the correct values for your installation.

[ req ]
 default_bits = 2048
 default_keyfile = rui.key
 distinguished_name = req_distinguished_name
 encrypt_key = no
 prompt = no
 string_mask = nombstr
 req_extensions = v3_req

[ v3_req ]
 basicConstraints = CA:FALSE
 keyUsage = digitalSignature, keyEncipherment, dataEncipherment
 extendedKeyUsage = serverAuth, clientAuth
 subjectAltName = DNS: ServerShortName, IP: ServerIPAddress, DNS: server.domain.com

[ req_distinguished_name ]
 countryName = Country
 stateOrProvinceName = State
 localityName = City
 0.organizationName = Company Name
 organizationalUnitName = Service
 commonName = server.domain.com

Now that we have all of our request templates (cfg files) we need to generate our Certificate Signing Request or CSR. Run the following two commands against each template.

openssl req -new -nodes -out ~/certs/serviceDirectory/rui.csr -keyout ~/certs/serviceDirectory/rui-orig.key -config ~/certs/serviceDirectory/service.cfg

openssl rsa -in ~/certs/serviceDirectory/rui-orig.key -out ~/certs/serviceDirectory/rui.key

In my case the commands for SSO would look like this:

openssl req -new -nodes -out ~/certs/sso/rui.csr -keyout ~/certs/sso/rui-orig.key -config ~/certs/sso/sso.cfg 

openssl rsa -in ~/certs/sso/rui-orig.key -out ~/certs/sso/rui.key 

In the sso folder you’ll see the files rui-orig.key, rui.csr, rui.key, sso.cfg. You’ll take the rui.csr file and use it to get your certificate.

When downloading your certificate from your CA you’ll want to download it in Base-64 format, save it to the service (sso, vCenter, etc..) folder as rui.crt.

Next we need to create a PKCS #12 file for each service, run the following command to generate it:

openssl pkcs12 -export -in ~/certs/serviceDirectory/rui.crt -inkey ~/certs/serviceDirectory/rui.key -certfile ~/certs/Root64.cer -name "rui" -passout pass:testpassword -out ~/certs/serviceDirectory/rui.pfx

In my case the command for SSO would look like this:

openssl pkcs12 -export -in ~/certs/sso/rui.crt -inkey ~/certs/sso/rui.key -certfile ~/certs/Root64.cer -name "rui" -passout pass:testpassword -out ~/certs/sso/rui.pfx 

It’s very important that you do not change the password, it must remain testpassword.

Lastly we need to create the Java Key Store for SSO, run the following commands to generate it:

keytool -v -importkeystore -srckeystore ~/certs/sso/rui.pfx -srcstoretype pkcs12 -srcstorepass testpassword -srcalias rui -destkeystore ~/certs/sso/root-trust.jks -deststoretype JKS -deststorepass testpassword -destkeypass testpassword

keytool -v -importcert -keystore ~/certs/sso/root-trust.jks -deststoretype JKS -storepass testpassword -keypass testpassword -file ~/certs/Root64.cer -alias root-ca

When you have run those two commands run the following to verify all certificates are in the store:

keytool -list -v -keystore ~/certs/sso/root-trust.jks

We now have all the certificates generated and are ready to replace the self signed certificates with our CA generated ones. Copy the folder under our certs folder to the corresponding server, for example the sso folder is copied to the SSO server.

In each of the following KB’s it’s assumed that you don’t have the Root and any Intermediate certificates installed, in most organizations this isn’t true and you can skip the steps about importing Root64.cer into your Trusted Root Certificate Authorities keystore, though verify your servers do have these certificates.

Replace the certificate for the SSO by following the steps in KB 2035011, I’ve found that since our Root64.cer already contains our Intermediate certificates steps 24-28 aren’t required. Also since we haven’t done the vSphere Web Client yet you can’t do step 29-37, instead run the cli command listed under step 37.

Next do the vCenter Inventory service by following the steps in KB 2035009.

Next do vCenter by following the steps in KB 2035005.

Next do the vSphere Web Client and Log Browser by following the steps in KB 2035010.

Lastly do vCenter Update Manager by following the steps in KB 2037581.

Since we used vCenter Heartbeat in our configuration if you want to change the certificate for it you can do so by following the steps in KB 2013041. The one thing with this is it states if you don intend to change the password you need to edit Server.xml. In vCenter HeartBeat 6.5 there is a randomly generated password that is in Server.xml, so you’ll need to open that file and note the password and use it in place of password.

And last but not least if you are planning on using SRM in this environment you must specify a CA generated certificate for it during install or your sites will not pair. KB 1008390 explains the requirements for the certificate but doesn’t give a good step by step, luckily this post in the VMware Communities sums it up pretty well.

And that sums up our install of vSphere 5.1 using vCenter Heartbeat and custom SSL certificates.

Script to use with SRM to clone a Domain Controller

Every time I go into a Site Recovery Manager engagement and we start talking about how the test functionality works I always get the question of how to make a domain controller available to the test network. I always say I have a script for that, so I thought I’d share it with anyone who might need it.

The script below is pretty simple, it just clones an existing domain controller, changes its network connection to the test network’s portgroup and then boots it up.

#Add PowerCLI Snapin
Add-Pssnapin VMware.VimAutomation.Core

#Connect to vCenter
connect-viserver <vCenter Name>

#Clone DC
new-vm -name <NewVMName> -datastore <DataStoreName> -VM <SourceVM> -ResourcePool <ClusterName>

#Change Portgroup
get-vm <NewVMName> | get-NetworkAdapter | Set-NetworkAdapter -NetworkName <IsolatedPortGroupName> -Confirm:$false

#Sleep for 1 Minute
start-sleep -s 60

#Power on DC
start-vm -VM <NewVMName>

<vCenter Name> is the name of the vCenter to connect to. If having SRM run this script make sure the service account SRM is running under has the required vCenter permissions.

<NewVMName> is the name you want to use for the cloned DC.

<DataStoreName> is the name of the datastore you want to store the clone in.

<SourceVM> is the DC you want to clone.

<ClusterName> is the cluster or resource pool you want to cloned DC to run in.

<IsolatedPortGroupName> is the name of the portgroup you want the cloned DC to use.

And that’s it, after running or having SRM run this short script you’ll have an isolated DC from your DR site connected to your isolated SRM test network. Just be sure and delete this DC when you’re done with your tests.

vCenter 4.1 to 5.0 Update 2 database upgrade failure

I’ve been working with a pretty big client the last couple weeks planning and upgrading around 20 vCenter 4.1 servers to 5.0 Update 2. This client runs all their databases on Oracle, currently version 10.2.0.4. I checked out the Interoperability Matrixes and found that 10.2.0.4 is supported for vCenter 5.0 Update 2. Matrix Pic

This client also uses the Oracle 11g client. I remember reading a Gotcha Chris Colotti blogged about so we obtained the client patch.

Now all should be good, but it’s not, upgrading the database failed. Looking through the VCDBUpgradeDatabase and the vpxd logs we’re seeing the same errors as KB 2010529 says were fixed in 5.0 Update 1, we’re installing Update 2 and hitting the bug. I made a comment on twitter last week about upgrading a large environment and dbuhelper.exe taking 22GB of RAM, turns out that’s not a good thing!

What’s happening is that when it crashes dbuhelper is updating all the url records in the database, a table at a time naturally, it does this by trying to load them all into memory in a contiguous block, when it can’t it crashes and the database upgrade fails.

The fix is quite easy but one that’s not documented anywhere I could find, so hopefully this helps someone out. First restore your 4.1 database from the backup you made before you started the upgrade then open up your vpxd.cfg file, on Windows 2008 this is stored in c:\programdata\VMware\VMware VirtualCenter find the <vpxd> section and paste this in somewhere before the closing </vpxd>:

<upgradedb>
     <urlBatchSize>1200</urlBatchSize>
</upgradedb>

This limits dbuhelper to only process 1200 records at a time which in turn keeps its memory usage low. Running this I never saw the RAM get above 25 MB where before I was seeing it range from 15-22 GB and more importantly the database was successfully upgraded.

vCenter 5.1, Multisite SSO, Linked Mode, Custom SSL certificates, protected by vCenter HeartBeat. Part 2

In my previous post I talked about the requirements this customer had and the high level architecture and products used to fulfill their requirements. In this post I’ll walk through the installation of each component and protecting it with vCenter Heartbeat.

In each site we’ll need a total of 15 public IP’s and 10 channel IP’s, by public IP I’m referring to an IP address that is routable on your network that other machines can access the services by and by channel IP I’m referring to a non-routable VLAN like you setup for vMotion but this one is dedicated to vCenter Heartbeat channel (or heartbeat) traffic.

Create 5 Windows 2008 R2 VM’s in the primary site. For simplicity I’m going to refer to these VM’s as sql1, vcenter1, sso1, vcweb1, and vum1. Each of these servers will need 2 NIC’s, one on the public network and one on the channel network. Our clones (don’t create these yet!) for vCenter Heartbeat will be sql2, vcenter2, sso2, vcweb2, and vum2. And lastly our shared names will be sql, vcenter, sso, vcweb, and vum.

Starting with sql1 assign the IP to be used for the shared name and the IP to be used for management to the public NIC and disable DNS registration.
On the channel NIC assign the channel IP and disable DNS registration and netbios.
Rename the computer account to the shared name, join it to AD if it’s not already and reboot.
Delete any dynamic DNS records for this machine from DNS and add static records for the shared name and both sql servers management hostnames.

Repeat this process for sso1, vcenter1, vcweb1 and vum1.
Now we should have 5 VM’s with the windows hostnames of sql, vcenter, sso, vcweb, and vum.

For anyone who hasn’t used vCenter Heartbeat before the above may be a little confusing so I’ll give a brief explanation as to what we’re accomplishing here, for full details read the vCenter Heartbeat documentation. vCenter Heartbeat works by synchronizing the file system and registry between 2 servers, these can be a pair of VM’s (this is the method used here), a pair of physical servers or one virtual and one physical server and using a network filter driver to hide the shared IP address on the server who is not running the shared service.
Each of the two servers is accessible on the network by its management IP and windows hostname, in vCenter Heartbeat this is referred to as non-identical mode, which allows for maintenance tasks, such as patching, to be done on the secondary node. The shared service can be moved between servers using the vCenter Heartbeat management console. When this happens manually the services are shutdown on the primary server, the shared IP is hidden from the network using the filter driver on the primary server then the shared IP is exposed on the secondary server by the filter driver and the services are started on the secondary server. The same happens automatically if the server running the shared service fails, except in a situation like a blue screen where it’s not possible to shutdown the services.

Since we can’t do anything without SQL we’ll start there. If you haven’t already configure the VM with the number vCPU’s and memory as you would for a production SQL server.
Install SQL as you normally would , use an AD account with local admin rights on the server as the service account for the database service. Set the database service and system attendant services to auto when prompted in the install.

Shut Down the server and clone it to sql2 with no customizations, if possible place it in a different cluster and on different storage controllers or at least a different datastore.
Disconnect both NICs on sql2.
Boot both servers up.
On sql2 change the Public IP settings, leave the shared IP but change the management IP to the correct IP for this server.
Change the channel IP to the correct IP for this server.
Connect the Private NIC, leave the Public NIC disconnected.

Rename the adapters in windows on both servers, name the Public adapters Public and Channel adapters Channel on both servers.

Start the HeartBeat install on SQL1:

  • Select Primary
  • Select Second Server is Virtual
  • Select LAN
  • Select your channel adapter, Click the Add button and select your IP. Click the Add button below and enter the channel IP of the 2nd server.
  • Choose your Public adapter, Click Add select the Shared IP and leave the default that both servers will use the same IP. Click Add choose your management IP, below click add and choose the 2nd servers management IP, click NO to the warning that the 2nd server can’t be reached this is normal since it’s adapter isn’t connected to the network.
  • Ensure SQL is selected to protect and leave the credentials for the vCenter plug-in blank.
  • Enter sql1 and sql2 when prompted for names to rename the computer accounts to.
  • For the location to copy the configuration to create a HB folder on C:\ and then enter \\localhost\c$\HB.
  • Finish the wizard by click Next as it completes each part of the install, reboot when prompted.

Once the first server is up move over to the 2nd server. You do not have Public network access, so copy the heartbeat install file by accessing the 1st servers private network i.e. \\Server1PrivateIP\c$.
Run the heartbeat install on SQL2:

  • Select Secondary, when prompted for the directory that contains the configuration enter \\Server1PrivateIP\c$\HB
  • Select your channel adapter
  • Connect your Public adapter to the network before continuing
  • Choose your public adapter and next through the rest of the install
  • Reboot when prompted

In Active Directory, find the service account you created to run SQL, go to properties and security and click advanced, click add, enter the same service account and then click allow under Write All Properties. Repeat this for each on SQL server computer objects.

Once both servers are up, open up the manage server heartbeat application and navigate to Applications and Tasks. Click the Users button and enter your SQL Server service account information, now edit the two NFsetspn tasks, change the user account from local system to the account you just added. On the server currently running the SQL Services (should be the primary server) right click the NFsetspn task and choose run now and ensure it completes with an exit code of 0.

Move the SQL resources to the 2nd node and verify everything is working and then move them back to the primary.

Next up is SSO.

Using the provided DB scripts create your SQL SSO database and the required _dba and _user accounts, I change the DB and user accounts to be SSO and SSO_ instead of the default RSA.

Ensure you’ve set the shared IP and management IP on the Public NIC and the Channel IP on its NIC and that you have DNS records created for the shared name and both SSO servers.
If not done, rename the computer account to the shared name to be used by SSO. Create a service account in AD to run the SSO service and grant it local admin rights to the SSO server. Install SSO, choose to create a primary node for a new installation, again choose to create the primary node (do not pick basic).

Shut Down the SSO server and clone it to sso2 with no customizations, if possible place it in a different cluster and on different storage controllers or at least a different datastore.
Disconnect the both NICs on sso2.
Boot both servers up.
On sso2 change the Public IP settings, leave the shared IP but change the management IP to the correct IP for this server.
Change the Channel IP to the correct IP for this server.
Connect the Channel NIC, leave the Public NIC disconnected.

Rename all adapters, name the Pulbic adapters Public and Channel adapter Channel on both servers.

Start the HeartBeat install on SSO1:

  • Select Primary
  • Select Second Server is Virtual
  • Select LAN
  • Select your channel adapter, Click the Add button and select your IP. Click the Add button below and enter the channel IP of the 2nd server.
  • Choose your Public adapter, Click Add select the Shared IP and leave the default that both servers will use the same IP. Click Add choose your management IP, below click add and choose the 2nd servers management IP, click NO to the warning that the 2nd server can’t be reached this is normal since it’s adapter isn’t connected to the network.
  • Ensure vCenter Services is selected to protect and leave the credentials for the vCenter plug-in blank.
  • Enter sso1 and sso2 when prompted for names to rename the computer accounts to.
  • For the location to copy the configuration to create a HB folder on C:\ and then enter \\localhost\c$\HB.
  • Finish the wizard by click Next as it completes each part of the install, reboot when prompted.

Once the first server is up move over to the 2nd server. You do not have Public network access, so copy the heartbeat install file by accessing the 1st servers private network i.e. \\Server1PrivateIP\c$.
Run the heartbeat install on SSO2:

  • Select Secondary, when prompted for the directory that contains the configuration enter \\Server1PrivateIP\c$\HB
  • Select your channel adapter
  • Connect your Public adapter to the network before continuing
  • Choose your public adapter and next through the rest of the install
  • Reboot when prompted

Attempt to move SSO to the 2nd node and verify everything is working (at this point you can only look in the logs as there is nothing using SSO) and then move them back to the primary.

Now that we have SSO installed, you may think about replacing the self-signed SSL certs with ones issued from a CA. Don’t do it! If you do when you go to install the Inventory Service, vCenter and/or the Web Client the installs will hang while it’s trying to pull the SSL certificate down from the SSO server. In the primary site I did this and had to kill openssl and/or msiexec processes until the installer moved on. It’s much easier to install all the components then come back and replace their SSL certs, which I’ll cover in my next post.

With SSO installed and still using its self-signed certificate move onto the vCenter Inventory Service and vCenter Server installs.

Ensure you’ve set the shared IP and management IP on the Public NIC and the Private (Channel) IP on its NIC and that you have DNS records created for the shared name and both sso servers. If not done, rename the computer account to the shared name to be used by vCenter and the Inventory Service.

Install the vCenter Inventory Service, when prompted register it with the SSO server, be sure and use the shared name and not the hostname name of either SSO servers.

vCenter requires a DB on our SQL server HeartBeat instance we created earlier, create the database and a SQL account to use for vCenter to authenticate with. Give the new account DBO rights to the vCenter Database, and the MSDB Database. For other options on the creation of the SQL DB see the vCenter Install documentation.

If not already part of your standard build install the SQL Native Client for the version of SQL you’re running.
Create a 64bit DSN that points to the shared SQL name, not either windows hostname running HeartBeat, use the SQL account you created and change your default DB to the one you created for vCenter.

Install vCenter, you’ll be prompted for the DSN info created earlier as well as SSO information, fill in the necessary information. Since the Inventory Service is installed on the same server the vCenter install will auto detect it, ensure the name displayed is the shared name and not either hostname, do the same for the vCenter FQDN when prompted.

Once the install is completed Shut Down the vCenter server and clone it to vcenter2 with no customizations, if possible place it in a different cluster and on different storage controllers or at least a different datastore.

Disconnect the both NICs on vcenter2.
Boot both servers up.
On vcenter2 change the Public IP settings, leave the shared IP but change the management IP to the correct IP for this server.
Change the Channel IP to the correct IP for this server.
Connect the Channel NIC, leave the Public NIC disconnected.

Rename all adapters, name the Pulbic adapters Public and Channel adapter Channel on both servers.

Start the HeartBeat install on vcenter1:

  • Select Primary
  • Select Second Server is Virtual
  • Select LAN
  • Select your channel adapter, Click the Add button and select your IP. Click the Add button below and enter the channel IP of the 2nd server.
  • Choose your Public adapter, Click Add select the Shared IP and leave the default that both servers will use the same IP. Click Add choose your management IP, below click add and choose the 2nd servers management IP, click NO to the warning that the 2nd server can’t be reached this is normal since it’s adapter isn’t connected to the network.
  • Ensure vCenter Services is selected to protect enter the credentials for the vCenter plug-in.
  • Enter vcenter1 and vcenter2 when prompted for names to rename the computer accounts to.
  • For the location to copy the configuration to create a HB folder on C:\ and then enter \\localhost\c$\HB.
  • Finish the wizard by click Next as it completes each part of the install, reboot when prompted.

Once the first server is up move over to the 2nd server. You do not have Public network access, so copy the heartbeat install file by accessing the 1st servers private network i.e. \\Server1PrivateIP\c$.
Run the heartbeat install on SSO2:

  • Select Secondary, when prompted for the directory that contains the configuration enter \\Server1PrivateIP\c$\HB
  • Select your channel adapter
  • Connect your Public adapter to the network before continuing
  • Choose your public adapter and next through the rest of the install
  • Reboot when prompted

Once both servers are up, open up the manage server heartbeat application and navigate to Applications and Plug-Ins. Right click on the vCenter Plug-in and choose edit, enter an account that can logon to vCenter.

Move the vCenter Services to the 2nd node and verify everything is working and then move them back to the primary.

vCenter and the Inventory Service is now protected by vCenter Heartbeat.

By now I’m sure you see a pattern, finish out the installs by repeating the process for the Web Client and VUM servers.

If you are unable to keep the first and second instance of each server in separate clusters be sure to create anti-affinity rules on your cluster to ensure they don’t end up running on the same host.

Repeat this process for the secondary site with the following exceptions:

  • When installing SSO you will choose to Join and existing SSO installation, then choose MultiSite and enter the FQDN and port number for the primary sites SSO instance (again used the shared name). Remember that MultiSite SSO does not perform replication so any changes you make in the primary site that you want reflected in the secondary site you must follow these instructions to accomplish.
  • When installing vCenter choose to join a linked mode configuration and point the install to the primary site’s vCenter.

In part 3 in this series I’ll go through the process of replacing the self-signed certificates with ones generated by your CA. The KB Articles from VMware are fairly accurate but seem to be written from the perspective that all services live on a single server so there are some differences when installing the services on separate servers such as we did.

vCenter 5.1, Multisite SSO, Linked Mode, Custom SSL certificates, protected by vCenter HeartBeat. Part 1

Over the past couple of weeks I got to work with a client who generally would have not used PSO for a vCenter upgrade but because of the new features, such as SSO, introduced in 5.1 they brought me in.

The customer, like many, didn’t really understand SSO and the different options to deploy it. Many customers I talk to initially want SSO in HA, not understanding that this requires a load balancer in front of the SSO servers. While this option is perfectly viable there are a couple caveats to it. Since the admin role is only installed on the first SSO server when this server is down services that are registered to SSO, like vCenter, the Inventory Service, and the Web Client, will not be able to start. They will keep running with the admin service down but if a restart of the server or service takes place they will not be able to start. Another caveat is no administration of SSO or registering of additional services can take place. So really in this first release of SSO the only thing that is HA in HA mode is authentication.

Once this conversation takes place customers ask how to protect it. The easiest is to just use HA that’s built into vSphere. This is a good option if you’re only worried about host failures, or BSod’s (when enabled HA can recover crashed guests). If you also want to protect against outages due to OS issues vCenter HeartBeat provides this in addition to doing maintenance and rebooting VM’s with only taking a small outage to move the service to the secondary node.

I am, and have been for years, a big fan of vCenter Heartbeat. When a customer has the requirement to keep vCenter and related services available such as when other components like vCD, View, SRM, etc. are in the environment or they just wish to always have vCenter available I recommend they implement vCenter HeartBeat.

We were also deploying SRM the week after the vCenter upgrade, which meant we needed a 2nd vCenter as they had been running a single vCenter to manage both sites.

The customer also requires all certificates to be issued from their CA and to not use any self-signed certificates. As I’ll detail in the following posts, this lead to some interesting discoveries that I haven’t seen documented elsewhere.

The diagram below is how I recommended this customer implement vSphere 5.1 and is the direction we went. In it you’ll see in each site the following servers:

  • SQL Server protected by vCenter Heartbeat
  • Single Sign On (SSO), configured in Multisite mode, protected by vCenter Heartbeat
  • vCenter and the Inventory Service protected by vCenter Heartbeat
  • vSphere Web Client protected by vCenter Heartbeat
  • Update Manager protected by vCenter Heartbeat (this server also hosts storage plug-ins such as NetApp’s VSC)

Architecture Image

Windows 2008 R2 Templates / Customization Specification / Local Administrator Password

I haven’t done much with templates in quite a while. Last week a client requested I assist with creating a Windows 2008 R2 template. We installed the base OS, did some minor configuration such as installing VMware Tools, enabling RDP and disabling the firewall using netsh, since this template would be used for various server types and we had no Internet access to patch the templates we stopped there. We shutdown the server and converted it to a template.

We then went and created a customization specification, the spec had an Administrator password set and had the server licensing set to per user, everything else was pretty straight forward.

To my surprise when creating a new VM from this template with this specification we couldn’t login, the Administrator password used in the template nor the one set in the specification worked.

When working with the client last week the only thing we got to work was a blank Administrator password in both the template and the customization specification. Not something I was happy about or recommended going to production with.

I came home and worked in my lab and had the exact same issues. After many tests scenarios the only thing that finally worked was enabling “Automatically logon as Administrator”.

2008R2 Template

After enabling this option having a password in the template and a different one in the customization specification the password in the customization specification worked.

I’m sure others have run into this but doing some searches, the only resolution I found was to use the blank passwords so I hope this helps someone who is seeing this same issue.

Issue with SSO and SQL Express

This past week I was working with a large Telco on their first vSphere deployment. This particular deployment was with vSphere 5.1 and was a lot of fun getting to use some new features in the real world such as Auto Deploy Statefull Install, enhancements in the distributed switch, among others.

That’s not to say we didn’t have any issues along the way, which is were this post comes in. This environment is for a particular set of apps that will be running across 18 VM’s, not a large number by any means but the service this provider is selling is quite cool, you can read more about it here.

The provider wanted to keep all vCenter services in one VM and to use SQL Express, they were over the host limit, 2 management and 10 resource nodes, VM wise they are way under so we decided SQL Express would meet their needs for now and if they grew out the environment they would move to full SQL at that time.

The issue we ran into is when installing SSO with SQL Express the SSO install sees the port SQL Express is using at the time of install and configures its connection string with that port. SQL Express installs with the dynamic ports option which when the server is rebooted causes SSO to not be able to connect to its database which in turns means vCenter won’t start. In the imsSystem.log file you’ll see.

java.sql.SQLException: Network error IOException: Connection refused: connect

This is easily fixed by setting SQL to use a fixed port and reconfiguring the connection string for SSO. To set SQL Express to use a static port open up SQL Server Configuration Manager, expand SQL Server Network Configuration, click on Protocols for VIM_SQLEXP, on the right hand pane double click TCP/IP.

SQL TCP/IP

In the properties delete the 0 from dynamic ports and enter a port to use under TCP Port, something like the default SQL port of 1433 or any other available TCP port would work fine.

SQL Ports

Once you’ve made those changes restart the SQL service. Now we need to update SSO’s connection string and configuration. This is a two step process, first run:

c:\program files\VMware\Infrastructure\SSOServer\utils\ssocli configure-riat -a configure-db –database-host <database server> –database-port <port number> -m <master password>

Next open the file c:\program files\VMware\Infrastructure\SSOServer\webapps\lookupservice\WEB-INF\classes\config.properites and change the portNumber= part of the file to the port you used for SQL.

SSO config.properties

Restart the SSO service and all services will be able to connect to SSO and start normally.