Virtual Machine Performance – CPU Ready

I have had feedback that navigation of my blog to find past posts is difficult, so I am aiming to solve this by creating new sections which will hopefully help with navigation.

This section is dedicated to posts I have written relating to CPU ready.

I am still seeing environments on a regular basis where virtual machines are not being sized appropriated during initial deployment and tools such as vCenter Operations (even where it has been deployed) not being used to optimize performance of virtual machines and vSphere cluster/s.

I have customers buying new hardware, where it is simply not required. The goal of this section is to make sure people get the best return on investment (ROI) out of their hardware and VMware licensing.

CPU Ready

1. VM Right Sizing – An Example of the benefits

2. How Much CPU Ready is OK?

3. Common Mistake – Using CPU Reservations to solve CPU Ready

4. High CPU Ready with Low CPU Utilization

5. More Coming soon

Related Articles

1. Determining if multiple vCPUs are causing performance problem (VMware KB)


vCenter Operations for View – Scalable Architecture for a 10,000 user Pod

Recently I was putting together a design for a vCenter Operations for View 1.0.x solution for a customer who has approx 6000 virtual desktops and it got me thinking, what would best way to implement vCenter Operations for View for a 10,000 user “Pod” for a View 5.0 or 5.1 environment?

Before we begin, I would like to clarify this solution is designed to work with a standard View “Pod” design. If your environment does not follow the VMware View Reference Architecture then I do not recommend this architecture. eg: Managing Server and Desktop workloads via the same vCentrer may cause issues for this solution.

Example: It is assumed if a customer is deploying a VMware View solution with greater than 2000 users, that a management cluster will be used, or that the View Management VMs (View Connection Server / View Security Server / View Composer etc) are hosted somewhere other than the View Blocks themselves.

For more details on why using a Management cluster for View Management VMs is preferred , see my post “Example Architectural Decision – Supporting VMware View Infrastructure Servers

The below is the basic concept of the View “Pod” which is made up of five (5) “Blocks”. Each block is a vSphere cluster which supports up to 2000 view users.


This above graphic is courtesy of John Dodge in his blog post Demystifying VMware View Large Scale Designs

In summary a 10,000 user View “Pod” is made up of

* Five (5) vCenter Servers
* Five (5) View Composer Servers (Note: Can be installed on the vCenter server)
* Seven (7) View Connection Servers (Brokers)
* Five (5) View “Blocks”

Now we need to confirm what is required to implement vCenter Operations for View.

Lets look at the system requirements for the vC Ops View Adapter server.


Reference: Pg 11 of vCenter Operations for View Integration Guide

So based on the above, to support a 10,000 user pod we would require a “Monster” VM with 20 vCPUs and 40gb Ram!


This would require an ESXi host with at least two physical CPUs w/ 10 cores each and the “Monster” VM would basically monopolize the host, so this doesn’t seem like a viable solution for the vast majority of customers.

Alternatively, if we take a scale out approach then we can use five (5) VMs with 4 cores and 8Gb ram. This sounds perfectly reasonable, and would fit within the majority of vSphere clusters currently deployed.

Next lets look at the requirements for the vCenter Operation Manager vApp.


Reference: Pg 13 of the vCenter Operations Manager 5.0 Installation Guide

Here we can see three examples, that support up to 1500, 3000 and 6000 virtual machines.

From these numbers, if we used a single vC Ops manager vApp it would require greater than 8vCPUs each for the UI and Analytics VMs, which could monopolize smaller management ESXi hosts and/or cause CPU scheduling difficulties or reduced consolidation ratios for the management cluster.

So similarly to the View Adapter server, I am proposing a “scale out” approach for the vC OPS Manager vApp.

In this case, I want to comfortably support 5 “Blocks” of 2000 virtual machines, therefore allowing for some head room, the “up to 3,000 virtual machine” solution appears to be the best option.

Therefore we will require five  vCenter Operations Manager vApp deployments for this solution.

It is also important to consider the storage capacity and performance requirements, which are shown below.


Reference: Pg 13 of the vCenter Operations Manager 5.0 Installation Guide

From a capacity/performance perspective, the solution for 10,000 users needs too be sized to support between

* 15,000 & 30,000 IOPS
* ~6TB & ~12TB

At this stage we have determined we require the following

5 x vCenter Operations for View Adapter Servers

5 x vCenter Operations Manager vApp’s

Next we need to work out how best to configure each View Adapter server.

If you review the vCenter Operations for View Integration Guide on page 19 you will see the below graphic which states “Enter the name of a View connection server in your VDI environment….”

So the question is, which View Connection Server (Broker) should we use?


Here I have come up with two Solutions which are overall very similar but have a couple of differences, which are

1. How many connection brokers are used to service user connections
2. What connection broker/s the View Adapter servers connect too.

Lets go over both solutions as well as a potential option 3 which needs further investigation. (I will follow up with another article on Option 3)

Solution 1 : Dedicated Connection Brokers for vCenter Operations for View Adapter Servers

The concept here is there is a total of 7 connection brokers servicing the 10,000 user “Pod”. We remove two (2) of the brokers (in this example, Number 6 and 7) from the round robin on the load balancer and configure the View Adapter servers to use either Connection Broker 6 or 7.

Here is a diagram showing the solution

DoHA vC OPS RA Dedicated Brokers for vCops View V0.1

Note: Even though Connection Brokers 6 & 7 are not included in the round robin to service user connections, they continue to replicate between all other brokers.

Here are the Pros and Cons for Solution 1


* Traffic from vC Ops for View does not impact the performance of the connection brokers which users connect too


* Only five (5) connection brokers are available to service user connections (Note: 5 should be sufficient as each broker can support 2000 connections)
* In the event one (or more) of the five connection brokers has an issue user connection times may be impacted

Solution 2 : One to One mappings between View Adapter Servers and Connection Brokers

The concept here is there is a total of 7 connection brokers and 5 View Adapter servers. Each View Adapter server is configured to connect on a one to one basis to a specific connection broker eg: View Adapter Server 1 connects to Connection Broker 1, and so on.

Here is a diagram showing the solution

DoHA vC OPS RA View Adapter Server 1to1 Mappings V0.1

Here are the Pros and Cons for Solution 2


* All seven (7) connection brokers are available to the load balancer to service user connections
* The performance of all the connection servers (Brokers) should be consistent as they have a fairly equal workload (except for 6 & 7 which dont service View Adapter traffic)


* Traffic from vC Ops for View may impact the performance of the connection brokers which users are using to connect

Solution 3 – Configuring the View Adapter servers to use the load balancer address

What about configuring the View Adapter servers to use the load balancer address, rather than connecting to a specific connection broker? I am currently investigating this option, and from some discussions with a number of the VMware EUC team, there may be reasons this wont work. I will post an update once I have further investigated and tested this option.

Moving on, Once you decide which of the above two Solutions suit your environment best, the following applies to all three of the above solutions.

For each “Block” (ran by a dedicated vCenter server), one (1) vC OPS for View Adapter server and one (1) vCenter Operations Manager vApp are deployed into the Management cluster.

For a complete 10,000 user “Pod”, this will mean a total of five (5) vC OPS for View Adapter server and five (5) vCenter Operations Manager vApps.

Each vCenter Operations Manager server will be configured on a one to one basis with one vCenter eg: Block 1 vC Ops vApp connects to Block 1 vCenter.

Next we need to ensure each vC Ops for View Adapter server isn’t monitoring all desktop pools in the pod (which it will by default). This is very important otherwise each View Adapter server will be saturated with all pools, and therefore managing  all desktops in the “Pod” (up to 10,000 desktops!). This would cause major performance issues and result in the vC Ops for View environment hitting some hard limits.

To avoid this issue we select the tick box “Specify desktop Pools” (shown below) and enter all pool names (separated by a “,”)  for the “block” that is being monitored by this View Adapter server, similar to the below.


Now each of the View Adapter servers configured to only monitor one block, so a maximum of 2000 users.

However, there is a catch, as the pool filter is configured on the View adapter server/s, there is still an ongoing overhead (both compute & network) on both the connection broker/s and the view adapter server/s as the View pods topology is still proceeded by the adapter server before being filtered out (by specifying the desktop pools as in the above screen capture).  It is important to understand this overhead does not impact the vC Ops Manager vApp in any way as it is associated with a vCenter which will only manage up to 2,000 users.

So in addition to the standard components of the 10,000 user View “Pod” discussed earlier, we now have

* Five (5) vC Ops for View Adapter Servers
* Five (5) vCenter Operations Manager vApps

In summary, here are the Pros and Cons of the overall concept.


1. Avoid’s the requirement for very large (>8vCPU) vC OPS UI / Analysis & View Adapter VMs to support the solution
2. Prevents large VMs from potentially monopolizing management cluster ESXi hosts
3. Prevents an increased CPU scheduling overhead in scheduling large vCPU VMs for the Management ESXi hosts
4. In the event of a View Connection Server (Broker) , View Adapter Server and/or vC OPS Manager vApp failure, only part of the monitoring solution is impacted.
5. Prevents an increased CPU scheduling overhead in scheduling large vCPU VMs for the Management ESXi hosts
6. The solution is scalable and can start from a smaller deployment of <2000 users (with one vC Ops vApp and one View Adapter Server) and easily scale with linear performance even beyond a 10,000 user pod as its based on a repeatable model supporting 2000 users. In short, this solution can scale basically without limit.
7. No single View Connection Server (Broker) is managing all vC OPS for View traffic
8. Increased DRS flexibility/efficiency for the management cluster as smaller VMs are easier for DRS to load balance
9. Flexibility in ESXi host hardware, ie: The ability to have smaller & potentially cheaper (2 socket / 8 core) servers for management


1. Additional installation / configuration time as five (5) View Adapter servers & five (5) vCenter Operations Manager vApps need to be deployed
2. Increased Microsoft Windows 2008 licensing – although if you can justify Windows 2008 Datacenter edition licenses on the management cluster/s allow unlimited windows VMs as such avoiding this issue.
3. Additional maintenance effort in patching/upgrading five (5) View Adapter servers & five (5) vCenter Operations Manager vApps instead of one.
4. You do not get a single pane of glass for monitoring the entire 10,000 user Pod, you will have five (5) separate vC OPS web interfaces, one per block.

In Conclusion, the above architecture is scalable and allows the deployment of vCenter Operations for View version 1.0.x which avoids a number of potential “gotchas”, some of which may degrade the performance of your View environment. Each View Adapter Server and vC Ops Manager vApp will only service one (1) block of 2000 users. When adding additional Blocks or Pods (new Pods are required for >10000 users) the solution will scale and support 2000 users at a time.

My advise would be to use Solution 1, as the main advantage is as the Adapter server still has to process all topology data before filtering it out, this solution ensures there is no impact on the connection brokers servicing user connections. This means the CPU/Network overhead (discussed earlier) only impacts the connection brokers not servicing users.

Looking forward to the upcoming version of vCops for View (1.5), it will bring increased scalability and is penciled in for late Q1 2013, so the above architecture is really an interim solution until the new product is released.

Note: In VMware View 5.2 (which will also be released first half of 2013) there are some major improvements in scalability which may change the “Block” and “Pod” architecture discussed in this post as well as some improvements to the View agent, again these changes will likely change the vC Ops for View architecture.

I will be following up this article, with a similar post on vC Ops for View 1.5 architecture closer to the release date.

I would like to Thank John Dodge , David Wooten , David Homoki & Tim Whiffen from VMware EUC team, as well as Michael Webster (@vcdxnz001) & Andre Leibovici (@andreleibovici) for there input into this article.

I hope this article has been helpful and I welcome any constructive feedback / comments etc.

Example Architectural Decision – Distributed Power Management (DPM) for Virtual Desktop Clusters

Problem Statement

In a VMware View (VDI) environment where the bulk of the workforce work between 8am and 6pm daily, how can vSphere be configured to minimize the power consumption without significant impact to the end user experience?


1. The bulk of the workforce work between 8am and 6pm daily
2. Most users login during a 2 hour window between 7:30 and 9:30 daily
3. Most users logoff during a 2 hour window between 4:30 and 6:30 daily
4. VMware View cluster maintains at least N+1 redundancy
5. VMware View cluster only runs desktop workloads
6. VMware View cluster size is >=5
7. VMware View cluster/s are configured with HA admission control policy of “Percentage of cluster resources reserved for HA” to avoid the potentially inefficient slot size calculation preventing hosts going into standby mode


1. Reduce the power consumption
2. Align with Green IT strategies
3. Reduce the datacenter costs
4. Reduce the carbon footprint

Architectural Decision

Configure and enable DPM on all ESXi hosts with the power management set to “Automatic” and the DPM threshold set to “Apply priority 3 or higher recommendations” and set hosts 1,2 and 3 in the cluster not to enter standby mode.


1. As the bulk of the users are inactive outside of normal business hours, a significant power saving can be achieved
2. The users do not all login at once, which allows DPM to gradually start ESXi hosts (which were put into standby mode by DPM previously)
3. In the event the workload is unusually low on a given day, power savings can be realized without significant impact to the end user experience
4. Where a large number of users login unexpectedly early one morning, the impact to users will be minimal
5. DPM is configured to ensure a minimum of three (3)  ESXi hosts remain on at all times. This number is expected to be able to support all desktops within the environment under low load (ie: 80% of desktops at idle). This number can be adjusted if required.


1. In the unlikely event a large number of users logon unexpectedly early one morning, the impact to users may be experienced for the time it takes for one or more ESXi hosts to exit maintenance mode. This is generally <10mins for most servers.
2. Out of band interfaces such as DRAC / iLO / RSA or IMM interfaces (depending on host hardware type) will need to be configured and be accessible to vCenter and the ESXi hosts to enable DPM to function
3. As the “Percentage of cluster resources reserved for HA” setting is static (not dynamically adjusted by DPM) in the event of a host failure while one or more hosts are in standby mode, in unlikely event a VM attempts to power on before a host has been able to successful exit standby mode, the VM may fail to power on.
4. Where large percentages of Memory reservations are used (see Example AD – Memory Reservation for VDI) then ability for the for DPM to put one or more hosts into standby will be reduced. Where DPM is expected to be used, no more than 50% memory reservation should be configured to ensure maximum  memory overcommitment can be achieved without placing a significant overhead on the shared storage for vSwap files
5. Monitoring solutions may need to be customized/modified not to trigger an alarm for a host that is put into standby mode


1. Set a lower number of hosts to remain on to maximize power savings – This may result in higher impact to users first thing in the morning in the event of high concurrent logins
2. Set a higher number of host to remain on, however this will minimize power savings and give less value to the added complexity of setting up DPM (and associated out of band management interfaces)
3. Set the DPM threshold more aggressive to maximize power savings – This would likely result in some impact to VMs due to increased physical cores being available to the CPU scheduler and physical memory being available for VMs which may result in swapping