High CPU Ready with Low CPU Utilization?

I have noticed an increasing amount of search engine terms which results in people accessing my blog similar to

* High CPU Ready Low CPU usage
* CPU ready and Low utilization
* CPU ready relationship to utilization

So I wanted to try and clear this issue up.

First lets define CPU Ready & CPU Utilization.

CPU ready (percentage) is the percentage of time a virtual machine is waiting to be scheduled onto a physical (or HT) core by the CPU scheduler.

CPU utilization measures the amount of Mhz or Ghz that is being used.

Next to find out how much CPU ready is ok, check out my post How Much CPU ready is OK?

CPU Ready and CPU utilization have very little to do with each other, high CPU utilization does not mean you will have high CPU ready, and vice versa.

So it is entirely possible to have either of the below scenarios

Scenario 1 : An ESXi host has 20% CPU utilization and VMs to suffer high CPU ready (>10%).
Scenario 2: An ESXi host has 95% CPU utilization and VMs to have little or no CPU ready (<2.5%)

How are the above two scenarios possible?

Scenario 1 may occur when

* One or more VMs are oversized (ie: not utilizing the resources they are assigned)
* The host (or cluster) is highly overcommited (either with or without right sized VMs)
* Where power management settings are set to Balanced / Low Power or custom

Scenario 2 may occur when

* VMs are correctly sized
* The ESXi hosts are well sized for the virtual machine workloads
* The VM to host ratio has been well architected

So the question on everyone lips, How can high CPU ready with Low CPU utilization be addressed/avoided?

If you have a situation where you are experiencing high CPU ready and low ESXi host utilization the following steps should be taken

* Right size your VMs

This is by far the most important thing to do. I Recommend using a tool such as vCenter Operations to assist with determining the correct size for VMs.

* Ensure your hosts/clusters are not excessively overcommited

I generally find 4:1 vCPU overcommitment is achievable with right sized VMs where the avg VM size is <4 vCPUs. The higher the vCPU per VM average, the lower CPU overcommitment you will achieve.)
If you have an average VM size of 8 vCPUs then you may only see <1.5:1 overcommitment before suffering contention (CPU ready).

* Use DRS affinity rules to keep complimentary workloads together
VMs with high CPU utilization and VMs with very low CPU utilization can work well together. You  also may have an environment where some servers are busy overnight and others are only busy during business hours, these are examples of workload to keep together.

* Use DRS anti-affinity rules to keep non-complimentary workloads apart

VMs with very high CPU utilization (assuming the high utilization is at the same time) can be spread over a number of hosts to avoid stress on the CPU scheduler.

* Ensure your ESXi hosts are chosen with the virtual machine workloads in mind
If your VMs are >=8vCPUs choose a CPU with >=8 cores per socket and more sockets per host, like 4 socket hosts as opposed to 2 socket hosts. If the bulk of your VMs are 1 or 2 vCPUs, then even older 2 socket 4 core processors should generally work well.

* Use Hyperthreading
Assuming you have a mix of workloads and not all VMs require large amounts of cores and Ghz, using hyper threading increases the efficiency of the CPU schedulure by effectively doubling the scheduling opportunities. Note: A HT core will generally give much less than half the performance of a pCore.

* Use “High Performance” for your Power Management Policy

The above seven (7) steps should resolve the vast majority of issues with CPU ready.

For an example of the benefits of right sizing your VMs, check out my earlier post – VM Right Sizing , An example of the benefits.

Also please note, using CPU reservations does not solve CPU ready, I have also written an article on this topic – Common Mistake – Using CPU reservations to solve CPU ready

I hope this helps clear up this issue.

vCenter Operations for View – Scalable Architecture for a 10,000 user Pod

Recently I was putting together a design for a vCenter Operations for View 1.0.x solution for a customer who has approx 6000 virtual desktops and it got me thinking, what would best way to implement vCenter Operations for View for a 10,000 user “Pod” for a View 5.0 or 5.1 environment?

Before we begin, I would like to clarify this solution is designed to work with a standard View “Pod” design. If your environment does not follow the VMware View Reference Architecture then I do not recommend this architecture. eg: Managing Server and Desktop workloads via the same vCentrer may cause issues for this solution.

Example: It is assumed if a customer is deploying a VMware View solution with greater than 2000 users, that a management cluster will be used, or that the View Management VMs (View Connection Server / View Security Server / View Composer etc) are hosted somewhere other than the View Blocks themselves.

For more details on why using a Management cluster for View Management VMs is preferred , see my post “Example Architectural Decision – Supporting VMware View Infrastructure Servers

The below is the basic concept of the View “Pod” which is made up of five (5) “Blocks”. Each block is a vSphere cluster which supports up to 2000 view users.

view_pod

This above graphic is courtesy of John Dodge in his blog post Demystifying VMware View Large Scale Designs

In summary a 10,000 user View “Pod” is made up of

* Five (5) vCenter Servers
* Five (5) View Composer Servers (Note: Can be installed on the vCenter server)
* Seven (7) View Connection Servers (Brokers)
* Five (5) View “Blocks”

Now we need to confirm what is required to implement vCenter Operations for View.

Lets look at the system requirements for the vC Ops View Adapter server.

vCOPS_ViewAdapter_Sizing

Reference: Pg 11 of vCenter Operations for View Integration Guide

So based on the above, to support a 10,000 user pod we would require a “Monster” VM with 20 vCPUs and 40gb Ram!

MelvinMonsterVM

This would require an ESXi host with at least two physical CPUs w/ 10 cores each and the “Monster” VM would basically monopolize the host, so this doesn’t seem like a viable solution for the vast majority of customers.

Alternatively, if we take a scale out approach then we can use five (5) VMs with 4 cores and 8Gb ram. This sounds perfectly reasonable, and would fit within the majority of vSphere clusters currently deployed.

Next lets look at the requirements for the vCenter Operation Manager vApp.

vCopsMgrSizingvApp

Reference: Pg 13 of the vCenter Operations Manager 5.0 Installation Guide

Here we can see three examples, that support up to 1500, 3000 and 6000 virtual machines.

From these numbers, if we used a single vC Ops manager vApp it would require greater than 8vCPUs each for the UI and Analytics VMs, which could monopolize smaller management ESXi hosts and/or cause CPU scheduling difficulties or reduced consolidation ratios for the management cluster.

So similarly to the View Adapter server, I am proposing a “scale out” approach for the vC OPS Manager vApp.

In this case, I want to comfortably support 5 “Blocks” of 2000 virtual machines, therefore allowing for some head room, the “up to 3,000 virtual machine” solution appears to be the best option.

Therefore we will require five  vCenter Operations Manager vApp deployments for this solution.

It is also important to consider the storage capacity and performance requirements, which are shown below.

vCOPS_sizing

Reference: Pg 13 of the vCenter Operations Manager 5.0 Installation Guide

From a capacity/performance perspective, the solution for 10,000 users needs too be sized to support between

* 15,000 & 30,000 IOPS
* ~6TB & ~12TB

At this stage we have determined we require the following

5 x vCenter Operations for View Adapter Servers

5 x vCenter Operations Manager vApp’s

Next we need to work out how best to configure each View Adapter server.

If you review the vCenter Operations for View Integration Guide on page 19 you will see the below graphic which states “Enter the name of a View connection server in your VDI environment….”

So the question is, which View Connection Server (Broker) should we use?

vCopsViewAdapterSettings

Here I have come up with two Solutions which are overall very similar but have a couple of differences, which are

1. How many connection brokers are used to service user connections
2. What connection broker/s the View Adapter servers connect too.

Lets go over both solutions as well as a potential option 3 which needs further investigation. (I will follow up with another article on Option 3)

Solution 1 : Dedicated Connection Brokers for vCenter Operations for View Adapter Servers

The concept here is there is a total of 7 connection brokers servicing the 10,000 user “Pod”. We remove two (2) of the brokers (in this example, Number 6 and 7) from the round robin on the load balancer and configure the View Adapter servers to use either Connection Broker 6 or 7.

Here is a diagram showing the solution

DoHA vC OPS RA Dedicated Brokers for vCops View V0.1

Note: Even though Connection Brokers 6 & 7 are not included in the round robin to service user connections, they continue to replicate between all other brokers.

Here are the Pros and Cons for Solution 1

PROS

* Traffic from vC Ops for View does not impact the performance of the connection brokers which users connect too

CONS

* Only five (5) connection brokers are available to service user connections (Note: 5 should be sufficient as each broker can support 2000 connections)
* In the event one (or more) of the five connection brokers has an issue user connection times may be impacted

Solution 2 : One to One mappings between View Adapter Servers and Connection Brokers

The concept here is there is a total of 7 connection brokers and 5 View Adapter servers. Each View Adapter server is configured to connect on a one to one basis to a specific connection broker eg: View Adapter Server 1 connects to Connection Broker 1, and so on.

Here is a diagram showing the solution

DoHA vC OPS RA View Adapter Server 1to1 Mappings V0.1

Here are the Pros and Cons for Solution 2

PROS

* All seven (7) connection brokers are available to the load balancer to service user connections
* The performance of all the connection servers (Brokers) should be consistent as they have a fairly equal workload (except for 6 & 7 which dont service View Adapter traffic)

CONS

* Traffic from vC Ops for View may impact the performance of the connection brokers which users are using to connect

Solution 3 – Configuring the View Adapter servers to use the load balancer address

What about configuring the View Adapter servers to use the load balancer address, rather than connecting to a specific connection broker? I am currently investigating this option, and from some discussions with a number of the VMware EUC team, there may be reasons this wont work. I will post an update once I have further investigated and tested this option.

Moving on, Once you decide which of the above two Solutions suit your environment best, the following applies to all three of the above solutions.

For each “Block” (ran by a dedicated vCenter server), one (1) vC OPS for View Adapter server and one (1) vCenter Operations Manager vApp are deployed into the Management cluster.

For a complete 10,000 user “Pod”, this will mean a total of five (5) vC OPS for View Adapter server and five (5) vCenter Operations Manager vApps.

Each vCenter Operations Manager server will be configured on a one to one basis with one vCenter eg: Block 1 vC Ops vApp connects to Block 1 vCenter.

Next we need to ensure each vC Ops for View Adapter server isn’t monitoring all desktop pools in the pod (which it will by default). This is very important otherwise each View Adapter server will be saturated with all pools, and therefore managing  all desktops in the “Pod” (up to 10,000 desktops!). This would cause major performance issues and result in the vC Ops for View environment hitting some hard limits.

To avoid this issue we select the tick box “Specify desktop Pools” (shown below) and enter all pool names (separated by a “,”)  for the “block” that is being monitored by this View Adapter server, similar to the below.

vcops_View_view_settings

Now each of the View Adapter servers configured to only monitor one block, so a maximum of 2000 users.

However, there is a catch, as the pool filter is configured on the View adapter server/s, there is still an ongoing overhead (both compute & network) on both the connection broker/s and the view adapter server/s as the View pods topology is still proceeded by the adapter server before being filtered out (by specifying the desktop pools as in the above screen capture).  It is important to understand this overhead does not impact the vC Ops Manager vApp in any way as it is associated with a vCenter which will only manage up to 2,000 users.

So in addition to the standard components of the 10,000 user View “Pod” discussed earlier, we now have

* Five (5) vC Ops for View Adapter Servers
* Five (5) vCenter Operations Manager vApps

In summary, here are the Pros and Cons of the overall concept.

PROS

1. Avoid’s the requirement for very large (>8vCPU) vC OPS UI / Analysis & View Adapter VMs to support the solution
2. Prevents large VMs from potentially monopolizing management cluster ESXi hosts
3. Prevents an increased CPU scheduling overhead in scheduling large vCPU VMs for the Management ESXi hosts
4. In the event of a View Connection Server (Broker) , View Adapter Server and/or vC OPS Manager vApp failure, only part of the monitoring solution is impacted.
5. Prevents an increased CPU scheduling overhead in scheduling large vCPU VMs for the Management ESXi hosts
6. The solution is scalable and can start from a smaller deployment of <2000 users (with one vC Ops vApp and one View Adapter Server) and easily scale with linear performance even beyond a 10,000 user pod as its based on a repeatable model supporting 2000 users. In short, this solution can scale basically without limit.
7. No single View Connection Server (Broker) is managing all vC OPS for View traffic
8. Increased DRS flexibility/efficiency for the management cluster as smaller VMs are easier for DRS to load balance
9. Flexibility in ESXi host hardware, ie: The ability to have smaller & potentially cheaper (2 socket / 8 core) servers for management

CONS

1. Additional installation / configuration time as five (5) View Adapter servers & five (5) vCenter Operations Manager vApps need to be deployed
2. Increased Microsoft Windows 2008 licensing – although if you can justify Windows 2008 Datacenter edition licenses on the management cluster/s allow unlimited windows VMs as such avoiding this issue.
3. Additional maintenance effort in patching/upgrading five (5) View Adapter servers & five (5) vCenter Operations Manager vApps instead of one.
4. You do not get a single pane of glass for monitoring the entire 10,000 user Pod, you will have five (5) separate vC OPS web interfaces, one per block.

In Conclusion, the above architecture is scalable and allows the deployment of vCenter Operations for View version 1.0.x which avoids a number of potential “gotchas”, some of which may degrade the performance of your View environment. Each View Adapter Server and vC Ops Manager vApp will only service one (1) block of 2000 users. When adding additional Blocks or Pods (new Pods are required for >10000 users) the solution will scale and support 2000 users at a time.

My advise would be to use Solution 1, as the main advantage is as the Adapter server still has to process all topology data before filtering it out, this solution ensures there is no impact on the connection brokers servicing user connections. This means the CPU/Network overhead (discussed earlier) only impacts the connection brokers not servicing users.

Looking forward to the upcoming version of vCops for View (1.5), it will bring increased scalability and is penciled in for late Q1 2013, so the above architecture is really an interim solution until the new product is released.

Note: In VMware View 5.2 (which will also be released first half of 2013) there are some major improvements in scalability which may change the “Block” and “Pod” architecture discussed in this post as well as some improvements to the View agent, again these changes will likely change the vC Ops for View architecture.

I will be following up this article, with a similar post on vC Ops for View 1.5 architecture closer to the release date.

I would like to Thank John Dodge , David Wooten , David Homoki & Tim Whiffen from VMware EUC team, as well as Michael Webster (@vcdxnz001) & Andre Leibovici (@andreleibovici) for there input into this article.

I hope this article has been helpful and I welcome any constructive feedback / comments etc.

VMware View 5.1 Local Mode Check Out fails with “Internal Error” at 10%

I had an interesting issue today, where a VMware View virtual desktop failed to check out (ie: Local mode) giving an error “VMware View Client – Internal Error.” (as shown below) at the 10% mark.

View_LocalMode_InitializingCheckOutPaused_InternalError

I went into Monitoring , and Events, and came across the following error while trying to diagnose the issue.

LocalModeEventError

Talk about a useless error message!!

However, as it turns out the solution was actually very simple, all you need to do is the following

Login to VMware View Administrator, under View Configuration, click “Servers”, select the “Transfer Servers” tab, highlight your Transfer server (as shown below) and press the “Enter Maintenance mode” button.

TransferServerEnterMaintenanceMode

Once the “Status” of the transfer server changes to Maintenance mode, perform a controlled shutdown of the VM.

Next edit the settings of the virtual machine and remove any/all Floppy Drives from the VM’s hardware.

Now power back on the VM. While the VMs is powering back on, you can return to the VMware View Administrator, browse back to the Transfer server tab, highlight your transfer server and press the “Exit Maintenance Mode” button. The status of the transfer server will be “Pending” and once the transfer server is back online its status will change to “Ready”.

Now retry your Check Out operation and you should see something similar to the below, where the Check Out process proceeds as it did previously to 10%

View_LocalMode_InitializingCheckOutProcess6%

Where the check out previously failed, it should now proceed similar to the below example.

LocalModeCheckingOutPogress

After the Check out process has completed you should see the message “Log on to Local desktop” (as shown below).

LocalMode_LogOnToLocalDesktop

So what do we learn from this?

It is always a good idea to remove unnecessary devices from your VMs. 😉