Simple IT made complex

Dryerr's blog

How vCPU works on the ESX host

with 9 comments

Update: The vCPU benchmark can be found here:

Having recently made a simple vCPU benchmark, I thought about posting the results here. But for those numbers, you really need to understand how vCPU works on the ESX host, or more important, how multiple vCPU works on the ESX host.

First, lets look at how a normal CPU works in a physical environment
The term CPU cycle is the key here. A CPU cycle happens with every Hz, so a 2GHz CPU can do up to 2 billion CPU cycles each second. If you then add another CPU to the mix, you can now do up to 4 billion CPU cycles per second.
So far, pretty straightforward, but, with 2 CPUs, each cycle now happens twice, once on CPU0 and once on CPU1. This doesn’t mean that both CPUs will do some work, the cycle might do something demanding for CPU0, but be empty for CPU1, leaving a free CPU for more work.

So what’s the interesting part here? Both CPUs must run a cycle at the same time. This is probably done to avoid synchronization problems or something similiar.

Now, in a virtual environment
Lets say you have a VM, this VM has 1 vCPU. With every cycle, the vCPU asks the hypervisor for some real CPU to do the work. The hypervisor then tries to reserve a physical CPU to take care of the cycle.
Now your VM has 2 vCPU, what happens then, is that the VM asks the hypervisor to do 2 CPU cycles, and the hypervisor then tries to reserve 2 physical CPUs to do the cycles. This is all very good and well, but it presents a problem. It’s harder to reserve 2 physical CPUs than 1 (oh really?).

For fun, imagine you have an ESX server with 8 physical CPUs available. You also have 5 VMs with 1 vCPU each. In this case, every VM shouldn’t really fight for CPU, as there is always one available. Now add another 2 VMs with 2 vCPU each, a total of 7 VMs, and 9 vCPUs. All the VMs with 2 vCPU will have a harder time reserving CPU time, because the 1 vCPU VMs can jump in at any free CPU. Imagine all the VMs is actively using a lot of CPU. This requires 9 physical CPUs, but we only have 8, so the ESX does what it’s best at, queuing the work and making our virtual environment doing it’s thing. However, the VMs with 2 vCPU will have a hard time reserving CPU time, because everytime a CPU frees up, a single vCPU jumps in and grabs it. I’m pretty sure ESX has a great way to reduce this kind of grabbing, but it does matter. A lot.

If you look on the performance charts on the VM, you have a graph called “CPU Ready”. I find the title a bit confusing, because what it means is, that’s the amount of time the vCPU was ready to do work, but couldn’t get time on a physical CPU. You can’t avoid some CPU ready delay, not when you’re doing virtual environments.

Written by dryerr

March 27, 2010 at 11:14

Posted in VMWare ESX

Tagged with , ,

9 Responses

Subscribe to comments with RSS.

  1. Hi,

    Thanks for the info, please keep writing.


    November 3, 2010 at 16:52

  2. Relly great information ,thankyou for writing the article.


    January 4, 2012 at 06:46

  3. U truly produced a lot of excellent tips inside ur blog
    post, “How vCPU works on the ESX host Simple IT made
    complex”. I’ll wind up heading back to ur web page soon enough. Thank you ,Mai


    February 18, 2013 at 22:17

  4. I love to share understanding that I have accrued through the year to assist
    improve group overall performance.


    July 29, 2013 at 02:44

  5. I have an excited analytical vision just for fine detail and may foresee difficulties prior to these people occur.


    July 31, 2013 at 09:13

  6. Good day! I just wish to give a huge thumbs up for the great
    info you have got here on this post. I will be coming again to your weblog
    for more soon.


    August 3, 2013 at 05:23

  7. Whats up! I simply want to give an enormous thumbs up for the nice data you have got here on this post.
    I shall be coming back to your blog for extra soon.


    August 4, 2013 at 09:56

  8. After I initially commented I appear to have clicked the -Notify me
    when new comments are added- checkbox and now each time a comment is added I recieve four emails with the exact same comment.
    Is there a way you can remove me from that service? Appreciate it!

    search engine

    September 11, 2014 at 00:39

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: