Back on the horse

Yeah, I finally posted again after a pretty long break. Checking the stats, I’m getting a suprisingly high amount of visitors each day, considering the only way in here is pretty much through some search engine, so I’m hoping the first posts has been of use to somebody.

The goal of this blog, is to post solutions and information on topics I’m having a hard time finding anything about, and that way help other people who is in a similar situation as I was at some point.
At the same time, I enjoy writing this stuff, but I try to keep my standards high, and because of that, it takes quite a while to write a post.

A few days a go, a guy called Praveen actually posted a comment on this blog. When I read it and thought about the reasons why I created this blog, it inspired me to pick up writing again, and try to be better at posting stuff here. While I’m not the guy to admit this easily, comments like that means the world to me, it’s nice to know that the work I put into something, was useful for someone else 🙂

I’m not sure if a lot of the hopefully upcoming posts will be about VMWare, as I usually do most of my work on Windows servers. I have some basic home made powershell scripts that I feel might be great to post more about in a near future.

Outlook 2011 not working with Exchange 2010

A colleague of mine was setting up a customer for Exchange 2010. Everything was set up like it should be, Outlook 2007 and Outlook 2010 clients worked like a charm. This customer also has a few Macs running Office 2011, where Outlook unfortunately refused to do anything (literally). When he put in the information for the Exchange server, nothing happened. It would just sit at an empty mailbox, no connection errors, no authentication errors, nothing. Let me be honest and say that at my company, we’re not exactly good when it comes to Macs, we’re actually quite horrible. Anyways, we struggled a long time as we kept getting no feedback on the process from the Mac. The solution? Keep reading!

The solution

When we finally found the solution, it was a fairly basic one. Outlook 2011 doesn’t use MAPI, it relies solely on EWS (Exchange Web Services). When we tried to browse https://my.exchange.server/EWS/exchange.asmx via a browser on the Mac, it wouldn’t prompt for any credentials, just show a blank page, where a Windows computer would actually show something useful. Then we checked the authentication allowed for EWS in IIS. It was set to allow Anonymous Authentication (this is required by the Exchange server) and Windows Authentication. Obviously you need to authenticate to pull anything useful out of this service, and, for perhaps pretty obvious reasons, the Mac wasn’t using Windows Authentication. We enabled Basic Authentication and it started working immediately on the Mac. If you want to make sure it’s updated fast, do a iisreset from a command line.

These images should tell you everything.

Fire up your IIS manager, find EWS (in this case in the Default Web Site), open Authentication.

 

Enable this bad boy and you're set. Do a iisreset if you're impatient... like me.

There’s a lot more to this story than just the above steps, it felt as an uphill battle the entire time, but I won’t bore you with all the troubleshooting we did (even though I already did a little).

Inactivity

So I got caught up with a lot of gaming since my last post, which has resulted in this newly created blog is already missing updates 🙂

I probably wont update it much the next month or so, but I’ll leave it up, I believe the info so far can be useful still.

vCPU performance degradation

This post will lightly touch the subject of vCPU performance degradation, when using multiple vCPU in a VM. This is not a scientific test, it’s just meant to help make decisions about how many vCPU you should use 🙂 

How is the test made?

It’s pretty simple. I got the program Super Pi, and for the single thread test, I ran one occurence of the program. For the multi thread test, I ran 10. All tests were made with the 2M calculation. The test is run on the same ESX host, which is in a production environment, in a low load period. Tests are run multiple times over the course of 3 hours, to adjust somewhat for inaccurancy in case of some sudden loads. 

The numbers, how should I read them?

Well, the number of vCPU is pretty self-explanatory. The number at the end of the bar though, it’s the average finish time (lets call that AFT from now on) of the threads. In the single thread test, that’s simply the average of how long Super Pi took to finish. In the multi thread test, it’s all the 10 finish times added together, then divided by 10. 

Single thread

Let me warn you, this first test is slightly weird, and I’m still unable to explain it. But have a look at the numbers first, then we’ll dive into it. 

 

1 vCPU takes 38,3 seconds to finish. Okay, noted, no problem.
2 vCPU takes 46,1 seconds to finish, hmm, that’s quite a performance hit, roughly 20%. 4 vCPU must be horrible then!
4 vCPU takes 39,3 seconds… Wait, what? That’s only 2% slower than 1 vCPU. 

These numbers are consistent, I ran this test on the same ESX host every 15-20min over 3 hours, and they kept ending up this way. If anyone reading has the sligthest idea why this happens, I would love to be enlightened. 

Anyway, this shows a slight performance degradation from 1 vCPU to 4 vCPU, 2% roughly. This means, that if your environment is able to reserve the cores needed to run a 4 vCPU VM, it wont really perform worse when it’s not multi threading. 

Multi thread

This is mainly why I made the test, to clarify how much multi threading you actually gain from more than one vCPU. Like the last time, take a look at the numbers. 

 

Notice how the 2 vCPU performs how you would expect when multi threading. 

1 vCPU is 10 times slower to calculate 10 threads than 1, makes sense.
2 vCPU is ~49,4% faster than 1 vCPU, pretty good deal.
4 vCPU is ~73,4% faster than 1 vCPU,  ~47,4% faster than 2 vCPU. Not as good a deal, but still pretty good. 

Putting it together

I’ve put the two graphs in the same chart, simply because I could, which gives me the opportunity to post one more. Behold! 

 

A final round of numbers. This is the average finish time per thread. The calculation is (AFT * Number of vCPU) / Number of threads. The number is in seconds.

 Single thread
1 vCPU: 38,3
2 vCPU: 46,1 (20,3% slower than 1 vCPU)
4 vCPU: 39,3 (2,6% slower than 1 vCPU) 

Multi thread
1 vCPU: 38,6 (0,08% percent slower than single thread)
2 vCPU: 39,1 (17,9% faster than 2 vCPU single thread, 5,1% faster than 4 vCPU multi thread)
4 vCPU: 41,1 (4,6% slower than 4 vCPU single thread) 

A note: I did also do the test with 8 vCPU, but because the ESX host only has 8 cores, it’s pretty hard for the hypervisor to reserve the CPU cores, and the tests were horrible 🙂 

What does this tell us? Well, if you’re not running a terminal server, examine if you need to use more than 1 vCPU. CPU resources aren’t the most expensive in a virtual environment, but if you just pop 4 vCPU in every VM you create, you will run into a CPU limit very fast. Imagine you have a ESX who can have 100 VMs with 1 vCPU (fictional numbers here), you might instead only be able to support 50-70 with 2 vCPU, or 10-20 with 4 vCPU. These numbers aren’t real, but what I’m trying to say, is that the curve is exponential, meaning you will loose more than you gain. 

As said, the above is not a fact sheet, but it should help you make decisions. Sure you might loose some extra CPU by creating a 4 vCPU VM instead of 2 VMs with 2 vCPU each. On the other hand, you might be able to save RAM and storage by just creating a single 4 vCPU VM, which might be a better choice for you, because those 2 resources might cost you more than CPU. Usually we run out of RAM before anything else 🙂

(Forced) Vacation

While I do like some time off every once in a while, I mostly never take any vacation, which now results in my second consecutive forced vacation. And with my Xbox360 just broken (no, not RROD), I suppose I have a chance to post a some of the stuff I want to here 🙂

Reading the CPU ready time

I was just messing around with the tags in the previous post, and noticed only one other person has a tag called CPU ready, which led me to a very interesting blog.

First of all, this is an excellent post about reading the CPU ready time in the performance tab. http://www.vmdamentals.com/?p=44

In general, the blog contains a lot of useful information.
http://www.vmdamentals.com

How vCPU works on the ESX host

Update: The vCPU benchmark can be found here: https://dryerr.wordpress.com/2010/04/02/vcpu-performance-degradation/

Having recently made a simple vCPU benchmark, I thought about posting the results here. But for those numbers, you really need to understand how vCPU works on the ESX host, or more important, how multiple vCPU works on the ESX host.

First, lets look at how a normal CPU works in a physical environment
The term CPU cycle is the key here. A CPU cycle happens with every Hz, so a 2GHz CPU can do up to 2 billion CPU cycles each second. If you then add another CPU to the mix, you can now do up to 4 billion CPU cycles per second.
So far, pretty straightforward, but, with 2 CPUs, each cycle now happens twice, once on CPU0 and once on CPU1. This doesn’t mean that both CPUs will do some work, the cycle might do something demanding for CPU0, but be empty for CPU1, leaving a free CPU for more work.

So what’s the interesting part here? Both CPUs must run a cycle at the same time. This is probably done to avoid synchronization problems or something similiar.

Now, in a virtual environment
Lets say you have a VM, this VM has 1 vCPU. With every cycle, the vCPU asks the hypervisor for some real CPU to do the work. The hypervisor then tries to reserve a physical CPU to take care of the cycle.
Now your VM has 2 vCPU, what happens then, is that the VM asks the hypervisor to do 2 CPU cycles, and the hypervisor then tries to reserve 2 physical CPUs to do the cycles. This is all very good and well, but it presents a problem. It’s harder to reserve 2 physical CPUs than 1 (oh really?).

For fun, imagine you have an ESX server with 8 physical CPUs available. You also have 5 VMs with 1 vCPU each. In this case, every VM shouldn’t really fight for CPU, as there is always one available. Now add another 2 VMs with 2 vCPU each, a total of 7 VMs, and 9 vCPUs. All the VMs with 2 vCPU will have a harder time reserving CPU time, because the 1 vCPU VMs can jump in at any free CPU. Imagine all the VMs is actively using a lot of CPU. This requires 9 physical CPUs, but we only have 8, so the ESX does what it’s best at, queuing the work and making our virtual environment doing it’s thing. However, the VMs with 2 vCPU will have a hard time reserving CPU time, because everytime a CPU frees up, a single vCPU jumps in and grabs it. I’m pretty sure ESX has a great way to reduce this kind of grabbing, but it does matter. A lot.

If you look on the performance charts on the VM, you have a graph called “CPU Ready”. I find the title a bit confusing, because what it means is, that’s the amount of time the vCPU was ready to do work, but couldn’t get time on a physical CPU. You can’t avoid some CPU ready delay, not when you’re doing virtual environments.

So it begins…

It’s kind of fascinating, the more techy (lets say nerdy) a person is, the less likely the chance of a Facebook account seems. How on earth does that make sense? I don’t know.

As you’ve probably guessed, I don’t have a Facebook account, but it might be time for a blog (wow, already!). Mostly to share solutions to the more annoying IT problems I run into. Primarily on Windows server platforms and some networking. In lack of a better place to make it public.