Common Mistake: Inefficient cluster sizes

Link

In my day job, I regularly come across environments which are running poorly and have inefficient designs.

One of the most common issues I see is VMware environments which cannot power on VMs due to being out of compute resources, but not for the reasons you may expect.

While the environments may have less than optimal HA settings / policies, the most common issues I see is customers (for whatever reason) having multiple clusters with only a few nodes. (ie: 2/3/4 etc)

Some of the time, there are corporate policies which may require this type of setup, but alot of the time, you can comply with these policies while still optimizing the environment.

It seems that even with virtualisation having been common place for many years, the basics are still mis-understood by a significant percentage of industry professionals. I have heard comments event recently saying you need 2 node clusters for maximum HA efficiency, They couldn’t be more Wrong!

So, why are small clusters a potential problem?

Depending on what HA setting you choose (Host failures cluster tolerates , Percentage of cluster resources reserved for HA, or Failover Host/s), the clusters have a large amount of “waste”.

What is “Waste”?

“Waste”, is the amount of the compute power within the cluster, that cannot be used to ensure in a HA event, VMs can be restarted on the remaining hosts.

Now at this stage, let me point out, some “Waste” is a good thing. We need to have some spare capacity for HA events, but the challenge is to minimize the waste without compromising HA.

So, in a recent environment I reviewed, there was 4 clusters using similar IBM x3850 Servers.

Cluster 1 : 2 Nodes

Cluster 2 : 2 Nodes

Cluster 3: 3 Nodes

Cluster 4 : 2 Nodes

In all clusters, HA was enabled (as it should be) and the HA admission control setting was “Percentage of Cluster resources reserved for HA” (which I prefer).

The 2 node clusters HA reservation percentage was set to 50%, and the 3 node cluster was 33%, which would be the settings I would choose if I had to stick with the 4 cluster design.

Because the environment (in its current state) was unable to host any more VMs, the customer wanted to purchase another 2 new Hosts, and form a new cluster.

At this stage we have the equivalent of 4 hosts of “waste” within the environment, and with a new cluster we would have 5 hosts “wasted”.

Now after a quick check of the VMware EVC KB: 1003212 all CPUs are compatible with EVC and support the EVC mode “Intelยฎ “Merom” Generation”.

So, we can form a single new cluster using the existing 9 hosts and maintain full cluster functionality by enabling EVC.

Lets assume the hosts are all in a cluster and we’re configuring HA, How do we ensure we have more available compute for the new virtual machines?

Simple, we Enable HA (as you always should), Enable admission control, and set the HA policy to “Percentage of Cluster resources reserved for HA, But what percentage should we choose?

Well, it depends of what level of redundancy you require.

Generally, I recommend for

<8 hosts = N+1 – Note: If you require N+1 during maintenance you need N+2

>8 hosts < 16 hosts = N+2

>16 hosts <24 hosts = N+3

>24 hosts = N+4

The reason for the above, is as you add more hosts, your chance of a host failure, and a subsequent host failure increases. Therefore the more hosts you have, the more redundancy you need, Similar concept to RAID.

So in this example, we’re right on the line in terms of N+1 or N+2.

Lets be conservative, and choose N+2, therefore setting “Percentage of Cluster resources reserved for HA” to 22% (N+2 is actually 22.5%, but we use round numbers).

So what have we achieved?

The previous setup had only N+1 and an average HA overhead of 45.75% (50%+50%+50%+33% divide 4).

The new 9 node cluster now with N+2 redundancy and only has an overhead of 22%. A NET gain of 23.75% of available compute resources without purchasing new hardware.

What else do we gain by having a single larger cluster:

1. Increased DRS flexibility

2. Increase redundancy (previously N+1, now N+2)

3. Less chance of contention

4. No need to purchase new hardware!!

The above is a simple example of how to increase efficiency within a VMware environment without purchasing new hardware.

Now for those of you wanting to know more about HA/DRS, this has been covered in great detail in other blogs, I would recommend you first have a read of the following blog and get a copy of “vSphere 5.0 Clustering technical deep dive” book.

Yellow Bricks (Duncan Epping) – HA Admission control Pros and Cons

My VCDX Journey

After deciding mid 2011 to take on the challenge of VCDX4 with a good friend of mine (James Wirth,VCDX#83), I went and sat the VCP5 exam when it was released. I passed VCP5 and assumed that would qualify me for the VCAP4 exams., WRONG! Therefore I had to go back and sit the VCP4 exam which i did two weeks later.

The VCP4 exam was surprising difficult as I had all the latest vSphere 5 configuration maximums and features etc fresh in my mind, so it made answering the “simple” questions much harder than it would have been if I had done VCP4 before doing VCP5.

None the less I passed VCP4 and submitted my authorization for both the VCAP4-DCD and VCAP4-DCA exams in November.

“#83” and I regularly got together to study in our labs as well as reviewing various material in preparation for DCD which we both agreed we should take first.

“#83” sneakily went ahead and booked his DCD without telling me, and I received a call one day from a very happy man advising me he was a VCAP4-DCD.

Not wanting to be left behind I booked my exam for Dec 3rd, and without to much more study, sat and passed the exam with the below score of 403, which I was reasonably happy with. I’m not sure how the grading works, but I felt comfortable during the exam, and was confident pressing “Finish” before the score shows up, so 400+ is pretty good i think.

Image

I found the DCD exam to be by far the best exam I had ever sat, as I feel it actually tests your knowledge and experience as an Architect rather than your ability to remember large amounts of information which in reality everyone just googles. I found the visio style questions to be excellent, although the UI is a little clunky.

I called “#83” and had a similar conversation to when he called me after his result, but as I scored higher than him, I made sure he knew it. lol

#83 and I had a few more study sessions in our labs and I was looking on the Pearson Vue website to schedule my DCA in mid-late January. For some reason I was only authorized to sit the exam until Dec 31st, and there was only one (1) spot available. What the?!?

I therefore booked the exam for Dec 13th (Tuesday) which I was less than impressed about, but hey.

At this point I should point out, #83 and I both agreed the VCAP-DCA was likely the most difficult part of the VCDX process for us, as neither of us have ever been “VMware Administrators”, although we both have years of hands on experience.

I took annual leave for the Friday and Monday before the exam, and went through the DCA blueprint with a fine tooth comb, and did all tasks in my lab several times.

Exam day comes and I left home around 7am (for a 9am exam) and the commute normally takes 45mins, and I planned to review my notes prior to the exam.

However, wonderful Melbourne traffic hits! There was a accident on the main freeway into the city, and it was closed. So I was stuck in a huge traffic jam going off the first off-ramp. Long story short, I arrived at the exam center stressed to the back teeth 5mins after 9. Not happy!

As this was the last spot for the year I couldn’t afford to miss it, but the exam center we’re fine and set me up in the room after taking my ID and a photo etc.

The Exam UI is not bad, but for whatever reason (probably due to me being in Australia and the Lab being in the USA?) there was latency issues at times.

The exam format is a live lab, so

Tip#1 Think before you do something, as you could (and I have heard of people) essentially locking themselves out of a host. If you do this, you will almost certainly fail.

Tip#2 Manage your time effectively, as you don’t have a lot of time.

The tasks your asked to perform range from simple, to more advanced where some design experience will assist greatly.

Tip#3 Be aware not completing some questions prevents you from completing future tasks, so avoid skipping questions unless your totally stuck, in which case, don’t waste time and move on.

Tip#4 Know the DCA Blueprint! The exam isn’t impossible, if you are comfortable with the content of the Blueprint, without following documentation every step of the way, you should be able to pass the exam.

At the end of the exam, I was very nervous as I couldn’t properly concentrate due to the events of the morning commute. But, i felt I had done enough to pass (just!).

I called #83 and let him know I sat the exam, and felt I would only just scrape through, but the next 10 days wait for the exam result was a killer.

I received an exam from certification@vmware.com on the 11th business day (by this time im stressing), and opened the email to find a PDF attachment with my exam score report.

I opened the PDF and it went to full screen (about 150%), and I couldn’t see the score, I had to scroll down to view it. I stopped for a moment, and had serious butterflies in my stomach, I scrolled down and saw the below.

Image

LOL!

Well, a Pass is a pass right?!?!

I must say, I didn’t really care about the score, I was stoked that I passed after a less than ideal preparation and getting the to exam stressed out.

I called #83 and boasted that I had passed, as he had been giving me a hard time over the pass two weeks while I was waiting for the exam result. He congratulated me, and asked what score I got, and I advised him, as expected, I just scraped through… and after a few seconds I told him I got 308… and we both laughed.

My mid quickly turned to the VCDX application / handbook, and my first tip for VCDX applications is

VCDX TIP # 1 – Ensure you documentation fully meets/exceeds the requirements set out in the handbook

The design I submitted is of a IaaS offering I designed for IBM, and it’s based on vSphere 5, vCloud 1.5 , vC OPS 5.0

The Architecture document I felt was of a high standard, but the supporting documents had not yet been completed for the project, So I had to create these in my own time, to submit the application by the due date. I would say I was 90% happy with my architectural document, but only 50% happy with the other documents.

VCDX TIP #2 – Submit a design for a solution which has been implemented, and all documentation is already completed to reduce the personal time you spend on your application.

I submitted my application (as did #83 & #93) and eagerly awaited a reply from the VCDX team.

I was very pleased to receive the below from Mark Brunstad (@MarkBrunstad on Twitter)

Image

Again I have no idea how the applications are scored, and I was not provided a score, so who knows if my application was just good enough, or passed the bar with flying colors. I guess it doesn’t matter either way as I was accepted.

I got together with #83 (James Wirth) and #93 (Frank Fan) who were both accepted, and organized several mock panels which we did over the next few weeks.

VCDX TIP #3 – Do as many mock panels as possible, with a number of different panelists of different expertise.

The experience of presenting and defending your design is invaluable. The better you know your exam, and the more comfortable you are answering questions relating to your design, the better chance you have of successfully defending.

At this stage I started thinking about travel arrangements, as I am a single dad, I needed to bring my 20 month old daughter with me., I called a good friend of mine and asked if she wanted an all expenses paid trip to Canada to care for my 20 month old daughter while i studied prior to the defense, and during the defense itself. She accepted, and I booked the flights / accommodation etc.

On arrival, I caught up with #83 & #93, and we each did a final presentation to each other. I also did a mock panel (over WebEx and Skype) with #83 and #93 for #88 Ramses (@rsmeyers on Twitter) and he presented well.

#83 did his defense on Wednesday Morning, and #93 was scheduled for Friday afternoon. My defense was Thursday afternoon.

The morning of the defense, I booked a meeting room and did a brief rehearsal (in front of the mirror, lol) and came to the realization that if i didn’t know my design by now, it was too late, so I called it quits and went and spent time with my friend and daughter. (Which i think was a great idea as it took my mind off the defense).

My defense was scheduled for 2PM, so I jumped in the car around 1:30 and drove to VMware’s office which was only a few KM’s away.

I arrived and met up with #89 Eiadโ€ (@VirtualizationT on Twitter) & #91 Brian (@bsmith9999 on Twitter) in reception. We had a brief chat and all went for the essential pre defense toilet stop. ๐Ÿ™‚

After a few mins, Mark Brunstad came and introduced himself, and after checking IDs and taking photos, led me into my defense room.

Mark was my Moderator, Andrew Mitchell (@amitchell01 on Twitter) was my Observer, and my three (3) panelists were Ben Lin โ€ (@blin23 on Twitter), Mike Brown โ€ (@vMikeBrown on Twitter) & Andrew Hald (@vmxl on Twitter).

I found my Panelists to be very good, they asked clear, concise questions. They are also all very good at not giving anything away during the defense, so dont play poker with these guys. ๐Ÿ˜‰

The first part was the 75min Design Defense. I plugged my USB stick into the laptop provided, and setup my power point presentation. The VCDX documentation advises you create a 15 min presentation, I did this, but also had a lot of support slides, including diagrams and my documented architectural decisions.

VCDX TIP #4 – Have support slide at the end of your 15 min presentation

As I had done numerous presentations of my design both for work, and during my mock panels, I felt that I knew my design pretty much back to front. I was comfortable with all my architectural decisions, and could speak to all of them without referring to the slides.

I was surprised my panel asked very few questions in the first hour, im not sure if this was a good sign or not, but I felt that I had explained my solution fairly well, so I guess I answered some of their questions, hence they didn’t need to ask.

About the 60min mark, I got a few questions from the panel relating to vCloud and vC OPS which are technically not part of the criteria, but as my design needed to support these components, It is expected you can explain how the solution supports those products.

VCDX TIP #5 – Anything in your design is fair game for the panel to ask you about. Know all components of your design.

After answering a few questions (not as many as i expected) , the iPad 75min timer flashed red, “00:00” and time was up.

I was escorted back to reception for a short break, before go back in the room and starting the Design Scenario.

The 30min design scenario starts by the Moderator presenting a PowerPoint, of a fictional scenario.

For me, as I do this on regular basis, I didn’t find it difficult, although, in 30mins, you need to ensure you demonstrate your expert level abilities, so ensuring key aspects of a design are covered, albeit at a high level, isn’t easy.

I used the whiteboard extensively, and talked the panel through my thought process. I believe this is key to the panel being able to assess your abilities.

VCDX TIP #6 Talk the panel through your thought process, and use the whiteboard during the design scenario.

The 30mins went past very quickly for me, and before i knew it, the iPAD timer was blinking red “00:00” again.

We then went into the “Troubleshooting” session which is only 15 mins.

I was given a scenario in a power point presentation, and then the session started.

By now im sure you have read other blogs, where a lot of people who have passed VCDX don’t get to the solution within the 15mins. I would say, the solution is not important, its what you eliminate which is important. The scenario I was given was complex, and have numerous factors not directory VMware related which needed to be addressed.

VCDX TIP # 7 – Whiteboard what you have investigated and eliminated.

Doing this will ensure you don’t repeat yourself and waste the little time you have.

VCDX TIP #8 – Ask questions of the panel (this goes for the Design Scenario too)

The more questions you ask, the more opportunity the panel has to see your expert knowledge.

Right at the end of the 15mins, in the dying seconds, I believe I was at the solution, but more importantly, I had eliminated probably a dozen or more potential issues.

At the end of the Troubleshooting session, Mark (The Moderator) asked if I had anything I wanted to say, which i basically said it was a great experience, pass or fail, and that I believe having to defend in person add a lot of value and credibility to the certification.

Mark escorted me back to reception where I met up with #89 & #91 at reception. We had a good chat about our experience and after 15 mins or so, left the office.

After returning home after the defense, I eagerly awaited the result.

I was woken up Thursday morning around 5am by my phone blinking away, I checked to see if i had a missed call and noticed it was Twitter, since I was now awake, I checked twitter and saw several tweets congratulating the new VCDXs.

I immediately opened my email (on the Phone) and sure enough, there was an email from Mark Brunstad with a subject of “Your VCDX Design Defense Results”.

I opened the email, and saw the PDF attachment, without reading the file name (VCDX4 Congratulations Letter – Josh Odgers.pdf) I opened the attachment and saw the below.

Image

I was over the moon! (and still am).

I am now a VCDX4, and because I sat and passed the VCAP5-DCD BETA, I am also VCDX5!

I found the VCDX experience start to finish to be a fantastic learning experience, and even now, as a VCDX, I still feel there is a hell of a lot of learn. I intend to continue learning from various sources including my VCDX colleagues as well as giving back to the VMware community by blogging, tweeting, and presenting at VMUGs and VMworld (hopefully 2013 as I have missed submissions for 2012).

VCDX # 90 out.

Related Articles

VMware Certification Pro’s Video Interview – Josh Odgers VCDX #90

Image