Competition Example Architectural Decision Entry 6 – Improve Performance for BCAs on Cisco UCS

Name: Anuj Modi
Title: Unified Computing & Virtualization Consultant @ Cisco
Twitter: @vConsultant
Blog: http://anujmodi.wordpress.com

Problem Statement

Most of the companies are migrating application workload to virtual infrastructure to take the advantages of virtual computing. With benefits of virtualizing the environment, the application still are facing I/O performance issue and end-users are not happy with response time for moving applications to physical servers. What are the ways to improve the performance for business critical applications in such environments?

Assumptions

1.      Cisco Unified Computing System
2.      VMware vSphere 5.x
3.      Cisco Virtual Interface Card M81/1240/1280
4.      Critical applications/databases

Constraints

1.      No impact on the applications production data
2.      Benefits of Virtual infrastructure features
3.      High Availability of Applications
Motivation

1.      Better performance and response time for business critical applications
2.      Reduce CPU cycles on ESXi Servers and offload the I/O load to hardware level.
3.      Improved I/O throughput for applications

Architectural Decision

Use the Cisco VN-Link in hardware with VMDirectPath to get better I/O performance for network traffic. All the traffic will be redirected through physical interface card and bypassing the vmkernel. This will provide better I/O performance as this will reduce the OS kernel layer to pass the network traffic to physical interface card.

VN-Link in Hardware with VMDirectPath

Alternatives

Cisco provides three different options for Virtual machine traffic on hypervisor. These options are listed below

1.      VN-Link is Software
2.      VN-Link in Hardware
3.      VN-Link in Hardware with VMDirectPath

The other two options can be used to improve the performance for virtual machine traffic.
In option1, Nexus 1000V switch can be used for network traffic forwarding. Virtual machine nic will directly connects to Nexus 1000V switch and Nexus 1000V switch uplinks will connect to Cisco virtual interface card. With this option, you can get benefits of Nexus 1000V advanced network features like ERSPA and Netflow and standardization of network switch management.

In option 2, UCSM will be used as Distributed switch and will integrated with vCenter server to control the virtual machine traffic. Each virtual machine nic will maps to a different virtual interface (VIF) on the UCS Fabric Interconnect and directly pass the traffic through it. This will give better I/O performance to network traffic and directs the I/O load to physical interface card.

Justification

Option 3 is selected with this solution to provide higher I/O performance for network traffic. Hypervisor bypass is the ability for a virtual machine to access PCIe adaptor hardware directly in order to reduce the overhead on host CPU.  Cisco UCS provide this feature with VN-Link in Hardware with VMDirectPath option and help to reduce the overhead for host CPU/memory for I/O virtualization. The virtual machine directly talks to Cisco virtual interface card and bypass the vmkernel to provide higher performance to network traffic. The current virtual interface card can scale up to 256 virtual interface cards, which means the most of the virtual machines can get PCIe adaptor on a single host.

Implications

1.The disadvantage is currently limited vMotion support on VMware hypervisor.

Back to Competition Main Page or Competition Submissions

Example Architectural Decision – Storage Protocol Choice for a Horizon View Environment

Problem Statement

What is the most suitable storage protocol for a Virtual Desktop (Horizon View) environment using Linked Clones?

Assumptions

1. VMware View 5.3 or later

Motivation

1. Minimize recompose (maintenance) window
2. Minimize impact on the storage array and HA/DRS cluster during recompose activities
3. Reduce storage costs where possible
4. Simplify the storage design eg: Number/size of Datastores / Storage Connectivity
5. Reduce the total solution cost eg: Number of Hosts required

Architectural Decision

Use Network File System (NFS)

Justification

1. Using native NFS snapshot (VCAI) offloads the creation of VMs to the array, therefore reducing the compute overhead on the ESXi hosts
2. Native NFS snapshots require much less disk space than traditional linked clones
3. Recomposition times are reduced due to the offloading of the cloning to the array
4. More virtual machines can be supported per NFS datastore compared to VMFS datastores (200+ for NFS compared to max recommended of 140, but it is generally recommended to design for much lower numbers eg: 64 per VMFS)
5. Recompositions/Refresh activities can be performed during business hours, or at Logoff (for Refresh) with minimal impact to the HA/DRS cluster, thus giving more flexibility to maintain the environment
6. Avoid’s potential VMFS locking issues – although this issue is not as important for environments using vSphere 4.1 onward with VAAI compatible arrays
7. When sizing your storage array, less capacity is required. Note: Performance sizing is also critical
8. The cost and complexity of a FC Storage Area Network can be avoided
9. Fewer ESXi hosts may be required as the compute overhead of driving cloning has been removed thus reducing cost
10. VCAI is fully supported feature in Horizon View 5.3

Implications

1. The Storage Array supports NFS native snapshot offload to enable the full benefit of NFS (VCAI clones) however all other benefits remain without VCAI support.

Alternatives

1. Use VMFS (block) based datastores via iSCSI or FC/FCoE and have more VMFS datastores – Note: Recompose activity will be driven by the host which adds an overhead to the cluster. (Not Recommended)

Competition Example Architectural Decision Entry 5 – New vSphere 5.x environment

Name: Anand Prakash
Title: Virtualization Specialist
Twitter: @ananprak32
Profile: VCA-WM, VCA-DCV , VCA-Cloud , VCP3, VCP-DCV 4,5

Problem Statement

This is new company which is virtualization to VMware vSphere 5.x environment. As they have been managing lot of physical environment, they want to carry forward those settings/practices in Virtual Environment:
1.Tiering of all disks allocated to virtual machines. Using different Tiers for different mount point inside Windows Guest. So that they can separate OS/Data/Application Log with different performance levels.
2.They want to use dedicated LUN’s for all the VMDK’s assigned to the Guest. Possibly using  RAID inside guest to enhance performance .
3.Having multiple of virtual NIC’s for Backup/Management/Data per virtual machine.
4.Creating larger machines with more memory and CPU so that applications runs faster. While creating virtual machines it is considered for future workloads (Minimum 5 years). This includes Memory/CPU/Storage.
5.Using traditional methods of DR/Backup in virtual environment. Having similar dedicated environment at DR site. The servers are running all the time so that they can used during and event of DR.
6.The environment has been designed and implemented to meet above requirements. Now they have are having multiple of performance issues and outages. The engineers spend most of their time in troubleshooting and fixing/reworking issues.
7.With so much investment they have hit the limit of consolidation. They have some legacy servers for which hardware warranty has expired and cannot be virtualized. Customer do not want to extend the warranty of these servers and it is difficult to find spares or upgrade them.

Assumptions

1.Environment is running VMware vSphere 5.0 (VMware vCenter 5.0 and ESXi 5.0)
2.There are 2 sites with 12 servers of following Specs:
HP ProLiant SL210t Gen8 Server with 512GB of Memory and 4 – 12 Core CPU, Embedded NIC – Intel’s x2 1G NIC on board
3.There are clusters made up 4 hosts each. There 2 clusters at primary site (1 Prod and 1 Non Prod). The other cluster at secondary site is dedicated to DR and not running any workload.
4.They are using Symmetrix VMAX storage array with VPLEX solution.
5.They have replication technologies to replicate data at secondary site.
6.Out of 2- 1 GB NIC’s, one is dedicated for vMotion, Management and other NIC is dedicated for virtual machine traffic.
7.Network is capable to supporting 10GB NIC’s.
8.The Non-Prod environment are not required to be running during DR.

Constraints

1.Organisations wants to remediate this issues with minimum investment and hardware purchases.
2.The Production and Non-Prod environment should be physically separate and should not communicate with each other.
Motivation

1.The environment should easy to manage and should not require too much time and effort to support it.
2.The environment should provide higher consolidation ratio and better performance.
3.They should be standard across environment and flexible to changing needs.
4.More network redundancy for failures and load balancing.
5.This should provide customer satisfaction and cost savings.
6.Provide better options to manage capacity.

Architectural Decision

1.Recommended to use of Automated Storage Tiering (AST) at Storage Array. There are policies which can be set at storage level of assign infrequently used data to slower, less-expensive SATA storage but allow it to be automatically moved to higher-performing SAS or solid-state drives (SSDs) as it becomes more active. We can create many different groups based on business need (Bronze/Silver/Gold and Replicated/Non-Replicated). Create Storage DRS cluster and simply deploy the virtual machines to that DRS cluster.
Note: Make sure you disable uncheck “Enable I/O metric for SDRS recommendations.” if you are using Automated Storage Tiering (AST) at Storage Array.
2.Recommended to use large 4TB LUN’s instead of using small LUN per vmdk. Move the virtual machines to appropriate SDRS Group. User single disks from datastore to create single drive.
3.Use single NIC’s per virtual machine and use network less backups method like VMware vSphere Data Recovery. No management and backup NIC’s.
4.Deploy VMware vCenter Operations in environment. Start analysis of whole environments and right size the virtual machine based on their active usage.  And right size machine based on analysis of vCenter Operations manager. Also if required use converter to right size disks (It is easy to increase it but you will need rework to decrease the disk allocation).
5.Buy additional 2 – Dual Port 10GB NIC’s and install them on all 12 servers. Then create 2 Addition virtual switches on ESXi hosts. Connect one uplink of each 10GB NIC to Prod Network and other uplink to Non-Prod Network. Have both onboard uplinks connected to Management and vMotion VMkernal interfaces. Use ESXi host profiles to standardize this configuration.
Once the above configuration are done to ESXi host we can run both Prod and Non-Prod load on Same ESXi host. Now instead of using backup/restore or data disk replication we can setup the replication of all large datastores. Move the Guest OS on these datastores which require DR. We will run the Non-Prod load on the cluster at DR sites. We can use VMware Site Recovery to configure DR Plan to shutdown Non-Prod Guests at DR site and power on the Production Guests.
Merge both Prod and Non-Prod Clusters in the same cluster at primary site. Setup high priority/Reservation to Production Guests.

Alternatives

1.You can use vSphere Storage Profiles to maintain Tiers or Storage I/O Control.
2.Leave as current configuration
3.Leave as current configuration
4.You can ask to use to use VMware Capacity Planner to review the environment.
5.Leave as current configuration

Justification

1.You can use vSphere Storage Profiles and SIOC in combination. But the use of storage profiles post implementation is tedious task. Also if you do not use storage profiles you will loose current configuration and location of disk tiers (During storage vMotion if advanced option is not used). This type of machines with Tiers are difficult to storage vmotion and recovery. When you take snapshot of this type of Virtual machine and all IO’s goes to working directory and the naming structure of snapshot files and base disks become non-standard. If there was the snapshot and virtual machine is crashes (Configuration is lost) it will not be easy to recover. Even if you dedicate the Gold tier (SSD Disk) for application you will not know if it is utilized fully.
2.If you use multiple disks with dedicated LUN you will hit the limit of LUN (256) and no. of paths (1024). Again every time you create a new disk you will loose some free space for VMFS Metadata. Also there will whole lot of free space which gets wasted. If you do not maintain free space your vCenter will be full of alerts. And it will be difficult to distinguish between genuine and false alerts. Hence you will be limited in how many virtual machines you can create. Creating RAID inside guest can invalidate the impact of RAID created at storage array.
3.When you configure the network less backup you do not need dedicated backup NIC. As data on all NIC’s travel through same uplink, there is no point in having dedicated virtual NIC’s for management and data. Also if have some data traffic issues which is unlikely on 10GB uplink you can manage the virtual machine using console. This is critical when you are using Nexus 1000v with limited no. of ports. You could hit the limit of port much before you hit the limit of memory and CPU
4.Using VMware Capacity Planner can help one but there is need for continuous capacity planing. So VMware Operations Manager is a better option. The virtual machines themselves are not aware that they are running virtual hardware and they think that all resource allocated to them are only meant for them and try to use it. But in reality they are shared with others (Hence the machine really need resources suffer). In virtual it is very easy to increase capacity so we only provision for current workload unlike physical.
5.With current configuration there is no redundancy for all the networks. As network is capable of 10GB network and onboard NIC’s are not capable of utilizing that. Also the current configuration requires dedicate host for prod and Non-Prod network. This setup will create standardization of ESXi hosts and make it possible to use for all Guests (Prod/Non-Prod). This can also give you flexibility to group similar (Window/Linux) family to group together based on host (If same type of Guest OS running then they can better utilize Memory TPS). Also if Guest OS are placed on same datastore back-end storage can make use of de-duplication and save lot of storage.

Implications

1.Simple easy management of Virtual machines as VMware admin does not require to spread machine over datastores and can use simple storage vmotion to move virtual machines. You will not need top maintain documentation or keep checking compliance of disk periodically. The storage array can manage load effectively by monitoring current usage. This will ensure all the Tiers are used efficiently and on need basis. Whenever there is low space on datastore cluster you add new datastore to cluster and it manages the re-balancing automatically.
2.More free space on the datastores. Combined free space of many machines results in more savings. There are no alerts in vCenter for free space usage.  More deployment of virtual machine and increased efficiency. Environment will have spare capacity to deploy more storage or larger disks machines if required.
3.This will increase the speed of backup and do not clog your network. This will also result in cost savings as you do need tradition agents for backing up individual virtual machine. This will also result in less utilization of memory and CPU of ESXi host. This will provide you more ports to deploy more virtual machines. This will increase consolidation ratio and effective use of physical resources. This will in turn help to recover initial cost of hardware early.
4.Based on analysis we can release the memory, CPU and storage from other virtual machines. This will result in additional resources for other virtual machines and reduced CPU wait times. This will also allow business to virtualize more servers and retire old hardware which cannot be upgraded. More savings for business which can be used to by 10GB NIC’s.
5.As we have host running plenty of memory and CPU, the network can become bottleneck. Deploying additional 10GB NIC’s will overcome this bottleneck. This will provided better performance and more consolidation and efficient use of hardware. This will make it possible to make use of hardware which was dedicated for DR and was using power and space. This will further result in cost saving and higher customer satisfaction.

Back to Competition Main Page or Competition Submissions