Sunday, April 04, 2010

Dell Server: PowerEdge R710 with Hyper-V will hang/freeze

Recently I encounter a case with below situation:

Product: Dell PowerEdge R710
RAM: 12GB ECC DIMM
Processor: 2x Quad Core Intel Processor
OS: Windows Server 2008 Enterprise R2 x64
Others: Hyper-V activated

Problem:
Customer mentioned that he have total 5 VMs running in this server. 4 of it will running all the time. He allocate 8.5GB for 5 VMs, and after 2-3 days, server will hang. He monitor this issue by his own, and found that Memory usage will reach 100% before it hang.
He submit this issue to me, and cause me study a lot of whitepaper, forum from Microsoft site to look for more information.

VMs Resource Allocation:
1. VM1 Memory=512MB HDD Space=80GB
2. VM2 Memory=2GB HDD Space=200GB
2. VM3 Memory=4GB HDD Space=300GB
2. VM4 Memory=2GB HDD Space=150GB

In a nutshell, he left 3.5GB RAM for Host which is totally insufficient to run in a optimal environment.
Maybe you'll ask: "Where is the 5th VMs?"
I ask the same to customer, but he told me the 5th VM in OFF mode all the time, will use when its needed. -.-"

Research:
After 1 day of studies, research, I manage to obtain the below info to customer. To configure Hyper-V with Optimal state, we have to follow the below formula:
To calculate the host RAM = (VMRamRequirementMB x #ofVMs) + (#of VMs x 32MBVMOverhead ) + (512MBHost)
To calculate VMRamRequirementMB, I found 1 statement from one of the Microsoft White Paper:

...the root partition must have sufficient memory (leave at least 512 MB available) to provide services such as I/O virtualization, snapshot, and management to support the child partitions.

A good standard for the memory overhead of each VM is 32 MB for the first 1 GB of virtual RAM plus another 8 MB for each additional GB of virtual RAM. This should be factored in the calculations of how many VMs to host on a physical server. The memory overhead varies depending on the actual load and amount of memory that is assigned to each VM.

Reference:
(Search for “Correct Memory Sizing”)

Therefore, I manage to come out the optimal resources allocation with above-mentioned formula:

1. VM1 = (1 x 32MB) + (79 x 8MB) = 664MB
2. VM1 = (1 x 32MB) + (198 x 8MB) = 1616MB
3. VM1 = (1 x 32MB) + (299 x 8MB) = 2424MB
4. VM1 = (1 x 32MB) + (149 x 8MB) = 1224MB

Total RAM for VMs = 5928MB
= 5.8GB

Host RAM = (5928MB) + (4 x 32MB) + 512MB
= 6.41GB

Result:

According to calculation, obviously server (host) doesn't have sufficient Memory because it need at least 6.4GB, and VMs already occupied 5.8GB (maybe more as I didn't count in the 5th VM that customer turn on when it is needed).

Temporary workout:

1. Reserved at least 6.4GB for Host. This can only avoid the Host from hang/freeze issue, but it may cause VMs couldn't run in Optimal state.

2. Reduce the size of HDD Space for the VMs.

Permanent solution:

1. Add in more RAMs as Widows Server 2008 Enterprise R2 able to support up to 32GB memory space.

2. Move at least 1 or 2 VMs to other Host to reduce the loads in this server.



0 comments: