The advent of virtualization, changed many rules of technology -- what may apply to the physical world could be detrimental to its virtual equivalent. This is likewise true in terms of performance. Optimizing the performance of one virtual machine impacts the performance of others. In the end, tweaking for performance not only pays off big time, but does that in magnitudes of two-, three-fold or even more.
The above statements are true due to sharing of resources in the hypervisor level. If resources are freed by performance tweaks, it makes them available to others waiting for the same resource to be available for use.
The first of these tweaks is to ensure that the disk used by the virtual machine is aligned. Why is this important? To better understand the underlying infrastructure of the virtual filesystem, refer to the screenshot. (Illustration from http://www.yellow-bricks.com)
As seen from the diagram, the misalignment of the array, to the virtual filesystem, and, to the guest disk is very, very significant. A big performance degradation will occur due to the misalignment. What happens here is that, what could be seen as a single I/O operation on the guest operating system could end up as multiple I/O operations on the physical disk or array.
Translate the above performance degradation to multiple virtual machines and you see the drift. According to VMware (Reference):
Still, I/O could be a potential performance bottleneck if the disk alignment is not kept in check.
While on the same topic of I/O bottlenecks, Linux virtual machines could benefit from the proper I/O scheduler. Again, this is another arena where the physical rules and virtual rules differ.
Modern Linux kernels default to the "completely fair queuing" (or CFQ) scheduler. This is how the kernel deals with optimizing I/O. It re-orders I/O in such a manner that when the head moves across the platter, it does that in an orderly, sequential manner rather than moving back and forth to complete requests. The best illustration to this is an elevator, where it drops people to floors in an orderly sequential manner. Instead of dropping passengers as they push the corresponding floor buttons, what happens is the elevator drops people off at 2, then 5, then 6, before going to 11. The kernel has different methods of doing this.
Imagine that the hypervisor has its own elevator, the guest has its own elevator and the array has its own. When the guest performs that optimization, the hypervisor does the same, and the array does the same. Who then decides what is the best re-ordering or optimization? The hypervisor has no insight on the guest operating system (this time a Linux machine), meaning it can't tell the method used for optimization, or how they got re-ordered, if they indeed got re-ordered or why it came to that decision.
This kernel optimization has no impact on the virtual environment and in some situations could have a negative effect. Why? The hypervisor has no way of telling how long an I/O has been outstanding or how it got re-ordered and why it was that way on the guest level. Add the hypervisor elevator and that adds further latency to the already outstanding I/O request.
It would be best to leave all optimizations on the hypervisor level and turn off the elevator on the kernel level. This is done by having the string "elevator=noop" on the kernel line of the grub.conf file of Linux. The other way of turning it off is by choosing the noop scheduler, as recommended by VMware.
This is the default setting.
If you have screensavers running, those are detrimental to the performance of virtual machines. Disable them at once. I have seen a virtual machine with a load of 5.0 reduced to 0.0 simply by disabling the screensaver!
The other tweak you could perform is to change to runlevel 3. Running on a GUI (runlevel 5), consumes a lot of resources that would otherwise be free if running at runlevel 3.
Permanently change it on the file /etc/inittab.
By tweaking, you squeeze the best performace out of your current infrastructure and free up resources otherwise sapped by useless unnecessary processes or simple misconfigurations. The combination of the tweaks above may yield better results than applying one tweak alone.
The above statements are true due to sharing of resources in the hypervisor level. If resources are freed by performance tweaks, it makes them available to others waiting for the same resource to be available for use.
The first of these tweaks is to ensure that the disk used by the virtual machine is aligned. Why is this important? To better understand the underlying infrastructure of the virtual filesystem, refer to the screenshot. (Illustration from http://www.yellow-bricks.com)
As seen from the diagram, the misalignment of the array, to the virtual filesystem, and, to the guest disk is very, very significant. A big performance degradation will occur due to the misalignment. What happens here is that, what could be seen as a single I/O operation on the guest operating system could end up as multiple I/O operations on the physical disk or array.
Translate the above performance degradation to multiple virtual machines and you see the drift. According to VMware (Reference):
The degree of improvement from alignment is highly dependent on workloads and array types. You might want to refer to the alignment recommendations from your array vendor for further information.
Still, I/O could be a potential performance bottleneck if the disk alignment is not kept in check.
While on the same topic of I/O bottlenecks, Linux virtual machines could benefit from the proper I/O scheduler. Again, this is another arena where the physical rules and virtual rules differ.
Modern Linux kernels default to the "completely fair queuing" (or CFQ) scheduler. This is how the kernel deals with optimizing I/O. It re-orders I/O in such a manner that when the head moves across the platter, it does that in an orderly, sequential manner rather than moving back and forth to complete requests. The best illustration to this is an elevator, where it drops people to floors in an orderly sequential manner. Instead of dropping passengers as they push the corresponding floor buttons, what happens is the elevator drops people off at 2, then 5, then 6, before going to 11. The kernel has different methods of doing this.
Imagine that the hypervisor has its own elevator, the guest has its own elevator and the array has its own. When the guest performs that optimization, the hypervisor does the same, and the array does the same. Who then decides what is the best re-ordering or optimization? The hypervisor has no insight on the guest operating system (this time a Linux machine), meaning it can't tell the method used for optimization, or how they got re-ordered, if they indeed got re-ordered or why it came to that decision.
This kernel optimization has no impact on the virtual environment and in some situations could have a negative effect. Why? The hypervisor has no way of telling how long an I/O has been outstanding or how it got re-ordered and why it was that way on the guest level. Add the hypervisor elevator and that adds further latency to the already outstanding I/O request.
It would be best to leave all optimizations on the hypervisor level and turn off the elevator on the kernel level. This is done by having the string "elevator=noop" on the kernel line of the grub.conf file of Linux. The other way of turning it off is by choosing the noop scheduler, as recommended by VMware.
This is the default setting.
Change it for every [DEVICE].# cat /sys/block/[DEVICE]/queue/scheduler noop deadline anticipatory [cfq]
# echo noop > /sys/block/[DEVICE]/queue/scheduler # cat /sys/block/[DEVICE]/queue/scheduler [noop] deadline anticipatory cfq
If you have screensavers running, those are detrimental to the performance of virtual machines. Disable them at once. I have seen a virtual machine with a load of 5.0 reduced to 0.0 simply by disabling the screensaver!
The other tweak you could perform is to change to runlevel 3. Running on a GUI (runlevel 5), consumes a lot of resources that would otherwise be free if running at runlevel 3.
Permanently change it on the file /etc/inittab.
id:3:initdefault
By tweaking, you squeeze the best performace out of your current infrastructure and free up resources otherwise sapped by useless unnecessary processes or simple misconfigurations. The combination of the tweaks above may yield better results than applying one tweak alone.
