How I reduced data center costs by 60%, increasing capacity by 100%
January 6, 2014 by Gustavo Franklin Nóbrega in: Web Hosting ArticlesThe quick answer: I moved my website from a big data center in the USA to iWeb, and virtualized every server with XenServer. Here’s how my team and the iWeb sysadmins did it.
I work at eSapiens Internet, a small IT company that runs a social network in Brazil. Today we have around 500,000 unique visitors per day, and 250,000,000 page views per month. Keeping all our web servers and databases servers running well together is not an easy task.
Before October 2013, we had more than 30 dedicated servers connected by a 1Gbps network and a few shared iSCSI storage volumes. All these servers ran CentOS with nGinx above it. Backend web servers ran php-fpm and backend database servers PostgreSQL.
All services were very well configured and we were getting the optimal performance that a dedicated server could provide. But we still thought this might not be the best solution we could get, so we looked to a new solution, which I’ll describe now.
A new solution
After some research we decided to virtualize all our dedicated servers using Citrix XenServer. We made the choice by doing some tests between vmWare and XenServer, finding the latter to have the best cost/benefit ratio. On the storage side, we chose DSSv7 from Open-E. It’s a great storage software solution, Linux based and fully compatible with XenServer. It provided us with replicated high availability iSCSI storage volumes and, even better than that, an active-active cluster solution.
This image shows our solution today. As you can see, the solution is a quite simple.
Basically we have six application servers (AppServers) running Citrix XenServer 6.2, two storage servers running Open-E DSS v7, a couple of firewalls and a VPN server. Connecting all this hardware we have two 10 Gigabits switches, configured with LACP protocol.
This solution design was a joint effort between teams from eSapiens and iWeb, and the support offered by iWeb was crucial to choosing them over another data center. We discussed the design with iWeb for several days before the project, as well as during the deployment of servers, and without the support of the iWeb team we would not have had the same success. Today we have high availability at so many levels (storage server, xenserver, firewalls, switches) that any hardware failure will not stop us.
Tips for success
Here are some tips we have to share about the configuration of some of ours virtual servers:
Since we have a lot of simultaneous access, we built a PostgreSQL cluster with seven VMs, using one master node and six slaves nodes. To replicate data we used Streaming Replication, which is natively PostgreSQL and works very well, and to balance the requests we use pgPool. Our database VMs are the largest that XenServer support, each one have 128GB RAM and 16 vCPUs.
The biggest challenge we had was IO. Because of the XenServer architecture, only vCPU0 process all IO at the VM level. To get the maximum performance we didn’t add iSCSI storage volumes directly to the VM from the storage server, but rather connected the iSCSI storage volume from storage to XenServer and then to VM. You might think that this would jeopardize performance because we add another element (XenServer) between storage server and VM, but this element has eight CPUs that balance IO thought IRQ and that’s what matters. Another tip is to use the taskset program (in Linux) to set a group of programs, like PostgreSQL process, to use all others vCPUs except vCPU0, reserving this vCPU for IO subsys only.
The results
In terms of results, this project reduced data center costs by 60% (even taking all XenServer and DssV7 licenses in to account) and we estimate that our capacity to serve web pages has increased by at least 2x (maybe 3x) that of the older dedicated structure. This result was only made possible by the great prices offered by iWeb for equipment and the great crew that helped us throughout the project.
Gustavo Franklin Nóbrega is Chief Operations Officer at eSapiens. Find him here on LinkedIn.