I decided to blog this interesting topic that was brought up by a friend some time ago. How do you manage servers on the cloud? A simple decision such as using either public or private IP addresses could end in a disaster if not well managed.

For the past decade, the use of private IP addresses have been increasingly popular as public IP addresses in IPv4 are running out. It has also become a common security requirement to have servers in a “private” subnet sitting behind a firewall.

That sounds pretty OK for maybe 5, 10, even up till 50 servers. But what happens when you have 1000 servers?

Sure, you have VPNs from California to your headquarters in New York, and another VPN from your branch office in Singapore and Malaysia to a regional HQ in Hong Kong, and then a huge VPN from NY to Hong Kong carrying all trans-atlantic traffic. Oh, we’ll also need some monitoring systems in Hong Kong to monitor those in the SEA region, and another bunch in NY for USA. We’ll also need to get some servers in Germany to monitor our public services because we have quite a bit of customers there, and it costs too much for us to put a Point of Presence (POP) in Europe.

As your network grows, the VPN mess grows, so does monitoring and management. You have private IP addresses assigned by all sorts of systems integrators all over the world, some use 10.0.0.0/8, some use 172.16.0.0/12, some 192.168.0.0/16.

One day the Singapore office calls up. They can’t hook up to a server California. Oh where did that 10.23.55.0/24 route go?

Let’s face it. VPNs are a mess. They carry IP in a tunnel, encrypted or not, in an existing IP payload. Nevermind the overhead, but that also means routing is done TWICE. Why pay thousands on one, no, TWO (for redundancy, heh) Cisco routers to handle routing for all that VPN traffic within your internal networks when the job of getting the packet across the globe has already been done by a huge router sitting in the ISP?

Why even pay thousands for one, no, I beg your pardon, TWO firewalls with lots of RAM to hold NAT state tables for your public-facing servers?

Costs aside, these additional equipments also contribute to environmental factors such as power consumption.

So if you are a new startup venturing into the “cloud”, make a conscious decision to have all servers run public IP address. When time comes for management, monitoring, or even an IPv6 migration, you will be on the right path.

On a final note, IPv6 needs more support, especially from the government; imagine the day we could all throw out that Linksys sitting in our bedroom and have a public IP address on each computer. No more NAT, no more port forwarding, no more DMZ bullshit. Finally, UDP could realize its full potential. Cut the overheads from VPNs; seriously, somebody should do a research paper on how much encryped VPN payload could be saved by IPv6.

THIS ARTICLE IS PARTIALLY COMPLETE, I WILL CONTINUE TO EDIT IT…

What I forsee in the near future:
* ISPs no longer maintain large server farms for caches and e-mails
* Our computers automatically participates in a large P2P network sharing compute, memory and storage for a single application (e.g. e-mail)
* Our home appliances (game consoles, TV) runs P2P
* IPv6 will start to prevail with the need for public IP addresses

RAID as most know it is a technology that allows the grouping for multiple physical hard drives into one logical drive for either redundancy, performance or both. But the technology is not without caveats.

I found a good blog entry that explains in quite a bit of detail why one should stay away from FAKERAID/HOSTRAID and use Software RAID instead. If you need to build a high performance system, stick to a good hardware RAID vendor. Here are two takeaway pointers from the blog entry above:

1. “The FRAID driver is both an interface and, more importantly, the RAID logic.” What this means is that without a proper vendor supplied driver (the Linux HOSTRAID driver is not a proper vendor supplied driver), you will likely be running a RAID without you even knowing if it works. In our tests, we unplugged a disk and then attempted to rebuild the array. With the Linux HOSTRAID driver, we were unable to find out if the rebuild of the array was actually happening.
2. “15MBps! Welcome back to i486-era Programmed I/O (PIO)! … You have disks today of 50-80+MBps, and you can’t even break old Programmed I/O (PIO) Mode 4 or Mode 5 …” In summary, don’t run software RAID if performance matters. Otherwise if you run software RAID, stick to mirror (RAID-1) or stripe (RAID-0) or a combination of both (RAID-10). Don’t run software RAID 5 because of the way RAID 5 works; a degraded array will greatly impact performance.

LVM is another great technology but just like RAID, it is not without caveats either.

Ever attempted recovery of a filesystem on a disk with LVM using another system that’s also running LVM? What about cloning a disk with LVM from one system to another?

In both cases, I’ve bumped into numerous problems trying to get LVM to work. You’ll have volume groups name conflict (VolGroup00 is the default in RHEL/CentOS and probably some other distros) and booting problems with your Linux kernel complaining about not able to mount /dev/root. The latter would be fixed by running mkinitrd, but this is not common knowledge to most level 1/2 sysadmins.

Evaluate carefully if you need LVM. Don’t use it if you are sure that your server will almost never need more disk space. If you run a growing storage server, consider running Solaris 10 with ZFS instead.

Lastly, I found out that all RHEL/CentOS sets up GRUB incorrectly on systems running software RAID. This was discovered during a DR test. Try this –  remove the first mirrored disk from the server and attempt to boot the server. The secondary disk will fail to boot the O/S and you will see a GRUB message saying “GRUB Hard Disk Error”. The disk ain’t corrupted, it’s just that GRUB can’t find your disk.

Most people keep a rescue CD handy and reinstall GRUB during such an event, but here’s an immediate fix to reduce your potential downtime:

1. Edit /boot/grub/device.map and make both disks appear as (hd0), i.e.

   (hd0)   /dev/sda
   (hd0)   /dev/sdb

2. (Re)install GRUB on your second disk, i.e.

   # grub
   grub> device (hd0) /dev/sdb
   grub> root (hd0,0)
   grub> setup (hd0)

Source: http://grub.enbug.org/MirroringRAID

Justin Lee is a freelance Web 2.0 and Systems Consultant for Securlogic Singapore and currently works closely with core ISP engineering teams in Singapore during his day job.

You’d be surprised how long these discussions can take. It went on for about two hours and one dude finally broke the tension and stood up, and we thought he was headed for the washroom but he said instead, “what’s the point of all these? With all these costs, will the service even sell?”

He hit the nail. The room was quiet for a moment.

I’ve been providing freelance systems and network consultation for about 10 years now. Most of my consultation during the early parts of my career were rendered to SMEs. After the meeting ended, I sat down and tried to recall when was HA ever mentioned as a requirement when I was consulting SMEs. I concluded that I was either getting old and forgetful (not true!) or it was never mentioned.

On the other side of the world, ISPs are getting nailed in their butts by government regulations to maintain service uptime, and thus every project I’ve worked on with ISPs had HA as a default requirement.

You won’t believe how much effort people put into designing HA that they forget the basic requirement was to keep the service running, not the device. The final aim? To reduce risks that translate to business losses. Whilst having every extra piece of a device is a good-to-have, HA is really all about (truckloads of) money and balancing the returns.

But SMEs shouldn’t do without HA. Here’s some cheap (and probably free) HA solutions that SMEs can leverage:

• Redundant Array of Inexpensive Disks (RAID). Disks are mechanical and probably make up the highest percentage of componet failure in computers. Since disks are so unbelivably cheap and massive these days, there’s no reason why you shouldn’t RAID your disks. I would expect future desktops and laptops to run RAID as well.If a RAID controller adds significant cost, use software RAID.
• Linux Ethernet Bonding. Most servers come with two network interfaces these days. To reduce the possibility of a port failure (either on your switch or server), use Ethernet bonding in active-standby mode.
• Redundant Power Supplies. Power supplies are the second most commonly failed component as they are subjected to environmental factors such as power surges. If a sever with redundant supply is over budget, consider buying a spare power supply on standby off eBay.
• Virtual Router Redundancy Protocol (VRRP). This is a great technology to keep your network running. If your network setup permits, run the Vyatta open source router instead of the typical junk SOHO router. In fact, run two copies of them on VMware and enable VRRP. When one device goes off, VRRP automatically swings you over to the active device.
• Virtualize, virtualize, virtualize. There’s a ton of virtualization solutions out there. Virtualization can help you reduce your recovery time ten folds. You don’t need to keep the same hardware  just to get a service up. In fact, you can even temporarily restore an important service into a Desktop PC!

Justin Lee is a freelance Web 2.0 and Systems Consultant for Securlogic Singapore and currently works closely with core ISP engineering teams in Singapore during his day job.

The Crisis of Credit Visualized from Jonathan Jarvis on Vimeo.

Meanwhile, the virtualization infrastructure is due for an upgrade (we’re still on VMware Server 1.x). I’m evaluating VMware ESX 3i, VMware Server 2.0 (on CentOS 5.2), Citrix XenServer and Sun xVM VirtualBox and I’ll just drop a few quick pointers for those who are going though the same process as I am.

• Both VMware ESX 3i and Citrix XenServer installs directly on bare metal and will only work if your server is listed in their Hardware Compatibility List (HCL).
• Both VMware ESX 3i and Citrix XenServer have small footprints that installs quickly on a server with minimal configuration. Most of the time network information was the only configuration needed.
• Both VMware ESX 3i and Citrix XenServer required an additional Windows node for installation of management utilities. This can be your desktop/laptop and shouldn’t be much of a problem to most people unless you’re a Mac guy.
• Both VMware ESX 3i and Citrix XenServer depends on your hardware RAID controller for RAID. They cannot perform software RAID and does not support FAKERAID/HOSTRAID controllers as well. (There is a workaround for Citrix but we did not test it as it is not an officially supported feature.)
• Both VMware ESX 3i and Citrix XenServer have awesome management tools that provide real-time information of memory usage, CPU usage, etc.
• oth VMware ESX 3i and Citrix XenServer could readily connect with an iSCSI SAN storage. Unfortunately, we did not have the hardware available to test how easily this could be done.
• Citrix XenServer is compatible with more “whitebox” hardware (i.e. non-branded server hardware) than VMware ESX 3i.
• VMware ESX 3i is supposedly capable of higher workloads* than VMware Server 2.0.
• Citrix XenServer is supposedly blazing fast* when used to run Linux guests.
• Citrix XenServer required a CIFS or NFS share to host ISO images for installation of guest O/S which we found very inconvenient. Isn’t virtualization supposed to reduce your hardware count?
• Citrix XenServer (being based off Xen) had to install a custom kernel into your Linux guest O/S. We did not have an AMD-VT capable processor and thus was unable to verify if this was the case for those with CPUs built with virtualization support.
• Citrix XenServer (being based off Xen) was unable to run Windows as a guest O/S since we did not have an AMD-VT capable processor.
• VMware Server 2.0 provides a web interface that allows remote management (including console access via a browser plug-in for Mozilla and IE) but this web interface is not compatible across different versions.
• VMware Server 2.0’s web interface requires a Java container (Apache Tomcat 6.x) to be installed and really sucks memory!
• Both xVM VirtualBox and VMware Server 2.0 requireed a base O/S. This required more time and effort to set up than VMware ESX 3i and Citrix XenServer.
• Both xVM VirtualBox and VMware Server 2.0 will be supported on most hardware that Linux or Windows supports except if your CPU is not in their HCL.
• xVM VirtualBox can be installed on Solaris 10 as a host O/S, allowing you to leverage the power of ZFS and possibly Solaris Zones (native performance).
• xVM VirtualBox requires development packages to be installed in RHEL5. (We didn’t try installing it on Solaris yet).
• xVM VirtualBox does not have the remote management capabilities of VMware ESX 3i or VMware Server 2.0. It works more like a client app and requires that your server runs a GUI (X11), somewhat like VMware Player.

Verdict? VMware Server 2.0 fits our requirements best. Here’s why:

1. No FAKERAID/HOSTRAID support. We need software RAID. Software RAID gives us control, and allows easy monitoring and data recovery. If you have a goofy FAKERAID/HOSTRAID controller like the nVidia MCP series on the Sun Fire X2100, you are usually better off with software RAID.
2. Data recovery. I am skeptical about data recovery on a failed VMware ESX 3i or Citrix XenServer. Since the disks are using proprietary file systems, data recovery can potentially be a big problem.
3. Remote management. The ability to manage from the web is a plus, although we had instances of the browser console plug-in failing across different VMware Server 2.0 builds, we hope this issue gets ironed out soon. The lack of any proper form of network-based management simply kicks Sun xVM VirtualBox out of the league. (VNC? No thanks.)

On a side note, we love the management tools provided by Citrix XenServer and VMware ESX 3i. If not for the lack of hardware compatibility, we would have probably gone with VMware ESX 3i and Citrix XenServer second in the league. The preference for VMware ESX 3i goes towards the ease of installation/deployment of a VM. Citrix XenServer required an additional CIFS/NFS/FTP/HTTP server for installation which was not a resource we had readily available for production use at the datacenter.

In conclusion, all four products have their own strengths. You should consider them carefully and decide which suits your environment best.

Edited on March 24, 2009: Added Citrix XenServer and more reasons why we choose VMware Server 2.0.

Justin Lee is a freelance Web 2.0 and Systems Consultant for Securlogic Singapore and currently works closely with core ISP engineering teams in Singapore during his day job.

So what makes the Internet different?

The Internet is a utility that is capable of muti-tasking; almost anything can be carried via the Internet. Voice over IP (VoIP) and Skype are replacing traditional phones, IPTVs and YouTube are replacing broadcast TV channels, Instant Messaging and e-mails are replacing post mail and faxes. Many traditional businesses that were once profitable are slowly going out of business over the past 10 years (think Digital Cameras and film developers). It’s a scary phenomenon.

If only power and water were carried on the Internet.

This exponetial growth of the Internet needs some form of control, and the dirty truth is that ISPs that are overselling their bandwidth are rate limiting subscribers based on applications and protocols. Remember the days when we had to configure our browsers to use a proxy server? Not anymore. We are forced (albeit unknowingly) to use a proxy, whether you like it or not.

The fact is, bandwidth is expensive, and no ISP will let you leech them off for free. However, the other unknown fact is that most people really don’t need that 12mbps plan they subscribed. A typical home user is sufficient with a 512kbps-1mbps broadband. The reason why we think our 12mbps broadband isn’t enough is that we never really get even near 10% of that bandwidth (at least in Singapore) during peak hours, all thanks to our neighbours leeching off YouTube and P2P networks!

Don’t get me wrong. P2P is a good thing when used for the right reasons - what’s better than having your neighbours send you a file than having it downloaded across the Atlantic? P2P is the Internet’s way of bandwidth recycling.

HTTP and FTP protocols are ancient. Just like how plain HTML has evolved into AJAX sites and how IRC evolved into modern-day Instant Messengers, it’s time IETF did something new for the upcoming decade.

With the proliferation of bandwidth demanding content, we cannot continue to rely on traditional protocols. The future in the Internet is in P2P, but the current P2P community is constantly working against the will of ISPs. ISPs are constantly investing in cache technologies to have content served up locally, only to have the next release of some P2P client with a new 128-bit RSA encryption to prevent you from getting caught downloading an illegal copy of the latest blockbuster.

Can P2P be our savior in 2010? I guess we’ll just have to wait and see.

Justin Lee is a freelance Web 2.0 and Systems Consultant for Securlogic Singapore and currently works closely with core ISP engineering teams in Singapore during his day job.