Derailment
A Train Of Thought That Jumped The Track

I'm getting excited about the speaker list at Surge 2010. Lots of top talent, and all will be talking about challenges faced, mistakes made, and lessons learned in building scalable systems. I am particularly looking forward to hearing Bryan Cantrill talk about building enterprise solutions from commodity components, which is a big part of what I do at OmniTI.

Bryan has some fascinating stories from his days at Sun Microsystems, and he's an amazing speaker who has been described as "Tigger on speed". I'm hoping for a reprise of his characterization of a hard drive that unexpectedly reset when a high LBA was requested.

As any admin knows, OS upgrades can be painful. Despite the best intentions and efforts by the vendor, bad things happen. This is especially true with kernel upgrades. It's one thing to run a blanket "yum update" or "apt-get upgrade" on your average Linux desktop, but even the tamest patch can ascend to truly ulcer-inducing levels when it applies to a critical system that can ill afford any extended downtime.

Read the rest of this story...

At the $DAYJOB, we have a cluster of build systems that we use to test source trees on various platforms, including FreeBSD, Linux and Solaris. Some of these machines are fairly old, and have relatively puny hard drives by current standards, yet they continue to do their job just fine. We've simply begun to run low on local storage, as the source trees grow and developers need to work with multiple branch and tag checkouts.

Rather than try to boost local storage with extra drives (not a surefire solution everywhere), we focused our attention on a shared storage solution. The idea was to reuse some decommissioned hardware and create enough storage for every build system to be able to mount its home directories and give developers the space they need to do their work.

Read the rest of this story...