Oh, and then there’s this.

It’s good to love what you love and even better to share it.

I got to thinking about the topic of backups and was curious as to the state of research as well as backup tools. A great intro to so-called “backup theory” is available on the ‘Backup’ Wikipedia article and others have written on the subject (Google will verify that this is true). As it turns out, advances in storage have recently offered many new opportunities for improving the way that a given backup process might work.

Several distributed fault-tolerant filesystems are coming along quite nicely. Though the Google File System and the equivalent HDFS project which is part of Apache’s Hadoop have gotten much attention, there are other options. GlusterFS (follow GlusterFS on Twitter, perhaps?) and Ceph are two excellent examples of Free/Open Source Software projects which offer the compelling combo of fault tolerance and distributed storage. They each employ replication and have similar architectures insofar as they abstract individual machines into “chunks” or “blocks” and manage replication automatically. One interesting difference is that while GlusterFS exports filesystems “as-is” (see the docs for an explanation), Ceph exports entire block devices.

So then what about the theory and goals of backup? In my opinion, backing up is not enough. Having a backup plan and executing it perfectly doesn’t mean a thing if the data can’t be recovered.  (In fact this issue relates strongly to a larger discussion of reliability. Rather than focusing exclusively on total time spent in a failure state, individuals concerned with reliability would also do well to consider how fast a system can recover from those failures. If a system can be up and running again after only 5 minutes, then that system can go down 12 times before reaching an hour of downtime. If another system takes 20 minutes to rebound from a failure, then that system can only go down 3 times!) I haven’t yet gotten to thinking about the problem of restoration following failure for anything other than plain old files. For example, I backup my personal data to servers in Philadelphia and California in addition to an external hard drive in my apartment. I make careful use of  old stand-byes like tar, gzip and rsync along with a checksum utility like md5sum or, more recently, something in the SHA family. Plus I use the git revision control system for code and etckeeper for config. By backing up all of my configuration data in addition to my personal files, I make it so that I can easily return a given system to a usable state, if not restoring it perfectly. I have successfully restored systems by doing little more than a reverse rsync.

To be fair, I just spoke about about the few systems under my personal control which constitute a small and limited case. I back up to different locations which is good practice but my local copy is certainly not sufficient for anything industrial. If the disk breaks, I’m out of luck. Considering a larger-scale system is when GlusterFS, Ceph and others would come in handy. Obviously there are a great many books written about this topic but, for the purposes of discussion, if I were to build a platform for the reliable storage of huge amounts of data, my project would look something like this…

First, I would round up spare computers with room for extra disks. They machines would not need to be particularly fast or possess large amounts of memory. I’m not sure of the exact hardware requirements for either GlusterFS (vague wiki entry) or Ceph but it’s hard to imagine that they’d require huge amounts of anything but disk space. Anyway, if my organization had old desktops or something which were being replaced then they might be perfect candidates.

Second comes storage. It seems that at the time of this writing, one can purchase a 1TB hard disk for around US$85. Let us assume that 20 desktop machines could be procured and each had two spare disk slots. For around US$3500 (figuring 2 US$85 disks per machine and a little more for tax+shipping, etc.) one could buy 20TB of storage. Now, it’s not quite that simple as the fault-tolerance scheme in both GlusterFS and Ceph relies on replication. Assuming the accepted replication factor of 3 (a norm adhered to by the Google File System), that would reduce the 20TB storage block by a third  leaving around 6 and two thirds terabytes of fault-tolerant storage. Filesystems of this type usually require cluster control processes which (ideally, I think) reside on dedicated machines so an extra machine or two would also be required. I got a good explanation of how metadata servers work in GFS/HDFS by reading the HDFS design document, actually. Metadata servers and other control processes serve similar functions. Ceph documents are very explicit about not having a single point of failure whereas GlusterFS is not quite so adamant. I need some help figuring out if GlusterFS is as fault-tolerant in that respect.

For under US$4000, one could theoretically build over 6TB of fault-tolerant distributed storage (provided that spare machines are plentiful, something which shouldn’t be a problem for organizations with a semi-regular hardware replacement cycle). Now, as for the usage of that storage system for backup, it’s a different piece of the discussion. I’ve seen quite a few setups where a single-but-very-large server provides networked storage to a large number of users via something like samba or NFS. In this case, the big server is usually blessed with some complicated RAID arrangement which people put entirely too much faith in. No one ever listens that RAID is not a backup strategy. Personally, I don’t believe in hardware reliability because it seems silly to spend money trying to prevent hardware from failing when you know it’s going to break (eventually) anyway. I’m not saying that RAID isn’t useful because it *is* useful and has a place. What I am saying is that it’s not backup. So if an organization has a big network storage server then where does that get backed up to? It’s hard to backup 6 TB off site but it can (certainly) be done. However, if an organization is lucky enough to have multiple buildings then a distributed storage cluster like I described earlier would be an excellent addition to the overall backup infrastructure. Putting a few machines in different buildings and using it as a place to shadow the main file server and whatever else needs to be backed up would grant an added measure of security.

Snow days are a good time for thinking and my backup jobs went smoothly. However, I feel as if I have begun a track of study which might yield some good results. Granted, it’s not just about the technology (humans screw things up) but the ability to build reliable and fault-tolerant storage systems for massive amounts of data using only commodity hardware is a huge boon to users everywhere. This concludes my backup rant.

In reality, this has been a big month for JavaScript with Google open sourcing its internal JS toolkit, Closure along with much attention being paid to to projects like NodeJS (for network stuff) and CommonJS (for everything else).

Enter, OpenStreetMap. Seeded with the geographical data made publicly available by various governments and public universities, OpenStreetMap provides Free (as in freedom, licensed under CC-BY-SA) map data to anyone who wants it. The data is usually in pretty good shape because the initial measurements are in good shape. However, things aren’t perfect. Lucky for the web, OpenStreetMap.org allows users to help improve the data in a number of ways.

First, users can upload GPS traces to help improve the quality of unmapped regions such as seriously-rural areas along with bike and hiking trails. Second, users are able to tweak the mapping data to correct errors. There are a number of ways to do this but OpenStreetMap.org has an online editor which lets you overlay OSM data onto sattelite imagery so you can move those roads, landmarks and the like into the right location. In about an hour, I had cleaned up much of my hometown and began to add local landmarks, parks and buildings. It’s quite easy.

The project itself seems off to a great start and the planet shows a fair bit of activity. In particular, I like the idea of mapping parties where people get together and work on a given area. This seems like a great way to give back to the community and I plan to float the idea at the next SCOSUG meeting.

Unfortunately, the building fire alarm went off around 12:30 forcing us out into the cold Connecticut air in a hustle of confusion. Out there we met another couple who lives far above us on the 17th floor. It turns out that there was some sort of fire in the penthouse and everything was resolved by New Haven’s finest. However,  that left us with the problem of trying to catch the elevator up when there were hundreds of humans in the crowd waiting to get lifted. We only live on the 4th floor so it was no problem for us to walk up to our floor but the other couple was left to fend for themselves being too tired (obviously) to climb 17 floors! I told them to come up to the 4th floor with us.

Therefore, I suggested that they try to catch an elevator moving in either direction from the 4th floor. I figured that the volume of people moving up would make the most hotly-saught-after commodity a space on an elevator so I figured that it’d better for them to get two spots on the way down and ride down so they could then ride back up. This sort of made sense to me but I could not forsee one major factor: drunk people coming down at the same time! So this morning, I found a few references to special elevator algorithms.

• Elevator algorithm – actually for disks but interesting
• The elevator algorithm – actually about elevators
• Heuristic algorithm to solve the multi-floor layout problem with the consideration of elevator utilization – paper

Was my gut instinct correct about catching a lift from the 4th floor?

The Google Chart API provides a rediculously clever way to get high-quality information graphics which are generated on the back of the clearly-amazing Google infrastructure. You just use the URL layout provided and it sort of just works. All types of charts can be created. It’s very nice if you’re willing to take the time to piece together the URLs in the proper format. There are some abstractions, though.

Then there’s flot (which I’m told is Swedish for “pretty”). Flot is a library written in JavaScript on top of jQuery which produces very nice charts inside a canvas element. The demos are quite gorgeous and it’s operation seems straightforward enough. As a side note, Lift has a built-in flot widget.