Once it was completed, I looked around (grepped around) for the output file and was dismayed to discover that there weren’t any. After tinkering around for about an hour, I finally figured out that you can request shared libraries be built by providing an option to the configure script:

./configure --enable-shared

Shortly thereafter, I got the shared library version I needed and was on the road again. Just a useful tip which I hope can be of help to someone.

I got to thinking about the topic of backups and was curious as to the state of research as well as backup tools. A great intro to so-called “backup theory” is available on the ‘Backup’ Wikipedia article and others have written on the subject (Google will verify that this is true). As it turns out, advances in storage have recently offered many new opportunities for improving the way that a given backup process might work.

Several distributed fault-tolerant filesystems are coming along quite nicely. Though the Google File System and the equivalent HDFS project which is part of Apache’s Hadoop have gotten much attention, there are other options. GlusterFS (follow GlusterFS on Twitter, perhaps?) and Ceph are two excellent examples of Free/Open Source Software projects which offer the compelling combo of fault tolerance and distributed storage. They each employ replication and have similar architectures insofar as they abstract individual machines into “chunks” or “blocks” and manage replication automatically. One interesting difference is that while GlusterFS exports filesystems “as-is” (see the docs for an explanation), Ceph exports entire block devices.

So then what about the theory and goals of backup? In my opinion, backing up is not enough. Having a backup plan and executing it perfectly doesn’t mean a thing if the data can’t be recovered.  (In fact this issue relates strongly to a larger discussion of reliability. Rather than focusing exclusively on total time spent in a failure state, individuals concerned with reliability would also do well to consider how fast a system can recover from those failures. If a system can be up and running again after only 5 minutes, then that system can go down 12 times before reaching an hour of downtime. If another system takes 20 minutes to rebound from a failure, then that system can only go down 3 times!) I haven’t yet gotten to thinking about the problem of restoration following failure for anything other than plain old files. For example, I backup my personal data to servers in Philadelphia and California in addition to an external hard drive in my apartment. I make careful use of  old stand-byes like tar, gzip and rsync along with a checksum utility like md5sum or, more recently, something in the SHA family. Plus I use the git revision control system for code and etckeeper for config. By backing up all of my configuration data in addition to my personal files, I make it so that I can easily return a given system to a usable state, if not restoring it perfectly. I have successfully restored systems by doing little more than a reverse rsync.

To be fair, I just spoke about about the few systems under my personal control which constitute a small and limited case. I back up to different locations which is good practice but my local copy is certainly not sufficient for anything industrial. If the disk breaks, I’m out of luck. Considering a larger-scale system is when GlusterFS, Ceph and others would come in handy. Obviously there are a great many books written about this topic but, for the purposes of discussion, if I were to build a platform for the reliable storage of huge amounts of data, my project would look something like this…

First, I would round up spare computers with room for extra disks. They machines would not need to be particularly fast or possess large amounts of memory. I’m not sure of the exact hardware requirements for either GlusterFS (vague wiki entry) or Ceph but it’s hard to imagine that they’d require huge amounts of anything but disk space. Anyway, if my organization had old desktops or something which were being replaced then they might be perfect candidates.

Second comes storage. It seems that at the time of this writing, one can purchase a 1TB hard disk for around US$85. Let us assume that 20 desktop machines could be procured and each had two spare disk slots. For around US$3500 (figuring 2 US$85 disks per machine and a little more for tax+shipping, etc.) one could buy 20TB of storage. Now, it’s not quite that simple as the fault-tolerance scheme in both GlusterFS and Ceph relies on replication. Assuming the accepted replication factor of 3 (a norm adhered to by the Google File System), that would reduce the 20TB storage block by a third  leaving around 6 and two thirds terabytes of fault-tolerant storage. Filesystems of this type usually require cluster control processes which (ideally, I think) reside on dedicated machines so an extra machine or two would also be required. I got a good explanation of how metadata servers work in GFS/HDFS by reading the HDFS design document, actually. Metadata servers and other control processes serve similar functions. Ceph documents are very explicit about not having a single point of failure whereas GlusterFS is not quite so adamant. I need some help figuring out if GlusterFS is as fault-tolerant in that respect.

For under US$4000, one could theoretically build over 6TB of fault-tolerant distributed storage (provided that spare machines are plentiful, something which shouldn’t be a problem for organizations with a semi-regular hardware replacement cycle). Now, as for the usage of that storage system for backup, it’s a different piece of the discussion. I’ve seen quite a few setups where a single-but-very-large server provides networked storage to a large number of users via something like samba or NFS. In this case, the big server is usually blessed with some complicated RAID arrangement which people put entirely too much faith in. No one ever listens that RAID is not a backup strategy. Personally, I don’t believe in hardware reliability because it seems silly to spend money trying to prevent hardware from failing when you know it’s going to break (eventually) anyway. I’m not saying that RAID isn’t useful because it *is* useful and has a place. What I am saying is that it’s not backup. So if an organization has a big network storage server then where does that get backed up to? It’s hard to backup 6 TB off site but it can (certainly) be done. However, if an organization is lucky enough to have multiple buildings then a distributed storage cluster like I described earlier would be an excellent addition to the overall backup infrastructure. Putting a few machines in different buildings and using it as a place to shadow the main file server and whatever else needs to be backed up would grant an added measure of security.

Snow days are a good time for thinking and my backup jobs went smoothly. However, I feel as if I have begun a track of study which might yield some good results. Granted, it’s not just about the technology (humans screw things up) but the ability to build reliable and fault-tolerant storage systems for massive amounts of data using only commodity hardware is a huge boon to users everywhere. This concludes my backup rant.

First and foremost, I enjoy the LKML Summary podcast put out by Jon Masters. He does an exceptional job of summarizing things and his side commentary is usually rather hilarious, actually.  I do wish he’d provide a touch more background on certain things as it can be hard to jump right into the high-level technical discussion on the mailing list. Masters’ radio-compatible voice is a pleasure to listen to in the car, even if each episode is only just under nine minutes long.

Next, I have Software Engineering Radio, which has some really neat discussions. The latest one is on APIs and then I’m going to listen to the one where they interview Odersky. The commentary on topics is quite lucid and I greatly appreciate the candor of guests when discussing specific issues. After all is said and done, the software community at large can be a political group (myself included) so it’s nice to have some really great technical discussions as well.

So far, that’s it. I want to also sample the OSNews podcast but haven’t gotten around to it yet. So with about 2 hours in the car every day, anyone know of anything else I should be listening to now that LugRadio is over? I’d be interested in some podcasts on the topics of emacs, Debian or Scala stuff.

• Emacs concurrency (or lack thereof)
• XSLT for fun and profit
• Programming with distributed objects in Ruby
• Unit testing JavaScript

That is all.

For those who might not know, MapReduce is a novel data-processing system developed by Google for internal usage and described in their publication entitled MapReduce: Simplified Data Processing on Large Clusters. For the enlightened out there, it should be clear that the name and mechanism are derived from Lisp’s map and reduce functions. In any case, though Google’s implementation is proprietary, there have been several implementations based on their paper both written in and geared toward a variety of programming languages. Unfortunately, none of these are available in the Debian repositories. In all fairness, Debian does include CouchDB which uses map and reduce functions for generating views. However, it’s not a solution aimed at sorting and processing huge amounts of data, though it is an interesting and capable piece of software.

So, to try and get things moving, I have filed three Debian RFPs (Request For Package) for a few seperate MapReduce implementations.

• Hadoop – Probably the most well-known of the Free/Open Source implementations. Includes a distributed filesystem (HDFS), scaleable distributed database (HBase) and tools to get you going from start to finish. Hadoop is written in Java though it can interoperate with other languages (Scala, too). It’s a top-level project of the Apache Software Foundation and licensed under the Apache License 2.0 – http://hadoop.apache.org
• Skynet – A MapReduce implementation written in Ruby. It’s designed to be fault-tolerant and distrubuted, just like the big boys. Originally written for use at Geni.com and licensed under the MIT License – http://skynet.rubyforge.org/
• Disco – Though the implementation is itself written in Erlang, thus providing excellent distributed fault-tolerance, Disco jobs can be written in Python. It was developed as an in-house tool for rapid data analysis at Nokia and they seem to be quite keen on it. Disco is licensed under a modified BSD License. Page at http://discoproject.org/ and code at http://github.com/tuulos/disco/tree/master

Ok, there might be a few objections to my choices. Why did I leave out neat projects like GridGain, FileMap and BashReduce? Well, for starters, GridGain is another Java implementation that doesn’t seem (at least to me) to have the same momentum Hadoop does. FileMap and BashReduce, while novel, useful and fascinating, are not designed for use in networked environments and are therefore unsuitable for cluster situations. So then whey not MapSharp? Well, primarily because of all the Debian Mono debates going on right now (Gnome’s fail!) . I’ve done work in C# and it’s got some neat features but cool stuff doesn’t and will not ensure that users are not liable from patent litigation.

Also, it seems like those RFPs have some mistakes, so if anyone figures out how to edit them, let me know so I can clean them up.

The thing should be less than 18mm thick and the logo moves when you reorient the screen. It runs Linux and boots directly into a custom WebKit-based browser.

It worked out for the better as I had been sort-of-almost-definately been meaning to rebuild that particular machine anyway. Since the hardware is fine (though the disk might die, soonish), I reinstalled Debian Squeeze from a nightly-build installer and took the opportunity to change a few things in my network’s setup.

The first major change I made was to switch all the machines on my network to use debtorrent instead of apt-proxy which had been behaving unreliably. In particular, apt-proxy had been randomly hanging after a few transfers, thus causing the machines using it to be unable to upgrade their packages or install new ones. So, in an attempt to fix it and give a little back to the community, I have installed debtorrent on my main server and configured my other machines to use it. So far, it’s working quite well, the download speed is rather fast and it caches packages so that other machines may download them.

The next thing I changed was to take a few more security measures than I normally do. I have been using various known strategies for some time now but a fresh start seemed like a good opportunity to tighten things up with a fresh install. First, the Securing Debian manual is required reading for any sysadmin and I re-read over it while waiting for lengthy processes to finish. Since my machines are already behind a firewall that only lets SSH traffic in and then only through to the server in question, my revised security checklist goes something like this:

1. Remove all RPC services: sudo aptitude --purge remove portmap nfs-common
2. Remove root login option (especially since I disable the root account) from /etc/ssh/sshd_config by making sure that the relevant line reads PermitRootLogin no and then restarting ssh
3. Installing some harden packages: sudo aptitude install harden-servers harden-clients harden-tools
4. Installing some helpful security packages: sudo aptitude install debsums logcheck denyhosts chkrootkit and then doing a dpkg-reconfigure on debsums to make sure it does a daily integrity check and altering the denyhosts config file to make it sync with the global denyhosts database (this helped cut down on automated ssh attacks tremendously)
5. One of the most important things to do is also to make sure that you get your local mail delivered so that you can see status reports. I do a sudo dpkg-reconfigure exim4-config to make certain everything is as I like it and that no holes are left open but I still get my system mail.
6. The last thing that I’ll do is to install nmap and scan myself to see what’s showing. For this particular box, I saw nothing but SSH and SMTP from the box itself and nothing but SSH from the outside. Good.

There might be a few other things which I do but I can’t recall them now. I would install SELinux but my understanding (according to the Debian Wiki) is that it’s still in the experimental stage so I won’t move on that just yet. Is there something huge and obvious that I’m forgetting security wise? Is a file-integrity checker going to be useful if I have constantly-changing and files and I am continually-updating packages?

The other major change is that I moved from the XFS filesystem back to ext3 with the intention of soon trying the upgrade-in-place features found in ext4 now that it’s got so many things which I liked about XFS. Since the Debian installer didn’t present me with an option to use ext4, this seemed like the best idea. Was I very wrong?

As a side note, I tried out weechat on the console for about an hour before immediately going back to ERC on Emacs because it integrates so well with my alltime favorite editor.