I got to thinking about the topic of backups and was curious as to the state of research as well as backup tools. A great intro to so-called “backup theory” is available on the ‘Backup’ Wikipedia article and others have written on the subject (Google will verify that this is true). As it turns out, advances in storage have recently offered many new opportunities for improving the way that a given backup process might work.

Several distributed fault-tolerant filesystems are coming along quite nicely. Though the Google File System and the equivalent HDFS project which is part of Apache’s Hadoop have gotten much attention, there are other options. GlusterFS (follow GlusterFS on Twitter, perhaps?) and Ceph are two excellent examples of Free/Open Source Software projects which offer the compelling combo of fault tolerance and distributed storage. They each employ replication and have similar architectures insofar as they abstract individual machines into “chunks” or “blocks” and manage replication automatically. One interesting difference is that while GlusterFS exports filesystems “as-is” (see the docs for an explanation), Ceph exports entire block devices.

So then what about the theory and goals of backup? In my opinion, backing up is not enough. Having a backup plan and executing it perfectly doesn’t mean a thing if the data can’t be recovered.  (In fact this issue relates strongly to a larger discussion of reliability. Rather than focusing exclusively on total time spent in a failure state, individuals concerned with reliability would also do well to consider how fast a system can recover from those failures. If a system can be up and running again after only 5 minutes, then that system can go down 12 times before reaching an hour of downtime. If another system takes 20 minutes to rebound from a failure, then that system can only go down 3 times!) I haven’t yet gotten to thinking about the problem of restoration following failure for anything other than plain old files. For example, I backup my personal data to servers in Philadelphia and California in addition to an external hard drive in my apartment. I make careful use of  old stand-byes like tar, gzip and rsync along with a checksum utility like md5sum or, more recently, something in the SHA family. Plus I use the git revision control system for code and etckeeper for config. By backing up all of my configuration data in addition to my personal files, I make it so that I can easily return a given system to a usable state, if not restoring it perfectly. I have successfully restored systems by doing little more than a reverse rsync.

To be fair, I just spoke about about the few systems under my personal control which constitute a small and limited case. I back up to different locations which is good practice but my local copy is certainly not sufficient for anything industrial. If the disk breaks, I’m out of luck. Considering a larger-scale system is when GlusterFS, Ceph and others would come in handy. Obviously there are a great many books written about this topic but, for the purposes of discussion, if I were to build a platform for the reliable storage of huge amounts of data, my project would look something like this…

First, I would round up spare computers with room for extra disks. They machines would not need to be particularly fast or possess large amounts of memory. I’m not sure of the exact hardware requirements for either GlusterFS (vague wiki entry) or Ceph but it’s hard to imagine that they’d require huge amounts of anything but disk space. Anyway, if my organization had old desktops or something which were being replaced then they might be perfect candidates.

Second comes storage. It seems that at the time of this writing, one can purchase a 1TB hard disk for around US$85. Let us assume that 20 desktop machines could be procured and each had two spare disk slots. For around US$3500 (figuring 2 US$85 disks per machine and a little more for tax+shipping, etc.) one could buy 20TB of storage. Now, it’s not quite that simple as the fault-tolerance scheme in both GlusterFS and Ceph relies on replication. Assuming the accepted replication factor of 3 (a norm adhered to by the Google File System), that would reduce the 20TB storage block by a third  leaving around 6 and two thirds terabytes of fault-tolerant storage. Filesystems of this type usually require cluster control processes which (ideally, I think) reside on dedicated machines so an extra machine or two would also be required. I got a good explanation of how metadata servers work in GFS/HDFS by reading the HDFS design document, actually. Metadata servers and other control processes serve similar functions. Ceph documents are very explicit about not having a single point of failure whereas GlusterFS is not quite so adamant. I need some help figuring out if GlusterFS is as fault-tolerant in that respect.

For under US$4000, one could theoretically build over 6TB of fault-tolerant distributed storage (provided that spare machines are plentiful, something which shouldn’t be a problem for organizations with a semi-regular hardware replacement cycle). Now, as for the usage of that storage system for backup, it’s a different piece of the discussion. I’ve seen quite a few setups where a single-but-very-large server provides networked storage to a large number of users via something like samba or NFS. In this case, the big server is usually blessed with some complicated RAID arrangement which people put entirely too much faith in. No one ever listens that RAID is not a backup strategy. Personally, I don’t believe in hardware reliability because it seems silly to spend money trying to prevent hardware from failing when you know it’s going to break (eventually) anyway. I’m not saying that RAID isn’t useful because it *is* useful and has a place. What I am saying is that it’s not backup. So if an organization has a big network storage server then where does that get backed up to? It’s hard to backup 6 TB off site but it can (certainly) be done. However, if an organization is lucky enough to have multiple buildings then a distributed storage cluster like I described earlier would be an excellent addition to the overall backup infrastructure. Putting a few machines in different buildings and using it as a place to shadow the main file server and whatever else needs to be backed up would grant an added measure of security.

Snow days are a good time for thinking and my backup jobs went smoothly. However, I feel as if I have begun a track of study which might yield some good results. Granted, it’s not just about the technology (humans screw things up) but the ability to build reliable and fault-tolerant storage systems for massive amounts of data using only commodity hardware is a huge boon to users everywhere. This concludes my backup rant.

Writing code

How best to write code? Are there IDEs which support or maybe even encourage Scala? Yes! Yes, there are!

• IDE plugins (Eclipe, NetBeans, IntelliJ)
• Emacs – and a post on how to put many Scala pieces together with Emacs.

Project management

How best to keep track of Scala projects? Well, everyone’s got their favorite way to build stuff (Make and cousins) and those will most likely work fine. However, if you want something a little more specialized there are several tools which can help.

• Simple Build Tool (sbt)
• Apache Maven

Testing

There is no one way to test Scala, and this is true of most langauges. However, Scala does include a test framework called SUnit. That being said, it’s slated for deprecation as soon as it can be replaced and removed. Also, the current thinking out there seems to be that SUnit sucks. So, you know what, kids? We aren’t going to let that bother us because there are many great frameworks out there which may be used to effectively test Scala. Since Scala compiles to Java bytecode, there shouldn’t be trouble testing Scala from Java or Java from Scala.

• ScalaTest
• Specs
• ScalaCheck (pseudo-clone of Haskell’s QuickCheck)
• JUnit – Oldie-but-goodie still works well from Java or from Scala (helpful list of assertions)

Documentation

Who doesn’t love documentation? I know that I sure do! To help satisfy my deep-seated urge to author code-centric exposition, there is scaladoc. Ever hear of javadoc? Yeah, it’s like that. Read about javadoc here and then check out the scaladoc manpage. If, for some reason you don’t like scaladoc’s output, you’re in luck because there is an alternative generator called vscaladoc.

Notes

This list is most likely incomplete and certainly not exhaustive. If someone would like me to add something, please provide the link and info in the comments.

The perils of clobbering namespaces for the sake of convenience in any language are well know. Now, in some languages, a use directive is not always the culprit when namespaces get clobbered. Languages which also have an as clause can be dangerous as well because they obscure the true origin or name of a given object.

Part of the reason is that using USE or one of it’s friends is dangerous due to the fact that programmers risk forgetting that they used it. This kind of error is not restricted to the USE statement, however. For example, in Lisp, global variables are typically enclosed in asterisks by convention. So what, right? Well, early on in Practical Common Lisp, Peter Seibel points out that redirecting something like *standard-output* to point to another source, it would be all too easy to forget to put it back! Just like it’s easy to screw up redirection by forgetting to put it back, USE in most languages makes it quite easy to make trivial mistakes.

Now, I am not suggesting that these examples are directly equivalent. I am, however, making the point that, in this day and age, we are well aware of how easy it is to make careless mistakes and USE directives only make it easier. Say we have an example instance of an SQL server and there are a couple of databases in there. We want to work with two of them (called panda and bamboo) each but we’ve got quite a lot to do with one. For an example such as this, databases like MySQL, provide USE only to be compatible with Sybase and even then only as a utility function.

In any case, we’re going back to the example. We’re at the prompt and we’re going to work with with our first database, panda. So to save ourselves trouble, we go forth with a USE panda; and then go about our business. Then, we’re ready to go ahead and delete a whole bunch of records from bamboo so we run our DELETE * FROM trees; and all should be well, right? WRONG! We forgot to switch to the right database by using USE bamboo; so now we’re in trouble. While this is a simplistic example, the world would be hard-pressed to find a competent programmer who’s never been tripped up by the most simple of errors.

It is for this reason that I am wary of USE and it’s friends. Unless there is a very good reason to USE USE, I suggest that programmers be very careful AS well AS when we USE AS. Together, they can be a tricky combo.

In my investigation, I came across some very enlightening sources of information on the overall discussion of language speed, code optimization and the tension between different paradigms. Please peruse the following:

• Compiling and Optimizing Scripting Languages – Google Tech Talk by one of the maintainers of phc, the PHP compiler.
• Compiling Dynamic Languages – Another Google Tech Talk focusing on optimization of Python.
• What To Know Before Debating Type Systems – From the neo-classics department, this essay by Chris Smith helped me gain a great deal of perspective with respect to type systems
• LLVM stuff – Google Tech Talk about LLVM 2.0, tons of LLVM-related papers and presentations
• A formally verified compiler back-end – Paper by Xavier Leroy that was too long for me to finish

Something missing from this list?

def test_ctime
  record = Record.new
  assert_equal(Time.now, record.ctime)
end

This failed because, believe it or not, the timers have too high a resolution! In fact, even evaluating (with the help of irb) Time.now == Time.now is false. This is evident by evaluating 5.times { puts Time.now.to_f } and seeing how the output differs by a tiny amount with each iteration.

I response to this, I decided to generalize and adapt my assertion to accept a diminished resolution with a tolerance of one second. assert_equal(Time.now.sec, record.ctime.to_i). This is still imprecise but it gets my point across.

I had resigned myself to accept this troublesome compromise until I went to write the test for mtime. My test for mtime updating looked like this:

def test_mtime_update
  record = Record.new
  record.update
  assert_operator(Time.now, :>, record.mtime)
end

I feel like I am compromising because I can’t make accurate comparisons and can’t figure out how to do this properly. It seems impossible and I don’t feel as if I have adequate coverage because I can’t verify timestampt at all. Perhaps I am doing something obviously wrong or testing for the wrong things. To make myself feel better I added an assert_instance_of(Time, record.ctime) and the same thing for mtime. Still, this has given me much frustration and I can’t imagine no one else has encountered this before.

I’m writing this as part of a multithreaded application where timestamp resolution could be quite crucial and I see my tests failing to give me proper coverage. Help?