Thanks for a great post!

Fill out the RSVP form and we’ll meet at 6pm EST at Blue State Coffee on Thursday,  July 1st. Bring a computer and let’s write some code.

View Larger Map

Problem

The Dream Castle Architectural Firm (D-CAF, among friends) wants a tool which it can use for very high-level prototyping of custom homes. They need it to be understandable by both computers and humans. It only needs to keep track of the following things:

• Houses – Each house has a name
• Floors – Each house has multiple floors. Each floor has a number
• Rooms – Each floor has multiple rooms and each room has a type

The only other requirement is that the DSL be translatable to plain English so that they can show it to customers. No problem.

Solution

The DSL for the high-level specification of custom houses will look like the following:

CustomHouse.build :home do
  floor(1) {
    room :den
    room :kitchen
  }

  floor(2) {
    room :bedroom
    room :bathroom
  }
end

That’s it. You specify that a house should be built, that it has a name, some floors and each floor has some rooms. Simple, easy and 100% pure Ruby.  Notice the cunning usage of both do/end and bracket notation for defining blocks. The outer block passed to CustomHouse#build uses do/end syntax while the blocks passed to House#floor use the bracket syntax. These could easily be reversed (or combined or whatever) but it makes it look pretty and helps to visually differentiate things so that you (the developer) and the user (the architect) can things more clearly. Plus, when you print the output (calling to_s on the instance of House which gets returned), you get the following wonderful text:

House named home has 2 floors.
Floor 1 has 2 rooms (den, kitchen)
Floor 2 has 2 rooms (bedroom, bathroom)

It’s a small house, don’t be a wiseacre. So, how can such a DSL be built?

Implementation

Let’s write the implementation for this together.  First, we’re going to start of with a module named CustomHouse.  In Ruby, modules are just classes so we’ll define a class method called build which will behave like a factory method.

module CustomHouse
  def self.build(name, &block)
    house = House.new(name)
    house.instance_eval(&block)
    return house
  end
end

As you can see, the method takes two params, a name and a block. The first thing that it does is create an instance of the House class (which we have not yet defined) passing in the name parameter. Second it calls instance_eval on the house passing in the block. This ventures into the territory of metaprogramming which is great fun but beyond the scope of this document, though I am sure others have used it for DSL construction (coincidentally, if you are interested in Ruby metaprogramming, buy this book). Suffice it to say that it executes the supplied block in the context of the House instance. Finally, the CustomHouse#build method returns the instance of House.

(There are those who believe that *any* type of eval is fundamentally evil. I have been taught this many times and I try to avoid using eval on an actualy string whenever possible. Still, someone with a better understanding of Ruby internals might be able to better explain if this in any better.)

Next, let’s define what the code for the House class looks like. As a note, it will be defined in the CustomHouse module, technically making the fully-namespaced name of the class CustomHouse::House.

class House
  attr_accessor :name, :floors

  def initialize(name = '')
    @name = name.to_s
    @floors = []
  end

  def floor(number, &block)
    fl = Floor.new(number)
    fl.instance_eval(&block)
    @floors < < fl
  end

  def to_s
    str = "House named #{@name} has #{@floors.length} floors.\n"
    @floors.each do |f|
      str << f.to_s
    end
    str
  end
end

The above code should appear relatively simple. The House class has only a few methods. First, it has a constructor function which takes a name and a block, setting the name as an instance variable and another instance variable, @floors, which is just an empty array. Next, it has a method called floor which takes a number and a block. The guts of this method should look familiar to you because it mimics almost exactly the build factory method defined on CustomHouse. Finally, it has a to_s method because of the requirement that the DSL be translatable to plain English for clients to check out.

If we can just dwell for a moment on the floor method, notice that, it too, takes a block and uses instance_eval. It then adds the newly-constructed instance of Floor to the array in @floor. Let’s look at the Floor class now.

class Floor
  attr_accessor :number, :rooms

  def initialize(number = 0)
    @number = number
    @rooms = []
  end

  def room(type)
    @rooms < < Room.new(type)
  end

  def to_s
    str = "Floor #{@number} has #{@rooms.length} rooms ("
    @rooms.each do |r|
      str += "#{r.type}, "
    end
    str.chop!.chop!
    str += ")\n"
    str
  end
end

There shouldn’t be anything confusing about the above code as it doesn’t use any sort of block eval. In fact, the only thing left to look at is the class for Room which is even less impressive.

class Room
  attr_reader :type

  def initialize(type)
    @type = type
  end
end

That’s it. Seriously, that’s the entire implementation of the DSL. You pass a block to CustomHouse#build which gets executed in the context of a new instance of House. The block calls the House#floor method with a block which in turn gets executed in the context of a new instance of Floor. The Floor#room method adds new Room instances to the class and that’s basically it.

Here’s all the code together with the example:

module CustomHouse

  def self.build(name, &block)
    house = House.new(name)
    house.instance_eval(&block)
    return house
  end

  class House
    attr_accessor :name, :floors

    def initialize(name = '')
      @name = name.to_s
      @floors = []
    end

    def floor(number, &block)
      fl = Floor.new(number)
      fl.instance_eval(&block)
      @floors << fl
    end

    def to_s
      str = "House named #{@name} has #{@floors.length} floors.\n"
      @floors.each do |f|
        str << f.to_s
      end
      str
    end
  end

  class Floor
    attr_accessor :number, :rooms

    def initialize(number = 0)
      @number = number
      @rooms = []
    end

    def room(type)
      @rooms << Room.new(type)
    end

    def to_s
      str = "Floor #{@number} has #{@rooms.length} rooms ("
      @rooms.each do |r|
        str += "#{r.type}, "
      end
      str.chop!.chop!
      str += ")\n"
      str
    end
  end

  class Room
    attr_reader :type

    def initialize(type)
      @type = type
    end
  end
end

h = CustomHouse.build :home do
  floor(1) {
    room :den
    room :kitchen
  }

  floor(2) {
    room :bedroom
    room :bathroom
  }
end

puts h

Try running it and see what happens! Then try writing other definitions for custom houses and experience the theoretical joy of the hypothetical architectural firm.

DSL construction techniques

For clarification and context, I’d like to share some other, smaller examples which build on this technique and demonstrate one more.

Behold, a DSL for feeding Pandas:

Panda.feed {
  nom :bamboo
  nom :chocolate
}

The implementation of this is both simple and straightforward.

class Panda
  def self.feed(&block)
    panda = Panda.new
    panda.instance_eval(&block)
  end

  def nom(food)
    #whatever
  end
end

Since the block is evaluated in the context of the new Panda instance, it has access to the Panda#nom method. For people deathly afraid of eval, there is this alternative syntax:

Panda.feed do |p|
  p.nom :bamboo
  p.nom :chocolate
end

Which is implemented with yield instead of instance_eval like so:

class Panda
  def self.feed
    yield Panda.new
  end

  def nom(food)
    # whatever
  end
end

For a wonderful and inspiring treatment of Ruby DSLs and associated patterns, see the most-excellent talk on the matter given by Jeremy McAnally at the 2009 Mid West Ruby Conference.

Conclusion

DSLs are a fantastic tool which can help to simplify complicated and repetitive tasks. Ruby is very good for creating DSLs but it is not the only good tool out there. I advise you look into the creation of DSLs with Scala and, the best DSL-creation tool there ever was, Lisp Macros.

I am interested in improving this tutorial for the benefit of those programmers who wish to learn about DSL construction but don’t know where to start.

Once it was completed, I looked around (grepped around) for the output file and was dismayed to discover that there weren’t any. After tinkering around for about an hour, I finally figured out that you can request shared libraries be built by providing an option to the configure script:

./configure --enable-shared

Shortly thereafter, I got the shared library version I needed and was on the road again. Just a useful tip which I hope can be of help to someone.

In HTML5, including any arbitrary attribute may be included in any element provided that it is prefixed with data- and doesn’t interfere with the rest of the standard. Anything. So I can do the following and have it be perfectly valid:

<div data-panda="bamboo"> Whatever <div>

How awesome is that?

Oh, and then there’s this.

I got to thinking about the topic of backups and was curious as to the state of research as well as backup tools. A great intro to so-called “backup theory” is available on the ‘Backup’ Wikipedia article and others have written on the subject (Google will verify that this is true). As it turns out, advances in storage have recently offered many new opportunities for improving the way that a given backup process might work.

Several distributed fault-tolerant filesystems are coming along quite nicely. Though the Google File System and the equivalent HDFS project which is part of Apache’s Hadoop have gotten much attention, there are other options. GlusterFS (follow GlusterFS on Twitter, perhaps?) and Ceph are two excellent examples of Free/Open Source Software projects which offer the compelling combo of fault tolerance and distributed storage. They each employ replication and have similar architectures insofar as they abstract individual machines into “chunks” or “blocks” and manage replication automatically. One interesting difference is that while GlusterFS exports filesystems “as-is” (see the docs for an explanation), Ceph exports entire block devices.

So then what about the theory and goals of backup? In my opinion, backing up is not enough. Having a backup plan and executing it perfectly doesn’t mean a thing if the data can’t be recovered.  (In fact this issue relates strongly to a larger discussion of reliability. Rather than focusing exclusively on total time spent in a failure state, individuals concerned with reliability would also do well to consider how fast a system can recover from those failures. If a system can be up and running again after only 5 minutes, then that system can go down 12 times before reaching an hour of downtime. If another system takes 20 minutes to rebound from a failure, then that system can only go down 3 times!) I haven’t yet gotten to thinking about the problem of restoration following failure for anything other than plain old files. For example, I backup my personal data to servers in Philadelphia and California in addition to an external hard drive in my apartment. I make careful use of  old stand-byes like tar, gzip and rsync along with a checksum utility like md5sum or, more recently, something in the SHA family. Plus I use the git revision control system for code and etckeeper for config. By backing up all of my configuration data in addition to my personal files, I make it so that I can easily return a given system to a usable state, if not restoring it perfectly. I have successfully restored systems by doing little more than a reverse rsync.

To be fair, I just spoke about about the few systems under my personal control which constitute a small and limited case. I back up to different locations which is good practice but my local copy is certainly not sufficient for anything industrial. If the disk breaks, I’m out of luck. Considering a larger-scale system is when GlusterFS, Ceph and others would come in handy. Obviously there are a great many books written about this topic but, for the purposes of discussion, if I were to build a platform for the reliable storage of huge amounts of data, my project would look something like this…

First, I would round up spare computers with room for extra disks. They machines would not need to be particularly fast or possess large amounts of memory. I’m not sure of the exact hardware requirements for either GlusterFS (vague wiki entry) or Ceph but it’s hard to imagine that they’d require huge amounts of anything but disk space. Anyway, if my organization had old desktops or something which were being replaced then they might be perfect candidates.

Second comes storage. It seems that at the time of this writing, one can purchase a 1TB hard disk for around US$85. Let us assume that 20 desktop machines could be procured and each had two spare disk slots. For around US$3500 (figuring 2 US$85 disks per machine and a little more for tax+shipping, etc.) one could buy 20TB of storage. Now, it’s not quite that simple as the fault-tolerance scheme in both GlusterFS and Ceph relies on replication. Assuming the accepted replication factor of 3 (a norm adhered to by the Google File System), that would reduce the 20TB storage block by a third  leaving around 6 and two thirds terabytes of fault-tolerant storage. Filesystems of this type usually require cluster control processes which (ideally, I think) reside on dedicated machines so an extra machine or two would also be required. I got a good explanation of how metadata servers work in GFS/HDFS by reading the HDFS design document, actually. Metadata servers and other control processes serve similar functions. Ceph documents are very explicit about not having a single point of failure whereas GlusterFS is not quite so adamant. I need some help figuring out if GlusterFS is as fault-tolerant in that respect.

For under US$4000, one could theoretically build over 6TB of fault-tolerant distributed storage (provided that spare machines are plentiful, something which shouldn’t be a problem for organizations with a semi-regular hardware replacement cycle). Now, as for the usage of that storage system for backup, it’s a different piece of the discussion. I’ve seen quite a few setups where a single-but-very-large server provides networked storage to a large number of users via something like samba or NFS. In this case, the big server is usually blessed with some complicated RAID arrangement which people put entirely too much faith in. No one ever listens that RAID is not a backup strategy. Personally, I don’t believe in hardware reliability because it seems silly to spend money trying to prevent hardware from failing when you know it’s going to break (eventually) anyway. I’m not saying that RAID isn’t useful because it *is* useful and has a place. What I am saying is that it’s not backup. So if an organization has a big network storage server then where does that get backed up to? It’s hard to backup 6 TB off site but it can (certainly) be done. However, if an organization is lucky enough to have multiple buildings then a distributed storage cluster like I described earlier would be an excellent addition to the overall backup infrastructure. Putting a few machines in different buildings and using it as a place to shadow the main file server and whatever else needs to be backed up would grant an added measure of security.

Snow days are a good time for thinking and my backup jobs went smoothly. However, I feel as if I have begun a track of study which might yield some good results. Granted, it’s not just about the technology (humans screw things up) but the ability to build reliable and fault-tolerant storage systems for massive amounts of data using only commodity hardware is a huge boon to users everywhere. This concludes my backup rant.

In reality, this has been a big month for JavaScript with Google open sourcing its internal JS toolkit, Closure along with much attention being paid to to projects like NodeJS (for network stuff) and CommonJS (for everything else).

The Google Chart API provides a rediculously clever way to get high-quality information graphics which are generated on the back of the clearly-amazing Google infrastructure. You just use the URL layout provided and it sort of just works. All types of charts can be created. It’s very nice if you’re willing to take the time to piece together the URLs in the proper format. There are some abstractions, though.

Then there’s flot (which I’m told is Swedish for “pretty”). Flot is a library written in JavaScript on top of jQuery which produces very nice charts inside a canvas element. The demos are quite gorgeous and it’s operation seems straightforward enough. As a side note, Lift has a built-in flot widget.