Fill out the RSVP form and we’ll meet at 6pm EST at Blue State Coffee on Thursday,  July 1st. Bring a computer and let’s write some code.

View Larger Map

Problem

The Dream Castle Architectural Firm (D-CAF, among friends) wants a tool which it can use for very high-level prototyping of custom homes. They need it to be understandable by both computers and humans. It only needs to keep track of the following things:

• Houses – Each house has a name
• Floors – Each house has multiple floors. Each floor has a number
• Rooms – Each floor has multiple rooms and each room has a type

The only other requirement is that the DSL be translatable to plain English so that they can show it to customers. No problem.

Solution

The DSL for the high-level specification of custom houses will look like the following:

CustomHouse.build :home do
  floor(1) {
    room :den
    room :kitchen
  }

  floor(2) {
    room :bedroom
    room :bathroom
  }
end

That’s it. You specify that a house should be built, that it has a name, some floors and each floor has some rooms. Simple, easy and 100% pure Ruby.  Notice the cunning usage of both do/end and bracket notation for defining blocks. The outer block passed to CustomHouse#build uses do/end syntax while the blocks passed to House#floor use the bracket syntax. These could easily be reversed (or combined or whatever) but it makes it look pretty and helps to visually differentiate things so that you (the developer) and the user (the architect) can things more clearly. Plus, when you print the output (calling to_s on the instance of House which gets returned), you get the following wonderful text:

House named home has 2 floors.
Floor 1 has 2 rooms (den, kitchen)
Floor 2 has 2 rooms (bedroom, bathroom)

It’s a small house, don’t be a wiseacre. So, how can such a DSL be built?

Implementation

Let’s write the implementation for this together.  First, we’re going to start of with a module named CustomHouse.  In Ruby, modules are just classes so we’ll define a class method called build which will behave like a factory method.

module CustomHouse
  def self.build(name, &block)
    house = House.new(name)
    house.instance_eval(&block)
    return house
  end
end

As you can see, the method takes two params, a name and a block. The first thing that it does is create an instance of the House class (which we have not yet defined) passing in the name parameter. Second it calls instance_eval on the house passing in the block. This ventures into the territory of metaprogramming which is great fun but beyond the scope of this document, though I am sure others have used it for DSL construction (coincidentally, if you are interested in Ruby metaprogramming, buy this book). Suffice it to say that it executes the supplied block in the context of the House instance. Finally, the CustomHouse#build method returns the instance of House.

(There are those who believe that *any* type of eval is fundamentally evil. I have been taught this many times and I try to avoid using eval on an actualy string whenever possible. Still, someone with a better understanding of Ruby internals might be able to better explain if this in any better.)

Next, let’s define what the code for the House class looks like. As a note, it will be defined in the CustomHouse module, technically making the fully-namespaced name of the class CustomHouse::House.

class House
  attr_accessor :name, :floors

  def initialize(name = '')
    @name = name.to_s
    @floors = []
  end

  def floor(number, &block)
    fl = Floor.new(number)
    fl.instance_eval(&block)
    @floors < < fl
  end

  def to_s
    str = "House named #{@name} has #{@floors.length} floors.\n"
    @floors.each do |f|
      str << f.to_s
    end
    str
  end
end

The above code should appear relatively simple. The House class has only a few methods. First, it has a constructor function which takes a name and a block, setting the name as an instance variable and another instance variable, @floors, which is just an empty array. Next, it has a method called floor which takes a number and a block. The guts of this method should look familiar to you because it mimics almost exactly the build factory method defined on CustomHouse. Finally, it has a to_s method because of the requirement that the DSL be translatable to plain English for clients to check out.

If we can just dwell for a moment on the floor method, notice that, it too, takes a block and uses instance_eval. It then adds the newly-constructed instance of Floor to the array in @floor. Let’s look at the Floor class now.

class Floor
  attr_accessor :number, :rooms

  def initialize(number = 0)
    @number = number
    @rooms = []
  end

  def room(type)
    @rooms < < Room.new(type)
  end

  def to_s
    str = "Floor #{@number} has #{@rooms.length} rooms ("
    @rooms.each do |r|
      str += "#{r.type}, "
    end
    str.chop!.chop!
    str += ")\n"
    str
  end
end

There shouldn’t be anything confusing about the above code as it doesn’t use any sort of block eval. In fact, the only thing left to look at is the class for Room which is even less impressive.

class Room
  attr_reader :type

  def initialize(type)
    @type = type
  end
end

That’s it. Seriously, that’s the entire implementation of the DSL. You pass a block to CustomHouse#build which gets executed in the context of a new instance of House. The block calls the House#floor method with a block which in turn gets executed in the context of a new instance of Floor. The Floor#room method adds new Room instances to the class and that’s basically it.

Here’s all the code together with the example:

module CustomHouse

  def self.build(name, &block)
    house = House.new(name)
    house.instance_eval(&block)
    return house
  end

  class House
    attr_accessor :name, :floors

    def initialize(name = '')
      @name = name.to_s
      @floors = []
    end

    def floor(number, &block)
      fl = Floor.new(number)
      fl.instance_eval(&block)
      @floors << fl
    end

    def to_s
      str = "House named #{@name} has #{@floors.length} floors.\n"
      @floors.each do |f|
        str << f.to_s
      end
      str
    end
  end

  class Floor
    attr_accessor :number, :rooms

    def initialize(number = 0)
      @number = number
      @rooms = []
    end

    def room(type)
      @rooms << Room.new(type)
    end

    def to_s
      str = "Floor #{@number} has #{@rooms.length} rooms ("
      @rooms.each do |r|
        str += "#{r.type}, "
      end
      str.chop!.chop!
      str += ")\n"
      str
    end
  end

  class Room
    attr_reader :type

    def initialize(type)
      @type = type
    end
  end
end

h = CustomHouse.build :home do
  floor(1) {
    room :den
    room :kitchen
  }

  floor(2) {
    room :bedroom
    room :bathroom
  }
end

puts h

Try running it and see what happens! Then try writing other definitions for custom houses and experience the theoretical joy of the hypothetical architectural firm.

DSL construction techniques

For clarification and context, I’d like to share some other, smaller examples which build on this technique and demonstrate one more.

Behold, a DSL for feeding Pandas:

Panda.feed {
  nom :bamboo
  nom :chocolate
}

The implementation of this is both simple and straightforward.

class Panda
  def self.feed(&block)
    panda = Panda.new
    panda.instance_eval(&block)
  end

  def nom(food)
    #whatever
  end
end

Since the block is evaluated in the context of the new Panda instance, it has access to the Panda#nom method. For people deathly afraid of eval, there is this alternative syntax:

Panda.feed do |p|
  p.nom :bamboo
  p.nom :chocolate
end

Which is implemented with yield instead of instance_eval like so:

class Panda
  def self.feed
    yield Panda.new
  end

  def nom(food)
    # whatever
  end
end

For a wonderful and inspiring treatment of Ruby DSLs and associated patterns, see the most-excellent talk on the matter given by Jeremy McAnally at the 2009 Mid West Ruby Conference.

Conclusion

DSLs are a fantastic tool which can help to simplify complicated and repetitive tasks. Ruby is very good for creating DSLs but it is not the only good tool out there. I advise you look into the creation of DSLs with Scala and, the best DSL-creation tool there ever was, Lisp Macros.

I am interested in improving this tutorial for the benefit of those programmers who wish to learn about DSL construction but don’t know where to start.

In HTML5, including any arbitrary attribute may be included in any element provided that it is prefixed with data- and doesn’t interfere with the rest of the standard. Anything. So I can do the following and have it be perfectly valid:

<div data-panda="bamboo"> Whatever <div>

How awesome is that?

Oh, and then there’s this.

• Emacs concurrency (or lack thereof)
• XSLT for fun and profit
• Programming with distributed objects in Ruby
• Unit testing JavaScript

That is all.

For those who might not know, MapReduce is a novel data-processing system developed by Google for internal usage and described in their publication entitled MapReduce: Simplified Data Processing on Large Clusters. For the enlightened out there, it should be clear that the name and mechanism are derived from Lisp’s map and reduce functions. In any case, though Google’s implementation is proprietary, there have been several implementations based on their paper both written in and geared toward a variety of programming languages. Unfortunately, none of these are available in the Debian repositories. In all fairness, Debian does include CouchDB which uses map and reduce functions for generating views. However, it’s not a solution aimed at sorting and processing huge amounts of data, though it is an interesting and capable piece of software.

So, to try and get things moving, I have filed three Debian RFPs (Request For Package) for a few seperate MapReduce implementations.

• Hadoop – Probably the most well-known of the Free/Open Source implementations. Includes a distributed filesystem (HDFS), scaleable distributed database (HBase) and tools to get you going from start to finish. Hadoop is written in Java though it can interoperate with other languages (Scala, too). It’s a top-level project of the Apache Software Foundation and licensed under the Apache License 2.0 – http://hadoop.apache.org
• Skynet – A MapReduce implementation written in Ruby. It’s designed to be fault-tolerant and distrubuted, just like the big boys. Originally written for use at Geni.com and licensed under the MIT License – http://skynet.rubyforge.org/
• Disco – Though the implementation is itself written in Erlang, thus providing excellent distributed fault-tolerance, Disco jobs can be written in Python. It was developed as an in-house tool for rapid data analysis at Nokia and they seem to be quite keen on it. Disco is licensed under a modified BSD License. Page at http://discoproject.org/ and code at http://github.com/tuulos/disco/tree/master

Ok, there might be a few objections to my choices. Why did I leave out neat projects like GridGain, FileMap and BashReduce? Well, for starters, GridGain is another Java implementation that doesn’t seem (at least to me) to have the same momentum Hadoop does. FileMap and BashReduce, while novel, useful and fascinating, are not designed for use in networked environments and are therefore unsuitable for cluster situations. So then whey not MapSharp? Well, primarily because of all the Debian Mono debates going on right now (Gnome’s fail!) . I’ve done work in C# and it’s got some neat features but cool stuff doesn’t and will not ensure that users are not liable from patent litigation.

Also, it seems like those RFPs have some mistakes, so if anyone figures out how to edit them, let me know so I can clean them up.

In my investigation, I came across some very enlightening sources of information on the overall discussion of language speed, code optimization and the tension between different paradigms. Please peruse the following:

• Compiling and Optimizing Scripting Languages – Google Tech Talk by one of the maintainers of phc, the PHP compiler.
• Compiling Dynamic Languages – Another Google Tech Talk focusing on optimization of Python.
• What To Know Before Debating Type Systems – From the neo-classics department, this essay by Chris Smith helped me gain a great deal of perspective with respect to type systems
• LLVM stuff – Google Tech Talk about LLVM 2.0, tons of LLVM-related papers and presentations
• A formally verified compiler back-end – Paper by Xavier Leroy that was too long for me to finish

Something missing from this list?

Today, when I sat down to work on a small software project for work that was taking way too long, I realized that I should most likely just use a framework. It being a web-app, I immediately figured I should use Rails. After about 20 minutes, I realized that the task at hand was too simple for Rails and that something else might do me better. After seeking advice from Trek, I installed the Sinatra web framework and my journey began. It was fifth grade all over again.

With Sinatra on the back  (fully supported by Dreamhost, apropros) and jQuery on the front, I was able to advance beyond what I had accomplished in one day in just 3 hours. I had never used Sinatra before but I certainly will in the future.

Also, because Sinatra gives you very little policy with respect to how must do things, I had no problem writing my business logic in Sinatra, leaving me free to write my data-handling code and page generation without worry.  Just so you know, when you keep your components seperate in the way I just described, you can get all excited and call it MVC.  For views, Sinatra supports HAML, ERB (my fav) and Builder while for models you may do pretty-much whatever you want (Datamapper, ActiveRecord, Sequel).

Sinatra is really worth a look and I know that I’ll be relying on it heavily in the next few months.

def test_ctime
  record = Record.new
  assert_equal(Time.now, record.ctime)
end

This failed because, believe it or not, the timers have too high a resolution! In fact, even evaluating (with the help of irb) Time.now == Time.now is false. This is evident by evaluating 5.times { puts Time.now.to_f } and seeing how the output differs by a tiny amount with each iteration.

I response to this, I decided to generalize and adapt my assertion to accept a diminished resolution with a tolerance of one second. assert_equal(Time.now.sec, record.ctime.to_i). This is still imprecise but it gets my point across.

I had resigned myself to accept this troublesome compromise until I went to write the test for mtime. My test for mtime updating looked like this:

def test_mtime_update
  record = Record.new
  record.update
  assert_operator(Time.now, :>, record.mtime)
end

I feel like I am compromising because I can’t make accurate comparisons and can’t figure out how to do this properly. It seems impossible and I don’t feel as if I have adequate coverage because I can’t verify timestampt at all. Perhaps I am doing something obviously wrong or testing for the wrong things. To make myself feel better I added an assert_instance_of(Time, record.ctime) and the same thing for mtime. Still, this has given me much frustration and I can’t imagine no one else has encountered this before.

I’m writing this as part of a multithreaded application where timestamp resolution could be quite crucial and I see my tests failing to give me proper coverage. Help?

Anyway, after rewriting things and shrinking the code base down to size, I’m please to release the first version of RadiateFS, a networked filesystem written in Ruby and making use of the most-excellent FUSE bindings. Other than Ruby (tested using Ruby 1.8.7 on Debian Lenny) and FUSE itself, it has no other external dependencies. It used the DRb distributed object code bundled with the standard Ruby distribution.

This would not have been possible without the help and inspiration of a few exceptional individuals:  Erik Ordway and the folks on freenode’s #ruby-lang channel. In particular drbrain and raggi.

You can download a tarball here (v0.2b) and read the mildly-hilarious FAQ below.

Does it work? Yes. Well, sort of. It works well enough that I feel comfortable releasing it in accordance with the principal of “release early, release often”. Now that I’ve done the “early” bit, let’s see how the “often” part goes.

How do I install it? You’ll need a few things. On Debian-friendly systems, you can install the pre-requisites like so: sudo aptitude install ruby1.8 fuse-utils libfusefs-ruby1.8 and that should pull in what you need. After that, you can download the tarball, unpack it and go to town. Of course, you’ll need to do this on any system on which you want to use RadiateFS, whether it be client or server.

How do I use it? It’s rather simple really. Once you’ve got the required packages installed and the tarball unpacked, you can share a directory on the server like so:
ruby radiate-server.rb -d source/ -l 0 -s host -p 54321 In this case, we’ere sharing a directory named “source” using a loglevel of 0 on a host named “host” listening on port 54321. In the directory you ran this in, you’ll see a log named ‘radiate-server.log’ which, with a loglevel of 0 (debug), will have a whole lot of stuff in it. Then, on the client, you can run this to mount the remote directory: ruby radiatefs.rb -d test -s host -p 54321 to mount the remote directory over a local directory named test.

Why do directories act funny? Remember that whole “release early” bit? Well, it’s early and not everything works right. Right now, you can get at files for which you have a proper pathname no matter how deep in a shared directory tree you are. The problem comes when trying to get a directory listing for directories below the top level. Patches welcome.

Is it secure? Absolutely not. I haven’t even completely implemented support for the ACL functionality which comes wit DRb. Don’t use this over the internet or any insecure network.

Why on earth would you write this? First, I was interested by the idea of putting DRb and FUSE together. Then, it was for the challenge. Looking back, it was mostly for the lolz.

What is your favorite food? Toss up between good olives and good curry.

Why is it read-only? The way things are set up right now, the instance of the class which specifies the FUSE mapping is shared directly. Therefore, all methods invoked on the client are actually executed on the server. While I *could* technically get write support working, I’d rather wait and rewrite this piece to have the FUSE directory objects live on the clients and connect to a broker on the server which implements proper threading and synchronization. If I’m imagining things correctly, it’ll add a degree of scaleability as well.

Is it documented? Yes. You can run rdoc on the source and it’ll generate the documentation. Other than that, you can certainly go source diving as this might be a neat way for people to learn about DRB and/or FUSE.

How is it licensed? GPL.

What’s next? Well, I’d like to fix directory support and add auto-discovery using Rinda. Should be neat to enable clients to automagically find the server and connect to it. Could be cool, right?

I found a bug. What now? Please contact me and let me know about it so I can fix the issue. If you help me fix it, I’ll gladly give you credit for your contribution.