Writing code

How best to write code? Are there IDEs which support or maybe even encourage Scala? Yes! Yes, there are!

• IDE plugins (Eclipe, NetBeans, IntelliJ)
• Emacs – and a post on how to put many Scala pieces together with Emacs.

Project management

How best to keep track of Scala projects? Well, everyone’s got their favorite way to build stuff (Make and cousins) and those will most likely work fine. However, if you want something a little more specialized there are several tools which can help.

• Simple Build Tool (sbt)
• Apache Maven

Testing

There is no one way to test Scala, and this is true of most langauges. However, Scala does include a test framework called SUnit. That being said, it’s slated for deprecation as soon as it can be replaced and removed. Also, the current thinking out there seems to be that SUnit sucks. So, you know what, kids? We aren’t going to let that bother us because there are many great frameworks out there which may be used to effectively test Scala. Since Scala compiles to Java bytecode, there shouldn’t be trouble testing Scala from Java or Java from Scala.

• ScalaTest
• Specs
• ScalaCheck (pseudo-clone of Haskell’s QuickCheck)
• JUnit – Oldie-but-goodie still works well from Java or from Scala (helpful list of assertions)

Documentation

Who doesn’t love documentation? I know that I sure do! To help satisfy my deep-seated urge to author code-centric exposition, there is scaladoc. Ever hear of javadoc? Yeah, it’s like that. Read about javadoc here and then check out the scaladoc manpage. If, for some reason you don’t like scaladoc’s output, you’re in luck because there is an alternative generator called vscaladoc.

Notes

This list is most likely incomplete and certainly not exhaustive. If someone would like me to add something, please provide the link and info in the comments.

For those who might not know, MapReduce is a novel data-processing system developed by Google for internal usage and described in their publication entitled MapReduce: Simplified Data Processing on Large Clusters. For the enlightened out there, it should be clear that the name and mechanism are derived from Lisp’s map and reduce functions. In any case, though Google’s implementation is proprietary, there have been several implementations based on their paper both written in and geared toward a variety of programming languages. Unfortunately, none of these are available in the Debian repositories. In all fairness, Debian does include CouchDB which uses map and reduce functions for generating views. However, it’s not a solution aimed at sorting and processing huge amounts of data, though it is an interesting and capable piece of software.

So, to try and get things moving, I have filed three Debian RFPs (Request For Package) for a few seperate MapReduce implementations.

• Hadoop – Probably the most well-known of the Free/Open Source implementations. Includes a distributed filesystem (HDFS), scaleable distributed database (HBase) and tools to get you going from start to finish. Hadoop is written in Java though it can interoperate with other languages (Scala, too). It’s a top-level project of the Apache Software Foundation and licensed under the Apache License 2.0 – http://hadoop.apache.org
• Skynet – A MapReduce implementation written in Ruby. It’s designed to be fault-tolerant and distrubuted, just like the big boys. Originally written for use at Geni.com and licensed under the MIT License – http://skynet.rubyforge.org/
• Disco – Though the implementation is itself written in Erlang, thus providing excellent distributed fault-tolerance, Disco jobs can be written in Python. It was developed as an in-house tool for rapid data analysis at Nokia and they seem to be quite keen on it. Disco is licensed under a modified BSD License. Page at http://discoproject.org/ and code at http://github.com/tuulos/disco/tree/master

Ok, there might be a few objections to my choices. Why did I leave out neat projects like GridGain, FileMap and BashReduce? Well, for starters, GridGain is another Java implementation that doesn’t seem (at least to me) to have the same momentum Hadoop does. FileMap and BashReduce, while novel, useful and fascinating, are not designed for use in networked environments and are therefore unsuitable for cluster situations. So then whey not MapSharp? Well, primarily because of all the Debian Mono debates going on right now (Gnome’s fail!) . I’ve done work in C# and it’s got some neat features but cool stuff doesn’t and will not ensure that users are not liable from patent litigation.

Also, it seems like those RFPs have some mistakes, so if anyone figures out how to edit them, let me know so I can clean them up.