Say you've implemented wordcount in MapReduce, and are looking to write unit tests against your workflow. The Hadoop consulting giant Cloudera defines current best practices in this article:

The current state of the art often involves writing a set of tests that each create a JobConf object, which is configured to use a mapper and reducer, and then set to use the LocalJobRunner (via JobConf.set(”mapred.job.tracker”, “local”)). A MapReduce job will then run in a single thread, reading its input from test files stored on the local filesystem and writing its output to another local directory.

This process provides a solid mechanism for end-to-end testing, but has several drawbacks. Developing new tests requires adding test inputs to files that are stored alongside one’s program. Validating correct output also requires filesystem access and parsing of the emitted data files. This involves writing a great deal of test harness code, which itself may contain subtle bugs. Finally, this process is slow. Each test requires several seconds to run. Users often find themselves aggregating several unrelated inputs into a single test (violating a unit testing principle of isolating unrelated tests) or performing less exhaustive testing due to the high barriers to test authorship.

This is insane. Written this way, unit tests meant to target application logic are dwarfed by the maintenance they impose.

Most real world MapReduce applications involve chained sequences of many queries; these flows are a nightmare to coordinate and test using the above methods. Perhaps for this reason, Cloudera recommends testing mappers and reducers separately:

your map() and reduce() calls should still be tested individually, as the composition of separate classes may cause unintended bugs to surface.

Again, this is totally batshit crazy. MapReduce queries are functions that operate on datasets. Imagine writing a Clojure function and testing its lines one at a time, in isolation; the approach makes no sense.

A well designed testing suite will examine each MapReduce query as the function that it is. Queries should be tested against a range of inputs, both in isolation and as components of more complicated workflows. It shouldn't matter how one's data is stored; query logic is independent of data storage, and should be tested as such.

I'd like to offer a more sane approach to MapReduce testing. In the following example, I'll write MapReduce query with Cascalog and test it using midje-cascalog, a thin wrapper I wrote over the Midje testing DSL.

Let's say you want to test a Cascalog workflow that examines your user datastore and returns the user with the greatest number of followers. Your workflow's top level query will generate a single tuple containing that user's name and follower-count. Here's the code:

(defn max-followers-query [datastore-path]
  (let [src (name-vars (complex-subquery datastore-path)
                       ["?user" "?follower-count"])]
    (cascalog.ops/first-n src 1 :sort ["?follower-count"] :reverse true)))

max-followers-query is a function that returns a Cascalog subquery. It works like this:

• The function accepts a path, (datastore-path) and passes it into a function called complex-subquery.
• complex-subquery returns a subquery that generates 2-tuples; this subquery is passed into name-vars.
  • name-vars binds this subquery to src after naming its output variables ?user and ?follower-count.
• first-n returns a subquery that
  • sorts tuples from src in reverse order by follower count, and
  • returns a single 2-tuple with the name and follower-count of our most popular user.

At a high level, the subquery returned by max-followers-query is responsible for a single piece of application logic:

• extracting the tuple with max ?follower-count from the tuples returned by (complex-subquery datastore-path).

A correct test of max-followers-query will test this piece of logic in isolation.