Word count took me four files and about forty lines of Java to get working, for a computation that is a single line in Scala: words.groupBy(identity).mapValues(_.size). My first thought was that MapReduce was simply a worse tool. My second thought, after I actually traced through what those forty lines were doing, was that the verbosity was the point. Every step Spark hides behind a one-line combinator, MapReduce makes me write out by hand, and that made the underlying mechanics impossible to ignore.
Word count is the “Hello World” of distributed computing for a reason that has nothing to do with word counting. It has exactly enough structure, a map step that transforms input and a reduce step that aggregates by key, to expose the shuffle and sort machinery that every other MapReduce job depends on, without any domain complexity getting in the way.
The shape of a MapReduce job
A MapReduce job has three phases that run in a fixed order, and understanding what happens between the map phase and the reduce phase is most of what there is to understand about the model.
Map. The framework splits the input into chunks (input splits, roughly aligned to HDFS block boundaries) and runs one Mapper task per split. Each Mapper reads its portion of the data and emits key-value pairs. For word count, the Mapper reads a line of text and emits (word, 1) for every word in that line, regardless of what other Mappers running on other splits are doing.
Shuffle and sort. This is the phase nobody writes code for and everybody depends on. The framework takes every key-value pair emitted by every Mapper, groups all values that share the same key, sorts the keys, and routes each key (with its full list of values) to exactly one Reducer. This is also the expensive part. It means data moving across the network from wherever it was mapped to wherever it needs to be reduced, and on a job processing a large corpus that shuffle can dominate the total runtime.
Reduce. Each Reducer receives one key at a time, along with every value emitted for that key across the entire input, and produces the final output. For word count, the Reducer receives ("fox", [1, 1, 1]) and sums the list to emit ("fox", 3).
The Mapper
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Mapper;
import java.io.IOException;
import java.util.StringTokenizer;
public class WordCountMapper extends Mapper<Object, Text, Text, IntWritable> {
private final static IntWritable one = new IntWritable(1);
private Text word = new Text();
public void map(Object key, Text value, Context context)
throws IOException, InterruptedException {
StringTokenizer itr = new StringTokenizer(value.toString());
while (itr.hasMoreTokens()) {
word.set(itr.nextToken());
context.write(word, one);
}
}
}The four generic type parameters on Mapper are input key, input value, output key, output value. The input key here is Object because for a plain text file it is just a byte offset nobody cares about; the input value is a Text holding one line. Text and IntWritable are Hadoop’s own serialisable wrapper types, not plain Java String and int, because everything a Mapper emits has to be serialised, shipped across the network during the shuffle, and deserialised on the other end, and Hadoop’s Writable interface exists specifically to make that fast and predictable. Reusing the same word and one objects across calls to map(), rather than allocating fresh ones every line, is a deliberate micro-optimisation. A Mapper on a large split calls map() millions of times, and allocation pressure at that scale is not imaginary.
The Reducer
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Reducer;
import java.io.IOException;
public class WordCountReducer extends Reducer<Text, IntWritable, Text, IntWritable> {
private IntWritable result = new IntWritable();
public void reduce(Text key, Iterable<IntWritable> values, Context context)
throws IOException, InterruptedException {
int sum = 0;
for (IntWritable val : values) {
sum += val.get();
}
result.set(sum);
context.write(key, result);
}
}The Iterable<IntWritable> values argument is the entire list of 1s the shuffle phase collected for one key, already grouped, already sorted by key across the whole job. The Reducer never sees a partial view. By the time reduce() gets called for “fox”, every 1 emitted for “fox” by every Mapper across the entire cluster has already arrived. That guarantee is what the shuffle phase exists to provide, and it is also exactly why the shuffle is the phase that costs the most: guaranteeing full grouping before reduction starts means waiting for the slowest Mapper to finish before any Reducer can be certain it has seen everything for its keys.
Wiring it together and running it
A third class, the driver, configures the job: which classes are the Mapper and Reducer, what the output types are, and where the input and output paths sit in HDFS.
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
public class WordCountDriver {
public static void main(String[] args) throws Exception {
Configuration conf = new Configuration();
Job job = Job.getInstance(conf, "word count");
job.setJarByClass(WordCountDriver.class);
job.setMapperClass(WordCountMapper.class);
job.setReducerClass(WordCountReducer.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
FileInputFormat.addInputPath(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));
System.exit(job.waitForCompletion(true) ? 0 : 1);
}
}Compiling and submitting is its own small ritual, and I got it wrong twice before it stuck.
javac -classpath $(hadoop classpath) -d wc_classes \
WordCountMapper.java WordCountReducer.java WordCountDriver.java
jar -cvf wordcount.jar -C wc_classes/ .
hadoop jar wordcount.jar WordCountDriver /data/raw/corpus.txt /data/output/wordcountIf /data/output/wordcount already exists from a previous run, the job refuses to start rather than overwrite it, which caught me the first time and felt like friction until I noticed it is the same protection that stops a six-hour cluster job from silently clobbering someone else’s output.
Why the verbosity earns its keep
Forty lines of Java to count words looks absurd next to a Scala one-liner, until the one-liner’s execution plan is opened up and it turns out to contain the exact same three phases, just generated by Spark’s planner instead of written by hand. A groupByKey in Spark still triggers a shuffle. It still groups by key across partitions. It still has to wait for every upstream task to finish before the grouping is complete, for the same reason a Hadoop Reducer cannot start summing until it has seen every 1 for its key.
What the Java version buys, at the cost of that verbosity, is that none of this is hidden. When a Spark job runs slower than expected, understanding why usually means asking “where is the shuffle, and how much data is crossing it,” and having written a Mapper and Reducer by hand, emitted the intermediate pairs myself, and watched them get grouped and sorted before a single line of reduce logic ran, that question stopped being abstract. It became a specific phase in a specific pipeline that I had already built once, badly, on purpose, to see how it worked.
The other thing the exercise made concrete was cost. Reading the job’s counters after a run, bytes read, bytes written, records shuffled, gave me actual numbers to reason about instead of a vague sense that “shuffles are slow.” Once I could see how many bytes moved between the map and reduce phases for a given corpus, tuning a job stopped being guesswork and started being a question of which knob, fewer Reducers, a combiner to pre-aggregate on the map side, actually reduced that number.