I installed Scala 2.11.6 expecting to skim the syntax over a weekend and get straight to Spark. That did not happen. An hour into the REPL, I typed var total = 0 out of ten years of habit, and the compiler let it through without complaint. Then I tried to reassign a val two lines later and it stopped me cold: reassignment to val. That one error message did more to change how I think about distributed processing than any Spark tutorial managed in the following month.
Coming from Python and R, I had never needed to treat mutability as a language-level contract. A variable was a variable, and whether it changed was a matter of style, not enforcement. In Scala, val and var split that assumption into two distinct things, and the split is not decorative. It is close to the same split that decides whether a computation can run safely across a hundred machines at once, or whether it has to run in one place, in order, because something somewhere depends on shared state changing underneath it.
Why the underlying language, not just the API
Spark is written in Scala. The Python API, PySpark, wraps a JVM process running Scala code, and calls through pyspark.rdd eventually cross into that JVM to execute. I could have gone straight to PySpark and skipped Scala entirely. Plenty of people do exactly that and get useful work done. But I wanted to understand what Spark’s execution model was built to optimise for, not just how to invoke it, and that meant meeting the model in the language it was designed around.
The habit I needed to unlearn first was the for-loop-with-an-accumulator reflex. In Python, I would count words by opening a mutable dictionary, looping over lines, splitting each one, and incrementing counts as I went. That pattern works fine on one machine because there is exactly one copy of the dictionary and exactly one thread touching it in a defined order. Hand the same loop to ten machines and the mutable dictionary becomes ten different dictionaries with no agreed way to merge them, or one shared dictionary with a locking problem nobody wants to own. Functional style sidesteps the question by never mutating shared state in the first place. Each transformation produces new data instead of changing old data, so there is nothing to coordinate.
The combinators Spark’s API is built from
Four functions carry most of the weight in both Scala and Spark’s RDD API: map, flatMap, filter, and reduce. Once these are second nature, the RDD documentation stops looking like a new API and starts looking like the same four operations wearing a distributed-systems costume.
val lines = List(
"the quick brown fox",
"jumps over the lazy dog",
"the fox runs fast"
)
val words = lines.flatMap(_.split(" "))
// List(the, quick, brown, fox, jumps, over, the, lazy, dog, the, fox, runs, fast)
val longWords = words.filter(_.length > 2)
// drops "the" (length 3, kept), keeps everything else length > 2
val wordCounts = longWords
.groupBy(identity)
.mapValues(_.size)
// Map(fox -> 2, quick -> 1, brown -> 1, jumps -> 1, over -> 1, lazy -> 1, dog -> 1, runs -> 1, fast -> 1, the -> 3)map applies a function to every element and returns one output per input. flatMap does the same thing but then flattens nested results, which is exactly what splitting a line into words needs: each line produces a list of words, and flatMap collapses the list of lists into one flat list rather than leaving me with a list of lists I would then need to flatten by hand. filter keeps elements that satisfy a predicate and drops the rest. reduce (and its close relative fold) combines all elements into a single result using an associative function.
The detail that mattered most to me was reduce requiring an associative combining function. 1 + 2 + 3 gives the same answer regardless of grouping: (1 + 2) + 3 or 1 + (2 + 3). That property is not a mathematical nicety. It is the precondition that lets Spark split a reduce across partitions, compute partial results independently on each executor, and then combine those partial results in any order, on any node, without changing the answer. A non-associative combining function would force every partition’s result through a single, ordered pipeline, and the whole point of splitting the data across a cluster falls apart.
Immutability is the part that carries over
val bindings cannot be reassigned. Most of the standard collection operations, map, filter, flatMap among them, return new collections instead of modifying the one they were called on. Once I stopped fighting this and started leaning on it, a few things about Spark stopped being surprising.
An RDD is immutable. Calling .map() on an RDD does not change it, it returns a new RDD describing the transformation, and the original stays exactly as it was. This is the same behaviour as calling .filter() on a Scala List. Spark did not invent this idea. It borrowed it wholesale from the language it is written in, because immutable data is safe to send across the network to a different machine without anyone worrying that some other thread is midway through changing it. Closures in Scala capture variables by reference in a way that makes mutable state genuinely dangerous once that closure gets shipped somewhere else to run, which is precisely what happens every time Spark serialises a function and sends it to an executor.
Tail recursion and why the JVM cares
Scala encourages recursion over explicit loops for anything that walks a structure, and the JVM has a fixed stack size that will overflow on deep, ordinary recursion. The @tailrec annotation tells the compiler to verify that a function is tail-recursive, meaning the recursive call is the last thing that happens, with no pending work stacked on top of it, so the compiler can rewrite it into a loop under the hood with no growing call stack.
import scala.annotation.tailrec
@tailrec
def sumTo(n: Int, acc: Long = 0): Long =
if (n == 0) acc else sumTo(n - 1, acc + n)If I write a recursive function that is not actually tail-recursive and mark it @tailrec anyway, the compiler refuses to build, which caught two of my own mistakes before they became a stack overflow on a dataset large enough to matter. This is a small language feature, but it says something about the mindset the whole ecosystem is built around: correctness checked before the job runs, not discovered four hours into a cluster job when the accumulated stack finally gives out.
What the imperative version costs you
I rewrote the word-count example the way I would have written it in Python or Java: a mutable HashMap[String, Int], a for loop over the lines, an if check for existence, an increment.
The imperative version is not wrong. It runs fine on one machine, on data that fits in memory, in a single thread. The problem is that none of those three conditions survives the move to a cluster. A mutable map cannot be split across ten JVMs. A for loop with side effects has no obvious boundary where Spark could insert a shuffle or a stage break. The functional chain, by contrast, is already expressed as a sequence of independent transformations, and Spark’s scheduler reads that structure directly: narrow transformations like map and filter that need no data movement between partitions, and wide transformations like groupBy that force a shuffle. The chain of combinators is not just a nicer way to write the same logic. It is the same shape the execution planner needs to see in order to decide what can run in parallel and what cannot.
The type system, which felt restrictive for the first two weeks, turned out to be doing the same job at compile time that the functional style does at runtime: catching a class of mistake before it reaches a cluster, rather than after a job has burned through six hours of executor time and failed on row four million with a type mismatch that Python would have let through silently until that exact moment.