Every batch job I had written up to this point started from the same unspoken assumption: the data is all there, sitting in a file or a table, waiting to be read in whatever order I choose. Spark Streaming asks a different question. What if the data keeps arriving, indefinitely, and the job has to produce answers before it knows what comes next? I had touched this shape of problem before with a Python and Spark pipeline over MongoDB, but that pipeline treated Spark as a scoring step at the end of a socket read, not as the thing actually managing the stream. This time I wanted to understand what Spark Streaming does with a continuous input on its own terms.
Batch, micro-batch, and true streaming are not the same trade-off
Before writing anything, it is worth being precise about what “streaming” even buys you, because the term gets used loosely. Batch processing reads a bounded, known dataset, processes all of it, and produces one result at the end. Latency is whatever the whole job takes, throughput is usually excellent because the scheduler can optimise across the entire dataset at once. True event-at-a-time streaming processes each record the instant it arrives, with latency measured in milliseconds, at the cost of far more overhead per record and a much harder consistency story if a downstream system needs exactly-once guarantees.
Micro-batch sits between the two. Spark Streaming does not process each event individually. It collects everything that arrived within a fixed interval, say five seconds, batches it into an RDD, and runs an ordinary Spark batch job against that RDD. Latency is bounded by the batch interval rather than by record-by-record processing, and because each micro-batch is a real RDD, all the batch machinery Spark already has, fault tolerance, the DAG scheduler, caching, applies without needing a separate streaming engine underneath.
DStream: a sequence of RDDs, nothing more exotic
The core abstraction in Spark Streaming is the DStream, discretised stream, and once I saw what it actually is underneath, the whole API stopped feeling like a separate system bolted onto Spark. A DStream is a sequence of RDDs, one per batch interval, and every operation you call on a DStream, map, filter, reduceByKey, gets applied to each of those RDDs individually as it arrives. There is no separate streaming runtime learning new operators. It is the RDD API, called repeatedly, on a schedule.
I set this up first against a raw TCP socket, using nc (netcat) on one terminal to feed lines in manually and a Spark Streaming job on the other end reading them.
import org.apache.spark.streaming.{StreamingContext, Seconds}
import org.apache.spark.SparkContext
val ssc = new StreamingContext(sc, Seconds(5))
val lines = ssc.socketTextStream("localhost", 9999)
val words = lines.flatMap(_.split(" "))
val wordCounts = words.map(word => (word, 1)).reduceByKey(_ + _)
wordCounts.print()
ssc.start()
ssc.awaitTermination()ssc.socketTextStream opens a TCP connection to the netcat listener and produces a DStream of lines, one micro-batch every five seconds containing whatever text arrived in that window. flatMap, map, and reduceByKey read exactly like the RDD versions from a fortnight earlier, because that is precisely what they are, applied per batch. wordCounts.print() is a DStream output operation, the streaming equivalent of an action, and it is what actually triggers each micro-batch to run rather than sit unevaluated. Nothing computes until ssc.start() sets the scheduler running, and awaitTermination() blocks the driver so the job keeps consuming batches instead of exiting after the first one.
Typing a handful of words into the netcat terminal and watching a count appear five seconds later, unprompted, was a different experience to anything I had run before. No cell to re-execute. No file to re-read. The job just sat there, doing work on a clock, whether I typed anything or not.
Watching a folder instead of a socket
The second setup pointed Spark Streaming at an HDFS directory instead of a socket, which behaves differently in a way that tripped me up initially.
val ssc2 = new StreamingContext(sc, Seconds(10))
val fileStream = ssc2.textFileStream("hdfs://localhost:9000/streaming-input")
val errorLines = fileStream.filter(_.contains("ERROR"))
errorLines.count().print()
ssc2.start()
ssc2.awaitTermination()textFileStream does not read files already present in the directory at start time. It only picks up files that appear after the streaming context starts, checked once per batch interval, and it identifies “new” by comparing file modification timestamps against the last check. Dropping a file into the watched directory before calling ssc2.start() meant that file was silently skipped, not backfilled, which cost me a confused ten minutes assuming the job was broken before I realised the file had simply arrived too early. The fix was mechanical, start the context first, then write files into the folder, but the underlying lesson mattered more: a streaming source has an implicit notion of “now,” and anything that existed before “now” is not part of the stream by definition.
What a micro-batch actually goes through
Underneath both examples, the architecture is the same regardless of source. A receiver, running as a long-lived task on one of the executors, pulls data from the source, whether that is a socket or a watched directory, and buffers it. At the end of each batch interval, everything the receiver collected in that window gets turned into one RDD, and that RDD is handed to the ordinary Spark engine as a job, the same DAG scheduler and task scheduler used for any batch RDD.
This is why fault tolerance in Spark Streaming looks like fault tolerance in ordinary Spark rather than something new. If a task processing one micro-batch’s RDD fails partway through, Spark can recompute it from the RDD’s lineage the same way it would recompute any batch RDD, provided the receiver’s data was reliably stored first, which is what StorageLevel.MEMORY_AND_DISK_SER_2 on the receiver is for. Nothing about DStream processing invents a new failure model. It reuses the one Spark already had and applies it once per batch interval instead of once per job submission.
Windowed operations, and why the batch interval is not the only knob
Beyond mapping and filtering each batch independently, DStreams support windowed operations that span multiple batch intervals, useful when a five-second batch is too fine-grained for the actual question being asked.
val windowedCounts = wordCounts.reduceByKeyAndWindow(
(a: Int, b: Int) => a + b,
windowDuration = Seconds(30),
slideDuration = Seconds(10)
)
windowedCounts.print()reduceByKeyAndWindow combines results across a thirty-second window, sliding every ten seconds, which means the output at any point reflects the last thirty seconds of data, not just the last batch interval. This decouples two things that are easy to conflate at first: how often Spark checks for new data, the batch interval, and how much history a given calculation should consider, the window duration. Setting the batch interval to five seconds for responsiveness while windowing over thirty seconds for a more stable rolling count is a normal combination, not a contradiction, once the two are understood as separate knobs.
The overall shift this exercise settled for me is less about the API and more about what “done” means. A batch job finishes. A streaming job does not, by design, and every decision, the batch interval, the window size, the storage level for received data, is a trade-off between how quickly an answer becomes available and how much that answer is allowed to be revised as more data proves it incomplete. Micro-batch does not solve that tension. It just gives you a dial to turn, and lets you reuse the batch engine you already trust while you turn it.