For decades, processing data meant running a job every night over what had happened the day before. Today, many businesses can't wait until tomorrow: fraud detection, recommendations, live dashboards. Understanding the difference between batch and real-time processing — and tools like Kafka — is core knowledge for a senior data professional.

Batch: processing in chunks

In batch, data is accumulated and processed in blocks, at set intervals — hourly, once a day. It's the model of traditional pipelines. It has high latency (you only see the result in the next cycle), but it's simple, cheap and easy to reprocess: if something goes wrong, you just run the batch again.

Streaming: event by event

In streaming, each event is processed as soon as it arrives — a purchase, a click, a sensor reading. Latency is seconds or less, allowing immediate reactions. In exchange, the system is more complex: it must run continuously, deal with out-of-order and late events, and it never "finishes".

The append-only event log

The heart of modern streaming is an append-only log: a sequence of events to which you only add at the end, never changing the past. Think of a bank statement — each transaction is a new line; you never erase a previous one. This durable log is what lets you re-read events and lets several consumers read the same stream independently.

Kafka: topic, partition, consumer

Kafka is the most widespread implementation of this log. Three concepts are enough to get started:

• Topic — a named channel of events ("orders", "clicks").
• Partition — each topic is split into partitions to scale (the same idea as partitioning); order is guaranteed within a partition.
• Consumer — whoever reads the topic, tracking its position (offset) to know how far it has processed.

Producers ──▶ Topic "orders"
                ├─ Partition 0: [e1][e4][e7]──▶ Consumer A
                ├─ Partition 1: [e2][e5][e8]──▶ Consumer B
                └─ Partition 2: [e3][e6][e9]──▶ Consumer C
                (each event append-only, order per partition)

Time windows

In an infinite stream, questions like "how many sales in the last minute?" require slicing time into windows. The most common: tumbling (fixed, non-overlapping windows, like "every 1 minute") and sliding (windows that slide and overlap). That's how you compute continuous metrics without waiting for the stream to "end" — because it never ends.

Delivery semantics

In distributed systems, messages can get lost or duplicated. There are three possible guarantees:

• At-most-once — may lose events, never duplicates. Fast, but risky.
• At-least-once — never loses, but may duplicate. The most common; it requires the consumer to be idempotent to tolerate repeats.
• Exactly-once — each event processed a single time. It's the ideal guarantee and the most expensive to implement.

The question isn't "which is better, batch or streaming", but "how fresh does the data need to be to create value?". If the decision can wait until the small hours, batch is simpler and cheaper. If every second counts, you pay the price of streaming.

In interviews they often ask you to explain at-least-once versus exactly-once and why idempotency matters in streaming. Day to day, this shows up when you decide whether a new feature truly needs real time or whether an hourly batch already does the job — and that choice defines much of the system's cost and complexity.

Conclusion

Batch processes chunks with high latency and simplicity; streaming processes event by event with low latency and more complexity. The append-only log — made concrete in Kafka with topics, partitions and consumers — is the foundation; windows slice time; and delivery semantics define which guarantees you get. Knowing how to choose between the two worlds, based on the freshness the business requires, is the crux of the decision.