Apache Kafka: The Backbone of Event-Driven Systems

A customer taps Place Order.

In that instant, four things have to happen. Payment gets captured. Stock gets decremented. A confirmation email goes out. Dashboards update. And tomorrow, someone from the fraud team will walk over and ask to be told about orders too.

So: how do those services actually find out? That one question is where most architecture arguments start, and it is the question Kafka answers unusually well. Here is the whole thing — why event streaming exists, the four concepts you actually need, a broker you can run in five minutes, and a design problem to argue about at the end.

The shape of the problem

The obvious answer is that order-service calls the other four. It works. It works right up until it doesn't.

Because now order-service knows about payment, inventory, email, and analytics. It waits on all four. If the email provider is having a slow morning, checkout is having a slow morning. If analytics falls over at 2am, does the order fail? Somebody has to decide, and whatever they decide becomes a branch in the order code.

Then the fraud team shows up, and you edit order-service again.

The bottom half is the entire pitch. order-service writes one event to a topic and goes back to work. Four services read it on their own schedule. The fifth subscribes next week and nobody touches the producer.

Monoliths, microservices, and the bill that comes due

Worth naming the structural version of this, because it is where the problem comes from.

A monolith is one codebase, one deployment. Simple to start, easy to debug, everything a function call away. The bill arrives later: scaling teams gets messy, and a change in one corner redeploys the world.

Microservices flip that. Independent services, each owning its database and its deploy cycle. Teams scale. But you have traded one problem for another.

Microservices don't remove complexity. They relocate it — out of the codebase and into the network between services.

That network is the thing you now have to design. There are four common ways to do it:

• REST / HTTP — simple, synchronous request-response. The default, and genuinely fine for a lot of things.
• gRPC — fast, typed, synchronous RPC. Same shape as REST, better contracts and performance.
• Message queue — asynchronous, point-to-point. A message is consumed once and then it is gone.
• Event streaming (Kafka) — asynchronous, publish-subscribe, and replayable. The event is still there after it is read.

That last word is the one that matters, and it comes from one design decision.

Kafka is a log. That's the whole trick.

Kafka is a distributed event streaming platform — born at LinkedIn in 2011, open-sourced through Apache. It does messaging, stream processing, and storage at once, which sounds like three products but falls out of a single idea:

An append-only log. Events are stored in the exact order they arrive. Nothing is ever edited. Nothing in the middle ever changes. New events go on the end.

Look at what the consumers are doing there. Kafka is not keeping track of who has seen what — each consumer remembers its own position, its offset, and that is the only bookkeeping in the system.

Once you see that, the rest of Kafka stops being a list of features:

• A slow consumer cannot block a fast one. They are just at different offsets.
• A consumer that was offline resumes exactly where it stopped. Its offset didn't move.
• A brand-new consumer can read the last two weeks of history. The events never left.
• Replaying after a bug is rewinding an offset, not restoring a backup.

A queue can't do any of those, because a queue forgets a message the moment somebody reads it.

Partitions: the part people get wrong

A topic is a named stream of related events, like orders. One log would cap you at whatever a single machine can do, so each topic splits into partitions.

Which partition does an event land in? That is the partition key's job — and picking it is the highest-leverage decision you will make about a topic.

Same key, same partition, every time. Which leads to the sentence worth memorising:

Order is guaranteed per key, not across the topic.

Every event for order #4 arrives in order relative to every other event for order #4. It says nothing about ordering against order #7. And that is almost always the guarantee you actually wanted — nobody needs unrelated orders globally sequenced. Giving up the guarantee you don't need is precisely what buys the horizontal scale you do.

Choose the key badly and it shows up fast. Key on something unique per event and you shred ordering entirely. Key on something with one dominant value and every event piles into one partition while the others idle.

Consumer groups: scale out, or fan out

Two rules, and between them they cover every Kafka topology you will draw.

Consumers in the same consumer group divide the partitions between them — each partition read by exactly one consumer in that group. That is how you scale reads: add consumers, up to the partition count. The fourth consumer on a three-partition topic sits idle, which surprises people once.

A separate group reading the same topic gets its own full copy of every partition, with its own offsets. That is the fan-out.

Back at the start of this post, email-service, inventory-service and analytics-service were three groups on one orders topic — each reading every event, each at its own speed, none aware of the others. In production a cluster of brokers also replicates each partition, so losing a broker doesn't lose data. For learning, one broker is plenty.

Five minutes to a running broker

Modern Kafka runs in KRaft mode: no ZooKeeper, no config file. Two commands and you have a broker.

docker pull apache/kafka:4.3.1

docker run -d -p 9092:9092 --name broker apache/kafka:4.3.1

Create the topic from inside the container, with three partitions:

docker exec -it broker /bin/bash
cd /opt/kafka/bin

./kafka-topics.sh --create --topic orders   --partitions 3 --bootstrap-server localhost:9092

Now two terminals side by side. Terminal 1 — the producer:

docker exec -it broker /bin/bash
cd /opt/kafka/bin && ./kafka-console-producer.sh   --topic orders --bootstrap-server localhost:9092

Terminal 2 — the consumer:

docker exec -it broker /bin/bash
cd /opt/kafka/bin && ./kafka-console-consumer.sh   --topic orders --from-beginning --bootstrap-server localhost:9092

Type into the first, watch it land in the second:

> {"orderId": 101, "item": "coffee"}

Then do the experiment that actually teaches something. Kill the consumer. Produce a few more messages into the void. Start it again without --from-beginning. Everything you sent while it was dead arrives immediately, in order, because its offset never moved and the events were never going anywhere. That is the durability story in one command, and it is much more convincing than reading about it.

Who runs on this

• LinkedIn — where Kafka was born, out of exactly this problem.
• Netflix — real-time monitoring.
• Uber — trip and pricing events.
• Airbnb — search and experimentation.

Closer to home: think of Contentstack Launch's build and deployment logs as a live event stream. Each log line publishes once. The live-tail viewer, monitoring, and alerting each tail it independently, at their own pace. Somebody watching a deploy on hotel wifi does not slow that deploy down. Publish once, react everywhere.

Your turn: live cricket scores

Worth genuinely thinking about before you scroll.

Every ball bowled is an event. It has to reach a live scoreboard, push notifications, and an analytics/leaderboard service — independently, none of them blocking another. Then it is India vs. Pakistan: ten times the traffic, and zero dropped balls.

• What is your topic, and what is the partition key?
• How do you survive the spike?
• The notification service dies for two minutes. What happens?
• Scoreboard and analytics — one consumer group, or two?

There is no single right answer. Here is one that holds up.

Key on matchId. Every ball in a match stays ordered, and different matches spread across partitions — so you scale across a full tournament without ever reordering a single over. Keying on a ball id would destroy the ordering you care about.

Beat the spike to it. Add partitions and brokers before the big match. Repartitioning a live topic under record load is not a place you want to find yourself.

The outage is a non-event. notify-svc down for two minutes means its events sit in the log. It comes back, resumes from its last committed offset, catches up. Nothing lost — and, more to the point, nothing else even noticed.

Three groups, not one. All three services need every ball. One group would split the partitions between them and each would see a third of the match.

The four things worth keeping

• Topic → partition → offset. An ordered, append-only log per partition.
• The partition key decides ordering. Same key, same partition. Per key, not topic-wide.
• Groups divide, separate groups duplicate. Scale inside a group, fan out across them.
• Producers and consumers never know about each other. Kafka sits durably in between.

And the part that's easy to skip on the way out: Kafka is not a queue, and it is the wrong tool when a user has to immediately read their own write. Add-to-cart hits the database. Event streaming is for everything that happens because of the write — not the write itself.

Reach for it when several independent things care about the same fact, and when you'd rather add the sixth consumer than redeploy the producer. Get the log model right and the rest genuinely does follow.