Kafka for .NET Developers: Why Distributed Systems Need It, and How to Start

Reading time: about 10 minutes

The problem Kafka solves

Picture a small online store. A customer clicks “Place Order,” and the Order service has to:

1. Reserve stock in the Inventory service
2. Charge the card through the Payment service
3. Send a confirmation through the Email service
4. Tell the Warehouse service to start packing
5. Push the sale into the Analytics system

The first version is usually the obvious one: the Order service calls each of these directly. It works, until the email provider has a slow afternoon and checkout starts timing out. Or until someone asks for a loyalty-points feature, and now the Order service, which should only care about orders, needs a sixth call and another deployment.

This is tight coupling. The Order service knows too much about everyone else, and any one failure can spread to the rest. Most “microservices” that turn out painful are really distributed monoliths with this problem.

Martin Fowler’s article What do you mean by “Event-Driven”? is worth reading to see how varied this idea is. The core is simple, though: instead of telling other services what to do, a service announces what happened (“Order #1042 was placed”) and lets others react.

Kafka is the most widely used platform for doing that at scale.

What makes Kafka different from a queue

If you’ve used RabbitMQ or Azure Service Bus, your first question is probably “isn’t this just another message queue?” The difference is that Kafka is built around a log.

In a traditional queue, a message is removed once someone consumes it. In Kafka, messages are appended to a log and stay there for a retention period you configure (days, weeks, or forever). Consumers just keep a bookmark of how far they’ve read.

That one design choice has big consequences:

• A new service can start from the beginning of history and catch up.
• If you ship a bug in a consumer, you can fix it and reprocess the events.
• Many different consumers can read the same events independently, without affecting each other.
• Reading is cheap, because it is sequential disk access, which is one reason Kafka handles huge volumes.

Jay Kreps, one of Kafka’s creators, explains the thinking well in The Log: What every software engineer should know about real-time data’s unifying abstraction. It’s long, but it is the best explanation of why Kafka exists.

Kafka compared with other options

For my own projects, if the need is “process this job once, reliably,” I reach for a queue like Azure Service Bus first. Kafka starts to win when many systems need the same stream of facts, or when replay matters.

I worked on a system with 13 microservices, where Kafka was the message broker and everything communicated through publish/subscribe. A service would publish an event once, and any other service that cared about it simply subscribed, so nobody had to know who else was listening. That is what kept 13 services from turning into a tangled web of direct calls.

The vocabulary, without the jargon

Think of Kafka as a very reliable notice board.

• Topic: one board for one kind of event, like orders.
• Producer: whoever pins a notice to the board.
• Consumer: whoever reads notices from it.
• Partition: the board is split into sections so many people can read at once.
• Offset: a bookmark that records how far a reader has got in a section.
• Consumer group: a team of readers who divide the sections among themselves, so each notice is handled once per team.
• Key: notices with the same key (say, the same order ID) always land in the same section, so their order is kept.
• Broker: a Kafka server. A cluster is several brokers working together, with copies of data on more than one.

Two rules will save you real pain:

1. Ordering is guaranteed only within a partition, not across the whole topic.
2. Within a consumer group, one partition is read by one consumer. If you have 3 partitions and 10 consumers, 7 of them sit idle.

Why it matters in system design

If you’re preparing for system design interviews or designing real architecture, here is where Kafka earns its place.

Absorbing traffic spikes. During a flash sale, orders arrive faster than downstream systems can handle. Kafka acts as a shock absorber: producers keep writing, and consumers work through the backlog at their own pace.

Surviving failures. If the Email service is down for an hour, events wait in the log. When it comes back, it picks up from its bookmark. Nothing is lost and the Order service never noticed.

Adding features without touching old code. A fraud-detection team can subscribe to orders next month without asking the Order team to change anything.

Keeping systems in sync. With change data capture, database changes can be streamed into search indexes, caches, and data warehouses.

Audit and debugging. Because events are kept, you can answer “what happened to order 1042 last Tuesday?” by reading the log.

When Kafka is the wrong choice

Be honest about this. Kafka brings real operational weight: cluster management, partition planning, monitoring, schema evolution. It’s probably overkill if:

• You have one or two services
• You only need simple background jobs
• Your team is small and nobody wants to own the infrastructure
• Your traffic is low and a database table or a simple queue would do

A lot of teams adopt Kafka because it’s fashionable and then spend months fighting it. Pick it because you need replay, fan-out to many consumers, or very high throughput.

Trying it locally

You don’t need a cluster to learn this. The quickest path:

1. Install Docker Desktop.
2. Run the official apache/kafka image. It starts a single-node Kafka in KRaft mode, which means no separate ZooKeeper to set up. The official Kafka quickstart walks through the exact commands.
3. Create a topic called orders with 3 partitions.
4. Add a browser UI such as Kafka UI. Seeing messages, partitions, and consumer lag visually makes the concepts click much faster than reading about them.

In a few minutes you’ll have Kafka on localhost:9092.

Using Kafka from .NET

The standard client is Confluent.Kafka, available on NuGet. It wraps librdkafka, a fast C library, which is why it performs well.

Publishing events

Create one producer and reuse it for the life of the app (register it as a singleton). Creating one per request is the single most common beginner mistake. Producers are thread-safe and expensive to build.

var config = new ProducerConfig
{
    BootstrapServers = "localhost:9092",
    Acks = Acks.All,           // wait for replicas to confirm
    EnableIdempotence = true   // retries won't create duplicates
};

using var producer = new ProducerBuilder<string, string>(config).Build();

await producer.ProduceAsync("orders", new Message<string, string>
{
    Key = orderId,             // same order => same partition => ordered
    Value = json
});

Those two settings, Acks.All and EnableIdempotence, cost almost nothing and protect you from lost or duplicated messages during network trouble. Turn them on from day one. The full list of options is in the librdkafka configuration reference.

Consuming events

Consuming is a long-running loop, so in ASP.NET Core it belongs in a BackgroundService. The loop subscribes to a topic with a group ID, reads messages, does the work, and then commits the offset.

var config = new ConsumerConfig
{
    BootstrapServers = "localhost:9092",
    GroupId = "email-service",
    AutoOffsetReset = AutoOffsetReset.Earliest,
    EnableAutoCommit = false   // we decide when a message counts as done
};

The important decision is when you commit:

Most real systems choose at-least-once and make the consumer idempotent: processing the same event twice does no harm. For example, before sending a confirmation email, check whether you already sent one for that order ID. Honestly, this one habit prevents more production incidents than any Kafka setting.

Patterns worth knowing

Dead-letter topic. If one malformed message fails repeatedly, don’t let it block the partition forever. After a few retries, move it to something like orders.dlq and keep going. Look at it later.

Transactional outbox. A classic trap: you save an order to the database and then publish to Kafka. If the app crashes between the two, your database and Kafka disagree. The outbox pattern writes the event to a table in the same database transaction, and a separate process publishes it. microservices.io has a clear write-up.

Schema management. Teams break each other when someone renames a field. Using Avro or Protobuf with a Schema Registry makes changes explicit and safe.

Mistakes I’d warn a new team about

• Creating a producer per request. Slow and leaks connections.
• Assuming global ordering. It only exists per partition. Pick your keys on purpose.
• Slow work inside the consume loop. If a consumer takes too long between polls, Kafka assumes it died and rebalances the group.
• Too few partitions. Parallelism is capped by partition count. You can add partitions later, but doing so changes which key maps to which partition.
• Ignoring consumer lag. Rising lag is the earliest sign your consumers can’t keep up. Monitor it before users notice.
• Not handling ProduceException. Failures happen; log them and decide what to do.

One lesson I learned the hard way: the same event can be delivered more than once. A service restarted before it committed its offset, and when it came back it processed the same message again. After that, I made a habit of writing consumers so that handling an event twice does no harm, for example by checking whether that event ID was already processed.

Frequently asked questions

Is Kafka a database?
Not really. It stores data durably and you can read history, but it isn’t built for ad-hoc queries the way a database is. Think of it as the pipe and the history between systems.

Do I need ZooKeeper?
Not anymore for new setups. Modern Kafka uses KRaft mode, which removes that dependency.

Can I use Kafka on Azure?
Yes. You can self-host it, or use Azure Event Hubs, which exposes a Kafka-compatible endpoint, so existing Kafka clients can connect with little change.

How many partitions should I start with?
Enough for your expected peak consumer parallelism, with some room to grow. For learning, 3 is plenty.

Is Kafka hard to learn?
The basics take an afternoon. Operating it well in production takes much longer. That’s why many teams use a managed service.

Where to keep learning

Official docs

• Apache Kafka documentation
• Confluent .NET client docs
• confluent-kafka-dotnet on GitHub, with runnable examples

Free courses

• Confluent Developer has free courses including Kafka 101

Books

• Kafka: The Definitive Guide by Gwen Shapira, Todd Palino, Rajini Sivaram, and Krit Petty
• Designing Data-Intensive Applications by Martin Kleppmann, the best book on the thinking behind logs, streams, and distributed data

Related reading

• Transactional outbox pattern
• Azure Service Bus overview, for comparing with a traditional broker

Final thoughts

Kafka changes how you think about services. Instead of “who do I need to call?”, the question becomes “what happened, and who might care?” That shift is what makes systems easier to extend and more forgiving when parts fail. Start small: one Docker container, one topic, one producer, one consumer. Break things on purpose, and watch what happens when a consumer dies mid-message. That teaches more than any article, including this one.

If something here is out of date or wrong, tell me in the comments and I’ll fix it.