Introduction to Apache Kafka for .NET Developers

Kafka is everywhere in modern architectures. If you’re coming from MSMQ or RabbitMQ, Kafka is a different beast. Here’s what .NET developers need to know.

What Makes Kafka Different

Kafka isn’t just a message queue—it’s a distributed commit log. Messages persist, consumers track their own position, and you can replay history. This changes how you think about messaging.

Core Concepts

• Topics: Named feeds of messages
• Partitions: Topics split for parallelism
• Consumer Groups: Logical grouping for load balancing
• Offsets: Position tracking within partitions

.NET Client: Confluent.Kafka

dotnet add package Confluent.Kafka

Producer

using Confluent.Kafka;

var config = new ProducerConfig { BootstrapServers = "localhost:9092" };

using var producer = new ProducerBuilder<string, string>(config).Build();

var result = await producer.ProduceAsync("orders", new Message<string, string>
{
    Key = order.Id.ToString(),
    Value = JsonSerializer.Serialize(order)
});

Console.WriteLine($"Delivered to {result.TopicPartitionOffset}");

Consumer

var config = new ConsumerConfig
{
    BootstrapServers = "localhost:9092",
    GroupId = "order-processor",
    AutoOffsetReset = AutoOffsetReset.Earliest
};

using var consumer = new ConsumerBuilder<string, string>(config).Build();
consumer.Subscribe("orders");

while (true)
{
    var result = consumer.Consume(TimeSpan.FromSeconds(1));
    if (result != null)
    {
        var order = JsonSerializer.Deserialize<Order>(result.Message.Value);
        await ProcessOrder(order);
        consumer.Commit(result);
    }
}

Key Differences from Traditional Queues

• Messages aren’t removed after consumption
• Multiple consumer groups can read independently
• Ordering is per-partition only
• High throughput is the default design

References

• Apache Kafka Documentation
• Confluent .NET Client