A consumer service reports ten thousand messages a second, every pod is green, CPU is at forty percent, and the on-call engineer is looking at a graph that says everything is fine. Meanwhile an order placed twenty minutes ago has not been shipped, because the message that would ship it is sitting behind eleven million others.
Throughput is a rate. It tells you how fast the consumer is running, and nothing about whether that is fast enough. Lag is a difference: how far the consumer's position sits behind the end of the log. It is the one number that reflects the producer and the consumer together, which is the only comparison that matters, and it is the number most dashboards leave off because it is not a property of the service that owns the dashboard.
This post is about how to read it, why it climbs, and the specific ways the fixes that look obvious make it climb faster.
#Lag is per partition, and the total hides everything
Kafka stores each topic as partitions, and each partition is an append-only log with a monotonically increasing offset. A consumer group assigns every partition to exactly one consumer in the group. That consumer reads from its committed offset forward and periodically commits how far it got.
For one partition:
lag = log end offset - committed offsetThe log end offset is where the next produced message will land. The committed offset is the last position the consumer said it had finished. The gap is the work that exists and has not been acknowledged.
Every group and topic dashboard sums these across partitions and shows one line. That sum is nearly useless, for the same reason an average latency is: it hides distribution. Twelve partitions with lag of a hundred each is a healthy group doing steady work. Eleven partitions at zero and one at twelve hundred is a hot partition, and the fix is completely different. The first needs nothing. The second needs a look at the partitioning key.
Read lag per partition, and read it as a shape over time, not a value. Lag that rises during the daily peak and drains to zero afterwards is a group that is provisioned for the average and paying for it at the peak. Lag that rises and never drains is a group whose consumers are slower than its producers, and no amount of time will fix that.
#Offsets are the wrong unit for an alert
Twelve hundred messages behind means nothing until you know how fast messages arrive. On a topic doing a hundred a second it is twelve seconds of delay. On a topic doing two a second it is ten minutes.
The number that maps to what users experience is lag in time: the age of the oldest unconsumed message. Every Kafka record carries a timestamp, so it is available. Read the record at the committed offset, subtract its timestamp from now, and you have the delay a message is currently experiencing. Alert on that, in seconds, with a threshold that comes from the product ("an order must ship within two minutes of being placed") rather than from an offset count that nobody can interpret at three in the morning.
Burrow, the exporters that ship with most Kafka distributions, and kafka-consumer-groups --describe all give you offsets. Converting to time takes one extra fetch and is worth doing yourself if the tooling does not.
#Adding consumers stops working at the partition count
The first response to lag is to scale the consumer group. It works, until it does not, and the point where it stops is fixed at topic creation time.
A partition is assigned to one consumer. Twelve partitions means at most twelve consumers doing work. The thirteenth joins the group, is assigned nothing, and sits idle. Scaling from twelve to twenty-four pods doubles your bill and changes nothing.
Partition count is the parallelism ceiling of every consumer group on that topic, forever, and it is decided by whoever created the topic, usually with the default. Changing it later is possible and painful: adding partitions changes which partition a given key hashes to, so ordering for a key breaks at the moment of the change, and every consumer that relied on "all events for order 123 are in partition 4" is now wrong for the events that were already there.
So pick the partition count for the throughput you will need in two years, not the one you have now. Over-partitioning costs some broker memory and file handles. Under-partitioning costs a migration.
And before adding consumers at all, look at the per-partition shape. If one partition carries the lag, more consumers will not touch it, because that partition is still assigned to one consumer.
#Hot partitions come from the key
Messages are assigned to partitions by hashing the key. A key with far more traffic than the others produces a partition with far more traffic than the others, and one consumer has to drain it alone.
Tenant id is the classic mistake. It gives you per-tenant ordering, which sounds right, and it gives your largest customer their own partition and their own permanently lagging consumer. Whatever your largest key is, that partition's consumer defines the group's lag.
The fix is a key with a flatter distribution that still preserves the ordering you actually need. Often that is a narrower entity: order id rather than customer id, device id rather than account id. Ordering per order is usually what the consumer required in the first place; ordering per customer was an accident of choosing the first id to hand.
A null key round-robins across partitions and gives perfect balance and no ordering at all. That is the right choice more often than people assume, for anything where each message stands alone.
#Rebalances are where lag comes from on a quiet day
Adding a consumer, removing one, a pod restart, a deploy, or a consumer that took too long to call poll() all trigger a rebalance: the group reassigns partitions among its members.
With the default eager protocol, every consumer stops, gives up every partition, waits for the new assignment, and starts again. For the duration, nothing in the group is consuming. A rolling deploy of twenty pods is twenty rebalances in a row. Each one stops the world. Lag spikes on every deploy, and the on-call engineer learns to ignore the spike, which means they will ignore the real one too.
Three settings decide how much of your life this takes:
partition.assignment.strategy set to CooperativeStickyAssignor uses incremental rebalancing. Only the partitions that actually move are revoked. The rest keep consuming. This is the single largest lag improvement available for free and it has been available since Kafka 2.4.
group.instance.id gives a consumer a static membership. A pod that restarts with the same id within session.timeout.ms rejoins with its old partitions and triggers no rebalance at all. Every deployment that uses stable pod names should set this.
max.poll.interval.ms is the one that bites. It is the maximum time between two calls to poll(), and the default is five minutes. If processing one batch takes longer, the broker decides the consumer is dead, removes it, and rebalances. The partitions go to another consumer, which starts from the last commit, which was before the slow batch. The slow batch is processed twice. And the original consumer, still alive and still working, finishes its batch and tries to commit, and fails, because it is no longer a member. This is the loop that turns a slow downstream into a group that reprocesses the same batch forever while lag climbs vertically.
Lower max.poll.records so a batch always finishes inside the interval with margin. If a single record can legitimately take longer than that, the record needs to be handed to something else, and the consumer needs to commit and move on.
#Commit after the work, and know what that costs
Where you commit decides what a crash does.
Auto-commit, the default, commits on a timer regardless of whether processing has finished. A crash between the commit and the end of processing loses those messages. That is at-most-once delivery, whatever the documentation implies.
Committing manually after the work is done gives at-least-once. A crash after the work and before the commit means the next consumer redoes it. This is the correct default for almost everything, and it means the work must be idempotent. The same discipline as for retried requests applies to redelivered messages: dedupe on a key that travels with the message, in the same transaction as the effect.
Exactly-once exists in Kafka, and it means something narrower than the name. Transactions make a consume-transform-produce loop atomic between two Kafka topics. They do nothing for a side effect that leaves Kafka, which is where nearly all the work is. If the consumer writes to a database and then to a topic, the database write is not in the transaction. The outbox pattern is how that gets solved, and it is at-least-once on the way out too.
#A poison message stops a partition
One message the consumer cannot process, because the payload is malformed or a downstream rejects it permanently, is a message that fails, is retried, fails again, and never commits. The partition behind it does not move. Lag on that one partition climbs at exactly the producer rate, forever, while eleven other partitions stay at zero.
Retrying it in place is the wrong move, for the reasons that retry storms are the wrong move everywhere: it does not fix the message and it holds up everything behind it. Attempt it a bounded number of times, write it to a dead letter topic with the error attached, commit past it, and alert. A partition that drains with one message parked is a system; a partition blocked behind one message is an outage with a good excuse.
#Slowing down on purpose
A consumer that pulls faster than it can process fills its own memory. The Kafka client is pull-based, so the natural throttle is simply not to call poll() until you have capacity, but that runs into max.poll.interval.ms above.
The client has pause() and resume() for this. Pause the partitions, keep calling poll() so the heartbeat and the group membership stay alive, and resume when the downstream has drained. This is backpressure applied at the source, and it is the right response to a slow database or a rate-limited API on the other side: lag rises in the broker, where it is durable and visible, instead of in the consumer's heap, where it is neither.
#The graph you actually want
Per partition, lag in seconds, with the deploy markers on the same axis. From that one panel you can read every failure in this post: a spike on every deploy is eager rebalancing, a single partition climbing alone is a hot key or a poison message, a slow steady climb across all of them is a group that needs more partitions and more consumers, and a vertical line is a consumer that exceeded its poll interval and is now fighting the group for its own partitions.
Throughput never showed any of that. It was ten thousand a second the whole time.
#The one-line summary
Lag is the gap between what has been produced and what has been acknowledged. Measure it per partition, in seconds, alert on it from a product number, and remember that no consumer count gets you past the partition count you chose on day one.
#References
- Kafka documentation: consumer configuration, in particular
max.poll.interval.ms,max.poll.records,group.instance.idandpartition.assignment.strategy - KIP-429: incremental cooperative rebalancing
- KIP-345: static membership
- Burrow, LinkedIn's consumer lag checker