The old Kafka dependency

For a long time, setting up a Kafka cluster meant setting up ZooKeeper too.

That raised an obvious question:

Kafka is already distributed. Why does it need another distributed system?

Because Kafka did not only need brokers that store logs. It also needed a consistent answer to cluster questions:

  • Which brokers are alive?
  • Which broker is the controller?
  • Which topics exist?
  • Which broker leads each partition?
  • Which replicas are in sync?

ZooKeeper gave Kafka a consensus-backed place to store and coordinate cluster metadata.

What ZooKeeper stored

In a ZooKeeper-backed Kafka cluster, brokers created and updated znodes under paths like:

/brokers/ids/1
/brokers/ids/2
/brokers/topics/topic-1
/brokers/topics/topic-1/partitions/0/state

Those paths were not random. They represented the metadata Kafka needed to keep the cluster coherent.

/brokers/ids/1 and /brokers/ids/2 told the cluster which brokers were registered. Topic paths stored topic and partition metadata. Partition state paths tracked leadership and replication state.

zookeeper namespacecluster metadata as small znodes

ZooKeeper's model fit this job well: a small hierarchical namespace, znodes with data, watches for changes, and ephemeral nodes that disappear when the session that created them ends.

The interesting part: partition leadership

Kafka partitions need leaders.

If topic-1 has three partitions and two brokers, each partition still needs exactly one leader at a time. Producers write to the leader. Followers replicate from the leader. Consumers read from the leader depending on configuration and version.

The partition state znode could contain information like:

{
  "leader": 1,
  "isr": [1, 2],
  "controller_epoch": 7
}

That little JSON object carries a lot of meaning.

  • leader says which broker currently owns writes for the partition.
  • isr lists replicas that are caught up enough to be considered in sync.
  • controller_epoch versions controller decisions so stale controllers do not overwrite newer state.

Without a consistent partition-leader view, Kafka risks split-brain behavior: two brokers believing different things about who owns a partition.

Why one controller mattered

Kafka also needed one active controller broker.

The controller was responsible for cluster-level decisions: reacting to broker failures, moving partition leadership, updating metadata, and telling brokers what changed.

ZooKeeper helped elect that controller and made sure there was one winner. This is the core coordination problem.

If two brokers both think they are controller, they can issue conflicting leadership changes.

If no broker is controller, failures do not get handled correctly.

If brokers disagree about controller epoch, old decisions can leak into new cluster state.

Failure detection was part of the deal

ZooKeeper sessions and ephemeral znodes gave Kafka a clean failure signal.

When a broker registered itself in ZooKeeper, that registration was tied to its ZooKeeper session. If the broker died or lost its session, the ephemeral znode disappeared. Other participants could observe the change and the controller could react.

That is why ZooKeeper was useful for more than static configuration.

It gave Kafka:

  • broker membership
  • controller election
  • change notification through watches
  • a consistent metadata namespace
  • failure detection through sessions and ephemeral znodes

ZooKeeper was Kafka's coordination substrate, not just a configuration database.

Why not just implement consensus inside Kafka?

Because consensus is hard.

ZooKeeper existed specifically to relieve distributed applications from implementing coordination from scratch. Its docs say coordination services are hard to get right because race conditions, deadlocks, and failure scenarios are easy to mishandle.

ZooKeeper used an atomic broadcast protocol to keep its servers in sync. Writes were ordered, replicated, and acknowledged by a quorum. That gave Kafka a proven external system for the hard part: agreeing on small pieces of critical state.

At the time, using ZooKeeper was pragmatic.

The alternative was every distributed system implementing its own Paxos-like or Raft-like metadata layer and discovering the same failure cases the hard way.

What changed with KRaft

KRaft moved the consensus layer into Kafka itself.

Instead of storing metadata in ZooKeeper, Kafka stores metadata changes in an internal metadata log managed by a controller quorum. The active controller writes metadata changes; other controllers replicate them; brokers fetch metadata updates.

coordination shiftexternal consensus becomes Kafka metadata log

This is the direction described in Kafka's KIP-500: replace ZooKeeper with a self-managed metadata quorum.

The motivation was not just "remove a dependency." It was also:

  • simpler deployment
  • fewer mismatches between controller memory and ZooKeeper state
  • metadata as an ordered event log
  • better scalability for large partition counts
  • one security/configuration model instead of two systems

KRaft makes Kafka responsible for its own metadata consensus.

The practical takeaway

ZooKeeper was there because Kafka needed agreement.

Not agreement about user messages. Kafka brokers already stored partition logs.

ZooKeeper handled agreement about the cluster itself:

  • who is alive
  • who is controller
  • what topics exist
  • who leads each partition
  • which replicas are safe
  • which metadata update is newer

KRaft did not remove the need for consensus. It moved consensus into Kafka.

That is the important mental model.

ZooKeeper went away as a dependency, but the coordination problem did not go away. Kafka absorbed it.

Sources