SolutionGigsSolutionGigs
  1. Home
  3. Data Engineering
  5. Kafka Partition & Retention Storage Estimator

Kafka Partition & Retention Storage Estimator

Enter message rate, payload size, retention, partitions, replication factor and compression — get a per-broker disk estimate, a ready kafka-topics.sh command and matching server.properties defaults.

✓ Per-Partition Accurate✓ Runs in Browser✓ No Signup Required✓ Deterministic Formula
Retention policy

Ingest (uncompressed)

9.77 MB/s

Ingest (compressed)

3.42 MB/s

Per-broker disk

29.6 TB

Uncompressed: 34.3 GB/hr · 824 GB/day. Compressed: 12.0 GB/hr · 288 GB/day. Total cluster storage (all replicas, with buffer): 88.7 TB.

kafka-topics.sh --create

kafka-topics.sh --create \
  --topic events \
  --partitions 12 \
  --replication-factor 3 \
  --config retention.ms=604800000 \
  --config compression.type=zstd \
  --bootstrap-server localhost:9092

server.properties (cluster defaults)

default.replication.factor=3
num.partitions=12
min.insync.replicas=2
compression.type=zstd
log.retention.hours=168
log.retention.check.interval.ms=300000

These are cluster-wide defaults — the topic-level config above always wins for this specific topic.

How to size Kafka disk storage

1

Describe the topic traffic

Message rate, average payload size, partitions, replication factor and how many brokers will host it.

2

Pick retention and compression

Time-based or size-based retention, plus a compression codec — each changes how much of that traffic actually stays on disk.

3

Copy the create command

Copy the kafka-topics.sh command with the right --config flags, or the server.properties defaults, straight into your cluster.

Frequently Asked Questions

How do you calculate Kafka disk space?

Multiply message rate by average payload size to get raw ingest bytes/sec, apply your compression ratio, multiply by the retention window in seconds to get bytes retained per partition, then multiply by partition count and replication factor for the cluster total. This tool does that math and adds an index/compaction buffer margin on top, since segment files and compaction never pack storage at exactly 100% efficiency.

Is log.retention.bytes a per-topic or per-partition limit?

Per-partition. Kafka applies retention.bytes to each partition's log individually, so a topic with 12 partitions and a 50 GB retention.bytes setting can hold up to 600 GB total, not 50 GB. This is the single most common Kafka storage-sizing mistake, and the reason this calculator asks for retention size per partition, not per topic.

Why can't replication factor exceed the broker count?

Each replica of a partition must live on a different broker — that's what makes replication survive a broker failure. A replication factor of 3 needs at least 3 brokers to place those copies on; asking for more replicas than you have brokers is a request Kafka will reject outright when you try to create the topic.

How much does compression actually save?

It depends heavily on payload shape, but for typical JSON or text-like event payloads: gzip and zstd usually cut 65-70%, LZ4 around 55%, and Snappy around 50% — with LZ4 and Snappy trading some ratio for much faster CPU-bound throughput. This tool uses those rough averages; measure your own topic if the estimate needs to be precise.

Does this account for replication traffic and network cost?

No — this estimates disk storage only: what a partition's log occupies after retention, replication and compression. Inter-broker replication traffic, producer/consumer network throughput and page-cache sizing are separate capacity-planning questions this tool does not model.