Topics, Partitions, Offsets, Ordering, and Retention
Build a solid Kafka foundation by understanding how records are organized, how ordering works, and how retained event history supports replay.
Inside this chapter
- Why Topics Matter
- Partitions and Parallelism
- Offsets
- Retention and Replay
- Key-Based Partitioning
- Real-Time Example
Series navigation
Study the chapters in order for the clearest path from Kafka basics and local setup to stream processing, platform operations, cloud usage, and advanced event-driven architecture thinking. Use the navigation at the bottom to move smoothly through the full tutorial series.
Why Topics Matter
A topic is a logical stream of records. Teams often create topics around important business events such as orders, payments, shipments, user-signups, device-metrics, or inventory-changes. Topic naming and ownership are important because they shape how events are discovered and used across the organization.
Partitions and Parallelism
Each topic can be split into partitions. A partition is an ordered append-only log. Kafka uses partitions to scale reads and writes horizontally.
Ordering is guaranteed within a partition, not automatically across all partitions of a topic. This is one of the most important concepts students must understand early.
Offsets
An offset is the position of a record inside a partition. Consumers track offsets so they know what they have already processed and where to resume after restart or failure.
Partition 0:
offset 0 -> order-created
offset 1 -> order-paid
offset 2 -> order-shippedRetention and Replay
Kafka keeps records for a configurable retention period or based on size limits. This is powerful because consumers can replay old events for recovery, debugging, backfill jobs, or onboarding new downstream systems.
Key-Based Partitioning
If a producer sends records with the same key, Kafka can route them consistently to the same partition. This is how teams preserve ordering for a specific entity such as order id, customer id, or account id.
Real-Time Example
In a bank, events for one account should usually stay ordered relative to that account. Key-based partitioning makes this possible while still allowing other accounts to be processed in parallel on different partitions.