Kafka in IoT — Ingesting Telemetry from Millions of Devices

The hard part of IoT is not connecting devices. It's what happens when a million of them wake up at once and start reporting. Every reading is small, arrives continuously, and is worthless on its own but valuable in aggregate — and the backend has to accept all of it without dropping data, while still answering real-time questions like "is this machine about to fail."

That shape of problem — high volume, continuous, decoupled producers and consumers — is exactly what Kafka is built for. This article walks through where Kafka sits in an IoT stack and the design decisions that determine whether the pipeline holds up.

Why Kafka fits IoT

Kafka's value in an IoT pipeline comes down to three properties, each matching one of IoT's headaches.

Decoupling. The devices producing data and the systems consuming it (stream processors, time-series databases, alerting) don't have to run at the same speed or even be up at the same time. A device burst doesn't stall the processing layer; it lands in the log and waits.

Buffering. Kafka is a durable, replicated log. When ten thousand devices report simultaneously, the data sits in partitions instead of hammering a database. The buffer absorbs the spike.

Replay. Because the log retains messages, you can go back. A bug in a downstream consumer doesn't lose data — you fix the consumer, reset the offset, and reprocess. You can also backfill a new analytics job over historical telemetry without re-querying the devices.

The third one is the least obvious and often the most valuable in practice. Devices don't retry on your schedule, so "we can replay the last week of readings" is the difference between a recoverable incident and lost data.

The reference architecture

A typical IoT stack looks like this:

devices ──MQTT──▶ MQTT broker ──bridge──▶ Kafka ──▶ consumers
                                               ├─ stream processing (Flink / Kafka Streams)
                                               ├─ time-series DB (TimescaleDB / ClickHouse)
                                               └─ alerting / dashboards

The left edge is MQTT, not Kafka, for a good reason: constrained devices run on battery and bandwidth, and MQTT is a lightweight pub/sub protocol built for that. Kafka's protocol is heavy and stateful — you don't run a Kafka client on a sensor. The bridge translates between the two worlds.

The right edge is where Kafka's properties pay off. Raw telemetry flows into Kafka, then fans out to as many consumers as you need — one writes to the time-series store, another evaluates alert thresholds, another feeds a dashboard.

MQTT and Kafka are different layers

A common confusion is treating MQTT and Kafka as competing message systems and asking which one to "choose." They solve different halves of the problem.

• MQTT is the device-communication layer. Lightweight pub/sub, Quality of Service levels (at-least-once, exactly-once), keep-alive and reconnection for flaky networks. It's designed to run on small devices.
• Kafka is the backend backbone. Durable, replicated, high-throughput, replayable. It's designed to run in a datacenter and absorb the combined output of all devices.

The bridge between them is deliberately thin. It holds a persistent connection to the broker, subscribes to device topics (usually with a wildcard like assets/+/telemetry), and publishes each message into Kafka. A good bridge is fire-and-forget on the Kafka side and buffers locally when Kafka is briefly unreachable, so a broker hiccup doesn't become data loss.

The core technique: key by device ID

Kafka guarantees ordering only within a partition, not across a topic. So the single most important decision in an IoT Kafka design is what you use as the message key. The answer is almost always the device ID.

When the device ID is the key, every reading from a given device hashes to the same partition, which means a device's readings stay in order while different devices are processed in parallel across partitions. That's the whole trade: per-device ordering plus horizontal parallelism.

ProducerRecord<String, byte[]> record =
    new ProducerRecord<>("iot.telemetry", deviceId, payload);
producer.send(record);

The deviceId here is the key; the hash partitioner routes it to a stable partition.

The anti-pattern worth naming: one topic per device. Ten thousand devices becomes ten thousand topics, and broker management collapses under the metadata. Shared topics with the device ID as the key are the standard at any real scale.

Partition count follows from your consumer parallelism. Size partitions so partitions >= consumer instances × (1.2 to 2), and for a fleet of a million devices a starting point in the tens of partitions is typical. The constraint to watch is hot-spotting: one device that reports ten times more often than its neighbors will overload its partition, so keep an eye on per-device throughput, not just fleet averages.

Topic design

With shared topics, the design effort moves to layering and per-topic configuration. A workable convention separates concerns by data class:

iot.{domain}.raw.telemetry    device readings as they arrive
iot.{domain}.raw.event        discrete events (unlock, tamper, door-open)
iot.{domain}.processed.alert  real-time alerts computed downstream
iot.{domain}.command          cloud → device instructions
iot.{domain}.status           device heartbeat / online state

The retention and compaction policy is per topic and follows from how the topic is consumed:

The status row is the subtle one. A heartbeat is only interesting as the latest value — you want to know each device's current state, not every heartbeat it ever sent. A compacted topic keeps one message per key (per device) and drops the rest, which is exactly right for current-state reconstruction. Raw telemetry, by contrast, is delete-policy because you want the full stream, at least for a while.

Creating the topic explicitly codifies these choices:

kafka-topics.sh --create \
  --topic iot.telemetry \
  --partitions 24 \
  --replication-factor 3 \
  --config cleanup.policy=delete \
  --config retention.ms=604800000

Telemetry vs. commands

Telemetry and commands look symmetric — one goes up, one goes down — but they're different data classes and should be separate topics.

• Telemetry / status is device → cloud, high volume, keyed by device, time-series in nature.
• Command is cloud → device, low volume, short-lived, and often keyed differently (you might address a command to a group of devices, or want it read by a specific gateway).

Mixing them in one topic forces every consumer to read both, and forces one retention policy onto two very different lifetimes. A command you sent last month is noise; a telemetry reading from last month might be data you still want. Separate topics, separate policies, separate consumers.

Backpressure and scale

IoT pipelines fail in two predictable places: the producer side gets overwhelmed, and the consumer side becomes the bottleneck. Both have standard mitigations.

On the producer side, tune for throughput and reliability together:

acks=all
enable.idempotence=true
batch.size=32768
linger.ms=100
compression.type=zstd

acks=all with idempotence gives you effectively-once delivery without per-message round trips, and batching plus compression turns thousands of tiny messages into efficient bulk writes.

Per-device rate limiting is the fix for a specific IoT failure mode: a malfunctioning device that starts reporting hundreds of times a second. Left alone, it consumes a disproportionate slice of the pipeline. Rate-limit at the broker or a stream stage, keyed by device ID, and record a rate-limit event when you drop — so the malfunction is visible instead of silent.

On the consumer side, remember that the consumer is usually the bottleneck. Cleaning, rule matching, and window aggregation are the expensive part, not the ingestion. The right pattern is manual offset commits after a batch is processed:

while (true) {
    ConsumerRecords<String, byte[]> records = consumer.poll(Duration.ofMillis(100));
    for (ConsumerRecord<String, byte[]> r : records) {
        process(r);
    }
    consumer.commitSync();   // commit only after the batch succeeds
}

enable.auto.commit=false is the key — you don't want offsets advanced before the work is actually done, or a crash loses processed data.

Time-series storage

The last mile is where a lot of IoT pipelines degrade: writing telemetry into storage without a plan.

Two rules keep it healthy. Never serve live queries from the write database. Write-path throughput and dashboard read latency are at odds; a hot query can stall the ingest. Keep a live-state cache (Redis or similar) for "what is this device doing right now," and let the time-series database handle history.

Batch your writes. A consumer that inserts one row per reading is wasting most of its time on per-statement overhead. Pull a batch, then do a single bulk INSERT into the time-series store. The difference on a high-volume pipeline is an order of magnitude.

Choosing the store itself is a separate decision — TimescaleDB if you want SQL and relational joins, ClickHouse for extreme write throughput, InfluxDB for a telemetry-native model — but the two rules above apply regardless of which one you pick.

The Bottom Line

Kafka earns its place in IoT the same way it earns it anywhere: it's a durable, replayable buffer between producers and consumers that don't run at the same speed. In IoT those producers are millions of devices that never slow down for you.

The decisions that make the difference are specific and mechanical: key by device ID to get per-device ordering for free, use shared layered topics instead of one per device, give each topic a retention policy that matches how it's consumed, separate telemetry from commands, and don't let the write path and the read path collide in storage.

The one habit to build from the start is telemetry for the pipeline: track ingestion lag, consumer lag, malformed-message rates, and per-device throughput. A pipeline this big fails in ways you can only see if you instrument it — and if you can't see it, you'll find out about it from your users.