ESP32 MQTT that survives bad networks: reconnects, LWT and offline buffering

Every ESP32 MQTT tutorial has the same shape: connect to Wi-Fi, connect to the broker, publish a number every few seconds. It works for ten minutes on your desk, and you ship it.

Three weeks later the device is silent. The router rebooted, the broker restarted, or the mobile network dropped for a minute. Your firmware hung in a blocking connect() loop, hammered the broker with reconnect attempts, or kept publishing into a dead socket and lost everything.

This post builds a client that handles those cases. It covers:

A unique client ID (a silent bug that kicks devices offline)
Keepalive, birth messages and Last Will, so you know a device’s real state
Non-blocking reconnect with exponential backoff and jitter
An offline buffer so readings survive an outage
What QoS and persistent sessions do for you

I’m using PubSubClient because most people have it. I point out its limits where they matter.

How MQTT links fail

A device on a real network will hit all of these:

Wi-Fi drops, and the TCP socket can stay “open” from your side for a long time.
The broker restarts or is briefly unreachable.
A NAT or carrier silently drops an idle connection. This is common on cellular links, where idle TCP sessions can be dropped after a few minutes.
Two devices share a client ID, so the broker keeps kicking one off for the other.
Everything reconnects at once after a power cut and floods the broker.

Each fix below addresses one of these.

Fix 1: a unique client ID

If two clients connect with the same ID, the broker disconnects the older one. Both devices then reconnect, and each kicks the other off in a loop. It looks like random instability and is miserable to debug.

Never hardcode "esp32-client". Derive the ID from the chip’s MAC:

char clientId[24];

void makeClientId() {
  uint64_t mac = ESP.getEfuseMac();
  snprintf(clientId, sizeof(clientId), "esp32-%04X%08X",
           (unsigned)(mac >> 32) & 0xFFFF, (unsigned)mac);
}

Fix 2: keepalive, birth message and Last Will

You want the broker to tell you two things: is the device alive, and did it die cleanly?

Keepalive is how often the client must send something. If the broker hears nothing for about 1.5x the keepalive interval, it declares the client dead. The default in PubSubClient is 15 s. On cellular or NAT-heavy links, use 30–60 s so a middlebox doesn’t drop you as idle.
Last Will and Testament (LWT) is a message you give the broker at connect time. If the connection dies without a proper disconnect, the broker publishes it for you.
The birth message is what you publish yourself right after connecting.

Together they give you a reliable status topic. Make both retained so any new subscriber sees the current state immediately:

// LWT: broker publishes "offline" (retained) if we vanish
mqtt.connect(clientId, MQTT_USER, MQTT_PASS,
             topicStatus, /*willQos*/ 1, /*willRetain*/ true, "offline",
             /*cleanSession*/ false);

// Birth: we publish "online" (retained) once connected
mqtt.publish(topicStatus, "online", true);

One gotcha: the LWT fires on unclean disconnects only. If you disconnect on purpose, for example before an OTA restart, publish "offline" yourself first.

Fix 3: non-blocking reconnect with backoff and jitter

The classic reconnect code looks like this:

while (!mqtt.connected()) {
  mqtt.connect(clientId);
  delay(5000);
}

This blocks the whole firmware, so sensors stop being read and the watchdog may bite. It also retries at a fixed interval, so a hundred devices after a power cut all hit the broker at the same moment.

Instead, treat the connection as a small state machine that never blocks for long, and space out retries with exponential backoff plus random jitter:

enum class Conn { WifiDown, Connecting, Online, Backoff };
Conn conn = Conn::WifiDown;

constexpr uint32_t BACKOFF_MIN = 1000;    // 1 s
constexpr uint32_t BACKOFF_MAX = 60000;   // 60 s
uint32_t backoffMs = BACKOFF_MIN;
uint32_t retryAt = 0;

void scheduleRetry() {
  uint32_t jitter = random(0, backoffMs / 4 + 1);   // up to +25%
  retryAt = millis() + backoffMs + jitter;
  backoffMs = min(backoffMs * 2, BACKOFF_MAX);
  conn = Conn::Backoff;
}

void connectionTick() {
  if (WiFi.status() != WL_CONNECTED) {
    conn = Conn::WifiDown;      // Wi-Fi auto-reconnect handles this
    return;
  }

  switch (conn) {
    case Conn::WifiDown:
      conn = Conn::Connecting;
      break;

    case Conn::Connecting:
      if (mqtt.connect(clientId, MQTT_USER, MQTT_PASS,
                       topicStatus, 1, true, "offline", false)) {
        mqtt.publish(topicStatus, "online", true);
        mqtt.subscribe(topicCmd, 1);
        backoffMs = BACKOFF_MIN;          // success: reset backoff
        conn = Conn::Online;
      } else {
        scheduleRetry();
      }
      break;

    case Conn::Online:
      if (!mqtt.connected()) { scheduleRetry(); break; }
      mqtt.loop();
      qFlushOne();                        // drain the offline buffer
      break;

    case Conn::Backoff:
      if ((int32_t)(millis() - retryAt) >= 0) conn = Conn::Connecting;
      break;
  }
}

Notes:

mqtt.connect() can still block up to the socket timeout. Lower it with mqtt.setSocketTimeout(5) (seconds) so a dead broker costs 5 s, not the default 15.
The (int32_t)(millis() - retryAt) >= 0 comparison stays correct when millis() wraps around after about 49 days.
Reset the backoff only on a successful connect, not on every attempt.
This lifecycle maps well onto an explicit state machine library. With a hierarchical one like PulseHSM, Online can own the publish and flush behavior while Backoff owns the timer.

Fix 4: an offline buffer

By default, a publish during an outage is dropped. For a sensor that logs every 10 seconds, a 2-minute outage means 12 lost readings.

Add a small ring buffer in RAM. When the link is down, queue messages. When it comes back, drain them gradually. When the buffer is full, drop the oldest, because fresh data usually matters more than stale data:

struct Msg { char topic[48]; char payload[96]; };

constexpr size_t QCAP = 32;
Msg q[QCAP];
size_t qHead = 0, qCount = 0;
uint32_t qDropped = 0;

void qPush(const char* topic, const char* payload) {
  if (qCount == QCAP) {              // full: drop oldest
    qHead = (qHead + 1) % QCAP;
    qCount--;
    qDropped++;
  }
  Msg& m = q[(qHead + qCount) % QCAP];
  strlcpy(m.topic, topic, sizeof(m.topic));
  strlcpy(m.payload, payload, sizeof(m.payload));
  qCount++;
}

bool qFlushOne() {
  if (qCount == 0) return false;
  Msg& m = q[qHead];
  if (!mqtt.publish(m.topic, m.payload)) return false;   // keep it, retry later
  qHead = (qHead + 1) % QCAP;
  qCount--;
  return true;
}

void sendTelemetry(const char* payload) {
  // Only publish directly if nothing is waiting, so ordering is preserved
  if (conn == Conn::Online && qCount == 0 &&
      mqtt.publish(topicTelemetry, payload)) return;
  qPush(topicTelemetry, payload);
}

Three details matter here:

Flush one message per loop pass, not all at once. Sending 32 messages in a burst right after reconnecting can overwhelm a weak link.
Put a timestamp inside the payload. A reading delivered 5 minutes late is useless if the server stamps it on arrival. Sync time with NTP and include the original measurement time.
Count the drops (qDropped) and publish that counter occasionally. It shows how bad the outages are.

If the data must survive a reboot too, move the buffer to flash (LittleFS or NVS), but mind the write wear. Batch writes instead of saving every message.

Putting it together

Setup configures Wi-Fi and the client once. The main loop ticks the connection and produces data:

void setup() {
  makeClientId();
  WiFi.mode(WIFI_STA);
  WiFi.setAutoReconnect(true);
  WiFi.begin(WIFI_SSID, WIFI_PASS);

  mqtt.setServer(MQTT_HOST, 1883);
  mqtt.setKeepAlive(30);
  mqtt.setSocketTimeout(5);
  mqtt.setBufferSize(512);
  mqtt.setCallback(onMessage);
}

void loop() {
  connectionTick();

  static uint32_t lastSample = 0;
  if (millis() - lastSample >= 10000) {
    lastSample = millis();
    sendTelemetry("{\"temp\":24.6}");   // add a timestamp in real code
  }
}

QoS and sessions

QoS 0: fire and forget. No acknowledgment.
QoS 1: delivered at least once, with an ack and possible duplicates.
QoS 2: exactly once, with a heavier handshake. Rarely worth it on a microcontroller.

The catch is that PubSubClient can only publish at QoS 0. It can subscribe at QoS 0 or 1, but it cannot send QoS 1 publishes. If you need acknowledged delivery from the device, use a library that supports it, such as espMqttClient or ESP-IDF’s esp-mqtt. Until then, the offline buffer gives you best-effort recovery, not a delivery guarantee.

Persistent sessions help in the other direction. Connect with cleanSession = false (as in the code above) and subscribe at QoS 1, and the broker queues commands for your device while it’s offline and delivers them when it returns. This needs a stable client ID, which is another reason for Fix 1.

TLS and the watchdog

TLS costs memory. A handshake with WiFiClientSecure needs tens of KB of free heap, so fragmentation can make reconnects fail after days of uptime. Log ESP.getFreeHeap() and ESP.getMaxAllocHeap() on every reconnect. If you validate certificates, sync the clock before connecting.

For the watchdog, keep every blocking call short (the socket timeout above) and feed it in the main loop. A device that reboots and recovers beats one that hangs forever.

Test it like the field will

Break it on purpose before you trust it:

Stop the broker for 2 minutes, restart it, and confirm the buffer drains in order.
Power-cycle the router and watch the backoff grow and reset.
Block port 1883 with a firewall rule so Wi-Fi is up but the broker is unreachable.
Pull power from the device and check that the retained status topic flips to offline after about 1.5x the keepalive.
Flash two devices with the same client ID once, just to see the kick loop you’re preventing.

Checklist

Unique client ID from the MAC
Keepalive set to suit your network (30–60 s for cellular)
Retained birth and LWT on a status topic
Reconnect that never blocks the main loop
Exponential backoff with jitter, reset on success
Offline ring buffer with drop-oldest and a drop counter
Timestamps inside payloads
Heap logging on reconnect