On the web I got used to a comfortable lie: if the request came back 200, the thing happened.
It's mostly true. The browser asks, the server answers, and the answer comes back on the same connection a moment later. Request and response are glued together. When something breaks, you usually find out right away.
Then I started building cloud-to-device messaging over MQTT, and the glue was gone.
With MQTT you don't send a message to a device. You publish it to a broker, on a topic, and the device subscribed to that topic gets it when it can. "When it can" is doing a lot of work in that sentence. The device might be asleep, on a weak network, rebooting, or halfway through an update.
MQTT's quality-of-service levels help, but it's easy to misread what they promise. At QoS 1 you get at-least-once delivery and an acknowledgement back. It's tempting to treat that acknowledgement like a 200. It isn't. It tells you the broker has the message. It says nothing about whether the device read it, understood it, or did anything with it.
Sent, held by the broker, received by the device, and acted on are four different states. Coming from the web, my instinct was to collapse them into one.
Acknowledge the work, not the wire
The fix is to stop asking the transport for meaning. If you need to know a command was carried out, the device has to say so, in a message of its own, about that specific command.
So give every command an identity:
{ "id": "c-7f3a", "type": "set_config", "expires_at": "…" }
Have the device reply on its own topic with the same id and an outcome. Then the backend can follow each command through a real lifecycle instead of a hope:
pending: created, not yet publishedsent: the broker has itacknowledged: the device says it arrivedappliedorfailed: the device says what happenedexpired: nobody answered in time
Expired is the one people forget.
Retries need a clock
No acknowledgement, so you retry. Obvious. The interesting part is everything around the retry.
Retry immediately and forever, and a device that comes back online gets buried in a pile of stale commands at once. Retry with backoff, and you still need a ceiling, or a command meant for this afternoon lands next week. Some commands are only correct inside a short window. Delivering them late can be worse than never delivering them.
So give commands an expiry, and make the retry loop respect it. When the window closes, the command becomes expired, the system should say so plainly, and a person or another process decides what happens next. Silence is not an outcome.
At least once means sometimes twice
At-least-once delivery has its cost written in the name. Sometimes the device gets the same command twice: an acknowledgement got lost, the retry fired, and both copies arrive.
If the command is "set this value to 5", twice is harmless. If it's "add 5", twice is a bug. The answer is idempotency. A device should remember recent command ids and treat a repeat as a repeat, replying with the original outcome instead of doing the work again.
I'd met this idea on the web, handling webhooks: verify the signature, then make sure processing the same event twice changes nothing. With devices it stops being good hygiene and becomes the job, because duplicates turn up every day.
What came back to the web with me:
- I name the states between "sent" and "done" instead of pretending there are two.
- Anything that can be retried gets an id, and whoever receives it deduplicates on that id.
- Every wait has a deadline, and "expired" is a real state the interface can show.
- I don't read a transport acknowledgement as a business result.
None of this is new. People who build distributed systems have been saying it for decades. I just hadn't felt it until there was a real device on a real network at the other end, asleep when the command went out.