Changing a message between two services: why the consumer ships first

A classic bug in event-driven systems: someone edits a record in the back office, and the change never reaches the client app. Both services are up, the queue is healthy, and nothing shows in the error logs. Here is how I approach it, and the rule that prevents it next time.

Prove it at every hop

Follow the data, not the code: the API request, the database row, the message on the queue, the consumer, the second database. Reproduce the problem on unmodified code at the exact library versions each service pins. A local setup with a newer shared library will happily prove the wrong thing.

In one case the field was dropped twice: the producer forwarded only one of the changed fields, and the consumer's message model would have rejected the rest anyway. Fixing only one side would have looked right in review and still failed in production.

Strict consumers go first

If the consumer validates messages strictly and the producer starts sending a new field first, the consumer rejects messages it does not understand. If it acknowledges messages even when validation fails, those messages are simply gone. So the order is: update and deploy the consumer so it accepts the new shape, then deploy the producer that sends it. Shared libraries are tagged and released before the services that use them.

Watch for echoes

When two systems sync both ways, an update can bounce back and overwrite the original change with stale values. Record where a change came from, and make sure a service does not send back what it has just received.

Write the deploy order down

Put it in the ticket: which service goes first, which library version, any permissions or settings to change. QA and whoever deploys should not have to rediscover it. It is a two-line note that prevents a two-hour incident.