The Fallback That Hid a Broken Form

When someone writes to us through the chat on our site, the message goes to our own inquiry API, which stores it in the dashboard where we track every project conversation. If that API cannot be reached, the site sends the same message by email instead, so nothing is ever lost.

try {
  const id = await createInquiry(fetch, inquiry);
  return json({ ok: true, id });
} catch (err) {
  console.error('Inquiry API unavailable', err);
}
return sendFallbackEmail(data);

That is a sensible design. For the first days after we moved to our own server, it was also the reason we did not notice that the API had never worked in production.

What was wrong

The server’s environment file held placeholder text where the API address and key should have been. The values had been copied from output that hides secrets, so the file contained the mask instead of the secret. Every inquiry failed the API call, was caught, and went out as an email.

The emails arrived, and nothing about them looked different. The dashboard simply never received anything. It took three days for us to notice, and the cause was a fallback doing exactly what it was built to do.

Making a fallback visible

A fallback that looks the same as success hides every failure behind it. We changed three things.

• Fallback emails now say so. The subject starts with fallback, and the body carries a note that the dashboard was unreachable.
• The two paths return different IDs. A stored inquiry returns a database ID of 24 hexadecimal characters. A fallback returns the email provider’s ID, which is a UUID. One test message tells you which path ran.
• After any change to the server’s environment file, we search it for placeholder text before restarting anything.

grep -n SENSITIVE .env

The fix itself was two lines: the API’s address on the same server, and the real key, copied on the server itself rather than through anything that might mask it.

What we are still unsure about

A label still depends on someone reading the subject line. An alert when the fallback runs would be better, and we have not built one yet. As far as we can tell, no inquiry was lost. The fallback did its job, and that is exactly why it took three days to see that it was doing all of the work.