A few weeks ago we spent several hours discussing what looked like a tiny issue in one of our AI voice agents. It wasn’t hallucinating, the latency was fine, conversations were coherent, but every now and then it would address a female patient using masculine forms.
From an engineering perspective, that’s an almost boring bug. It’s easy to look at it and think, “We’ll fix it eventually.” But that’s not how users experience software. They don’t evaluate prompt quality, model architecture or response latency. They simply notice that something feels wrong, and once trust is gone, they stop caring how technically impressive the system is.
It reminded me that engineering quality and product quality aren’t always the same thing. We spend a lot of time optimizing metrics that users never see, while the details they notice are often the ones that determine whether they trust the product at all.