The Knobs Nobody Asked For: Learning to Build for the Customer, Not the Engineer

What pilot sessions with real researchers taught me about the difference between an interface an engineer wants and one a customer needs.

The first version of the platform worked exactly the way I wanted it to. That turned out to be the problem.

The tool lets domain experts review AI-suggested labels on text, correct its mistakes, and steadily improve the system’s accuracy on their task. Its users are researchers and annotators from wildly different fields. Our pilots ranged from mental health research to wildlife preservation: smart people, expert in their domains, and mostly not engineers.

The version I shipped to the first pilots didn’t reflect that last part at all.

Version one

My first UI showed the algorithm’s output, a set of required inputs, and a panel of knobs for tuning how the underlying algorithms behaved. To me this was obviously the right interface: every parameter that affected the system was exposed, so the user had full control.

I knew what every knob did, what “good” looked like in the output, and what step came next at every point in the pipeline, because I built the thing. It didn’t occur to me that this knowledge was the only reason the interface felt self-explanatory.

The pilot sessions

Then the pilots started, with researchers from domains as different as mental health and wildlife preservation. That range mattered more than I expected, because the same three questions dominated every session regardless of field:

• “What does this setting do?”, asked about nearly every knob I’d been so proud of.
• “How do I know the AI is doing well?” The output was all there, but nothing told them whether to trust it.
• “Where do I go next?” There was general confusion about what to do and in what order.

My first instinct was to treat these as documentation gaps: write a better guide, add tooltips, move on. But when users from completely unrelated fields hit the same walls in the same places, the problem clearly isn’t any one audience’s background. The questions were symptoms of a design built from the wrong point of view.

I had built the interface an engineer wants: maximum control, maximum visibility into parameters. What the users needed was the interface a customer wants, which answers a different set of questions: can I trust this, am I doing it right, and what do I do next. None of my knobs answered any of those.

The rework

I went back and rebuilt the experience around the users’ questions rather than the system’s parameters.

The trust question got a direct answer. The platform now shows one headline metric, the current false negative rate, that tells annotators how well the system is performing right now. I chose that number deliberately: it maps to the users’ real worry (what is the system missing?) rather than to a metric that’s natural to an ML engineer but meaningless to an annotator. Picking which single number to put in front of a user is a product decision wearing a statistics costume.

The “where do I go next” confusion got addressed before it could come up. First-time users now get a walkthrough that hand-holds them through the pipeline: what each section is, what to do at each stage, and what comes next. The pipeline didn’t change; the tool just stopped assuming you already knew it.

And most of the knobs went away, replaced by sensible defaults with the details tucked out of the critical path. The full-control panel had only ever been serving me.

In the sessions that followed the rework, feedback turned positive across every domain we piloted, and users specifically called out that there was less confusion in the product, which was the part I cared about most. The three questions that had dominated the first pilots largely disappeared, and sessions shifted from “what is this?” to annotating. For a tool whose whole premise is saving experts time, that shift was the real success metric.

The takeaway

The rework itself is not the interesting part. What changed was where I stood when making design decisions. An engineer looks at a system from the inside out: here are the capabilities, so expose them. A customer experiences it from the outside in: here is my goal, my uncertainty, and my limited patience. The two views look at the same product and produce almost opposite interfaces.

A few habits came out of this. I watch real users as early as possible and treat their questions as bug reports against the design rather than gaps in their knowledge; if three people ask what a control does, the control is wrong. I treat user trust as something the product has to earn by showing, honestly and continuously, how well the AI is doing, because nobody should rely on a system just because its output looks plausible. And I think hard before exposing a parameter, since every knob hands a decision to the person least equipped to make it.

The builder can never experience their own product cold, which is why pilots with real users belong at the start of design rather than at the end as a validation step. I came into this project thinking my job was to make the system capable. By the end, the job looked more like making the user capable, with the system as the means.