A framework for judging generated answers on two new AI chat surfaces at TOPdesk, a self-service assistant (VSA) and an operator-facing Copilot, and the product changes it drove.
My role: I designed the evaluation, built the review interface, ran the sessions, and wrote the analysis.
Two AI chat surfaces were being built on retrieval-augmented generation, and there was no systematic way to say whether a generated answer was helpful, safe to send, or correctly declining to answer.
That is the setup where a feature ships on the strength of a few good demo answers, and the fabricated citation or the mishandled safety-ambiguous query surfaces only when a customer finds it. What was missing was not more sample answers. It was a framework that made those judgments repeatable, and a review setting that let people make them well.
A note on confidentiality: this is TOPdesk product work. Participant quotes, incident references, prompt text and model outputs are TOPdesk-internal; findings are stated at the level of pattern rather than instance. The screenshots use fabricated sample data built for this write-up, and the answer and ticket text in them is blurred.
I built the review interface to look like the product itself, because judging a conversation out of context is a different task from judging it where it actually happens.
Both surfaces are conversational: the thing under evaluation is a turn in a conversation, not a paragraph in a spreadsheet. A spreadsheet of question-and-answer pairs asks a reviewer to imagine the surrounding product and then judge what they imagined. So the tool loads the case workbook and renders each turn inside a recreation of TOPdesk's own incident screen, with the AI panel docked exactly where it sits in the real product, and with the retrieved knowledge items and the ones actually cited sitting alongside the answer. Operators judged in the context they know from daily use, which kept their reactions about the answer rather than about the format they were reading it in. Each case takes a verdict and a free-text note, and notes persist across re-runs, so a revised answer set can be compared against the previous round without losing reviewer history.
Self-service portal, AI Mode
The test-case viewer rendering each question inside the portal itself, with the answer and its sources.
Incident view, Ask AI panel
The same viewer for the operator side, with the AI panel docked where it appears in the product and the retrieved knowledge items below the answer.
Both surfaces sit on the same retrieval stack, but they are not the same product and could not be judged the same way. Four rules did most of the work of keeping the results honest.
The end-user assistant was judged on helpfulness, tone and appropriateness, but not on correctness, since no reviewer owned the knowledge base. For the operator assistant, a pass meant the draft was useful to the operator, not that it was ready to send to a caller. Different bars, different people, different case sets: the two rates are reported side by side and never pooled or compared.
The operator questions were ours, written to probe, including deliberately awkward phrasings. When an objection was really about how we worded the question, it graded the test design rather than the product, and was excluded. Robustness to a badly formed prompt stayed in: a real prompt can contain anything.
Every finding carries how many participants raised it, on how many cases, and whether it was volunteered or drawn out by a moderator question. A strongly worded reaction from one session stays a single reaction rather than being promoted into a theme.
Passes needed a stated reason too, not just failures. Each failure was also tagged with a root cause plus, where relevant, a contributing second one, so the taxonomy shows what drives failures instead of listing symptoms flat.
| Set | Participants | Judged | Pass | Fail | Rate |
|---|---|---|---|---|---|
| End-user assistant (VSA) | 3 | 95 | 65 | 30 | 68.4% |
| Operator assistant (Copilot) | 2 | 52 | 45 | 7 | 86.5% |
Qualitative research on a small sample: every number indicates direction rather than measurement, and the two rates answer different questions.
Failures clustered into seven root causes, each carrying its own kind of fix: conversation design, interface, retrieval, prompt, or knowledge-base content.
Fifteen findings, each with a recommendation and a confidence label. The strongest were the ones participants reached independently, without being asked.
The evaluation fed straight back into the product rather than stopping at a report. Findings were labelled safe to act on now or directional, so the team could tell which ones carried enough evidence to build on, and they drove three kinds of change:
Two limits decide how far each finding can be pushed, and both are stated in the report rather than left for a reader to infer:
Both limits point the same way, and they set the follow-up work below.
Every tester so far works at TOPdesk. Repeat both halves with customer service desk agents and customer end users, reviewing answers drawn from their own knowledge base.
The argument for it is commercial, about customers whose knowledge base is too small to carry the feature. Internal testing will not answer it, and the people who would enable the setting are the ones to ask.
The study did not ask whether the answers are true. A customer reviewing answers drawn from their own knowledge base can assess whether an answer is faithful to it, so this fits inside the customer sessions rather than beside them.