A Bot Nobody Can Pass Is a Broken Brief

I rolled out the practice simulator to a client's customer-facing team — about a dozen people. Over eleven days they made 124 recorded practice calls. The number went up every single day, and I reported it upward with some satisfaction.

Then I decomposed it, and the decomposition says something close to the opposite of the aggregate.

What was underneath the 124

• One scenario had thirteen attempts across five people and was never passed. Not passed badly — of ten scoring criteria, two were ever satisfied, by anyone, in thirteen tries.
• Nine of thirty-four reviewed calls were under a minute. A sixty-second call cannot contain a discovery question and a close, so those nine contributed to the count and to nothing else.
• Openers scored around forty per cent; the full three-part call scored twelve and a half. The thing being practised was the first ninety seconds, over and over.
• Not one person passed the expansion criterion — the one the entire programme exists to teach, since the whole motion is about widening existing accounts rather than cold pitching.

So: activity high, one criterion universally failed, and the aggregate rising smoothly over the top of it.

The specification bug

My instinct when a rep fails a roleplay is to give them another rep. That instinct is right for individual variance and wrong for what I was looking at, and the difference is diagnostic:

Individual failure is noise. Collective failure at the same criterion is a specification bug.

If two people out of twelve cannot get past a scenario, that is a skill distribution and the answer is coaching. If nobody can — if thirteen consecutive attempts by five different people all die on the same two criteria — then the far likelier explanation is that the scenario is unwinnable as configured, or that the team was never told what winning it requires. Both of those are mine, not theirs.

I had built that bot deliberately cold, as a corrective to earlier personas that were too accommodating. Cold was the right call. What I did not do was write down what a good response to a cold refusal actually sounds like — the eight questions that open it up, the model answer to a flat no — and then hand that to the team before asking them to try again. I shipped the exam without the syllabus and read the failures as evidence about the students.

The corollary about the sub-minute calls is simpler and more embarrassing: any volume target with no floor gets satisfied at the floor. Four calls a day, three minutes each, was the stated standard. The standard with teeth is the one the scoring enforces, and a thirty-second call scored as a call. People are not gaming this; they are responding accurately to what I chose to count.

What I should have measured

One criterion, moving.

Not "calls completed" — that measures whether the tool is being used, which is a procurement question, not a competence one. The useful measure is: pick the criterion that is failing hardest, brief the team specifically on it, retest the same scenario, and compare criterion-level pass rates before and after. That produces a number that means something whether it goes up or down. Volume only ever goes up.

And it would have given me the thing I actually needed, which was evidence for a keep-or-drop decision on the tool itself.

The pilot clock is not the evidence clock

The second half of this, and the part that cost the most calendar time: I spent the first week of a time-boxed trial debugging it rather than evaluating it. Logins that did not arrive for three participants. Recording that failed after a successful sign-in. Short calls timing out. Two accounts under my own name making call attribution unreliable.

I had asked for and received a trial extension, and I treated the extension as a win. It was not a win; it was a week.

Pilot access buys evaluation time, not adoption evidence. The trial clock starts when access is granted and runs whether or not anyone is learning anything, and the infrastructure — invitations, login, recording, feedback delivery — is not a precondition to the evaluation. It is part of what is being evaluated, and it consumes the window that was supposed to produce the decision. The test I should run on day one of any trial, before the deck and before the cohort: get one person through the entire loop — invited, signed in, recorded, scored, feedback read — and only then invite the rest.

Where this goes wrong

Maybe the bot was right. It is entirely possible that twelve people genuinely cannot yet do expansion discovery against a cold buyer, and that the universal failure is an accurate reading of a real gap rather than a broken scenario. I cannot distinguish those two from the data I have. The distinguishing test is cheap and I have not run it: brief the team properly on that one scenario and retest. If the criterion moves, it was a brief problem. If it does not, it was a skill problem and I owe them real teaching, not more attempts.

Criterion-level coaching slides straight into teaching the rubric. I have written this caution before — see 03-The Simulator Told Me What No Real Call Would — and coaching to a named criterion makes it sharper, not softer. If I brief the team on exactly what the scorer rewards, scores will rise and may mean nothing. The only antidote I know is to keep checking the criterion against real calls rather than against the rubric's view of a good call.

And recognising participation rewarded the volume error socially. Four people were named for hitting the call count. None of them had passed the criterion that matters. The recognition was deserved — they showed up — but I should be clear-eyed that I publicly reinforced the measure I had just discovered was hollow, in the same week I discovered it.

The line I want to keep: when the whole cohort fails identically, stop coaching the cohort and go read your own brief.