You cannot roll back a phone call

A research engineer at Ufonia gave a talk this week about a voice agent that phones patients for pre op checks and post op follow ups. He says they have run around 200,000 real clinical calls across 20 hospitals in the UK and are contracted to reach a million patients in two years. His job, as he describes it, is proving the thing is safe before a patient ever hears it.

The constraints he opens with are the good part. You cannot run an A/B test on patients, because randomizing someone into the worse variant is unethical and often illegal. You cannot undo a call, because it has already been said. And a vendor benchmark number is not a defense at an incident review.

Notice what that takes away. Ship to five percent, watch the dashboard, roll back if it goes red. That playbook is not safe because it is careful. It is safe because being wrong stays cheap for a little while. When the dashboard going red means somebody was harmed, the loop is gone.

What they built instead is a simulator, with one model playing the patient and another judging the transcript against hazards written down with clinicians. He says the judge was checked against ten clinicians on 240 labelled examples and scored an F1 of 0.96 with near perfect sensitivity. He is also careful to say that passing simulation proves nothing on its own. It earns the right to try on real people, in stages, with clinicians watching.

The vault’s reliability notes treat evals as the thing that decides whether a workflow is safe enough to ship at all, which is a heavier job than testing.

His own summary beats mine. You do not ship the model. You ship the evidence.