What Clinicians Actually Need From an AI Scribe

"AI scribe" products are having a moment, and the pitch is straightforward: record the visit, transcribe it, done. That pitch undersells the actual problem, and it's why a lot of these tools produce something clinicians don't trust enough to use unedited. Having built an AI-powered clinical visit documentation system that clinicians actually use as part of their daily workflow, transcription turned out to be the smallest part of the engineering problem.

The conversation isn't the document

A clinical visit, transcribed word-for-word, is not a clinical note. It's a meandering conversation — small talk, the doctor thinking out loud, the patient describing symptoms in plain language rather than clinical terminology, both people talking over each other occasionally. A useful AI scribe has to do real work between "here's the transcript" and "here's the note": identify which speaker is the clinician versus the patient, extract the clinically relevant content from the surrounding conversation, and structure it into the sections a real chart expects — chief complaint, history of present illness, assessment, plan — rather than handing back an unstructured wall of text with the small talk still in it.

This is a genuinely different problem from transcription accuracy. You can have a perfect, word-for-word transcript and still produce a useless clinical note if the system can't tell the difference between "the patient mentioned they're allergic to penicillin" and "the doctor was just asking a routine screening question."

Real-time capture changes the interaction, so it has to be unobtrusive

The alternative to real-time capture is a clinician documenting after the fact from memory, which is worse for accuracy and adds administrative time to every single visit. But real-time audio capture only works if it doesn't change how the visit feels for the doctor or the patient — nobody wants a visit that feels like being recorded for a podcast. That constraint pushed the engineering toward capturing audio directly through the browser via the Web Audio API and a custom AudioWorklet, streaming compressed audio to the backend over WebSocket as the conversation happens, rather than requiring a separate recording device or an awkward "please wait while this uploads" step at the end of the visit.

Clinicians need a review step, not an autopilot

The single most important design decision on this kind of system isn't a model choice — it's the workflow around the model's output. An AI-generated clinical note that goes straight into the patient record without a human checking it is a liability, both because models occasionally make mistakes and because the clinician, not the software, is legally and professionally responsible for what's in that chart.

What worked: a structured review-and-submit workflow where the clinician sees the AI-generated documentation organized into clear, editable sections — tabbed by document type, since a single visit can generate several distinct outputs (the visit note, a patient instruction summary, in some cases billing-relevant codes) — before anything is finalized. The AI does the drafting; the clinician does the reviewing and signs off. That framing, not "the AI writes your notes," is what actually gets adopted, because it respects that the clinician remains accountable for the record.

Configuration matters more than model choice

Two clinicians using the same underlying AI model want meaningfully different output — different note formats, different levels of detail, different terminology conventions depending on specialty. Rather than one-size-fits-all prompting, giving each clinician configurable note-generation settings (format preferences, which sections to auto-populate, terminology defaults) made the tool feel like it adapted to how they already practiced, instead of asking them to adapt to the software. That configurability turned out to matter more for actual daily adoption than any incremental improvement in the underlying language model.

The bar is trust, not novelty

Clinical documentation is one of the domains where "the AI got it mostly right" isn't good enough, because "mostly right" clinical notes create real risk. The systems that actually get used daily aren't the ones with the flashiest AI — they're the ones that make it fast and natural for a clinician to review, correct, and confidently sign off on what the AI drafted, every single time, without exception.