Andrea Vallevand, University of Houston Tilman J. Fertitta Family College of Medicine
Cynthia Burns, Wake Forest University School of Medicine
Kimberly McDonough, Wake Forest University School of Medicine
Efrem Violato, Wake Forest University School of Medicine
Purpose
The application of artificial intelligence (AI) for building Objective Structured Clinical Examination (OSCE) cases shows promise but is considered early in the development phase. Our purpose was to investigate whether a "built from scratch" (BFS) OSCE case could be comparably created using AI.
Methods
Post-operative fever management has been rated essential for surgical clerkship curricular inclusion. Using simulation development guidelines and the medical literature (5Ws mnemonic), the BFS OSCE case was administered after an eight-week Surgery rotation. Clerkship students examined a standardized patient and then documented the patient’s note (history, physical examination [PE], differentials and workup plan). To investigate whether AI could create a comparable case and assessment checklist, two “zero-shot prompts” (presenting a task without providing training on how to perform the task) and one “few-shot prompt” (providing a small number of relevant examples to guide the output) were employed. The “few-shot prompt” method utilized the BFS OSCE case standardized patient and physician rater training materials.
Results
BFS scenario triggers for patient note history/PE documentation included
- dry cough/diminished breath sounds in lower left lung (wind),
- burning/increased urination (water),
- not walking much/denies pain/stiffness in legs (walking),
- dry/noninflamed surgical incision (wound), and
- received blood transfusion (wonder drugs).
All three AI-generated cases/materials included both history and PE wound checklist items and a PE checklist item for wind. Only the “few-shot prompt” generated materials included both history and PE walking checklist items. None of the generated cases included checklist items addressing wonder drugs.
Conclusions
The “zero-shot prompt” and “few-shot prompt” methods quickly generated foundational materials for case presentation, standardized patient training, and assessment checklist refinement. This was particularly notable with the “few-shot prompt” result. The absence of some case-specific information emphasizes the importance of expert review to ensure content validity and to ensure that the clinical scenario is designed with assessment objectives and scenario triggers in mind.