Presented By: Mary Kate Worden, University of Virginia School of Medicine
Co-Authors: Johanna Craig, University of Virginia School of Medicine
Karen Knight, University of Virginia School of Medicine
Eli Williams, University of Virginia School of Medicine

Purpose 
Artificial intelligence-powered chatbots, like ChatGTP, have shown potential in a variety of medical contexts including research, diagnosis, and patient monitoring.  However, these bots make errors, and their use can raise ethical and legal concerns.  To help first-year medical students recognize the strengths and limitations of using artificial intelligence (AI) as a source for biomedical information students were asked to critique the output of ChatGTP in 300-500 words. 

Methods 
A geneticist wrote twelve brief prompts for ChatGTP that included a clinical vignette about a patient with high suspicion of genetic disease as well as four clinical questions related to the patient presentation.  ChatGTP returned 12 text responses.  Each medical student (n=152) was given one of the prompts and asked to critique the associated response from ChatGTP with respect to its accuracy, completeness, and clarity using the biomedical literature as a reference source. Students then met in small groups with genetic counselors to come to consensus on their critiques and give oral presentations on the diseases. Faculty evaluated the essays and genetic counselors gave students narrative feedback on the quality of their oral presentations. The educational impact of this exercise was evaluated from student responses to a query on the course evaluation. 

Results 
Students reported a deeper understanding of both the benefits and pitfalls of their own use of AI in their role as future physicians.  Additionally, many students reported that the exercise encouraged them to consider how their patients might be using AI and the implications that has for clinical practice.  

Conclusions 
Asking students to analyze sets of known ChatGTP output greatly enhanced the feasibility of this exercise by simplifying the faculty effort required to evaluate student critiques of AI-powered answers to clinical questions.