Innovations in Instruction & Assessment

Abstracts are in the order they will be presented for both Lightning Talk blocks on Monday, December 5

Esther Dale
University of Minnesota Medical School

Medical education emphasizes integration of basic and clinical science, but little guidance exists for how to develop educational activities that effectively support cognitive integration, and build on authentic connections between pre-clerkship and clerkship curricula. The Aquifer Sciences initiative is a collaboration between approximately 200 faculty from IAMSE and the Aquifer consortium that has generated a robust curriculum database that is publicly accessible and can be used for the above purposes. The database houses a set of ~105 core concepts from 11 basic science disciplines, across 15 organ systems. The database also contains a set of ~800 'integrated' learning objectives (ILOs) that connect key clinical conditions to those core basic science concepts through relevant clinical decisions. Each ILO is paired with a basic science teaching point and a 'harm statement,' directed at elucidating how this knowledge helps the practitioner prevent harm to the patient. Use of this database is intended to ensure that learners' basic science knowledge is effectively organized for clinical practice and life-long learning. Because there is no national medical curriculum with cross-cutting concepts, we used the Aquifer Sciences database, an open educational resource, as a proxy. The University of Minnesota Medical School's Office of Assessment and Evaluation used the database as one of three resources (Aquifer database, USMLE Content Guide, UMN course syllabi with learning objectives) to create a bank of 10,000 preclinical multiple-choice items written by advanced medical students. Questions are comprehensive, covering all of 11 basic sciences at different cognitive levels including knowledge, comprehension, and application. While most questions were clinically based, they were tagged to one or more Aquifer core concepts, demonstrating the utility of a core basic science concept in clinical practice. In addition, student writers used the Aquifer learning objectives in various ways. Since the questions were written for preclinical students, the questions focused mostly on diagnosis. If a learning objective went beyond diagnosis to management, prevention, or workup, student writers broke down the learning objective into enabling objectives and created a set of multiple-choice questions that could be used in scenario-based learning. The tagging system and learning objectives encouraged writers to view a concept or system via the lens of different disciplinary approaches and points in clinical care (e.g., diagnosis, management). To date, 66 medical students and three medical professionals (specialized in areas of curricular need) generated 9400 of the 10,000 items for approximately $23 an item. The creation of 10,000 preclinical items is a milestone, however we see potential to generate many more thousands of preclinical and clinical items informed by the Aquifer database.

Caitlin Howe
Drexel University College of Medicine

When incorporating new technology into medical curricula, it is essential to evaluate student success and preference of resources. Our team created a database of 3D scanned dissections for students to use while studying Gross Anatomy. Incoming first-year medical students were recruited to examine the effectiveness and preference of this resource. The study was conducted in four parts: a pre-test using physical dissections and images of 3D scanned dissections, a teaching session using physical dissections or interactive 3D scanned dissections, a post-test identical to the pre-test, and a post-test survey. The post-study survey included background, Likert-scale (1=strongly disagree, 5=strongly agree), multiple choice and slider type questions, in addition to an open area for additional comments. 29 students participated in this study (physical dissections teaching group=15; 3D scanned dissections teaching group= 14). Exam scores significantly increased in both groups (physical= 42.6% ± 17.9%; 3D= 44.3% ± 24.0; P<.001) and no difference was found between groups or past gross anatomy experience. Students taught using the 3D scans were more likely to agree that they "would be able to sufficiently learn anatomy using 3D scans" (physical= 3.0 ± 0.8; 3D= 4.1 ± 1.1; P<.01). In addition, the students taught using the 3D scans were more likely to agree that they "enjoy using 3D scans to explore anatomical regions" (physical= 4.1 ± 0.9; 3D= 5.0 ± 0.0; P<.01) and that they "would be able to sufficiently learn anatomy using 3D scans" (physical= 3.0 ± 0.8; 3D= 4.1 ± 1.1; P<.01). Students preferred interactive features such as identification (count=20) and practical-style (count=24) quiz questions. Regardless of teaching group, students disagreed that they "would have a similar lab experience if I learned using 3D scans instead of dissection" (physical= 2.1 ± 0.6; 3D= 2.5 ± 0.8; P>.05) but agreed that they would use the 3D scans to prepare for the dissection lab and exams (physical= 4.5 ± 0.8; 3D= 4.9 ± 0.3; P>.05). This study demonstrates that 3D scans are a comparable resource to physical dissections in anatomy learning but students do not support replacing physical dissections with 3D scans.

Urvashi Vaid
Sidney Kimmel Medical College at Thomas Jefferson University

The clinical learning environment (CLE) is fraught with uncertainty that manifests from intersecting elements that are in constant flux- organizational, physical, social, influence of diversity, situational and emotional. Medical students are exposed to the CLE with little formal training in managing uncertainty, moving from black and white standardized tests in the pre-clinical curricular years to being immersed in patient care replete with all elements of uncertainty. A lack of experience in embracing uncertainty leads to medical student discomfort and can lead to over-prescription of diagnostics and treatments by future clinicians. This uncertainty of everyday clinical practice was brought to the forefront by the COVID-19 pandemic and truly illuminated the gap in training students and physicians receive to function in uncertain environments. We aim to address managing uncertainty in the pre-clinical years using our Case-based learning curriculum. In 2017, our medical college underwent a curriculum transformation and deployed a horizontally and vertically integrated curriculum comprising of basic, clinical and health system sciences anchored by small-group learning. Students meet in groups of 10 with a facilitator and analyze a clinical case as it unfolds over 4 hours every week. The transition from this classroom approach to the clinical learning environment poses students with specific types of clinical uncertainty, including uncertainty with diagnosis, interpretation of tests, management, prognosis, communication, and care coordination. Our goal is to insert specific elements of uncertainty in the cases to foster a dialogue amongst the students, increase their comfort with these elements and formulate frameworks to help them embrace working through uncertainty and complexity. One such curricular intervention planned is the creation of a case where the diagnosis is uncertain but management decisions must be made to proceed with patient care. Prior to this case-based intervention, the curriculum provides a theoretical basis to approaching uncertainty via lectures and modules. The case-based group is prompted to engage in: 1) sense-making using the Cynefin framework; 2) naming the type of uncertainty at hand; and 3) identifying appropriate problem-solving strategies. We aim to capture and assess student responses through pre and post questionnaires and facilitator observations. A subsequent survey in their clinical years can attest to long term retention or value of this early training. Our fundamental goal is to normalize uncertainty intrinsic to clinical practice and provide our students with the tools to address uncertainty in the CLE. By introducing these concepts early and in a clinical framework, we hope it will feel familiar and not paralyzing when they enter the clinical realm.

David Mullins
Geisel School of Medicine at Dartmouth College

Objective: As we emerge from a paradigm-shifting global pandemic, there is increasing interest in preparing our future medical professionals to identify and respond to emerging infectious disease (ID) threats quickly and effectively. However, the traditional approach to ID education in the undergraduate medical curriculum has tasked students to memorize long tables of "bugs-and-drugs" while largely ignoring the influence of geography, political structures, social practices, and economic limitations. The address this deficit, we developed a short course, based on viral hemorrhagic fevers, as a vehicle to explore the multifaceted and interdisciplinary responses to emerging disease outbreaks.

Introduction: Few diseases inspire panic like hemorrhagic fever viruses: international ID surveillance systems are constantly searching for signs of the next emerging disease, yet few medical professionals are knowledgeable of the interdisciplinary nature of an effective response. In this course, students study major hemorrhagic virus outbreaks at the intersection of foundational science, clinical medicine, environmental science, anthropology, and epidemiology. Embedded themes included geopolitical (in)stability, environmental change, culture, health equity, diversity, and finally, the host-pathogen-environment relationship.

Methods:  During each session, students divide into five "departments:" Epidemic Intelligence, Environmental Science, Cultural Anthropology, Clinical Science, or Epidemiology. Each Department is provided with unique information, requiring students to work together as a team under time pressure to construct a "narrative arc" to explain the source and projected progression of real-life historical hemorrhagic fever epidemics. All evidence are from primary sources and ranged from ancient artwork to animal migration charts. Only after solving the case is the pathogen revealed. Although we used historical outbreaks, the course structure is broadly applicable and provides a framework for critically assessing other epidemics, including COVID-19. Strengths of the approach include: 1) Modular session format is adaptable to any infectious outbreak; 2) Every student has a significant role in solving the case; 3) Students appreciate the interdisciplinary nature of ID surveillance and response; 4) The comprehensive session materials and active learning approach obviate the need for faculty with expertise in each interdisciplinary field, allowing for widespread adoption.

Results: In the initial offering, students evaluated the course 5.0/5.0 for enjoyment/engagement and 4. 7/5.0 for overall quality. Many students identified the collaborative nature of the task and their excitement to learn more about IDs as major drivers of satisfaction. We are in the process of preparing the materials for open-access publication.


Andrew Cuyegkeng
University of California, Irvine, School of Medicine

Medical student learning and success are two of the most important aspects of the University of California, Irvine School of Medicine (UCISOM) curriculum as it prepares the next generation of highly trained physicians. To ensure equal opportunity for medical students to achieve success, the Collaborative Learning Communities with Medical Students as Teachers (CLC-MSAT) was created in 2020. Drawing from a myriad of studies in holistic education, cognitive neuroscience, and positive psychology, the program provides an inclusive, learner-centered framework that supports long-term learning strategies, exam preparation, and professional growth. In our CLC-MSAT model, the more senior student cohorts provide guidance and learning strategies to the more junior student cohorts via exam review tutorials, biweekly case-based and board-style question collaborative learning communities, academic mentorship, and small group tutoring -- all at no cost to students. To assess medical students' perceptions of the CLC-MSAT model, a 29-question survey instrument was created in which questions were categorized into sections including overall experience, tutor knowledge, classroom environment, tutor instruction, professional responsibilities, and perceived impact. The survey was created using Qualtrics XM and distributed to the 1st and 2nd year medical student class (MS1 and MS2, respectively) via URL through class listserv three times during the course of the 2021-2022 academic school year; October 2021, December 2021, and February 2022. For the October 2021 survey, a total of 51 entries were analyzed, with 33 responses from MS1s and 18 from MS2s. 34 entries were analyzed for the December 2021 survey, with 19 of those being from MS1s and 15 from MS2s. Finally, 38 entries were analyzed for the February 2022 survey, with 29 of those being from MS1s and 9 being from MS2s. Throughout the year, more than 94% of students found the CLC-MSAT program to be valuable, and 93% of students also noticed positive results towards their personal goals after engaging with the program. Many students reported significant positive impact of the program on their learning, and 100% of survey respondents in February 2022 stated that the program increased their confidence in feeling prepared for exams. Conversely, less than half of students stated that the program helped them build stronger peer relationships and become better organized with their work. Given these results, the CLC-MSAT program has shown promise in cultivating academic excellence in medical students, despite its infancy. These findings will be used to better inform developments within the program, with the hopes that the CLC-MSAT framework can be utilized by other institutions, allowing for a strong positive impact on academic preparedness and achievement.

Audrea Burns
Baylor College of Medicine

Introduction: Residency training serves as a formative period of clinical and professional development for physicians. Notably, the Accreditation Council for Graduate Medical Education (ACGME) has listed professionalism as one of six core competencies and is a required assessment competency2. Despite advances in defining, teaching, and, providing formative and summative assessments for resident trainees within the professionalism domain, numerous studies have extensively demonstrated the high frequency of incidences of unprofessional behavior that compromise patient safety and increase health care costs3. Furthermore, trainee observations of lapses in professionalism during residency report little overlap between the values stated by accreditation standards in the United States and observed behavior during training4. The theory of threshold concepts can serve as a paradigm-shifting framework in understanding learner difficulties or otherwise termed troublesome knowledge in continuous modeling of professional behaviors. Recently, only a few studies have elucidated threshold concepts in medical students and residents. The purpose of our qualitative research study was to identify a framework for understanding threshold concepts in professionalism amongst residents in pediatric residency training through identifying a key component of threshold concepts, areas of troublesome knowledge. Our overarching goal is to better align national requirements in professionalism and observed behavior during residency training. Methods: To determine what threshold concepts pediatric residents experience during training in the domain of professionalism, we sought to identify difficult areas of knowledge, or troublesome knowledge, which is an essential component in identifying threshold concepts. We conducted a qualitative research study using constructivist approach to grounded theory to identify troublesome knowledge in professionalism. Eleven focus groups were conducted with residents from PGY1-PGY3 years at a large university-based pediatric hospital in the United States. To gain reflection from pediatricians who had "crossed" the threshold of learning professionalism skills during residency, we also conducted focus groups with BCM pediatric fellows across subspecialties and resident clinician-educators from across the United States. All interviews were conducted virtually, and video recordings were transcribed and coded. The methodology was the conceptual synthesis of the codes using Threshold Concepts as a sensitizing lens and themes were generated.

Results: To unpack domains of troublesome knowledge, we first sought to understand where they learned and how they defined professionalism. Five themes emanated from residents' definition of professionalism including (1) "interaction with others with subthemes of "ro


Brian Yuen
University of Central Florida, College of Medicine

Self-Efficacy is the conviction that a person has in themselves to complete tasks even under challenging circumstances. Current research suggests that self-efficacy is a key correlate to academic success in medical school. Therefore, understanding the role that self-efficacy plays in medical education can be a key step in helping to improve the quality of education for students and increasing performance. One key aspect of self-efficacy is that it is domain specific, meaning that a person's self-efficacy can change depending on the environment/context they are in. For example, a student who is confident in their clinical skills while practicing with a patient actor may have difficulty performing those same skills in a real clinic. Our understanding of which contexts or "domains of functioning" are most pertinent to medical students is limited. Although medical curricula have long defined the core competencies students must demonstrate, there is a gap in the literature regarding what the learners perceive as key domains in which they need to develop self-efficacy and personal agency, and moreover what aspects of curriculum facilitate or impede their development. The goal of this study was to close this gap by surveying students directly busing focus groups in a cross-sectional study using the Nominal Group Technique (NGT). The NGT is a standardized approach to generate a consensus among cohorts. The NGT requires four broad steps: a silent idea generation phase in which answers to a research question are brainstormed alone, an idea sharing phase, a discussion phase, and a ranking phase. We utilized the NGT to generate a rank-ordered list of the top 5 most important domains for medical education (i.e. the top 5 aspects of medical school that required the most self-efficacy). We also performed a second NGT to identify the top 5 aspects of the medical school curriculum that increased self-efficacy. The population surveyed to date included first, second-, and fourth-year classes of medical students. By spreading our study by year, we also captured information regarding how the domains changed throughout one's medical education. We found that students consistently ranked patient interactions (such as performing physical exams and interviewing patients) at the top 1 or 2 for most important domain of functioning. We also noticed that performing well in front of preceptors and attendings emerged during the second year and in the fourth year became the domain that students believed required the most self-efficacy. Another factor that remained consistent between cohorts was the value of being a self-advocate for learning; in other words, students believed that being unafraid to ask questions to learn also took a high degree of self-efficacy. Specific aspects of the curriculum that students agreed best boosted their self-efficacy was programs that gave clinical exposure (like standardized patients), feedback from faculty and support from peers/mentors.

Stephen Schneid
University of California, San Diego

Core Topics in Biomedical Sciences (CTBS) is a nearly eight week summer prematriculation course offered to all incoming medical students and second-year pharmacy students. CTBS underwent major changes ahead of several curricular renewal efforts within the School of Medicine to inform discussions and energize the effort. All class sessions were virtual using Zoom with optional on-campus anatomy/histology sessions for interested students. The decision to make it entirely virtual was to increase access and reduce financial burden on the students. There were no live lectures, but rather an outline, learning objectives, and instructional videos were provided to the students to learn from prior to class the following week. The topics were in weekly themes (Foundations, Neuroscience/Neurology, Autonomic Nervous System, Cardiovascular System, Renal System, Immunology, and Multi-Systems Integration). Team-based learning (TBL) was the pedagogical approach, two days per week. Weekly quizzes were used give feedback on learning progress. Health systems science (HSS) was incorporated on a weekly basis with a final project proposal turned in by the students at the end of the course. Senior medical students and a pharmacy resident served as academic coaches who met weekly with the two to three students. The coaches and students took the professional identity formation essay (PIE) and received detailed feedback. Students discussed professional identity formation with their coaches and created a professional identity formation development plan that was submitted at the end of the course. Students wrote over 20 self-reflections throughout the course that counted towards their final grade along with HSS, quizzes, coaching, TBL, and a cumulative final comprised of multiple-choice questions and open-ended questions. Of the 12 students who have filled out the course evaluations thus far, results were 5/5 for course organization, 4.91/5 for facilitation of learning, and 4.91 for overall course excellence. The highest rated elements were the assessment structure and TBL, both 5/5. The redesigned course is currently being used as model shared with the curricular renewal working groups.

Morgan Wilbanks
Medical College of Wisconsin

Introduction: Bias exists in medical student evaluations and can be associated with gender, race, nationality, and other factors. Many faculty and departments utilize individual faculty evaluations of students to write letters of recommendation for students as they apply for residency. The Linguistic Inquiry and Word Count software is a validated tool that has previously been used to evaluate for bias in medical student evaluations. We analyzed the on-shift evaluation comments of fourth-year students rotating in our large, urban emergency department for bias along student and faculty gender using this program. We analyzed the previous academic year and are collecting data currently for this academic year.

Methods: Student shift evaluation data was compiled over 4 months. Data was coded and anonymized for analysis. Content of evaluations was edited for gross misspellings and to remove student names. Data was analyzed using the LIWC program, with the initial analysis focusing on the length (Words Per Comment, WPC) of separate responses to prompts asking what the student did well (postive comments) and what they could improve upon (improvement comments). The results of data analysis were shared with faculty prior to the current academic year. Data collection is ongoing currently and will be added for analysis when current cohort finishes their rotations.

Results: There were 18 female, 16 male students, and 427 total shift evaluations included. Female faculty were more likely to leave positive comments (68%) than male faculty (55%). Female faculty were more likely to leave improvement comments (45%) than male faculty (34%). Overall, comments from male faculty were longer than those of female faculty (18.45 vs 14.36 words per comment). Female faculty were more likely to write longer positive comments for male students (16.4 WPC vs 12.29), and longer improvement comments for female students (14.07 vs 11.58). Male faculty were more likely to write longer positive comments for female students (19.02 vs 17.79), and there was no difference in length of improvement comments from male faculty between student gender.

Conclusions: Our data shows a gender bias in the amount and type of comments student receive on their performance evaluations, which are ultimately used to write letters of recommendation submitted with students' residency applications. This data was shared with faculty prior to the current academic year, and we intend to show the effect that intervention had on these trends in the current academic year. We hope to model how feedback related to gender bias in evaluation can be effective in addressing this problem.