Purpose
Medical schools across the United States have an ongoing demand for a robust bank of National Board of Medical Examiners (NBME) style multiple choice questions (MCQs) or ‘items’ to evaluate knowledge application. These MCQs are valuable for active learning sessions like flipped classrooms and as highly-preferred practice problems for students preparing for exams.

Methods
We postulated that training medical students in item writing could be an effective strategy to support their development as students and future medical educators while also expanding the school’s question bank. To this end, we created an 'item writing immersive experience,' a fully virtual summer course for third-year medical students interested in medical education careers. The course was designed to leverage peer learning and deliberate practice to provide experiential learning in item writing to these medical students while they were on clinical rotations across the country.

Results
Fourteen students participated and generated 42 new items spanning a variety of topics, which were added to the school’s question bank. After the course, all students reported feeling confident about writing their own NBME-style MCQs. They acknowledged the course's benefit in learning the content, felt more engaged in their educational journey, and reported a better understanding of navigating MCQs.

Conclusion
Item writing offers medical students a unique opportunity to simultaneously deepen their clinical knowledge and develop pedagogical skills. Engaging in this activity helps students develop skills that support their future careers as educators and enhance their institution's educational resources. Additionally, this experience can foster trust and build stronger alliances between students and faculty. The fully virtual format of the course enables participation of the students when they are away for clinical rotations. Further research is needed to identify whether teaching item writing to medical students effects their performance as test takers.

Purpose
Students in educational degree programs must demonstrate sufficient mastery of knowledge to pass summative assessments to advance further in training toward a health science career. Setting an examination passing standard is crucial to provide a means for classifying students as sufficiently competent to move forward and obtain a degree. We investigated the outcome of academic success in passing a pre-clinical curriculum using standard setting cut-off scores derived from the Yes-No Angoff method which yields passing standards set by faculty experts assessing levels of student performance. 

Methods
We analyzed exam performance for three student cohorts across four pre-clinical semesters with application of the Yes-No Angoff procedure.  Cut-off scores using an Angoff faculty-derived standard were compared to a historic single cutoff score, and to lowest performing students.  We compared these cut-off scores to pre-matriculation admissions qualifying examination scores predicting academic success.   We applied binary logistic regression and receiver-operator characteristic (ROC) analyses to inform passing standards that predicted academic success.

Results
We found the Yes-No Angoff yielded the best predictor for academic success in the first year of the curriculum.  Our school’s historical 65% passing standard was the best predictor for the second year of the curriculum.  ROC analysis showed excellent or outstanding prediction results for 11 of 12 semesters studied.

Conclusion
The Yes-No Angoff standard setting method yielded cut-off pass scores comparable to the school’s historic pass score, and provided acceptable to outstanding ROC classification results for predicting academic success aligned to pre-matriculation qualifying examination results for admission to medical school.

Purpose
Workplace-based assessments (WBAs) play a pivotal role in evaluating the competency of medical students in the ever-evolving technological landscape of medical education. However, variations in assessment instruments across clerkships can intentionally or unintentionally lead to differences in responses, thereby affecting the assessments' validity, reliability, and comparability. This lightning talk will comprehensively review best practices for evaluating WBAs in medical schools. The focus will be on identifying elements in the assessment instruments that may cause differential responses. 

Methods
Through an analysis of case studies of 5 medical schools’ WBA data collection instruments. Using these, the talk will delve into the methodologies for reviewing these instruments, the potential sources of variations, and their impact on the assessment outcomes. It will also discuss strategies to mitigate the effects of these variations and enhance the fairness and accuracy of WBAs while recognizing the importance of clerkships’ specific information needs. Of particular focus will be the roles that technological advances (AI, LLMs, etc.) can have on these issues. 

Results
Commonalties among schools – both challenges and successes – will be shared, along with the emerging best practices in assessing the assessments that have been developed as a result of this project. This will promote a reflective discussion among schools on their own assessment review processes among participants.

Conclusion
This talk aims to foster a dialogue about the importance of regular review and calibration of WBAs at the institution and clerkship level in medical education. We believe this will contribute to the ongoing efforts to improve the quality of medical education and training and that addressing these issues is even more important as assessment turns to automated, technological solutions.

Purpose
Feedback is essential for learning and developing competence. A common myth of competency-based assessment is that all feedback must be positive. In this study, we examined narrative feedback on workplace-based assessment forms (FieldNotes) to identify patterns in feedback about resident performance gaps. 

Methods
We used learning analytics through secondary data analysis to examine seven years (July 2016 to June 2023) of de-identified FieldNotes at one mid-sized Canadian Family Medicine residency program. FieldNotes include a narrative summary of feedback shared with a resident after direct observation, as well as an indication of resident competence (Progress Level). We extracted FieldNotes where the indicated Progress Level was “Stop, Important Correction” (“Stop”) for analysis. 

Results
Of all FieldNotes examined (N=42,383), 2% were designated as “Stop”. Within any academic year, 30% of learners received a “Stop” FieldNote. The majority of “Stop” FieldNotes included feedback about communication skills, equally split between verbal and written communication. The most frequently addressed topics in the feedback on “Stop” FieldNotes were identifying a sick patient with respect to vital sign abnormalities, abdominal pain, trauma surveys, neurological physical examination, delivering infants, gynecologic procedures, suturing, intubation, charting issues (e.g., incomplete documentation of treatment plan), and time management efficiency in clinic.

Conclusion
Our results suggest that feedback in competency-based assessment does include corrective feedback identifying gaps in competence. Sharing feedback with learners about areas to improve is beneficial to the development of competence and should be normalized in training programs as good assessment practice that does not reflect poorly on a learner.

The purpose of this study is to explore the influence of the COVID-19 pandemic on student test anxiety with an aim to identify if specified factors pose a risk for higher anxiety perceived by students and to identify gaps in student support to inform further actions to enhance student success.

Methods
Students were invited to participate in a voluntary, confidential online survey consisting of multiple-choice, Likert scale, and qualitative items. Survey questions include demographics, student resources, comparisons prior to and during the pandemic related to residence, sleep, exercise, academic preparation, and perceived anxiety. Integrated into the survey is a validated Online Test Anxiety Inventory developed by Abolghasemi et. al and a separate Coronavirus Anxiety Scale.  Dependent variables include psychological, physiological, and online anxiety. Independent variables include faculty and staff support, caregiver status, residence change, race, gender, undergraduate versus graduate status, cumulative GPA, and coronavirus anxiety. A stepwise regression model was employed and qualitative comments reviewed for themes.

Results
66 students (79% response rate) participated in this study.   
32.2% variance of students’ psychological anxiety and 32.5% variance of students’ physiological anxiety are explained by 2 independent variables (cumulative GPA, coronavirus anxiety). When controlling other variables, with 1 point increase on student’s GPA, psychological anxiety will reduce 6.815 units; with 1 unit increase on coronavirus anxiety, the psychological anxiety will increase 0.649 units; with 1 point increase in students’ GPA, the physiological anxiety will decrease 3.606 units; and with 1 unit increase in coronavirus anxiety, the physiological anxiety will increase 0.699 units.  No independent variables were selected into the final model for online anxiety based on stepwise regression.  

Qualitative data themes include student requests for practice questions and study guides, and anxiety related to online proctoring systems.  

Conclusion
Recognizing the influence of the pandemic on student academic performance is crucial to providing student-centric support to foster student wellness. It is essential to note that students’ lived experiences vary, and the need exists to explore individual challenges faced by students to construct support structures that nurture the individual.  Students’ return to in-person learning and assessment methods at the end of the pandemic presents an opportunity to investigate factors that contribute to student success in readjusting to the in-person classroom.

Purpose
Healer’s Art (HART), an elective typically offered in-person at over 70 medical and health professions schools for three decades, has been shown to foster Professional Identity Formation (PIF), however, no study has fully defined the emergent subthemes of PIF resultant of virtual HART instruction. We delivered the HART virtually, and have defined nine PIF dimensions students gained after engaging in virtual HART. 

PIF is a complex process guided by social construction and reflection,​​ and in medicine, includes aspects of self as a professional and self as healer.​  While most medical school instruction is focused on the professional and cognitive domains of medicine, HART is an elective course that offers non-cognitive education and validates physicians as healers. HART has been shown to decrease burnout, increase feelings of community, and foster development of intrinsic professional identity.

While previous research has shown that HART students are better able to define professionalism and incorporate community and authenticity into their framework, the research does not fully explicate the dimensions of PIF, nor does it address the validity of virtual instruction to support PIF.​​ Our research supplements this work by defining nine subthemes in HART PIF and demonstrating PIF in a virtual setting. 

Methods
Using end of course surveys from 25 schools and 570 students, we employed constant comparative analysis to define nine unique subthemes that students expressed as they discussed valuable lessons learned. Two independent coders analyzed student responses and achieved at least 85% interrater reliability.  

Results
Of the 465 student responses to the prompt of “most valuable lesson learned,” 211 (41.4%) indicated PIF. When asked the benefits of having a virtual HART experience (n=444), 114 (29.5%) students specifically identified PIF. Overall, there were 726 unique references to PIF across all student responses. The most common subtheme was Listening (n=115), followed by Humanity (n=82), and Compassion (n=76). One student recalled they gained an “understanding that silence gives another person a chance to comprehend what has just been said (e.g. when giving bad news),” thus shifting their identity as a healthcare provider from doctor to listener. Other themes include Authentic Self, Non-Judgment, Humility, Respect, Service, and Vulnerability.  

Conclusions
Our study highlights the importance and value of HART and non-cognitive emphasis in medical education and demonstrates the effectiveness of virtual HART instruction.  Further, we show nine emergent dimensions of PIF. Whether through convenience or necessity, learners can develop PIF in an online setting, thus reinforcing how virtual HART can still maintain impact.

Purpose
Knowledge of anatomy is a foundational building block in any medical curriculum. At the Larner College of Medicine (LCOM), anatomy is taught 1st semester but is not revisited following this early exposure. Our monthly review sessions aim to provide medical students the opportunity to practice spaced retrieval of anatomical concepts to maintain and enhance competency of the material to aid in Step1 preparation and clinical clerkships. Recently, we have begun implementing student-led sessions. Here, we sought to explore student knowledge retention and satisfaction after the session.

Methods
Our 1-hour monthly sessions are designed to align relevant anatomical material with the medical school curriculum. Our most recent peer-led session was on the Brachial Plexus, targeted to first year medical students but open to students of all levels. The session consisted of an 8-question pre-session quiz to evaluate students’ existing knowledge, followed by the workshop, which consisted of a review of relevant material and concluded by multiple-choice case-based questions. Students were given 60-70 seconds to answer each question and encouraged to work in groups. At the end of the session, they were given 10 minutes to complete the post-session quiz to evaluate retention of material. Students also completed a quality and improvement survey to gauge satisfaction of the sessions. A paired t-test was used to evaluate statistical significance between pre- and post-session quiz scores. Qualtrics XM was used to administer quizzes and the survey. All responses were anonymous.

Results
17 people completed the pre- and post-session quizzes for the Brachial Plexus session. The mean score on the post-session quiz increased by 20.6% (r2=0.8, p<0.01). 6 people completed the post-session survey. 5 of the 6 students responded that they were “extremely satisfied” with the session, that they would be extremely likely to recommend a peer attend the sessions and that they would use this material to study for Step 1. Overall, students enjoyed the visual presentation and interactivity of the session, though some mentioned that the pace was too fast and that they would have liked to have the session available virtually and recorded.

Conclusion
The anatomy review sessions have been well received by our 1st class of medical students as evidenced by the positive comments on the survey. The post-session quiz data supports that students are increasing their short-term retention of anatomical knowledge in these sessions. Going forward, we hope to retain attendance from the pilot class while also expanding to include the new class of medical students. Our future aims are to transition towards a predominantly student-led structure, expand accessibility to include an option to attend and access materials from the sessions virtually, and eventually take steps to incorporate anatomy review sessions into the curriculum to increase anatomical knowledge retention.

Purpose
Anatomy remains a foundational and critical component of health science education. Teaching donor dissection largely relies on paper-back, spiral-bound manuals that are discipline-non-specific. In-house structure lists lack context and guided instruction necessary to perform pre-requisite steps to identify target structures. Thus, there is a need to look ahead in this virtual digital era. We propose a simple, time- and cost-efficient way to enhance learning and discuss the outcomes and feedback gained during the creation and revision of an interactive “eDissection” manual.

Methods
An eDissection manual was created using Rise 360 within Articulate 360 and exported as an HTML file hosted on SharePoint. Multiple cohorts of health science learners were encouraged to access SharePoint and use the eDissection manual. Short Qualtrics evaluation surveys requesting a ranking of each course from 0 (not at all helpful) to 10 (extremely helpful) as well as two free text responses: what was most helpful and what needs to be changed were used to guide revision to the manual.

Results
Responses indicated that learners found the modules helpful in guiding dissection (8.8/10 ± 1.3) and understanding content (8.6/10 ± 1.5). The modules were easier to follow than traditional manuals and were preferred to alleviate prep anxiety and provide knowledge about specific expectations. Improvement suggestions included more videos, tagged structures on donors that the students could identify on their own, and more active learning and retrieval practice activities. We updated the eDissection manual with interactive Storyline 360 content and knowledge checks to continuously improve the learner experience.

Conclusion
It is possible to innovate in one of the oldest, most fundamental courses in health science education. This eDissection manual tailored to the specific learning objectives for our institution is the first that we have seen described. This innovation is shareable and can be tailored to other institutions thus enhancing teaching in the virtual digital era.