Qing Zhong    
Rocky Vista University    

Purpose
Online teaching for medical schools was predominant during the coronavirus-2 pandemic of 2020. Medical students' academic performances in online learning were better, not different, or decreased, compared to those in face-face mode teaching. Although similar or higher academic performance, students rated online learning lower than face-face learning. Since 2021, many medical schools have adopted hybrid teaching, which combines online teaching and face-face mode teaching. There are no studies on the evaluation of the lecturers from medical students or comparison of the ratings of faculty among different teaching modalities during the pandemic so far. The purpose of this study is to compare the academic performance, course rating from students, and faculty rating from students on the cardiovascular system II course from online teaching in 2020 and hybrid teaching in 2021 and 2022.

Methods
Rocky Vista University College of Osteopathic Medicine has had two campuses, Parker, Colorado, and Ivins, Utah since 2017. We transferred face-face mode teaching to online teaching in 2020, and hybrid teaching in 2021 and 2022. Academic performances, course evaluation, and evaluation of faculty for the Cardiovascular System II course from 2020 to 2022 were retrieved. Data were analyzed with a student t-test. 

Results
Students' final grades were not significantly different from 2020 to 2022. The rating of the course was lowest in 2020 with online teaching and was significantly elevated in 2021 and 2022 with hybrid teaching. A similar pattern was found in faculty ratings. The average ratings of faculty members from students were lowest in 2020, slightly increased in 2021 with hybrid teaching, and significantly increased in 2022 with more hybrid teaching. 

Conclusions
Although academic performances are similar, hybrid teaching is more welcomed by students than online teaching. Hybrid teaching combines the advantages of face-face mode teaching and online teaching.

Kyle Bauckman    
Nova Southeastern University    

Purpose
Assessment of educational competencies is a required task for medical schools. Traditional competencies like medical knowledge are practical to assess while professionalism is more open to interpretation. Faculty tasked with assessing professionalism are challenged by the subjective nature and the need for methodical documentation. To identify and provide consistent and meaningful feedback on professionalism across the curriculum, a standardized and real-time recording process is critical. We present a methodology to track and assess professionalism via a modified Angoff method. 

Methods
We implemented a professional development plan to optimize the tracking and reporting of professionalism incidences within the pre-clerkship curriculum. Course directors (CDs) were surveyed on common professionalism violations occurring within a course. The Angoff method was utilized to determine CD perception on whether various incidents would merit academic consequences. Survey results were aggregated to identify expectations for a minimally competent student in professionalism. Outcomes were used to train CDs to encourage more uniform reporting and assessment. 

Results
20 of 22 CDs participated in the survey. A threshold of 75% of CDs agreeing was a standardized threshold for a negative grade outcome for any given professionalism violation. CD responses showed higher thresholds for smaller incidences like late assignments but minimal tolerance for disrespectful behavior and unexcused absences. CDs were advised to use this data to help gauge and conform final grades for professionalism in upcoming courses.

Conclusion
Assessment of professionalism has shown to be highly variable between CDs. CDs previously expressed uncertainty on what actions merit monitoring and the frequency needed to deserve failing grades. Our survey and training provide a model for institutions to standardized core professionalism incidences and provide recommendations for thresholds that merit institutional action. Our approach standardizes the process and ensures CDs are institutionally supported when opting to deliver a failure in professionalism.

Justin Wang    
Boston University Chobanian & Avedisian School of Medicine    

Purpose
There has been growing interest in incorporating point-of-care ultrasound (POCUS) into medical school curricula. Previous studies have established a rubric on proper POCUS principles but there is not yet a standardized assessment of proficiency. We created a condensed version of the 4-view cardiac exam to assess POCUS proficiency in pre-clerkship medical students at Boston University Chobanian & Avedisian School of Medicine. This abstract describes the creation and consistency of this assessment tool.

Methods
Criteria regarding POCUS principles were adapted from the Objective Structured Assessment of Ultrasound Skills (OSAUS). A condensed assessment was created based on previous studies using the 4-view transthoracic echocardiogram. In this IRB-approved study, medical students between their first and second years completed the assessment before and after participation in a two-week anatomy and ultrasound summer program. Evaluators trained on the assessment format included professors and emergency medicine fellows specializing in ultrasound. Internal consistency was evaluated using Cronbach's alpha. 

Results
Identification of four structures were included in the criteria for each cardiac view for their clinical relevance. The parasternal and apical 4-chamber views allow measurement of cardiac function, left ventricular ejection fraction and valvular pathologies. The subxiphoid view can assess for pericardial effusion or other life-threatening diseases and is part of the FAST exam. Eighteen medical students completed the pre-assessment and 13 students completed the post-assessment. There was high internal consistency for both the pre-assessment and post-assessment when criteria were pooled together (pre: ? = 0.84; post: ?  = 0.78).

Conclusion
A continued increase in the use of POCUS across clinical specialities and in medical education accentuates the need for standardized assessment of ultrasound proficiency. This project outlined the creation and consistency of an assessment tool of POCUS proficiency for future clinicians.

Bruce Newton    
Campbell University School of Osteopathic Medicine

Purpose
There are numerous cross-sectional and longitudinal studies that have examined the affective and cognitive empathy scores of allopathic and osteopathic medical students as they progress through their undergraduate medical education. However, there is a paucity of similar data concerning Doctor of Physical Therapy students. This study helps to fill this knowledge gap.

Methods
The study used the Balanced Emotional Empathy Scale (BEES; measures affective empathy) and Jefferson Scale of Empathy Health Professions Student version (JSEHPS; measures cognitive empathy). Surveys were given at the beginning and end of each academic year (DPT1 – DPT3) to the 2022-2024 classes. Sex and desired specialty choice (Cardiopulmonary, Geriatrics, Musculoskeletal, Neurology, Pediatrics) was also gathered. DPT1-3 n = 137: 53 men and 74 women.

Results
For combined male and female students, at the start of each academic year, both affective and cognitive scores steadily increased with each successive class. BEES  DPT1-3 = 39.98 ± 34.61, 51.84 ± 27.91 and 56.23 ± 36.50, respectively. The BEES normal population score is 45 ± 24. JSEHPS DPT1-3 = 114.29 ± 9.34, 116.74 ± 10.14 and 117.08±10.06, respectively. Currently, there is no physical therapy student population norm for the JSEHPS.

Conclusions
At the beginning of each of the three academic years, the BEES and JSEHPS scores steadily increased. Although the  DPT1 BEES score is slightly lower than the population norm, it rose above the norm for the second and third year DPT students. Data collection for each class at the end of each academic year will begin in December of 2022 to determine how each academic year impacted BEES and/or JSEHPS scores.  Complete statistical analyses will be provided at the meeting.

Bruce Newton    
Campbell University School of Osteopathic Medicine   

Purpose
Although the BEES was not specifically designed for health professionals, it remains a well-established survey instrument measuring affective/vicarious empathy. Prior studies of this cohort showed a significant decline in BEES scores over students' four years of medical education. Because four of the 30 questions can particularly pertain to health professionals, we wanted to determine how the cohort answered these statements, and if their scores changed from entrance to medical school as compared to the start of year four.

Methods
The study used the BEES which was given at the beginning of each academic year (M1-M4 timepoints) to the 2017-2020 graduating classes. Desired specialty choice was also gathered. Specialty choice was divided into "people-oriented"(Core) and "technical/procedure-oriented" (Non-Core) groups. Data over 4 of the 30 BEES questions that pertained to physicians were compared at the M1 and M4 timepoints with regard to specialty choice.

Results
Overall, students selecting Non-Core specialties had a greater decrease in affective empathy scores than those selecting Core specialties. Students selecting Core specialties showed a larger decreased "sensitivity" to having to deliver bad news vs. those selecting Non-Core specialties. Students selecting Core specialties were more sensitive to the suffering of patients. For a question about self-inflicted serious illnesses, students selecting Core or Non-Core specialties had almost equal decreasing levels of sympathy.

Conclusions
Overall, the CUSOM curriculum had a positive impact on students being more competent/comfortable when delivering bad news to a patient. Students became more cognizant of patient suffering. Sympathy levels decreased for both groups, and this may benefit patients who do not want sympathy, but empathy.

Rebecca Lustfield    
University of South Dakota Sanford School of Medicine    

Purpose
Academic coaching in medical education is an increasingly popular curricular program implemented across broad medical education institutions. The utility of coaching is unique and separate from other academic roles such as mentor or advisor. The goal of an academic coach is to help facilitate learners to achieve their fullest potential, through the review of objective assessments, and identifying student needs within a plan the student owns. However, the literature unfolds programmatic differences in who serves as a medical education coach and the faculty training provided and relative outcomes. Our Midwest undergraduate medical education institution retained statewide faculty physicians to further engage with students and enhance their working knowledge to promote student professional identity and academic success. The current project aims to share the importance of assessing academic coach strengths and gaps to boost faculty development to ultimately support working interactions with students.

Methods
The development of our coaching curriculum serves as a primer to familiarize with coaching principles and monitor associated growth. Initially, 9 faculty members responded to the call to serve as a coach. Faculty were required to attend 2-hour monthly development sessions. A pre/post survey focused on level of comfort to serve as a coach role over time, including demographics, professional background, medical specialty, and range of abilities to serve as academic coaches.

Results
Collected data would be analyzed and used to demonstrate various aspects of faculty coach development in a medical school curriculum.

Conclusion
As coaching is emerging within medical education, it is essential to assess for adequacy in faculty training that aligns with institutional and programmatic goals and objectives.

Ali Aijaz    
Boston University Chobanian & Avedisian School of Medicine    

Purpose
Ultrasound has become an increasingly prevalent component of medical school curricula. In our elective at Boston University Chobanian & Avedisian School of Medicine, we analyzed the impact of low-stakes assessments on students’ confidence with ultrasound techniques. This elective included peer learning guided by trained physicians, practice opportunities, and targeted assessments.

Methods
Second-year medical and graduate students participated in a two-week intensive anatomy and ultrasound fellowship. Before and after the fellowship, students completed a hands-on assessment of cardiac ultrasound ability and a Likert survey regarding confidence in scanning cardiac and non-cardiac structures. The fellowship included five sessions covering cardiac, abdominal, head and neck, upper extremity, and lower extremity ultrasound. A Wilcoxon signed-rank test was used to assess overall confidence improvements, while a Friedman Test was used to assess differences in confidence improvement between cardiac and non-cardiac structures, with ?=0.05.

Results
Thirteen of 17 students completed both surveys and at least one assessment. Confidence with scanning three cardiac structures – heart as a whole, mitral valve, and left ventricle – and three non-cardiac structures – liver, right kidney, and thyroid – was analyzed. While there were significant confidence improvements after the fellowship (Heart: Z=-2.98, p=0.003; Mitral valve: Z=-2.88, p=0.004; Left ventricle: Z =-2.85, p=0.004; Liver: Z=-2.98, p=0.003; Right kidney: Z=-2.97, p=0.003; Thyroid: Z=-3.09, p=0.002), there was no significant difference in the confidence improvements between the cardiac and non-cardiac structures [?²Friedman(5)=1.51, p=0.912].

Conclusion
Our curriculum significantly improved student confidence with scanning all six structures studied, whether or not students' ability to perform that scan was assessed. However, low-stakes assessments played a minimal role in selectively improving confidence. This suggests that guided learning and practice opportunities are more consequential aspects of ultrasound education and should be emphasized over assessments.

Reshma Fateh    
Avalon University School of Medicine    

Purpose
Feedback is defined as the regular mechanism where the effect of an action is to modify and improve the future action. Feedback is essential for developing students' competencies in the clinical workplace and their future work as a professional. The attention of feedback shifted from teachers' feedback techniques to learners' goals, acceptance, and assimilation of feedback and impact-focused approaches. This study explored the views of medical students and faculty regarding the importance of constructive feedback and the process of feedback in medical education.

Methods
An explanatory, sequential, mixed method approach was used, beginning with a survey followed by interviews. It was conducted at DSMA, Myanmar, and AUSOM, Curacao, from November 2021 to October 2022. For the quantitative Phase, 75 students of Phase I, M.B.B.S program and 28 faculty from DSMA and 63 students of the MD program, and 13 faculty from Avalon University used a questionnaire. For the qualitative Phase, 10 students and 10 faculty from each university used In-depth Interviews. The thematic analysis was done by using MAXQDA software.

Findings
Questionnaire results show that most faculty and students strongly agree that feedback is essential for learning and should highlight both strengths and weaknesses of performance. A thematic analysis of the IDI data identified the perception of feedback, barriers to achieving constructive feedback, how the students use feedback in future careers, and the teachers apply it in a real setting to give constructive feedback. The students wanted immediate after the exam and preferred one-to-one instead of group feedback. Still, the faculty were concerned about time limitations in providing constructive one-to-one feedback.

Conclusion
The students and faculty agree that constructive feedback is essential to improve performance. The students preferred one-on-one feedback to have précised comments about their strengths and weakness to improve their performance. The barrier they faced around giving and receiving feedback was time.

Rebecca Lustfield    
University of South Dakota Sanford School of Medicine    

Purpose
Extensive research has identified significantly higher stressors for students transitioning through the medical curriculum. To alter adverse stressors, wellness programs and supportive resources have provided diverse outcomes primarily based on larger institutional-supported initiatives and little attention to monitoring of student wellness achievements in smaller medical institutions. The aim of this project is to monitor student wellness across the later years of the medical student curricula. This topic is important given the need to combat global mental illness, substance use, and burnout which is especially important in aspiring physicians to extend wellness within rural, tribal, and frontier communities.

Methods
A midwestern medical institution (rural and urban) within a Longitunitnal Integrated Curriculum (LIC) queried student wellness through self-reported survey questions over 4 years, primarily during the COVID-19 pandemic. Longitudinal data comprised a five-point rubric with specific emphasis on student balance within a LIC and time set aside for including holistic elements of wellness. Student responses range from highly agree to disagree strongly. Data accounts for changes in wellness from years three to four.

Results
Current data suggests early monitoring for academic and personal strains positively supports facets of student wellness.

Conclusions
This quantitative data highlights student wellness supported the bi-lateral movement in competency and aspiring resiliency.

Felise Milan    
Albert Einstein College of Medicine    

Purpose
The cancellation of Step2-CS created a vacuum regarding a standardized approach to clinical skills assessment (CSA). This has implications across the medical education continuum and for patient care. The GEA has spearheaded a national effort to reexamine these issues across the continuum from UME to CPD.

Methods
In August 2021, calls for participation resulted in over 400 responses including students, residents and faculty from allopathic and osteopathic medical schools representing 46 states and 156 institutions. The project is organized into 12 Taskforces each named for different questions relating to CSA. Groups are exploring the Why, What, How, Who, When, and Where, questions in sequence. A 13th Taskforce examines the role of Technology in CSA. Due to the enormous interest in participation, a unique structure was designed. Each question area (WHY, HOW, etc) has two task forces that work independently, then present their mid-point report to two Advisory Boards to help guide their work. This project is unique in its scope, structure and size.

Results
Work is ongoing although there are preliminary recommendations. An annotated bibliography with >600 references has been created. Task force members are gathering data on best practices and expert opinion to supplement the literature. A recent presentation at the AAMC Learn Serve Lead Annual Meeting included >800 attendees.

Conclusions
Creation of a national standard for CSA needs to be done with full transparency, accountability, fairness, equity and mitigation of bias. No framework or list of clinical skills should be assessed by one high stakes exam. Programs of longitudinal assessment of learners by multiple assessors in a variety of settings are needed. CSA is highly resource intensive requiring consideration for equity across institutions.  Research is needed to link CSA to patient outcomes. Existing regional OSCE consortia holds promise as we move towards a national CSA standard.

Daniel Zahra    
Peninsula Medical School, University of Plymouth    

Purpose
Single-best-answer (SBA) multiple-choice-questions (MCQs) form a mainstay of assessment in many domains of education. However, correct-incorrect and even negative-marking have a tendency to drive assessment-focussed strategies and rote-learning, rather than encouraging broader understanding and synthesis of concepts across topics. The work presented here provides a comparison of elimination marking, correct-incorrect, and negative-marking, with respect to both student performance, assessment behaviour, and student perceptions.

Methods
Correct-incorrect and negative marking award marks for selection of correct responses from a set of possible answers, and penalise the selection of incorrect responses in negative marking, whereas elimination marking requires learners to remove options which they think are incorrect, and awards a proportional mark relative to the number of remaining options. All students on each of the five stages of our Bachelor of Medicine / Bachelor of Surgery were given the opportunity to complete a formative MCQ-based assessment of medical knowledge. For each item on the assessment, rather than the usual SBA MCQ correct-incorrect format, they were asked how they would respond under elimination marking conditions, correct-incorrect, and negative marking conditions. They were also asked to provide their views on a number of aspects of these marking formats with respect to learning, assessment, and knowledge development.

Results and Conclusions
Student response patterns across the different marking formats were compared, and are presented in relation to their overall and item scores. Analysis of reliability as well as domain specific performance and differential impact across demographics groups are also considered. Student perceptions of the formats are presented, and ways of reconciling pedagogic aims and student expectations discussed. These finding present the first direct comparison of these three marking formats and the first to integrate the logistical and educational considerations with student perceptions of their learning and assessments.

Christine Prater    
Texas Tech University Health Sciences Center    

Purpose
A challenge for medical educators is choosing a method that best evaluates preclinical students' performance in preparation for Step 1. In previous years, block directors (BDs) of the neuroscience course at Texas Tech University Health Sciences Center issued three locally developed, faculty-written (LFW) examinations during the course with a NBME subject examination at the end. In 2022, BDs replaced LFW examinations with two National Board of Medical Examiners (NBME) custom examinations. The rationale being that the customized NBME exams would better reflect the national neuroscience curriculum and enhance student preparedness.

Methods
LFW examinations (2018-2021) were created by the faculty in the neuroscience course and reviewed by BDs. The number of LFW questions that needed to be bonused back to students were quantified using performance metrics determined by Office of Curriculum. In contrast, questions that best aligned with the material covered for the 2022 course were selected by BDs using MyNBMESM Services Portal. The custom questions selected are assigned a "difficulty" score by NBME, generating a predicted national average score.

Results
Concerning LFW examinations, 3.08 + 1.68 questions (out of 60) were bonused due to unsatisfactory performance metrics and 30.17 + 10.12 questions received written student commentary/complaints per exam. In contrast, NBME examinations were created with the predicted National Average set as 76% and 81%. The class averages were 76.6% and 83.2%, respectively. While no formal commentary/complaint system was in place following the NBME exams, post-exam meetings with class leadership produced no complaints on content of the exams. Indeed, students expressed a preference for the customized NBME exams as opposed to faculty generated exams.

Conclusions
Building customized assessments through MyNBMESM Services Portal was found to be useful and preferable for evaluating student performance.

Helen Kaiser    
University of South Carolina School of Medicine Greenville    

Purpose
Narrative feedback plays a vital role in the clinical setting. LCME standards for medical schools require that narrative feedback be incorporated into the formative assessment of student performance in lab settings, but provides no guidance on frequency or methodology, leading to a wide range of methods of delivering feedback across the field. During preliminary data collection, students at the UofSCSOMG reported high preference for feedback that is timely. The purpose of this study is to determine if students deem narrative feedback given within 48 hours of observation as more effective than feedback given at the end of the module.

Methods
M1 students receive narrative feedback regarding teamwork, professionalism, and work ethic for anatomy lab sessions. Half of the 110 students were randomly assigned to receive this feedback at the end of the 9-week module, while the other half received feedback within 48 hours of the observed lab session. After the module, all students will be asked to complete a survey assessing their opinions on the feedback using Likert-style questions. These responses will be compared to the time the feedback was delivered, as well as the nature of the feedback.

Results
When the study ends in March 2023, functional analysis will be performed to compare timeliness and the nature of feedback to student perception. Any common themes in open-ended questions will be categorized to create a "lessons learned" plan that will be used to compare this study to the results of the previous study from Summer 2022.

Conclusions
Medical students will receive narrative feedback throughout their career, and utilizing this feedback is a critical skill for any physician. The results of this study will further elucidate specific components of effective narrative feedback to more efficiently provide useful feedback to medical students in the anatomy laboratory.

Kelsey Dougherty    
University of Colorado Physician Assistant/Child Health Associate Program    

Purpose
Clinical assessments such as Objective Structured Clinical Examinations (OSCE's) are important tools in assessing clinical competence of future medical providers. Medical programs must ensure that learners are on the path towards achieving clinical competence by utilizing competencies and milestones. This abstract describes how one program mapped their clinical assessments to competencies and milestones that led to the refinement and standardization of these tools for more effectively assessing clinical competence.

Methods
Our program implemented a curricular revision to reflect current medical learning theories (self-directed, reflective and experiential learning) to assist learners in building a framework for clinical reasoning skills. This revision also included the creation of competencies and milestones to ensure that learners were meeting developmentally appropriate benchmarks across the curriculum, as well as capturing students who were not meeting these benchmarks. The final step was mapping formative and summative clinical assessments to milestones and competencies to ensure clinical competence upon completion of the program. This process has facilitated the refinement and standardization of cases that assess the developmental progression of learners in a variety of skills including communication, history taking, physical examination, clinical reasoning and documentation. This process is transferable to any medical program.

Results
All clinical assessments were mapped to specific program milestones and competencies across the curriculum. Through this process, our program was able to refine and standardize the outcomes of these assessments to more effectively evaluate learner progression through the program. It also allowed for more targeted support and feedback to learners when they were not meeting appropriate benchmarks.

Conclusions
Mapping clinical assessments (OSCE's) to program milestones and competencies led to refinement and standardization of these tools for more effective assessment of clinical competency in medical learners. 

Paul Megee    
Oakland University William Beaumont School of Medicine    

Purpose
Formative assessment facilitates learning in medical education. Course directors observed that students performing poorly on summative assessments in a two-part foundational biomedical sciences course are less likely to utilize formative assessment. Programmatic review was initiated to determine whether assignment of nominal course value to weekly quizzes altered formative assessment usage and whether its usage and overall performance on summative assessments could be correlated.

Methods
Cohorts of first-year medical students in two successive academic years were offered a total of 18 weekly formative assessments, with no course value assigned in the 2021-2022 academic year and nominal course value (2% total, with timed performance and a deadline) assigned in the 2022-2023 academic year. Course learning management system access logs and quiz performance data were analyzed, as were performance data for 4 high-stakes summative assessments (midterm and final exams in each part of the course). Formative and summative assessments are similar in difficulty and format.

Results
Assignment of nominal course value to weekly quizzes correlates with increased formative assessment utilization in the 2022-2023 cohort compared to the earlier cohort, including learners who ultimately performed poorly on high-stakes summative assessments. Notably, the percentage of students requiring retesting to demonstrate competency was reduced in the 2022-2023 academic year. Moreover, increased utilization of a separate ungraded quiz was observed in the 2022-2023 cohort, suggesting that assignment of course value to some formative assessment contributes to broader utilization of all formative assessment opportunities.

Conclusion
A course grading change assigning nominal value to format assessments increased its utilization and cohort high-stakes examination performance. Focus groups may reveal whether graded quiz deadlines also promoted timely progression through course content that contributed to improved high-stakes assessment performance. 

Inaya Hajj Hussein     
Oakland University William Beaumont School of Medicine    

Purpose
Student engagement and success are highly intertwined and among the most pressing issues for academic institutions worldwide. Hence, fostering student engagement has long been a main goal for institutions of higher education. While traditional pedagogy depict students as consummates of knowledge, recent psychological and education theories and empirical data support the notion that collaborative and students-as-partners approaches, broadly defined as co-creation, could be an efficient toll to enhance student engagement, their sense of belonging, academic self-efficacy, and well-being. This presentation introduces a co-creation project that was implemented to foster students' engagement in process of developing in-house exam questions for the assessment of medical students' knowledge and performance in pre-clinical years.

Methods
Using a qualitative approach, the contents of students' feedback and comments on instructor written exam questions were analyzed and a rubric was developed to revise and improve the quality of the questions in terms of clarity, complexity, rational, length, closeness to the correct answer, plausibility of responses, and alignment with the content of the lecture. Subsequently, the initial and revised exam questions were rated by two experts.

Results
Preliminary results showed that the quality of the revised internal instructor written NBME-style exam questions increased significantly.

Conclusion
Involving students in the process of co-creating of exam questions is an economic, feasible, and pedagogically meaningful approach to foster students' engagement and improve the quality of examinations materials.

Taylor Arnason    
Loyola University Chicago Stritch School of Medicine    

Purpose
Medical students are expected to graduate with cultural competence, which improves the quality of care of individuals across different minority groups and may help reduce health disparities. Incorporation of diversity, equity, and inclusion into medical teaching is critical to the development of culturally competent physicians; however, centuries of medical pedagogy have tied certain pathologies to racial and sexual/gender minority patients. This not only encourages stereotyping to arrive at the correct answer but also narrows the differential diagnosis developed by the future physician. We are surveying US and non-US third- and fourth-year medical students and first-year residents to demonstrate to what extent inclusion of race, sexual orientation, and gender identity data in question stems influences diagnosis and clinical decision-making.

Methods
A survey has been created and divided into three sections. The first section will collect demographic information, including age, race, gender identity, region, type of medical degree, current level of training, completion of examinations, and MCAT score. The second section will present one of two sets of twelve USMLE style questions with (experimental) or without (control) key demographic data, including race, sexual orientation, and gender identity. The third section will present four questions in order to assess participants' understanding of the higher burden of hypertension among Black patients and attitudes regarding the role of their medical education in the development of biases. Participants will be randomized to the set of twelve questions in the second section. Statistical analyses will be item-level comparisons of each set of questions.

Results
Results from a pilot study completed at Loyola University Chicago and/or preliminary results will be presented at the conference. 

Conclusions
An intervention in medical education is needed to minimize stereotyping, expand the differential diagnosis, and reduce health disparities. Our project represents a significant area of study because many medical students focus their studying around multiple-choice questions.

Anna Blenda    
University of South Carolina School of Medicine Greenville    

Purpose
Student evaluations of teaching (SET) are ubiquitously used by medical institutions as metrics to evaluate faculty, teaching effectiveness, and medical education courses. However, recent reviews have established that SET lack reliability and validity, and best practices for improvement remain unspecified. As such, this study aims to identify evidence-based recommendations, which medical institutions can implement, to improve the validity of SET and their utility as measures of teaching effectiveness.

Methods
A comprehensive review of the literature was conducted using PubMed, ProQuest, Web of Science, and Google Scholar databases for peer-reviewed, full-text articles in English published between 1990-2022. The search yielded 1,436 articles, which were then independently evaluated for inclusion criteria based on the relevance to SETs in the pre-clerkship curriculum at medical institutions. A total of 41 articles met inclusion criteria and were systematically analyzed.

Results
SET questionnaire items must undergo factor analysis to ensure construct validity and must be distinguished prior to utilization. Current end-of-course administration of SET is significantly impacted by student attributes and primarily, satisfaction with end-of-course examinations. Therefore, institutions should consider concurrent post-lecture evaluations and supplementing SET with additional measures, especially for multi-instructor courses. For example, recent studies demonstrate that pre- and post-course cognitive self-assessments of students provide a more valid measure of teaching effectiveness and overall course rating compared to traditional SET. Finally, teaching medical students how to provide constructive feedback via formal training significantly improved course evaluation quality.

Conclusions
Medical institutions can improve the validity and reliability of the current use of SET by (1) changing the timing of SET administration, (2) supplementing SET with multiple instruments to measure teaching effectiveness more robustly, and (3) integrating formal feedback literacy training to enhance the quality of student feedback.

Brian Keisler    
University of South Carolina School of Medicine    

Purpose
With Step 1 moving to a pass/fail format, it is reasonable to assume that the stakes may be higher for students on the Step 2 exam. Additionally, the previous step 1 score could serve as a predictor for Step 2 results. We want to see if the results of clinical NBME scores can serve as a good predictor of Step 2 scores going forward for our students.

Methods
We are looking at data collected over the past few years at our medical school to look at how previous Step 1 scores correlated with Step 2 scores. We will be comparing this information with data that shows how clinical NBME scores during the M3 year correlate with Step 2 scores.

Results
Our results are not final, but we do know from past data that Step 1 and Step 2 scores at our school are well correlated. Our initial data in looking at clinical NBME scores in the M3 year suggests a strong correlation with Step 2 scores as well.

Conclusions
We do not have final conclusions as of yet as we are still collecting data from the class of 2023 as it pertains to Step 2 scores. This data will be complete for this abstract by the time of the IAMSE meeting in June. We do believe that M3 NBME scores will correlate well with Step 2 results. If this is true, this could provide important predictive information for our students as they get ready to prepare for their Step 2 exam - particularly in light of the fact that they no longer will have a 3-digit Step 1 score.

Leanne M Chrisman-Khawam    
Ohio University Heritage College of Medicine    

Purpose
A wicked problem in medical education is the hand-off between medical school and residency. Individual learners all possess strengths and weaknesses which could be addressed earlier in graduate medical education with adequate hand-off documents. The traditional Medical Student Performance Evaluation (MSPE), tends to be utilized in the selection process and is often a pale view of an individual's progress across training.

Methods
An accelerated 3-year medical school to family medicine residency program conducts an end-of-semester competency review and anticipatory learning plan and guidance. Each learner creates a self-assessment and learning plan. This is compared and merged with the 360 clinical competency evaluations, case logs, self-reflection journals, clinical and written examinations, papers, and presentations in an ongoing assessment and learning plan. The culminating semesters' competency documents show the learner's growth and trouble areas. The final document is shared with the respective program director who is invested in their future resident's learning and areas of need.

Results
Eight students annually have conducted semester competency reviews akin to residency biannual reviews. The activity helps to prepare them for the work of the next semester and helps them focus on their individual growth and learning in a master adaptive learner way. Program directors appreciate the early insight as to resident learner needs and professional goals.

Conclusion
Improved competency monitoring and proof of attainment require a sustained and mentoring approach in the master adaptive learning model. By formalizing the end-of-semester review and preparing the student for the coming semester, the student is developing and demonstrating the skills of a lifelong learner. Utilizing a separate assessment for learning needs and residency selection may be time-consuming and necessary for improved undergraduate medical education to graduate medical education transitions. Examples of these self-assessments and reviews will be shared.

Mily Kannarkat    
Eastern Virginia Medical School    

Purpose
Student feedback and observed differences in assessment practices across instructors demonstrated an opportunity to improve our approach to exam construction, communication of exam content, and post-exam feedback and review processes. This study describes strategies used to improve perceived and actual fairness of the assessment of medical knowledge.

Methods
The program initially implemented new fairness review strategies with faculty-written exams in response to the COVID pandemic. The fairness review strategies included: a pre-exam review, a post-exam student feedback session, and a post-exam review with content and assessment experts. Score adjustments were made if exam questions did not meet pre-defined benchmarks for fairness. The program also used these strategies with National Board of Medical Examiners exams. A test blueprint to standardize communication from faculty to students regarding testable content was later added to the assessment process.

Results
Data indicates that implementation of strategies were valuable to improving fairness and equity of the assessment process. Student feedback and item statistics were useful in identifying problematic questions, warranting score adjustments. 20 questions were removed from scoring over 2.5 years. Lab practical exams were more likely to have concerns related to fairness. Although exceedingly rare, the review did result in a change in student outcomes (from fail to pass).

Conclusion
Implementing fairness review strategies helped identify opportunities to improve fairness of the assessment process, thereby enhanced the validity of score inferences about student learning. These strategies promoted alignment between curriculum, formative assessments, and summative assessments and increased insights into student learning. Fairness reviews were implemented by leveraging existing resources, stakeholder buy-in, and support. Efforts from the past 2.5 years highlighted the importance of fairness reviews regardless of the exam format and institutional context.

Antoinette Polito    
Elon University Department of PA Studies    

Purpose
At some point in the educational process, most medical science educators utilize single-best-answer multiple choice exam items in their assessment of student learning. Best practices based on psychometric evidence exist, and valid resources for developing these exam items are available to educators. Making use of these sources will translate into better quality assessments and improved student outcomes.

Methods
We have both facilitated training workshops in item writing over the past decade. Medical science educators of all disciplines can learn the foundations of item writing and the techniques necessary to continue to improve their skills. Instruction includes examining the "anatomy" of a multiple-choice item, dissecting its parts, and creating original items on a given topic. Common errors are discussed, and pitfalls to avoid are described. The peer-review process will be outlined; its use allows educators to continually improve their own skills and to teach these techniques to colleagues and new faculty.

Results
While it may seem easy to write these items, it is actually quite difficult to write excellent ones. The better the exam item, the better we are able to understand our students' grasp of the content knowledge and in turn the success of our curriculum. We will outline how to write, analyze, and peer-review multiple choice exam items in a stepwise, evidence-based fashion.

Conclusion
Any discipline can benefit from careful analysis of how to write better exam items. From simple formatting best practices to initial data analysis, getting better at this type of assessment makes us better educators. Our students are ultimately the ones who benefit most when we are able to write exam items that focus precisely on their learning objectives and outcomes. Valid resource tools will be listed on the poster itself.

Danielle Dickey    
Texas A&M School of Medicine    

Purpose
Clerkship Narratives are written for each medical student to be included into their final MSPE. When applying to residency strong weight is put into these narratives especially as more  curriculums change to a Pass/Fail curriculum and Step 1 no longer issues a point score. If bias is found in clerkship narratives, it could influence the whole projection of a student’s career. Perhaps not matching into the specialty of choice or affecting program prestige  or location.  At Texas A&M School of Medicine, we choose to review our clerkship narratives for gender bias.

Methods
1 years’ worth of student narratives were examined through the core clerkship rotations including, Family Medicine, Internal Medicine, Surgery, Psychiatry, Pediatrics and Obstetrics and Gynecology. Narratives were deidentified and run through a software created to identify gender bias in recommendations letters.  Adjectives were labeled  “feminine” or “masculine” using the software.  Then we compared the narratives based on the gender of the student to identify where bias occurred.

Results
Results are pending completion of the study, but the results will be used to identify specific interventions in the narrative writing process.  Although there has been work in this area within disciplines, this study will help identify the specific places and language where bias appears which will allow for tailored faculty development and narrative templates to be created.  Additionally, narratives can be reviewed for gender bias language before final submission to the student and Medical Student Performance Evaluation (MSPE)

Conclusion
We know gender bias occurs often implicitly.  We want to find strategies, interventions, development, and templates to help mitigate gender bias and strengthen clerkship narratives, MSPE’s and feedback.

Leslie Ruffalo    
Medical College of Wisconsin    

Purpose
Assessing medical students’ competency is a challenging yet desirable goal in undergraduate medical training. Many medical schools lack a consistent approach to share competency progression data with students in a formative way. In response, our institution uses its Continuous Professional Development (CPD) Course to track medical students’ progress along eight institutional global competencies. The goal of this session is to share our process for generating and sharing student competency data.

Methods
The CPD course spans years three and four of medical school with the goal of ensuring that each student’s professional development meets institutional global competencies. Students are assigned a CPD faculty director who monitors and tracks their competency progression. Each competency is rated on a Likert scale of 0-5 (0=no competency, 5= full competency). The CPD team works with a data analyst to synthesize competency data. This results in a visual “spider graph” depicting three types of competency data: 1) an average of clinical preceptors’ objective ratings of the learner for each competency, 2) the class average on each competency, and 3) the student’s self-reported score on each competency.

Results
Our standardized process to assess competency in undergraduate education has enhanced CPD advisors’ ability to normalize student experiences, identify areas of relative strength and weakness, compare individual students to peers, and assess trends in student self-assessment. The resulting visual “spider graph” deftly depicts a learner’s progress, which is important for both learners and advisors.

Conclusions
Competency-based education in undergraduate education is on the rise. Our CPD course highlights a method by which students’ competency data informs formative feedback as learners progress across all clinical rotations.