Presented By: Rebecca Sullivan, Lewis Katz School of Medicine at Temple University
Co-Authors: Erin Bruce, University of Florida College of Medicine
Marisol Lopez, Boston University Chobanian & Avedisian School of Medicine

Purpose 
When writing multiple-choice questions (MCQ) for assessments, it is essential to follow best practices to remove potential inequities and/or achievement gaps. Students from groups underrepresented in medicine (URiM) and/or English language learners (ELLs) are particularly impacted by flawed items. Studies show that simplifying language while maintaining content difficulty increases linguistic accessibility of test items and is helpful to ELLs. Cognitive load theory can be used as a conceptual framework to prove the impact of item flaws on student academic performance. A greater number of elements for a student to process in working memory (extraneous cognitive load) to successfully answer the question, leaves less working memory capacity (intrinsic cognitive load) available to demonstrate content knowledge. We propose a universal rubric will address flawed MCQ items in an objective manner across institutions. 

Methods 
We have developed an evaluation instrument (rubric) to score the presence of technical flaws that provide irrelevant difficulty and/or provide an advantage to test-wise examinees based on previous literature and the "Item-Writing Guide" from the National Board of Medical Education (NBME®). 

Results 
We have validated the rubric using 26 MCQs obtained from physiology exams at three different institutions in a variety of programs including Dental, Medical, and Physician Assistant programs. After analyzing the questions, we returned as a group and compared our individual ratings, clarified any discrepancies and modified the rubric accordingly. After scoring inter-rater reliability, this rubric will be used to analyze the full question banks for physiology exams at each institution. 

Conclusion 
The data obtained by assessing our MCQs will indicate the prevalence of item flaws in Physiology exams. We will then investigate how identified flaws affect item performance. This rubric can be used by faculty in diverse institutions to identify problematic MCQs and modify them by removing technical flaws, ensuring equitable assessments.