Purpose
The Multiple Mini Interview (MMI) is an increasingly adopted tool used in medical school admissions for assessing non-cognitive attributes critical to healthcare practice. Its use has expanded to other health professions, including Physician Assistant (PA) programs, as this growing profession becomes increasingly integrated into global health systems. However, little is known about the reliability of MMI scores when used in PA admissions. This study evaluates the reliability of MMI scores from a 5-station MMI with two independent raters per station, used during the inaugural admissions cycle of the University of Calgary’s Master of Physician Assistant Studies (MPAS) program.

Methods
Using Generalizability Theory (G-Theory), we analyzed sources of variability in MMI scores from 80 applicants assessed across five stations by multiple raters over two days in May 2024. Variance components attributable to applicants, stations, raters, and their interactions were computed and analyzed. A Decision Study (D-study) simulated varying configurations of stations and raters to examine reliability.

Results
The generalizability coefficient was 0.783, indicating moderate reliability. Sources of variability were largely interactions between applicants*stations, reflecting how applicants performed differently across stations, as well as raters*stations, reflecting variability in how raters score across different stations and station difficulty. D-study simulations showed that increasing the number of stations and raters incrementally improved reliability, with the highest reliability modelled using 10 stations and two raters per station.

Conclusions
A 5-station MMI with two independent raters per station demonstrated moderate reliability, meeting acceptable thresholds for high-stakes decisions like admissions. Increasing the number of stations and raters can further improve reliability, but the 5-station format with two raters could be appropriate in contexts with resource constraints. These insights will guide the optimization of the MMI process in future admission cycles, supporting fair and reliable assessment of PA applicants.