PURPOSE
In medical education research, statistical significance (or the likelihood
that a result occurred by chance) is frequently relied upon as the primary
marker that an educational intervention was successful. However, with the
sample sizes currently used in health professions education research, an
intervention having statistical significance does not necessarily indicate it
is practically applicable from the instructor’s perspective (i.e. having
practical significance). The objective of this project was to describe how
often medical education research studies consider practical significance in
their data analysis.
METHODS
A literature review was conducted of the Methods section from quantitative
research studies published between July 2024 and June 2025 in the journals
Academic Medicine, Medical Teacher, and BMC Medical Education for definitions
or other considerations of practical significance.
RESULTS
303 quantitative primary studies were identified, of which only 2
considered practical significance in their analysis (0.7%). Both studies used
performance on a test by physicians as their main outcome. In one of the
studies, practical significance was defined a priori as a Cohen’s d value
greater than 0.5. The second study considered whether their findings were
practically significant, but did not establish any prior threshold values.
CONCLUSION
Based on this literature review, practical significance is rarely
considered in the data analysis of quantitative research studies in medical
education, and the vast majority of studies still rely solely on statistical
significance. Among the few studies that considered practical significance,
there is no current consensus on its numerical definition. This is a gap in
the literature that should be addressed to improve the validity of studies in
medical education.