Purpose
Content analysis, while a cornerstone of qualitative educational research,
is an arduous process. Artificial intelligence has been explored as a method
of primary content analysis, but its feasibility as a method of testing the
reliability of a codebook has not been extensively studied. This project
explores the use of AI to enhance codebook refinement and finalization.
Methods
Primary analysis of 1280 de-identified peer evaluation comments in a
medical anatomy course was conducted in NVivo using descriptive, open-coding
methods to elucidate specific behaviors that students identified as favorable
or unfavorable to dissection teamwork. Primary analysis was conducted by a
coder familiar with the course from which the data originated. This analysis
is part of a larger study examining team dynamics within medical student
dissection teams. The researcher-generated codebook from this analysis and a
sample of the original, uncoded dataset were uploaded to Google Gemini and
Microsoft CoPilot, and then prompted to apply the provided codes to the
dataset.
Results
Analysis of the output yielded a Cohen’s kappa of 0.667 between Gemini and
CoPilot. Reviewing differences in the applications of codes between Gemini
and CoPilot revealed overall consistency in the broad categories of the codes
applied and exposed areas where the definitions of the codebook could be
refined by the researchers for clarity.
Conclusions
The use of AI as a tool for content analysis appears promising, although
future studies are needed to explore distinctions between different large
language models and the feasibility of handling large datasets. Utilizing AI
to aid content analysis could reduce the time and effort involved in
qualitative research studies, which are widely used across all areas of
medical science education, as well as improve the accuracy of such studies.