Purpose
In recent years, artificial intelligence (AI) systems in our everyday lives have grown exponentially, with one prominent example being ChatGPT. This machine learning model utilizes deep neural networks to understand and generate human-like texts based on user input. The purpose of this study is to evaluate the effectiveness and accuracy of ChatGPT in generating multiple-choice test questions for higher education, specifically in the field of immunology. This evaluation included comparing the performance of two medical students — one with prior immunology knowledge and one without — in using ChatGPT to create educational content.

Methods
Two medical students were tasked with generating 4-6 multiple-choice questions per learning objective (n=188-282) using ChatGPT. Both students used the same set of lecture materials and employed their own prompts to instruct ChatGPT. To mitigate the risk of generating false or deceptive information, the lecture materials were uploaded into ChatGPT as supplementary PDF files, and the AI was instructed to focus solely on the provided content. All generated questions were subsequently reviewed by an immunology faculty member for accuracy and quality, and to assess the validity of ChatGPT as a knowledge learning tool.

Results
The study found that the student with prior subject knowledge from previous enrollment in the Medical Health Sciences Masters Program was able to apply more effective prompts, resulting in more relevant and accurate questions. Additionally, the inclusion of lecture materials significantly improved the quality and accuracy of the questions generated by ChatGPT. This combination of improved prompts and supplementary lecture materials allowed ChatGPT to create high-quality, hallucination-free assessments.

Conclusion
ChatGPT, when supplemented with specific lecture materials and guided by effective user prompts, can serve as a valuable tool for generating accurate and relevant multiple-choice questions in higher education. This approach can enhance the learning experience of medical students by providing high-quality educational assessments and reducing the risk of AI-generated misinformation. ChatGPT has validity as a knowledge learning tool.