LLM-Generated Multiple Choice Practice Quizzes for Pre-Clinical Medical Students; Prevalence of Item Writing Flaws

Area of Science:

• Medical Education
• Artificial Intelligence in Healthcare
• Assessment and Evaluation

Background:

• Multiple choice questions (MCQs) are vital for medical student assessment.
• Automated MCQ generation can reduce faculty workload and enhance student practice materials.
• The USMLE/COMLEX-USA board exams require specific question formats.

Purpose of the Study:

• To assess the feasibility of using ChatGPT for generating USMLE/COMLEX-USA-style practice questions.
• To evaluate the quality and accuracy of AI-generated medical assessment items.
• To determine the utility of LLM-generated questions as study aids for medical students.

Main Methods:

• Second-year medical students used ChatGPT-generated practice quizzes on renal physiology.
• Independent experts evaluated the generated questions for quality and adherence to guidelines (NBME/NBOME).
• Analysis focused on item writing flaws, factual errors, and suitability for revision.

Main Results:

• 49% of AI-generated questions had item writing flaws.
• 22% contained factual or conceptual errors.
• 91% of questions were deemed a reasonable starting point for expert revision.

Conclusions:

• Large language models show feasibility for generating medical education practice questions.
• Expert supervision and revision are essential for ensuring accuracy and quality.
• AI-generated questions can serve as valuable study aids when properly validated.