Summative assessments often lead students to focus on rote memorization rather than deep understanding. Frequently they are not designed to provide detailed feedback for improvement. That is why faculty should emphasize that rather to pursue approving standardized exams, it should also align with the objectives of the courses taught and also promote effective strategies to comprehend complex medical subjects.

General objectives:

• Introduce summative assessments in pre-clerkship years such as single-best answer multiple choice, short answer, and essay-type questions.
• Focus on the purpose and structure of summative MCQ assessments.
• Promote techniques for constructing quality assessments for continuous improvement of the summative items.

Workshop Schedule:

• 12:30 - 1:30 (1 hour) Aligning course objectives and NBME/NBOME topics to elaborate MCQ items
  • Establish a common ground between course objectives and assessments using NBME/NBOME format.
  • Design multiple-choice questions (MCQs) that foster reasoning over memorization
• 1:30 - 2:30 (1 hour) Purpose of summative assessment in learner progression
  • Identify how institutions are using summative assess to gauge learning outcomes
  • Identify how summative assessment help to track learner progress
• 2:30 - 3:30 (1 hour) Critique basic lower-level questions and transform those questions to high-order question items
  • Evaluate quality of the MCQ item.
  • Criteria of a high-order question

Program evaluation is the systematic assessment of educational initiatives, policies, programs, or interventions to determine their effectiveness, efficiency, relevance, and impact on trainees, educators, and communities. It involves collecting and analyzing data to provide evidence-based feedback on the strengths and weaknesses of the program, progress toward achieving intended outcomes, and recommendations for improvement or modification. A strategic approach to evaluation is essential to achieve the long-term success and sustainability of programs. One such approach is to develop a logic model. Logic models visually connect intended outcomes with the resources and activities that go into a program, depicting relationships among program inputs, planned work, measurable outputs or products, and short and long-term outcomes or impacts for students or other learners. A well-constructed logic model can guide decision-making about data collection, assessment timepoints and overall program evaluation, supporting clear alignment between these measures and the desired outcomes. In this workshop, an experienced investigation and program evaluation team will guide participants through the elements and development process of a logic model, identify connections between the logic model and formative and summative evaluation components, and provide templates, strategies, and practice to foster excellence in their educational programs. Workshop

The workshop emphasizes the importance of program evaluation for the long-term success and sustainability of educational programs. As a result of participating in this session, participants will be able to:

• describe the elements of a logic model, and
• apply a logic model to refine evaluation and assessment strategies in the context of their own programs.

Who Should Attend
The workshop will be relevant for a wide range of conference participants who are involved in program design and are either familiar or less familiar with program evaluation.

Workshop Schedule:

• 12:30 - 12:40 (10 min) Introductions, session goals, agenda, and distribution of resources
• 12:40 - 12:50 (10 min) Session Warm-Up: and think pair share discussion of questions and experiences related to planning and evaluating an educational program.
• 12:50 - 1:10 (20 min) Overview of a logic model, with examples.
• 1:10 - 1:20 (10 min) Guided demonstration of using a logic model, starting with development of measurable outcomes, and showing the importance of alignment across components of a program.
• 1:20 - 2:20 (60 min) Individual and Small Group Activities: Individuals will select a project that is relevant to their work and use a logic model to identify outcomes and impacts, plan program activities and outputs, and identify resources and other “inputs”. Between each step, mentored small groups will discuss and troubleshoot challenges. Session leaders will act as program mentors.
• 2:20 - 2:40 (20 min) Large Group Activity: Presentation and discussion of program evaluations created.
• 2:40 - 2:50 (10 min) Overview of measuring outcomes
• 2:50 - 3:10 (20 min) Individual and Small Group Activities: Individuals begin to create formative and summative assessments of their outputs and outcomes. Mentored small groups will discuss and troubleshoot challenges. Session leaders will act as program mentors.
• 3:10 - 3:20 (10 min) Large Group Activity: Presentation and discussion of assessment tools.
• 3:20 - 3:30 (10 min) Group Discussion and Questions: What were the most challenging aspects of implementing a program evaluation? How might you use a program evaluation?

Participants will be provided with definitions of the different research methods used in medical education, the type of tools involved in collecting data, their analysis and possible interpretations. They will also be given examples of qualitative and mixed research methods that may be compatible with developing and implementing a research project at their own institution.

Workshop Schedule:

1. Introductions & Audience Perspectives on Qualitative Research vs. Quantitative research in general – 15 minutes
2. In-depth qualitative research methods and analysis – 60 minutes
   • Differentiate between different types of qualitative methods.
   • Identify the steps necessary to organize and manage data collection.
   • Describe the qualitative data coding process.
   • Discuss the different considerations that may determine the credibility of the data analysis process.
   • Explain the importance of checking and challenging qualitative data analysis.
   • Outline what must be considered before qualitative data analysis can be accurately represented and appropriate conclusions can be reached.
3. In-depth analysis with Mixed methods (includes an AI aspect) – 60 minutes
   • Describe mixed-method approaches such as Q-Sort, Nominal Group Technique, Delphi Technique, Focus Groups
   • Discuss the strengths and weaknesses of using a mixed method approach in research
   • Use a tool to help quantify qualitative data in a mixed methods approach
   • Design an appropriate mixed method approach for their home institution
4. Comparison between Quantitative and Qualitative research (Table) – 30 min
5. List of topics that can be suitable for qualitative research and discussion on their feasibility with the audience – 15 minutes