Purpose
Workplace-based assessments (WBAs) play a pivotal role in evaluating the competency of medical students in the ever-evolving technological landscape of medical education. However, variations in assessment instruments across clerkships can intentionally or unintentionally lead to differences in responses, thereby affecting the assessments' validity, reliability, and comparability. This lightning talk will comprehensively review best practices for evaluating WBAs in medical schools. The focus will be on identifying elements in the assessment instruments that may cause differential responses. 

Methods
Through an analysis of case studies of 5 medical schools’ WBA data collection instruments. Using these, the talk will delve into the methodologies for reviewing these instruments, the potential sources of variations, and their impact on the assessment outcomes. It will also discuss strategies to mitigate the effects of these variations and enhance the fairness and accuracy of WBAs while recognizing the importance of clerkships’ specific information needs. Of particular focus will be the roles that technological advances (AI, LLMs, etc.) can have on these issues. 

Results
Commonalties among schools – both challenges and successes – will be shared, along with the emerging best practices in assessing the assessments that have been developed as a result of this project. This will promote a reflective discussion among schools on their own assessment review processes among participants.

Conclusion
This talk aims to foster a dialogue about the importance of regular review and calibration of WBAs at the institution and clerkship level in medical education. We believe this will contribute to the ongoing efforts to improve the quality of medical education and training and that addressing these issues is even more important as assessment turns to automated, technological solutions.