Introduction
Assessment drives learning thereby ensuring that learners are prepared for the next stage of training. Assessment guides student studying and the results provide feedback. This cell is designed to give you an overview of important issues surrounding workplace-based assessment.
Learning Objectives (what you can reasonably expect to learn in the next 15 minutes):
- Describe the uses and potential benefits of workplace-based assessment.
- Given an assessment situation (mini-case) identify the weakness or weaknesses: Time, Lack of standardization, Variability in assessor judgements, Systematic bias or biases (BONUS: identify the bias).
To what extent are you now able to meet the objectives above? Please record your self-assessment. (0 is Not at all and 5 is Completely)
To get started, please take a few moments to list your thoughts on the potential benefits and challenges of workplace-based assessment:
Now proceed to the rest of this CORAL Cell.
An Introduction to Workplace-based assessment
Workplace-based assessment represents the “Does” level of Millar’s pyramid, i.e. what learners do as part of their actual work. Learners can be assessed performing a wide variety of tasks, such as taking a history, performing a procedure, eliciting informed consent, counselling a patient, presenting a case to their supervisor, consigning findings to the chart, preparing orders, writing a referral or discharge letter, handing-over cases to colleagues, contributing to multidisciplinary rounds, .… The assessment can use different types of evidence such as direct observation of learner performance, written evidence, or evidence from discussions. It can be conducted on specific events or longitudinally aggregating multiple events. Some examples of workplace-based assessment methods are provided in the table below:
| Evidence through direct observation | Other sources of evidence | |
| Single Event |
Informal assessment and feedback Clinical Evaluation Exercise (e.g. Mini CEX, PMEX, O-SCORE, TMEX..) Video assessment Case presentation |
Case-based discussion Chart review Discharge/referral letter Critical incident Reflective paper |
| Aggregate of multiple events |
End of rotation form (ITER) Multisource feedback |
Chart audit Encounter log Portfolio |
Potential benefits
Workplace-based assessment has high face validity, in that it represents what learners are actually doing and what they will likely do in their future practices1,2. As such, it enables the assessment of performance in its real context, with all of its complexities and challenges. It also has the potential of providing rich feedback that learners often attend to with great diligence1,2.
Challenges
Time
The imperative of patient care means that it may be difficult for assessors to take the time to observe performance, document their findings, and provide feedback.
Lack of standardisation
The assessment tasks and settings will not be standardised for all learners. Cases will be different and clinical performance is highly case-specific, meaning clinical and workplace performance varies from one case to the other, from one situation to another, and from one CanMEDS role to another. Contextual factors may vary from quiet to busy, functional to dysfunctional teams, and so on.
Variability in assessor judgements
Assessing complex performance requires judgement on the part of the assessors, and assessors vary in their judgements, even when based on the same evidence.
Some of this variation is meaningful. Specifically, different assessors may focus on different aspects of performance, and may value different things as indicators of good care, thus providing a richer overall picture of performance. Having multiple assessors, including assessors with different professional backgrounds and roles can therefore be beneficial.
Some of this variation can be viewed as random error, or ‘noise’. For example, different assessors may interpret a learner’s behaviour in different ways. For example, a learner crossing their arms might be interpreted as nervous versus un-empathetic. Although these inferences are largely subconscious, assessors should be encouraged to consider initial judgements as hypotheses that require testing. For example, assessors could discuss what they saw with the learners and ask questions about what they were thinking or feeling at the time. When assessors supervise learners over a prolonged period, patterns of behaviour under different circumstances may become clearer.
Another way in which assessors generate ‘noise’ in their judgements is in their varying interpretation of scale points – they may agree on a narrative description of a learner and how it ranks compared to descriptions of other learners, yet vary when the judgements are translated into numbers. Narrative descriptions may therefore be more beneficial, not just because they are easier for learners to understand and thus provide better feedback, but also because they may be interpreted by other assessors in a more consistent way. A compromise way of representing judgements is the use of rubrics such as milestones, which include clear descriptions of performance categories. They are easier to combine than purely narrative judgements, yet reduce the variability of interpretation involved in rating scales. Finally, faculty development may foster increased agreement about performance levels.
Systematic bias or biases
Finally, some of the variation can be systematic bias. Here are a few of the biases involved in workplace-based assessment:
- the hawk/dove effect (some assessors tend to be more stringent/lenient than others)
- the halo/horn effect (one good/bad aspect of performance leads to good/bad opinion of the performance as a whole)
- availability and saliency biases when assessors judge overall performance based on an episode that they can easily or more vividly remember because it is recent, because something else triggered its recall, or because it was inherently more memorable (often because it was in some way extreme)
- contrast/assimilation effects (when one learner’s performance is judged as unduly different/similar to that of other learners assessed at around the same time)
- gender/ethnic/racial biases
Having multiple assessors is one way to mitigate biases. Encouraging assessors to document their findings is also an important strategy to mitigate availability and saliency biases. By keeping a record of their findings, assessors can also provide more detailed feedback, thus bolstering its credibility. Field notes combine the advantages of multiple observations, narrative descriptions, and regular documentation.
Issues surrounding the provision of feedback will be addressed in a distinct Cell. Suffice it to say that learners are more likely to value and act upon feedback based in credible evidence provided by credible assessors whom they feel have their best interest at heart, and who can provide recommendations on next steps constructively.
Table 1: Methods of workplace-based assessment
Table 1 highlights the strengths and weaknesses of a few common workplace-based assessment methods.
|
Method |
Use to assess |
Strengths |
Limitations |
|---|---|---|---|
Single Event |
|||
|
Clinical Evaluation Exercises (e.g. mini-CEX, P-MEX, T-MEX, O-SCORE) Assess learner performances repeatedly using 15-20 minute observations of real patient interactions |
Clinical performance |
|
|
|
Field notes Narrative description of learner performance over several hours (e.g. in a clinic, during a shift)
|
Clinical performance |
|
|
Aggregated |
|||
|
In-training Evaluation Reports (ITER) Scales may be accompanied by narrative descriptions of strength and weaknesses |
Performance in the workplace |
|
|
|
360o assessment = multisource feedback (MSF) Measurement tools completed by multiple people in person’s sphere of influence (superiors, peers, subordinates, patients, families). Often associated with self- assessment |
Communication Collaboration |
|
|
|
Chart Review Review of data recorded in patient charts and progress notes |
Record-keeping Clinical reasoning |
|
|
| Portfolio Assessment Learner develops a portfolio of qualitative or quantitative evidence of performance over time, accompanied by reflection on the learning journey |
Clinical performance Reflective skills |
|
|
Check for Understanding
Self-assessment
Please complete the following very short self-assessment on the objectives of this CORAL cell.
To what extent are you NOW able to meet the following objectives? (0 is not at all and 5 is completely)
To what extent WERE you able the day before beginning this CORAL Cell to meet the following objectives? (0 is not at all and 5 is completely)
Thank you for completing this CORAL Cell.
We are interested in improving this and other cells and would like to use your answers (anonymously of course) along with the following descriptive questions as part of our evaluation data.
Thanks again, and come back soon!
The CORAL Cell Team
References:
1 van der Vleuten CPM, Schuwirth LWT, Scheele F, Driessen EW, Hodges B. The assessment of professional competence: building blocks for theory development. Best Practice & Research Clinical Obstetrics & Gynaecology. 2010;24(6):703-19.
2 Holmboe ES, Sherbino J, Long DM, Swing SR, Frank JR. The role of assessment in competency-based medical education. Medical Teacher. 2010;32(8):676-82.
Further reading:
Gingerich A, Kogan J, Yeates P, Govaerts M, Holmboe E. Seeing the 'black box' differently: assessor cognition from three research perspectives. Med Educ. 2014;48(11):1055-68.
Gauthier G, St-Onge C, Tavares W. Rater cognition: review and integration of research findings. Med Educ. 2016;50(5):511-22.
Hodges B. Assessment in the post-psychometric era: Learning to love the subjective and collective. Medical Teacher. 2013:1-5.
Credits:
Author: Valérie Dory, McGill University
Reviewer/consultant:
Series Editor: Marcel D’Eon, University of Saskatchewan