Assessment Redesign for Critical AI Literacy: A USask Example
In an introductory Pharmacy course, Dr. Ed Krol and Dr. Anas El-Aneed redesigned a group assignment to help students critically evaluate AI-generated feedback and reflect on its strengths and limitations for learning.
By Gwenna Moss Centre for Teaching and Learning
As generative artificial intelligence (AI) tools become increasingly available to students, many instructors are considering how these tools might affect student learning. In one introductory Pharmaceutical Sciences course, the goal was not to avoid AI, but to create an opportunity for students to build AI literacy by evaluating its strengths and weaknesses.
"We wanted to be proactive in having students understand the strengths and weaknesses of AI in learning," explains Dr. Ed Krol, Professor and Interim Dean, College of Pharmacy and Nutrition. "We wanted to acknowledge the fact that AI was there and explore how to get value from it to support student learning."
To do this, Ed and his colleague, Dr. Anas El-Aneed, adapted a published approach to incorporate an AI component into an existing group project.
A Familiar Assignment, Redesigned
The course is an introductory Pharmaceutical Sciences course for pharmacy students. One of the major assignments requires students to apply their learning from the course to explain the chemical and physical properties of an approved drug of their choice.
Students work in groups and submit a report that is limited to five double-spaced pages. Reports are assessed on the clarity of their hypothesis, their focus on drug structure, the quality of their conclusions, and their use of appropriate references.
"We added an AI component so that students could assess for themselves what the potential advantages and drawbacks were of using AI," says Ed.
To support this goal, students were instructed not to use AI while initially researching or writing the report. Once the report was initially completed and submitted for grading, each group participated in a mandatory in-class session focused on AI.
Evaluating AI Feedback
During the session, groups were instructed to prompt Microsoft Copilot using both their report and the grading rubric. The goal was to use AI to determine the extent to which their report met the assignment requirements and identify ways that it could be improved.
Students were encouraged to use detailed prompts and refine those prompts to obtain additional feedback. Rather than relying on a single response, they explored how different prompts influenced the feedback they received.
Following the AI activity, each group submitted a two-page reflection describing how they prompted Copilot, what feedback they received, what changes they would have made to their report, and which suggestions they would not have used and why.
"Students generally found the AI assessment exercise to be valuable," says Ed. "Students noted that Copilot frequently recommended additions that were outside the scope of the assignment, but the most useful suggestions were typically related to improving clarity and being more concise."
Because students had already completed the report and could compare the feedback against the rubric, they were able to evaluate which AI suggestions were useful and which were not. The activity encouraged students to think critically about AI-generated feedback rather than simply accepting it.
For Ed, this was an important outcome. The exercise gave students first-hand experience with both the strengths and weaknesses of AI in an academic setting.
Lessons Learned and Next Steps
The assessment redesign also highlighted some challenges.
"A major drawback was that we were unable to confirm that students did not use AI at an earlier stage of the assignment," says Ed. While students were instructed not to use AI during the research and writing stages, there was no reliable way to verify this.
At the same time, the experience prompted new questions about how AI might be used productively as part of the learning process. "Going forward, we will continue to refine this aspect and investigate how use of AI at earlier stages could be useful, for example to develop a search strategy and evaluate references," says Ed. He also plans to add requirements related to AI disclosure and documentation.
Ed has already begun using a similar approach in a second-year elective course. In that course, student groups submit a second report with track changes and comments showing how they would have implemented suggestions from Copilot.
Looking back on the assignment, Ed sees value in helping students develop a more critical understanding of AI. "We wanted students to understand the strengths and weaknesses of AI in learning," he says. "We wanted to acknowledge the fact that AI was there and explore how to get value from it to support student learning."
Apply in Your Context
If you want support exploring your own assessment redesign, contact GMCTL@usask.ca for consultation and workshop opportunities.
Title image credit: GMCTL stock image (D.Stobbe)
This article was created with the assistance of AI tools, as described in the GMCTL AI Disclosure Statement.