Logic Models for Program Evaluation 2: Creating and Using

CanMedEd-ipedia: The CORAL Collection. Concepts as Online Resources for Accelerated Learning.

Gwenna Moss Centre for Teaching and Learning

Introduction

A sound and realistic logic model can guide us in developing and implementing program evaluations that produce relevant, evidence-based information about a program’s effectiveness.

Learning Objectives (what you can reasonably expect to learn in the next 15 minutes):

  1. Describe the important steps in creating logic models.
  2. Critique a simple logic model.

To what extent are you now able to meet the above objectives? Please record your self-assessments. (0 is not at all and 5 is completely)

Describe the important steps in creating logic models.
Critique a simple logic model.

To start, please take a few moments to ask yourself the following questions about an educational program that you know well. In the interest of respecting your time, try to pick a small, simple program. In broad strokes, how would you describe:

  1. What this program is intended to achieve (it’s expected outcomes)?
  2. What steps are taken to achieve these expected outcomes (inputs, activities, outputs)?

Write out your thoughts here or somewhere handy so you can refer to them later:

Logic Models for Program Evaluation 2: Creating and Using

Here is a quick recap of the CORAL Cell “Logic Models for Program Evaluation 1: Purpose and Parts”. This section is included to remind, refresh, and perhaps re-teach.

A program logic model is a visual representation of how we anticipate that the resources invested in a program will lead to the results we intend to achieve.

By illustrating our expectations about how a program will be effective, logic models allow us to:

  • Verify that our expectations for the program’s effectiveness are sound and realistic.
  • Communicate information about our program to internal and external stakeholders in a reader-friendly, visual manner.
  • Select relevant performance measures, aligned with our expectations of the program as depicted in the logic model.
  • Interpret program data and their implications regarding the extent of program effectiveness.

Logic models come in various formats: the direction in which the logical model is presented vary (e.g., left-to-right, top-down). There is no “one size fits all” approach, which provides flexibility in developing a logic model that best suits your program.

Keeping this flexibility in mind, logic models typically depict the relationships between the following program components in a logical or reasonable sequence:

  • Inputs: The resources invested/available to deliver the program.
  • Activities: The key activities/components of the program, using the inputs.
  • Outputs: The products and services that are delivered as a result of the program activities.
  • Outcomes: The changes that are expected to occur, both near/short and longer-term, as a consequence of the program’s activities and outputs.

The template below shows how logic model components can be visually presented in a logic model diagram. The arrows depicted in a logic model illustrate:

  • the direction in which the logic model is intended to be read, and
  • our expectations of how program components at one level (e.g., an output) will influence one or more components at the next level (e.g., immediate outcomes), and eventually lead to one or more ultimate outcomes.

Essentially, the arrows represent our logical and reasonable assumptions and expectations about how the program will achieve each of its outcomes.

Tips for Developing Useful Logic Models

  • Keep it simple. A parsimonious logic model includes only as many inputs, activities, outputs, and outcomes as are necessary to provide a complete and accurate overview of the program. For example, similar deliverables (pamphlets, posters, and webpages) could be grouped into one broad category of outputs (communication materials).

  • Check your logic. Is it reasonable to expect that (a) the activities and outputs will contribute to achieving the outcomes and (b) each of the outcomes is within the program’s sphere of influence?

  • Encourage stakeholder participation. Include program stakeholders in the development of your logic model. Ask if their contributions/interests in the program are accurately reflected.

  • Start early (if you can). Developing a logic model when the program itself is being developed can help you ensure that the logical assumptions about the program’s effectiveness are reasonably sound, foster stakeholder agreement and buy-in from the beginning, and allow you to select and collect relevant program performance data over time.

  • Plan ahead. The components contained in your logic model will guide the selection of performance indicators and measures used to evaluate your program’s effectiveness.
    • Your program’s strategic objectives and mission statement should be reflected in the logic model if you intend to evaluate progress in achieving them over time. These will likely be intermediate or ultimate outcomes.
    • Outcomes that clearly indicate the direction of expected changes (e.g., increase engagement, enhance learning, reduce adversity, maintain enrollment, improve patient care) will later help you interpret program data and determine whether the outcomes have been achieved.

Check for Understanding

Which of the following statements are True/False about developing useful logic models?

1. The activities included in a logic model must be described with sufficient detail for the program to be replicated by others.

2. Involving program stakeholders in the development of a logic model can help verify that the logic model is sound and foster stakeholder engagement in the program.

3. Logic models are only useful if they are developed at the same time as the program itself is being developed.

4. In a logic model, each individual program output must be connected by arrows to all the immediate outcomes.

5. Outcomes that clearly state the direction of the expected change are useful for interpreting performance measurement data about the program’s effectiveness.

6. If you intend to evaluate progress in achieving a program’s strategic objectives then the strategic objectives should be represented in the logic model’s outcomes.

7. Adding as many details as possible to a logic model increases its usability.

8. All the outcomes included in a logic model must be directly or indirectly attributable to program inputs, activities, and outputs.

Check your answers

Can you spot the weaknesses of this logic model for a Family Physician Hospitalist Clerkship 6-week rotation?

(We’ve identified 4 weaknesses, though perhaps you can spot more.)

Weaknesses:

  1. Increasing knowledge is not an output delivered by the program, but rather an outcome that we expect will occur as a result of students being trained. Therefore, “Students’ knowledge of providing care to admitted patients is increased through teaching sessions” should be moved to the immediate outcome level and a new output should be included to indicate that the activity of delivering teaching sessions produces students trained in providing care to admitted patients.

  2. Training students through teaching sessions is not likely to increase the extent of students’ experience with providing care to admitted patients. Therefore, the arrow connecting the output of teaching sessions and the immediate outcome of increased experience with providing care should be removed.

  3. The direction (increase, decrease, or stay the same) of the expected change is missing in the immediate outcome “Students’ satisfaction with the clerkship rotation.” If satisfaction levels have historically been high then the expected outcome may be to maintain these levels. On the other hand, if satisfaction levels have been low, the expected outcome may be to increase them (which would require activities and outputs geared toward increasing student satisfaction).

  4. None of the activities or outputs depicted in this logic model could reasonably be expected to lead to the intermediate outcome “Enhance quality of clerkship hospitalist elective.” If this is an expected outcome of the program then the logic model should illustrate how we anticipate that it will be achieved through perhaps faculty development, quality improvement, or participatory action research. Also, this outcome may be more appropriately placed at the immediate outcome level, as it is would result directly from the delivery of the clerkship program.

Think back to the educational program you wrote about earlier. Using the information presented in this CORAL cell, how would you now describe what this program is intended to achieve (it’s expected outcomes) and what steps are taken to achieve these expected outcomes? Consider in your response the main components of a logic model (inputs, activities, outputs, outcomes) and how these components could be used to organize the information you wrote down earlier.

Self-Assessment

To what extent are you NOW able to meet the following objectives? (1 is not at all and 5 is completely)

Objectives
1. Describe the important steps in creating logic models.
2. Critique a simple logic model.

To what extent WERE you able just before beginning this Cell to meet the following objectives? (1 is not at all and 5 is completely)

Objectives
1. Describe the important steps in creating logic models.
2. Critique a simple logic model.
Concept map of related coral cells: Program Evaluation
Concept map of related coral cells: Program Evaluation

Thank you for completing this CORAL Cell. We are interested in improving this and other cells and would like to use your answers (anonymously of course) along with the following descriptive questions as part of our evaluation data.

Provide feedback on module

Thanks again, and come back soon!

The CORAL Cell Team

Credits:

Author: Vanessa Anastasopoulos, McGill University
Series Editor: Marcel D’Eon, University of Saskatchewan
Reviewer/consultant: Derek Wilson, University of British Colombia

Back to the CORAL collection