Scoping an Evaluation: Begin with the Purpose
This article is rated as:
Recently I made the decision to buy a bike. Like many others, I’ve put on some COVID weight and thought incorporating some exercise into my daily routine would help. I didn’t have a bike, so I started shopping for one.
Google is good, but not that good; it could bring up a bunch of bikes when I searched but I still needed to filter my search in order to figure out which bike to pick. What type of bike? (e.g. commuter, electric, road, etc.) Which brand of bike? What size of bike? How much did I want to pay? What features and accessories did I want? Admittedly, I know little about bikes, but what I did know was what I wanted to use it for. I wanted a bike to commute 5 km to and from work, on a non-hilly bike lane. In addition to that, I didn’t want to spend more than $750.
I ended up choosing a 3-speed commuter bike, with hand brakes, and a step-through frame. I added a bell and a box that sits on a carrier on the back of the bike. Two months in and there are no regrets with the purchase – it fits my needs to a tee. I can wear a skirt to work, carry my laptop back and forth, and switch gears if needed when my legs are feeling tired.
The two most important questions when scoping an evaluation
When you are asked to conduct an evaluation for a program it can be like shopping for a bike - there are various types, sizes and budgets. Designing an evaluation that meets your stakeholders’ needs begins with a scoping process. There are a number of questions that you should ask to scope an evaluation (refer to our scoping guide for a complete list of questions), but ultimately scoping an evaluation focuses on purpose.
1. What does the initiative do (and for what purpose)?
2. Who wants to know what (and for what purpose)?
What does the program do and for what purpose?
With scoping you want to get a detailed understanding of the program by asking stakeholders about the program’s:
history
stage of implementation
activities
intended results
target population
scale
budget
measurement activities
When I ask stakeholders about the program, I am listening for details that help me understand the scope and complexity of the program. However, what is equally important during the scoping process is listening for any disagreements or gaps in what is known about the program and its purpose. Does the program’s theory of change pass the sniff test? In other words, is the program designed to bring about its intended results? Listening for these clues are important to note, as they will help guide your conversation for the next scoping question.
Who wants to know what and for what purpose?
There are many different reasons and uses for a bike. Similarly, there can be many reasons for conducting an evaluation and how those evaluation findings get used. However, the more focused you can be on whose needs you are meeting through the evaluation, the more useable the evaluation will be.
When I ask stakeholders about the evaluation and its purpose, I am listening for details that help me understand who needs what information. However, I am also listening to see if expectations for the evaluation align with the details that were given regarding the program. For example, if you discover the primary purpose for the evaluation is to determine the value and future of the program then a summative evaluation is likely the right type of evaluation. For a summative evaluation we need a strong theory of change and program data to monitor changes over time. However, if you heard through your discussion to the first scoping question that the program does not have a clear, established theory of change and has not collected data then there is likely a discrepancy in expectations that you should address. Addressing the discrepancy means a more detailed discussions with stakeholders questioning them about the readiness of the program for a summative evaluation and advising on more appropriate evaluation support given the stage of the program.
Focusing your scoping process on understanding the program and evaluation purpose will equip you with the knowledge you need to develop a quote and/or an evaluation design. However, it also provides a framework for expectation setting with stakeholders that will result in more accurate quotes and appropriate, focused and useable evaluation designs.
Check out our guide to program scoping here