Program evaluation will demonstrate your understanding of your program and its impact, as well as eliminate guesswork and support sound decision-making going forward.
Some evaluations can be very complex. However, there are some simple steps you can take to ensure you have some form of evaluation in place. Read on…
Step One: Determine the main goal or goals of your program or event
If you don’t have a goal or multiple goals, you will not be able to tell whether your event or program will be addressing a particular need for your organization or the people you serve. Ask yourself, what are you really trying to achieve? That is your goal.
Identify what you are hoping to achieve:
- Teach or share a new skill or interest;
- Engage new participants;
- Increase inclusiveness;
- Build social connections;
- Entertain participants;
- Solicit support from community;
- Change a behaviour or ways of thinking;
- Or, all of the above!
Step Two: Determine key questions that will help you identify success or challenges
How will you know if you are moving towards your goal? If you have been running the program for some time, you are probably aware of what is going well and what areas might need some work. Questions will help narrow down what you should measure.
Identify what you need to know about your programs:
- What makes this program successful? Or, unsuccessful?
- Did the program have the impact intended?
- Is the program meeting the needs of participants?
- Is the program reaching the right participants?
- How can I get more participants involved? Increase diversity?
- What attracts participants to this program?
- Are delivery mechanisms as effective and efficient as possible?
Step Three: Identify key indicators that will help answer your questions
Identify what you want to evaluate and select the appropriate indicators. Measure areas that will help you find answers to your questions. This will inform your decision-making about changes needed going forward.
Collect data that may help you measure:
- Numbers of participants attending
- Levels of learning by participants: relevance and growth;
- Types and numbers of participants: attendance by target groups
- Reasons why participants decided to attend;
- Satisfaction levels of participants;
- Other benefits for participants;
- Efficiency of program and/or effectiveness of delivery;
- Willingness to return or recommend to others;
- Return rates;
- Amount of donations; and/or
- Changes in behaviour.
Step Four: Determine how you are going to measure change.
Once you know what types of data to collect, select the appropriate tool. Some tools work better to collect certain types of information. Remember to use tools that eliminate bias – you do not want to just collect feedback that you already know or want to hear. You want to gather the true reflections from a wide range of people. According to the Oxford Dictionary, bias is considered prejudice in favour or, or against one thing, person, or group, compared with another. The right tool will be the one that eliminates the most bias in the data.
Different tools to use in evaluation:
| Interview | A face-to-face conversation with participants where they are asked questions about their experience. This is useful for identifying change in skill levels or change in behavior, as well as collecting reactions and feelings. |
| Questionnaire/Survey | A series of structured questions that measure participants’ responses, either in-person or online, following their participation. This is useful to collect both qualitative and quantitative data, cost-effectively from a large sample of individuals, by using fixed alternative scales or open-ended questions. |
| Observation | Observe and analyze conditions or behaviours that can be counted, classified or rated. This is useful if you are measuring indicators that you don’t want to ask people directly. |
| Focus Groups | A group of participants brought together, usually in person, to provide their feelings, opinions, and perspectives on a subject to a moderator. This is useful if you want to explore more in-depth feedback, including reactions and feelings from participants. |
| Real-time Survey | Collect input from individuals as they are participating - usually through the use of a handheld device or a cell phone. This is useful for collecting real-time data that can be shared immediately with a group of participants. |
| Registration Data | Collect information through the registration process. This is useful to easily identify audiences, participation rates, return rates and trends. |
| Existing Data | Identify data that already exists from similar programs or activities to use as a benchmark, or as a comparison, to achieving program goals. |
Step Five: Analyze your data
Once you’ve collected your data, return to your program goals and evaluation questions. Be clear on what you hope to ascertain from this data – you will need the right data to answer your specific questions. Do a basic analysis of all quantitative data (all the numbers) and then, analyze all the qualitative data (verbal answers). Qualitative data is important too – you may have to summarize testimonials and feedback for easier review. Use this data to formulate answers to your key evaluation questions.
Identify the data that will answer your key questions:
- Does the data show that your program met its goals and objectives? If applicable, did you engage new participants? Did you interest participants in pursuing a new skill? Did you entertain or create a sense of belonging?
- Can you summarize your data numerically? For example, two out of ten (20%) of the participants were new to the program.
- What aspects of the program worked and what didn’t’?
- Do the results match your expectations?
- Was your budget and timeline realistic?
- What needs to change for next time?
Step Six: Write a report that will support future planning.
A final, and extremely important, part of the evaluation process is the creation of a report that outlines what you found out and recommendations for going forward. To determine what type of report to prepare, consider who the audience is for this information. If it’s an internal document, you can write it in point form. If it’s for your board or funders, ensure that you include the information needed to explain your process and recommendations.
Key parts of reporting evaluation results:
- Executive Summary – summarize your situation conclusions and recommendations;
- Background – describe your organization and the program(s) under review, including key goals and identified issues;
- Review Process – describe evaluation goals, methods and analysis procedures;
- Key findings – provide results and interpretations of data collected to answer evaluation questions;
- Recommendations – provide key actions needed to move the program forward;
- Conclusion – summarize next steps for the program and/or organization; and
- Appendix – describe the instruments used to collect data, such as the survey used, and raw data collected in tabular format.
Program evaluation is not always an easy process, but it is extremely worthwhile. It will demonstrate your understanding of your program and its impact, as well as eliminate guess work and support sound decision-making going forward.
Summarized By Diane Ell, Communications Manager, SaskCulture, 2022.
Dive deeper on this topic
-
GuideA Basic Guide to Program Evaluation, 2017Carter McNarmara, MBA, PHD
-
GuideA Guide to Planning and Conducting Program Evaluation, 2009.Fraser Health
-
ArticleEvaluating Outcomes of the Program, from Getting to Outcomes for Home Visiting, 2013Teryn Mattos, Sarah B. Hunter, M. Rebecca Kilburn, Shelley H. Wiseman