Measuring Impact

Measuring or evaluating your programs and services, or even your entire organization, has many benefits. It provides you with insight into what is working, existing or emerging challenges, as well as identifying change needed going forward.

Measurement also gives you important data to use when applying or following up on your grants. For each of your programs and services you may consider assessing certain indicators, such as attendance and/or satisfaction, to determine progress.

Evaluation is the bigger process of interpreting the data compiled and making decisions on the information learned. Cultural groups may want to consider a Value-Based Approach to Evaluation that focuses mainly on how an organization’s work and programs align with its core values.

What to learn more? Here are some topics to explore:

• Understanding your Participants
• Measuring Equity and Inclusion
• Understanding Value-Based Evaluation
• Wise Practices in Evaluation
• Suggested Evaluation Policies & Guidelines

Understanding YOUR PARTICIPANTS

Your group may offer programs and services designed to reach a wide range of participants. Besides members or returning participants, you may be hoping to reach new people. To help you better understand who attends your programs and who may in the future, it’s useful to establish some indicators, such as demographics, satisfaction levels, and program support. Below are some considerations:

What are your current participation numbers/levels? What is your expectation and what change is needed? Keeping track of participation helps you understand how well your programs and/or services are being utilized. Are you meeting a community need? If you are offering a new event/activity, you can estimate how many people you expect to participate.

Who are your current participants? Who do you hope will participate? Of all participants, you can break them down into various demographics by: Age, Income Level, Geographic Area, Gender and Cultural Background.

Who is helping to lead programs/services? Do you require a certain level of support to be successful? Participation may include the delivery of your programs, including paid and unpaid support, which would include staff, contracts and volunteers.

Measurement Tools: Registration Form Questions, Attendance/Support counts and After-event surveys

Understanding SATISFACTION with Programs/Services

How do your participants feel about your programs/services? It’s important to find out if your programs and services are meeting expectations of those involved. Regular check-ins on satisfaction with your programs/services is an important way to determine if status quo is okay, what changes are needed and evidence for future funding.

Satisfaction can be measured by:

• Programs/Events rated – Very Satisfied to Not Satisfied, and everything in between
• Will your participants return?
• Will they tell others to come to your events/activities?
• Newsletter sign-ups/social media follows?

Measurement Tools: Follow-up Survey, Real-time questions/polls

Measuring EQUITY & INCLUSION

Are your programs/services inclusive? Are you aware of any barriers to participation or access? Inclusive programs ensure that everyone involved has a sense of belonging, acceptance and recognition, as valued and contributing members. It involves ensuring all people, particularly from equity-deserving groups, can participate in a way that makes them feel comfortable, confident to be themselves, in a way that suits them and makes them feel valued. Many times returning participants and satisfaction levels can provide insight, however, understanding the diverse needs of participants and preparing for accommodations is recommended.

Are you reaching equity-deserving populations? Equity-deserving groups include Indigenous peoples, racialized communities, LGBTQ2S+ individuals, people with disabilities, people living in remote areas and low-income households. You can assess this through actual numbers of participants, but you can also measure by the programming directed to a particular equity-deserving group of people. (i.e. a Ribbon skirt-making class promoted to local Band members)

Are you offering equitable programs? Equity involves going a step further. Supporting equity is about treating individuals according to their diverse needs, in a way that enables all people to participate, perform and engage to the same extent. (i.e. offering a pay-what-you-can option or reduced rates or special free offers)

Measurement Tools: Registration Form Questions; Focus groups - assess potential barriers, partnerships; and Survey of key targeted attendees.

Understanding Value-based Evaluation and Measurement

A values-based evaluation approach is more than numbers and traditional metrics. It focuses mainly on how an organization's actions and work align with its core values. Identifying a core set of values can not only help inform how we measure, but also why and what we measure. According to Reciprocal Consulting, a values-based approach in evaluation can:

• help clarify and lead program development,
• facilitate a way for an organization and program participants to communicate about a program's goals and intentions,
• provide a critical lens for interpreting outcomes and impacts, and
• stand the test of time despite changing circumstances.

Most organizations already have set values and a mission/vision, so a matching evaluation approach that leans on these existing values makes the most sense. Assessing the impact of living your values can help provide better accessibility, understanding, and translation of any data collected.

Reciprocal Consulting also notes that these values can be used as ways to interpret feedback and guide funding or program choices and philosophies. This is possible if well-defined, meaningful, and intentional values are used. Identifying values can also help facilitate discussions on the outright and intentions of a given program and help determine how and why a certain activity needs to take place. The evaluation of values can inform the foundation of a flexible, yet systematic, measurement framework that can be used across contexts and future work.

The Principles Focused Evaluation (PFE), outlines how to assess values and principles as an evaluation approach:

• provide direction but not detailed prescription;
• are grounded in values about what matters;
• are based in evidence about how to be effective;
• must be interpreted and applied contextually;
• require judgement in application;
• inform choices at forks in the road;
• are the rudder for navigating complex dynamic systems;
• point to outcomes and impacts; and
• can be evaluated for both process and results.

Values-based evaluation can help you gain deeper insights into how your organization’s programs resonate with the community and create a purpose-driven narrative around what success truly looks like while showing stakeholders and the community the real difference your organization is making.

Wise Practices in Evaluation

Evaluation can take several forms, but some best practices remain consistent. In its 2023 report, Reciprocal Consulting provided some key “wise practices” to consider in evaluation:

• As your program or project evolves, allow flexibility within the evaluation based on evolving conditions or feedback. This also applies to evaluation tools and processes. This allows the evaluation to remain relevant even when circumstances change.
• Think about ways and how the work you are evaluating connects to things beyond the current project.
• Recognize that one size does not fit all. Evaluations should be designed to fit the unique needs, goals, and context of the project or program being evaluated. Also, note that people are on different levels as they come in with you during the evaluation process.
• Share lessons learned and valuable insights as you go. Don’t wait until the end to discuss what’s working and what’s not.
• At the early stages of the program or project, be sure to include participatory approaches to ensure that all parties are and feel involved in the work. This also applies to information-sharing and reporting options.
• Don’t forget to include opportunities for relationship-building as you move along your evaluation stages.
• Ensure all voices, most importantly, those most affected by the program or work, are well represented.
• Ensure you uphold high ethical standards in the evaluation practices and that all practices respect the rights of participants.

Suggested Evaluation Policies & Guidelines

When conducting an evaluation, it's important to establish policies and clear guidelines to ensure the process is effective, ethical, and reliable. Reciprocal Consulting provided some policies for cultural groups to consider:

• Policy on Reciprocal Accountability

This policy emphasizes that everyone involved in the evaluation process must be accountable to each other so that everyone can learn from the process and make valuable decisions to improve the project or program. Always share back any learning from the data collected. Also, remember the data should be collected with the right purpose and must serve a need.

• Policy on Data Stewardship

The Data Stewardship policy ensures that the data and information collected are treated with respect and that the identities and information are protected. Consider the following when developing a data stewardship policy:

• Policy on Competencies Required to Steward Evaluations

Managing evaluations effectively requires certain skills and knowledge. This means that there will be some levels of human involvement in the evaluation process. Remember, data is an extension of an individual and should be treated with the same respect as the physical person. This policy will help ensure that evaluations are conducted fairly, which would lead to useful and valuable insights.

When developing a policy on the competencies required to implement evaluation activities, consider the soft skills required, the technical skills required, as well as the experience and expertise required (i.e., extent of training on evaluation) of those who will be interacting with those in the evaluation.

• Policy on Evaluation Professional Development

This policy encourages ongoing learning for people who are working on the evaluation. The learning can be through training sessions, workshops, and conferences. This policy helps to keep evaluators updated on the latest methods and best practices in evaluation.

• Policy on Contracting Evaluations to a Third Party versus Conducting Evaluations Internally

Contracting evaluations to a third party can bring fresh ideas and expert knowledge, which may lead to more objective results. This is why a policy is needed to support when your organization decides to conduct evaluations internally or when to use a third party. When developing this policy, you should consider factors such as: capacity; budget; expertise; objectivity; and, the need to maintain and preserve sensitive information.

• Policy on Use of Incentives and Honouraria

Having a Use of Incentives and Honouraria policy in place will help honour the time someone takes to provide input. When developing a policy on the use of incentives and honouraria, consider these questions:

1. What constitutes an incentive;
2. What constitutes an honouraria;
3. Which evaluation participants qualify to receive an honouraria or incentive, and when;
4. Who should be excluded from receiving an honouraria or incentive, why, and under what circumstances; and,
5. What constitutes an appropriate incentive or honouraria (i.e., $100 for a 15-minute survey could be perceived as coercion/compellence).

• Policy on Breach of Evaluation Policies

There are examples of when a breach happens during an evaluation. This can happen in cases such as when evaluators fail to protect participants' confidentiality or when participants' consent is not properly obtained. When outlined in detail, this policy would help to know what happens if someone does not follow an evaluation's established agreement or conditions and suggest recommendations to help if harm has been done.

• Policy on Engaging Youth in Evaluation Activities

This policy addresses the engagement of youth (under the age 19) in evaluation activities. If you choose to involve this group in your evaluation process, it is essential to establish guidelines that ensure parental consent and/or youth assent is obtained.

• Policy on Self-Disclosures in Data

This policy ensures systems are in place to handle situations when someone shares information about harm or trauma during an evaluation process. You can consider what disclosures would constitute a breach of confidentiality (i.e., risk to self, to others), and whom to report statements to (i.e., management at your organization, the Board of Directors, police, etc.).

• Inclusion, Diversity, Equity and Access Evaluation Policy

The Inclusion, Diversity, Equity, and Access (IDEA) Evaluation Policy ensures that everyone is treated fairly and included in the evaluation process. It focuses on creating a welcoming environment where everyone’s voice is heard and valued. This includes ensuring evaluation approaches are inclusive and accessible in terms of:

Technologically accessible and aligned with The Accessible Saskatchewan Act. On November 15, 2022, The Accessible Saskatchewan Act was introduced in the Legislative Assembly. If passed, the new legislation will remove and prevent accessibility barriers for persons with disabilities.

This policy could also focus on ensuring evaluation activities engage a diversity of participants, such as seeking input from:

1. Participants from multiple regions in Saskatchewan;
2. Participants with multiple and intersecting identities; and,
3. Participants with ranging involvement with your organization.