Evaluation of Communications

Evaluation is a very important part of the communication process and it is usually guided by the goals and objectives you set early in your planning. If you have clear measurable benchmarks for your activities, plans or strategies, then evaluation may be quite straightforward. The main reasons for evaluating come down to accountability — you can show what you have achieved and justify the expenditure of time, energy and money — as well as improvement, to determine what worked well and what did not. Evaluation is an important way to find mistakes and improve your communication efforts into the future.

IMPORTANT THINGS TO KNOW ABOUT COMMUNICATION EVALUATION

Evaluation should be planned for at the beginning of the communication effort with the development of measurable objectives.

1. Evaluation is an ongoing, systematic process. It should be included in all communication planning to evaluate activities, plans and strategies. It should not be an optional activity.
2. Evaluation is forward-looking. It helps an organization learn how to improve future performance. It reduces uncertainties, improves effectiveness and enhances decision-making.
3. Evaluation is not the same as research. Research is a strategic tool that feeds into planning, implementation and evaluation. Evaluation is a management process that can be done informally or formally to help determine movement forward.
4. Evaluation should support strategic alignment. The strategy involves various levels of planning and implementation. Planning is rolled down and evaluation rolls up on the same strategic line.
5. Evaluation should be planned for at the beginning of the communication effort with the development of measurable objectives.
6. Evaluation may cost money. One rule of thumb is to add in 7-10 per cent to your communication budget to cover evaluation costs. But, it can also be done informally by your staff for very little extra cost (i.e. evaluation questionnaire).

STEPS IN EVALUATING COMMUNICATION

Your evaluation should give you the information to show progress on meeting your objectives, and whether or not you are closer to achieving your goal.

1. REVIEW YOUR GOALS AND OBJECTIVES

As stated earlier, evaluation is usually addressed in the early stages of communication planning, when you identify your overall goal (where you want to be) and your objectives (SMART). Your evaluation should give you the information to show progress on meeting your objectives, and whether or not you are closer to achieving your goal. The result of the evaluation should provide you with a progress report that you can share with your board or senior management. Example: Increase awareness of our new program among our members by 20 per cent by 2014.

2. WHAT DO YOU NEED TO MEASURE?

The goals and objectives help identify the targets which results can be measured against. From these goals, particularly the objectives, you can identify the benchmarks or indicators that will showcase success or needed change. Your objectives should be focused on the results you want to achieve on the way to meeting your goal: advancing an issue with policymakers, changing people’s attitudes, securing new funding streams from donors and so on.

A communication plan or strategy tries to create an effect per target audience. It may be: Cognitive (thoughts); Affective (emotions); and/or Conative (motives). Measurement of these areas involves determining if a key audience group actually received the messages directed to them, is paying attention to them, understanding and/or comprehending the messages, and whether they are retaining the messages and can read them. The two main areas of measurement are:

Effectiveness – measure the accomplished results, the achieved goals and objectives or realized impact. These may include:

• outputs (i.e. participation levels),
• outtakes (i.e. levels of favorability, understanding, recall and retention)
• outcomes (i.e. changes in opinion), and
• outgrowths (i.e. movement in reputation)

Efficiency – the ability to bring about the desired result without wasting energy, resources, effort, time or money. It measures greater efficiency that is achieved with same standard of output, outcome or outgrowth. These may include measuring technical, process or cost efficiency.

Sometimes measurement is done over time. It is good practice to set some measureable milestones that you can reach along the way. Sometimes these milestones are checkpoints that allow you to make adjustments to your strategy.

3. POSE YOUR EVALUATION QUESTIONS.

What are the key areas of questioning required to obtain the data to support movement on your objectives, and/or goals? An evaluation question identifies what you need to find out. It will serve as a guide for what you need to ask, who you need to ask, and if you have baselines in place.

Example evaluation questions:

• What is the audience’s initial response to my spokespeople? Are they the right messengers?
• Where have my messages first appeared? Local press? Web sites? Word of mouth?
• What evidence do I have that my messages are being absorbed by my audience?
• Is my audience engaged with the issue?
• Based on participation, do I need to change my audience? Change my message?

4. WHAT BASELINES DO YOU HAVE IN PLACE?

You may already have some measurements done on your communications. Maybe you completed this same evaluation last year and have a direct comparison. Or, another organization has completed a similar evaluation with which you could compare your organization. If not, identify what data you may have that can be used as a baseline (or benchmark).

Examples of baselines:

• Past participation levels
• Common misconceptions people have about your organization
• Media pick up on news releases
• Readership levels of newsletter.

5. WHO DO YOU NEED TO ASK?

Identify the stakeholders or target groups you need to question in some way to get the input you need to show results. The individuals, or groups, you identify are you samples. They can be selected because they fall into a particular category or group, or you can select them randomly from the population (or a segment of the population). A random sample from the population, which you need to evaluate, will give you the most reliable and valid evidence. However, random sampling is mainly done in more formal research, and is not always needed in an evaluation process.

Examples of samples:

• Organizational members
• Program participants
• Newsletter readers
• Ticket Subscribers

Approximately, 1,200 respondents, randomly selected from Saskatchewan residents will give you a fair representation of the entire adult population (with a standard error or 95 per cent).

Keep in mind when you select groups, you must have a way to reach them all, or a good portion of them. Do not plan to send out a general survey, just to reach a subset.

6. SELECT YOUR EVALUATION TECHNIQUES

It is important to determine you data collection methods based on how appropriate they are for answering your key evaluation questions and for achieving the ultimate purpose of the evaluation.

All of the information that you have compiled so far should be enough to help you determine which type of evaluation techniques will get you the data you require. You may decide that you need outside expertise to help you with particular evaluation techniques, or you may be able to move forward on your own. Keep in mind that all evaluation techniques have advantages and disadvantages. Some are more appropriate than others for collecting different types of data.

Surveys: Surveys are the go-to for most evaluation purposes. They are a written document that can be done in person, over the phone, or online. Surveys work best when respondents can “checkbox” answers that garner quantitative data. Surveys can collect qualitative data through open-ended questions; however, these introduce different types of interpretation bias into the overall results.

When to use: Surveys are useful for collecting and categorizing data at any given time; and usually, work best for gathering close-ended resopnses. They provide a wealth of data that can be manipulated to provide evidence towards particular behaviours or preferences.

Interviews: Interviews consist of selecting a handful of individuals who represent your target group and asking them a series of open-ended questions. This method gives you insight into how people are responding to your communication and helps you determine if something is really working the way it should. Interviewers usually take notes or ask their informants/respondents if they may use a tape recorder. Some use group interviews.

When to use: When you have identified a group of knowledgeable informants — participants, staff members, community members, media — an interview is a helpful tool in providing you with in-depth and detailed information about your program, the experience and/or changes needed.

Focus Groups: Focus groups involve bringing together a group of people from your target group (usually 10-20 people) who can have a discussion about your communication activities. The discussion is usually moderated by an impartial facilitator. The group shares opinions on your communication activities and bounce ideas off one another in the form of discussion.

When to use: Focus groups are especially useful in pre-testing of programs, services and even surveys. You are able to test specific messages, tactics or approaches and evaluate the results of the group.

Observation: Observation is used to simply observe individuals or groups to see how they are responding to certain messages.

When to use: Observation may be an easy way to measure the effectiveness of a communication tool or program. With the appropriate access to users, you can simply watch others interact with your communications.

Content Analysis: Content analysis is usually done to assess the quality and tone of your media coverage, or to review the content of specific programming that reaches your audience. This process requires some form of coding and well-trained coders to assess large quantities of data.

When to use: Use when you are trying to identify a positive or negative flow to media coverage on a particular issue, or if you are trying to assess the messaging in letters to the editor over a period of time.

Usage Tracking: Simply tracking usage, such as web site hits, length of visits, online newsletter opens, Facebook likes, gives you some simple measurements to help evaluate. Tracking provides you with a very superficial form of evaluation, that does not often showcase engagement or understanding, but it is often useful as a baseline.

When to use: Useful for setting a baseline for communication activities, plans and strategies, that you can compare on an annual basis. This works when just getting more people to your site, or as Friends/Fans on Facebook, is useful to your overarching purposes.

It is important to determine you data collection methods based on how appropriate they are for answering your key evaluation questions and for achieving the ultimate purpose of the evaluation. Tie method selection to available resources. Chose methods based on what is appropriate for the target population and project participants.

7. ESTIMATE YOU BUDGET

After you have selected your evaluation techniques, you will be in a better position to estimate your budget. However, in some cases, your budget may determine which techniques you are able to do. You should be able to find some form of the technique that will fit into most budgets. For example, instead of a phone survey by an external research firm, use a free online survey. Or, instead of bringing together people for a focus group, take your questions to a scheduled meeting.

Budget areas include:

• Staff Time;
• External consultant fees;
• Cost of selected evaluation techniques;
• Travel and incidentals; and
• Publishing and mail costs.

At this point, you should have an evaluation plan ready for implementation. Remember to remain somewhat flexible to your timelines to get the results you require. If you have a deadline, you can summarize your findings at a particular point, but leave the evaluation process open to ongoing input.

BENEFITS OF EVALUATION

Besides being the only way you can tell if your communication has truly succeeded, evaluation:

• Improves the effectiveness of your communication;
• Helps engage your audience;
• Helps your organization deal with situational change — support for strategic and tactical decisions; and
• Helps your organization allocate resources more effectively.

Also, there are others that can benefit from your research, so be prepared to share. Your newsletter readership survey, or event attendance survey, could be used as a baseline for another organization. This gives your organization credibility among your members and demonstrates your commitment to ongoing improvement.