Evaluation requires an honest look at what you are doing with consumers, and reviewing which consumer activities are working and which aren’t. Evaluation will help you understand if consumer involvement has:
- improved the quality and safety of your health organisation or service
- had a positive impact on the design of research agendas and priorities
- led to more consumer-focused policy.
If an ineffective consumer involvement activity can be improved, do so, but if not, it should be stopped. Consumer involvement that is ineffective or poorly managed, or where there is a skill mismatch can be damaging for the consumer and for future consumer involvement efforts.
Evaluations can be conducted by internal or external parties. You will need to decide which you prefer. You may decide to allocate funding for an external evaluation, which has the advantage of being totally independent.
Ideally, the evaluation process will involve both consumers and staff. Invite consumers early in the process and remember to let them know the results of the evaluation.
The evaluation cycle
Evaluation is a dynamic process that happens throughout the lifecycle of the consumer‑involvement activity. Information is collected and decisions are made throughout the cycle to ensure that consumer-involvement is effective. In addition, there is a dedicated stage in the cycle for formal evaluation, which contributes to improving the consumer-involvement process.
Evaluation typically focuses on 3 areas: process, impact and outcomes. These categories can be used to evaluate consumer-involvement activities. They can be evaluated sequentially, as each area usually builds on the former.
Evaluation of the process
Evaluating the process looks at all the processes involved in the activity and the immediate effects, such as:
- How was the experience for consumers and what were the immediate effects for them?
- Who were the consumers and how well did they represent the target group?
- What was the consumers’ level of satisfaction with the project?
- What additional activities were undertaken in response to the needs of consumers?
Evaluation of the impact
Evaluation of the impact focuses on the short-term impact of the activity. It is usually related to the objectives of the activity, and it normally includes unanticipated effects.
- Were the purpose, aims and objectives achieved?
- Were the consumer involvement activities effective to gather the required information?
- Was the implementation of the activity achieved within budget and on time?
- Were peoples’ skills sufficient to implement the project adequately?
- Were there any unintended effects of the activity?
- Were the results disseminated as planned?
Evaluation of outcomes
Evaluation of the outcomes identifies the long-term outcomes of an activity that are frequently related to the purpose and aims of a broader goal, such as a consumer participation policy.
- Has the program has achieved its main aim?
- Do we have the conditions required to implement this aim successfully in the future?
- Can we reproduce these conditions to achieve the aim in the future?
Evaluation tools
The tools you use for evaluation depend on the aims of the evaluation, timeframe, resources and staff skills. Tools include:
- documentation, such as minutes, reports, newsletters
- story-telling
- focus group interviews and discussions
- informal feedback
- ‘exit’ interviews
- observation
- face-to-face or telephone interviews
- self-completed questionnaires
- photos
- statistics
- feedback forms.
