Are good evaluations used more than bad ones?
Author: Pirmin Bundi, Kathrin Frey and Thomas Widmer
This post was originally published by the Evidence & Policy blog on 25th August, 2021.
We have re-issued the article that has already been published by the Evidence & Policy blog. We would like to express gratitude to the kind offer of the editorial board of the Evidence & Policy blog.
Original article URL: https://bit.ly/3DjSlEz
This blog post is based on the Evidence & Policy article, ‘Does evaluation quality enhance evaluation use?‘
Pirmin Bundi, Kathrin Frey and Thomas Widmer
Evaluations provide important information to improve public services, but only if they yield valid and reliable findings – so we believed for a long time. Evaluation communities have therefore established certain criteria that should define evaluation quality. Yet against prior studies on evaluation utilisation, we show that evaluation quality measured by the criteria is not necessarily associated with evaluation use, but rather linked to the perception of quality and impact of the evaluation. Evaluators should adjust their communications strategies accordingly.
You wouldn’t eat partially baked bread, would you? Probably not, but what if the bread looks perfect from the outside? Baked golden brown to the perfection with a crispy crust? It happens to the best bakers that a bread already taken out of the oven is not yet completely cooked. Even though it looks tasty, it is still indigestible. Consequently, the quality of the bread’s production is central to its consumption.
Much like baking bread, the process of doing evaluation impacts on the quality of the evaluation. Many studies have made a connection between evaluation’s quality and use. Literature argues that high-quality evaluations not only generate more useful findings, but also better engage stakeholders so that they make superior use of the evaluation results. Yet hardly any studies have investigated whether this connection upholds empirically. Are better evaluations really used more often? Albeit a higher frequency does not mean that they have more impact, it is nevertheless an indicator of decision-makers’ engagements in evaluation.
The quality of an evaluation can be assessed in two distinct ways. On the one hand, we can simply ask stakeholders whether they perceive the evaluation to be of high quality. This ‘subjective’ evaluation quality is often grasped by the perception of the involved actors with the help of a survey or other interview techniques. On the other hand, evaluation communities have defined sets of criteria for assessing t