This article was originally published by the Benckiser Stiftung Zukunft.
Organizations in the social sector should review the impact of their programs, or arrange such a review. Not to keep their donors happy but to check whether they really are doing the best they can to reach their target group. And to ensure they are reaching as many members of the target group as possible. External assistance can be very useful in demonstrating impact. But external evaluations also imply a lot of in-house cost and effort. What do social organizations need to consider to be sure the process pays off?
To measure the impact of social intervention, it’s essential for the players to have a precise understanding of their target group and also have clearly defined objectives, which must be detailed, comprehensive and broken down into short-term, medium-term and long-term effects. They also need a defined method of checking and demonstrating their success in achieving these objectives. It can therefore be useful to draw on the services of a third party to perform an evaluation, in combination with in-house methods of assessing impact.
The quality of external evaluations varies greatly when it comes to social programs. Not all results mean the same thing. The scope may range from a simple query about participant satisfaction through to randomized controlled trials (RCT), the “gold standard” that can trace the causative influence of social intervention on the impacts achieved in the target group. Our colleague Professor Armin Falk of the Behaviour and Inequality Research Institute (briq) at the University of Bonn takes the firm view that “There is no point in arranging an evaluation at all if an RCT cannot be performed.” While we understand his position, the bar must be set a little lower in the social sector, since RCTs will simply be out of the question for many organizations.
Even below gold-standard level, external evaluations can offer tips on being effective. What do you need to consider while cooperating with an external scientific partner?
Is our external partner the right one?
In addition to its area of expertise, the external partner’s familiarity with the program that has to be evaluated, and its experience with impact analyses, are of key importance. Reading previous studies can help establish a picture of how the potential partner entity operates.
Who’s in charge? How important is our evaluation for the external partner entity?
Certainly, many research institutes need assistants to be able to process several projects in parallel. But assistants should play only a supporting role, also in an impact analysis. The main work belongs in the hands of the experienced staff. Otherwise, everyone – the social organization in particular – will pay dearly for errors and omissions. In a worst-case scenario, errors in study design or execution will make the evaluation worthless.
Do the external partner’s research design and approach take our actual situation at the coal face into account?
Of course, it’s up to the external partner to determine the design of the study and the approach to be adopted in the surveys. But good co-operation implies the ability to call things into question. For example, if a group survey setting creates a risk that the parties being surveyed may be able to influence each other and therefore potentially alter the responses, the social organization should be able to intervene and insist on changes for the sake of the overall study quality.
Does the analysis deal sufficiently with our target group and program objectives?
Even if analyzing the generated data is the responsibility of the external partners in the first instance, discussion about the data, or working on it together, can be valuable. After all, it is not uncommon for data evaluation to look at the target group as a whole, which can eliminate the differences between the extremes. Social organizations know their target group and their objectives best, and so they should demand an analysis that is as highly differentiated as possible, to ensure the results are meaningful and informative.
Do the approach and scope of the study justify our expenditure on it?
In the best case, an impact analysis of the social organization will provide a useful demonstration that “the program works.” But it also costs money and means additional work – for the social organization, too. An impact analysis requires patience and tenacity, sometimes even courage. If there are any doubts about whether an external evaluation is really worth the cost and effort, it may be more practical to start by investing in in-house measures for outcome orientation.
If a social organization opts for an external impact analysis, then it must be done with full commitment and confidence, since maximum effectiveness should – indeed, must – be demanded even when effectiveness, i.e. impact, is being analyzed.
Originally published May 3, 2017
