Preparing an abstract for the GTZ Conference ‘Systemic Approaches in Evaluation’ on January 25 and 26, 2011, I went through some of the work by Bob Williams. He did some interesting work for the FASID “Beyond Logframe; Using Systems Concepts in Evaluation” (2010). In chapter 3 ‘Using Systems Concepts in Evaluation. A dialogue with Patricia Rogers and Bob Williams’, he is explaining the three core concepts that express the principles of systems thinking: Inter-relationships, perspectives and boundaries.
When discussing systems thinking and one of the DAC Evaluation Criteria ‘relevance’ he stirrs up one of my favourite topics: When evaluating, do we take a project-centric view or a participant/beneficiary view? He states: Whilst I may think the benefits are important and indeed people may be benefiting, I still need to consider just how important they are in relation to all the other things that are going on in beneficiaries’ lives. We have a very project-centric way of looking at the world. Going back to Churchman’s comment – just stepping out and looking back at the situation from the participants’ point of view is critical. Just how worthwhile is the project to them in the grand scheme of things? Could we have done something more worthwhile with our money and their time? (pp 70-71)
Since I think best in schemes and pictures, I call this issue the double funnel issue.
So do we research the results (outcome/impact) of an intervention (project-centric) and/or do we start with looking at changes and then try to ‘plausibly associate’ the results with our intervention, and thus look at the relative importance of these results vis-a-vis other trends, influences etc.?
My experience is that, although I have the opinion that both views should complement each other, in practice, many evaluators (including me) have a (rather) pragmatic bias towards the project-centric view, because it is usually easier to follow this view and in general what the donor or project is asking for only.
While coaching an evaluation of the Kulturhus-concept (what is the effect of the Kulturhus-concept on the quality of life/ social sustainablity?), this double funnel issue was one of the concerns of the evaluators. Through the MSC method, we could explain how the concept plausibly contributed to the quality of living, but to research what the relative importance was of the concept vis-a-vis other trends and interventions was -methodologically- near to undoable and would require a survey and in-depth study, for which there was no time nor money. The best we could do, was to make a set of assumptions of how other trends and interventions plausibly affected the quality of living as well.
The question basically is, how far and how precise can you venture without the attribution issue slapping you in the face? In general I let myself be lead by my common sense and acknowledging there are boundaries that are sometimes impossible to cross.