This study is intended to broaden the range of impact evaluation (IE) designs particularly for difficult to evaluate programmes. The study considered existing IE practice, reviewed methodological literatures and assessed how state-of-the art evaluation designs and methods might be applied.

The study has concluded that most development interventions are ‘contributory causes’. They ‘work’ as part of a causal package in combination with other ‘helping factors’ such as stakeholder behaviour, related programmes and policies, institutional capacities, cultural factors or socio-economic trends. Designs and methods for IE need to be able to unpick these causal packages.

Development programmes that are difficult to evaluate such as those concerned with governance, democracy strengthening or climate change mitigation, are often described as ‘complex’. Instead of classifying programmes into how complicated they are, attributes of programmes were identified on the basis of literature reviews and analysing a portfolio of current DFID programmes. Tailored evaluation strategies are needed to respond to these attributes.

There is often a trade-off between the scope of a programme and strength of causal inference. It is easier to make strong causal claims for narrowly defined interventions and more difficult to do so for broadly defined programmes. The temptation to break programmes down into sub-parts is therefore strong, however this risks failing to evaluate synergies between programme parts and basing claims of success or failure on incomplete analysis.

The study concluded that a common framework could be applied across different designs and methods. Standards such as validity, reliability, rigour and transparency have therefore been incorporated into a three part quality assurance framework covering: the conduct of the evaluation; the technical quality of methods used and normative aspects appropriate to IE in an international development setting.