“The Magenta Book is HM Treasury guidance on evaluation for Central Government, but will also be useful for all policy makers, including in local government, charities and the voluntary sectors. It sets out the key issues to consider when designing and managing evaluations, and the presentation and interpretation of evaluation results. It describes why thinking about evaluation before and during the policy design phase can help to improve the quality of evaluation results without needing to hinder the policy process.
The book is divided into two parts.
Part A is designed for policy makers. It sets out what evaluation is, and what the benefits of good evaluation are. It explains in simple terms the requirements for good evaluation, and some straightforward steps that policy makers can take to make a good evaluation of their intervention more feasible.
Key issues in policy evaluation
What is evaluation and what benefits can it bring?
What factors affect how a policy should be evaluated?
How evaluation fits into the policy cycle
Identifying the right evaluation for the policy
How was the policy delivered? Process evaluation
What difference did the policy make? Impact evaluation
Did the benefits justify the costs? Economic evaluation
Why did what happened occur?
What type of evaluation for the policy?
How do evaluation questions relate to the underlying “logic” of the intervention?
Factors affecting the choice of evaluation approach
Building impact evaluation into policy design
Thinking about impact evaluation when designing the policy
The role of comparison groups in identifying the impact of a policy
What practical issues need to be taken into account when designing an evaluation
The main steps in the evaluation process
How to ensure an evaluation meets the requirements: governance and quality control
Timing of the evaluation
What types of resources are likely to be needed?
What level of resource should be dedicated to the evaluation
Part B is more technical, and is aimed at analysts and interested policy makers. It discusses in more detail the key steps to follow when planning and undertaking an evaluation and how to answer evaluation research questions using different evaluation research designs. It also discusses approaches to the interpretation and assimilation of evaluation evidence.
The stages of an evaluation
The steps involved in planning and undertaking an evaluation
Setting out the evaluation framework
Reviewing the existing evidence
Rapid evidence assessment
Meta-evaluation and meta-analysis
Making sense of existing and new evidence: simulation modelling
What is monitoring data and how can it contribute to evaluation?
New data collection
Designing data collection tools
Ethical and data protection considerations
Process evaluation, action research and case studies
Evaluation to understand the implementation and delivery of policy
Why undertake a process evaluation, action research or case study?
Research methods to support process evaluation, action research and case studies
Choosing research methods
Empirical impact evaluation
Introducing empirical impact evaluation
When are empirical approaches possible?
Designing policies for effective evaluation
Power of design
Strategies for analysing quasi experimental data
Thinking critically about the textbook techniques
Drawing together and reporting evaluation evidence 125
How evaluation evidence may be used
Drawing together the evaluation evidence
Setting the evaluation results in a broader context
Future decisions and roll-out; scaling-up
Implications for evaluation planning
Reporting and disseminating findings
The Magenta Book will be supported by a wide range of forthcoming supplementary guidance containing more detailed guidance on particular issues, such as statistical analysis and sampling. Until these are available please refer to the relevant chapters of the original Magenta Book.