This document outlines the objectives and approach of a meta-evaluation of the impacts and legacy of the 2012 London Olympic and Paralympic Games. The meta-evaluation will comprehensively assess tangible and intangible effects across various areas such as sport, economic, social, and regeneration of East London. It will consider outputs, outcomes and the value for money of legacy investments. The approach involves developing logic models and research questions for each theme, drawing on multiple data sources including national evaluations, economic modeling, and some primary research. Challenges include attributing impacts and assessing additionality and sustainability of effects.

1. 450
400
Average house prices (£ 000's)
England
350
Five Host Boroughs
300
Greenwich
250 Hackney
200 London
Newham
150
Tower Hamlets
100
Waltham Forest
50
0
1995-01
1996-01
1997-01
1998-01
1999-01
2000-01
2001-01
2002-01
2003-01
2004-01
2005-01
2006-01
2007-01
2008-01
2009-01
2010-01
Meta-Evaluation of the Impacts and Legacy of the
London 2012 Olympic Games and Paralympic Games
Jonathan France and Anna Woodham
June 2nd 2011

2. Contents
• Overview of the meta-evaluation
– Objectives
– Phases
– Methods
• Planning and prioritisation
• The evaluation approach
• Key areas for further research
2

3. Study objectives
Comprehensive and systematic meta-evaluation of the Impacts and Legacy
of the London 2012 Olympic Games and Paralympic Games
• Considering Sport, Economic, Social and East London, plus Disability
and Sustainability
• Outputs, intermediate results, and outcomes, and the additional impact
and VFM of legacy investments
• Tangible and intangible/intended and unintended effects
• Lessons learnt around planning and delivery of a lasting legacy
• Influencing role (interim evaluation)
• Inform and advance the evidence base on hosting mega-events
• To develop and enhance methods of meta-evaluation (ESRC supported)
3

4. Overview of the 3 phases
Phase 1: May 2010 – April 2011 Phase 2 : Feb 2011 – March 2012 Phase 3: April 2012 – March 2013
Project Initiation Report 3 (Baseline)
• Project Initiation Document • Baseline Report 6 (Post Games Evaluation)
• Project Plan • Counterfactual • Baseline & counterfactual update
• Project set-up tasks • Evaluation synthesis
Report 4 (Interim Evaluation) • Primary research
Report 1 (Scope) • Evaluation synthesis • Economic modelling
• Identification of project scope • Primary research • VFM
• Research questions • Initial modelling • Lessons learnt
• Logic chains • Impacts and lessons learnt to date • Longer-term impacts
• Data strategy • Meta-evaluation methods
Report 2 (Methodology)
• Meta-evaluation methodology
• Lessons learnt
Phase 4 planned for (March 2013-2020)
4

5. Contents
• Overview of meta-evaluation
• Setting up and planning the meta-evaluation
– Project scope
– Logic chains
– Research questions
• The evaluation approach
• Key areas for further research
5

6. Defining scope
• Legacy initiatives:
– New public programmes and investments developed as a result of
Games
– Existing programmes and investments refocused, expanded, or
accelerated
– Private and third sector investments inspired by the Games
• Distributional impacts:
– Geography (East London, Nations and Regions)
– Groups (disabled, BME, young people, gender)
• Temporal impacts:
– Pre, during and post Games
 Thoughts refined through desk research, DCMS workshop, stakeholder
consultation and logic chains.
6

7. Logic chains
Rationale and Objectives Outcomes/impacts Gross to Net Conversion
London 2012 involves a major regeneration programme providing jobs, homes, - Economic benefits – -Need to take account of additionality -
infrastructure, training and opportunities for UK businesses. This helps overcome increased GVA, job extent to which opportunities would not
a coordination failure whereby firms and individuals are unwilling to invest in the creation and jobs have been provided or not be available
area even though if they all did together they would benefit. from other sources in the counterfactual
safeguarded
scenario.
The Games will also: - Better health
- act as a showcase for British expertise and capabilities, overcoming similar outcomes -Need some consideration of extent to
problems with coordinated promotion to encourage inward investment, tourism - Better quality of life/ which programme of intervention is
and improve export potential; improved wellbeing responsible for observed outcomes.
- provide a focal point for national identity, representing Britain and British culture to
- More cohesive and
the world and creating civic and national pride that inspires new attitudes and -Adjustments for
behaviours; inclusive communities displacement, leakage, substitution, cro
- inspire greater participation in sport, providing positive externalities through - More sustainable wding out and multiplier effects, as
improved health. communities. relevant.
Activities Outputs Results
- Sport - Harnessing the UK's passion for - Engagement with individuals - New participants in sport and cultural activity
sport (sports, culture and employment - People into jobs and gaining qualifications
support) - More new businesses and more contracts traded with
- Economy - Exploiting opportunities for existing businesses
economic growth - Improvements to sports
- New inward investment to East London and UK
infrastructure
- Social - Promoting social engagement - Increased visitor numbers to London and UK
- Engagement with businesses - Greater sense of community cohesion and inclusion
and participation
- Creation of opportunities - More sustainable behaviour of individuals and
- East London - Driving the regeneration (volunteering, disabled people etc) businesses
of East London. - Marketing and awareness campaigns - Improved accessibility for disabled people in accessing
- Building of venues according to new sport, culture employment and transport.
sustainable construction standards.
7

8. Research questions
Designed to tease out what the impacts have been in legacy areas
participation in sport and physical CROSS CUTTING
activity the development of
competitive and elite sport Changes in attitudes to disability; increased participation of
disabled people in sport, economy, volunteering and culture
economic Contribution to sustainable development
impacts, particularly in Effects on well-being
terms of employment
Effects on the international profile and reputation of the UK,
and GVA
London and East London
Benefits to target groups/communities
social impacts, particularly
in terms of volunteering
and development of the
'big society' value for money
Whether impacts have been sustained
Lessons learned about how to maximise the benefits to
contribution to the
the host country and city from the staging of mega-events,
regeneration of
particularly in terms of organisational lessons and change
East London

9. This is a meta-evaluation
• Requires robust programme evaluations, that meet needs of Meta-Evaluation,
so…
1. Identify gaps or lack of granularity
2. Influence evaluations
• Evaluation Steering Group
• Work with Legacy Boards
• Internal and external seminars
3. Ongoing assessment of relevance and robustness of project-level evaluations
4. Last resort develop own plans for primary research
9

10. Contents
• Overview of meta-evaluation
• Setting up and planning the meta-evaluation
• The evaluation approach
– Framework
– Value for money
– Challenges
• Key areas for further research
10

11. Framework for assessing impact
• Builds upon existing evaluation frameworks:
– Treasury Green Book input – output – outcome model
– Regional Development Agency Impact Assessment Framework
– Magenta Guidance
– PWC 2012 Games Evaluation Framework
• Each logic model grounded in theory of change and common outcome
measures, which can then be tested out through evaluation evidence
• Issues of deadweight must be tested and will apply to all themes;
displacement and substitution more to economic themes
• Relevant impact assessment frameworks then devised for each
theme, but top down and bottom up analysis required in all cases
11

12. Sport: Logic chain
Rationale and Objectives Outcomes/impacts Gross to Net Conversion
- Increased participation in Need to take account of additionality -
The 2012 Games provides an unparalleled opportunity to catalyse investment in new
in active sport extent to which opportunities would not
and improved sports infrastructure, as well as inspiring people to do more and
- Increased life expectancy have been provided or not be available
achieve more in sport. Social and economic benefits can be derived from health
and better health from other sources in the counterfactual
benefits. Elite achievement in sport can deliver important benefits in terms of
- Increased excellence and scenario.
boosting national pride, providing sport role models, encouraging community and
elite participation in sport, and improving the UK's reputation and influence abroad. elite sport achievements
- Increased sporting Need some consideration of extent to
influence abroad which programme of intervention is
The GOE’s Sports Legacy Strategy Paper, set out five promises:
responsible for observed outcomes.
- Places: Transforming the places where people play sport, making the benefits of
London 2012 visible in cities, towns and villages across the country
- People: Inspiring people to make sport happen at the local level
- Play: create the sporting opportunities and challenges that give everyone the
chance to become part of the mass participation legacy.
Activities Outputs Results
Legacy activity is to take place in three areas: - Increased access to sport opportunities, sport
- Youths involved in Olympic and Paralympic satisfaction, club membership
style school sport competitions - New participants in sport and less drop-off
-Places: Upgrading up to a thousand local sports clubs - Increased confidence of pupils, higher school
- Adults and young people involved in
and facilities;· investing in iconic multi-sport facilities; attendance, and academic motivation
community sport and physical activity
protecting and improving hundreds of playing fields. - Better pathways and support to elite sport
programmes
-People: Recruiting, training and deploying 40,000 - Higher level of sport participation for disabled
- Coaches trained to support sports
sports leaders - Increased access to sport volunteering and
- Hours of volunteering for sports
-Play: Motivate over 100,000 adults to test themselves higher interest and commitment in volunteering
- Investment in elite sport and NGBs
in multiple Olympic and Paralympic sports;; investment - Increased influence abroad
- Young people engaged in international sport
in sport for disabled people. - Mega sporting event bids won
- Bids for other sporting events
- Support for Elite and Internationalsport
12

13. NATIONAL EVALUATIONS
Data sources: National evaluations with a regional dimension
eg. CompeteFor
collation; QA;
synthesis; and PRIMARY AND SECONDARY DATA
aggregation Secure additional questions in surveys
Commission studies if needed
PROJECT MONITORING
Project/programme monitoring information
Input data per project
MACRO ECONOMIC MODELLING
Impacts of construction expenditure
Defining counterfactual
Estimating secondary (indirect and induced) impacts
QUALITATIVE DATA
Case studies
13

14. Examples
• Sport and social themes:
 Amendments to national culture and sport surveys in
England, facilitating statistical tests of association between sports
participation and variables measuring engagement in the Games;
medals performance; time-series data/scenarios for volunteering
 Bottom-up evidence from programme evaluations; post Games
venue planning; selective primary research for Games Makers
(survey), disability (media analysis) and sustainability (case studies)
• Economy and East London
• Mix of statistical techniques (differences in differences/control
areas; economic modelling); amendments to national tourism
surveys; UK trade and investment data
• Bottom-up evidence from programme evaluations; stakeholder
interviews (SAV); academic studies; resident and business surveys
14

15. Valuing benefits
2012 Games
Techniques include: Benefits/Outputs (with legacy)
• Benefits transfer (other studies) 2012 Games
(Without legacy)
• Econometric modelling No 2012 Games
Baseline
• Revealed preference techniques
• Stated preference techniques
On benefits transfer, some benchmarks available through the DCMS CASE
programme:
• Doing sport once a week generated Subjective Well Being equivalent to
£11,000 increase in household income;
• Health cost savings of doing sport are £5,000 for a 30-39 year old.
15

16. Thematic challenges
• Sport
– Measuring the sustainability of participation (or use of proxies)
– Methodologies to analyse elite sport networks are underdeveloped
– Top-down sport for development highly criticized and under-evaluated
• Social
– Lack of participant research – access to LOCOG Games Maker database
– Wider influences, e.g. Big Society
• Economy
– Aggregating the short-term and ongoing impact of the Games on economy
– Assessing the longer-term influence of the Games on image/competitiveness;
– need to influence tourism agencies to capture impacts over time
– wider impact of the Games on inward investment and export trends
(longitudinal approach preferred)
16

17. Broader challenges
• Questions have been prioritised, but scope still vast and complex
• Internal organisation and systems
• What is additional?
• Influencing, building support (e.g. LOCOG, academic community), access
• Synthesis or aggregation of outcome data?
• Assessing additionality
• Developing counterfactuals without evaluations
• Developing counterfactuals with evaluations
• Geographical and E&D sensitivities
• Mega-events have not previously been measured in such a
comprehensive way: no real precedent
17

18. Contents
• Overview of meta-evaluation
• Planning the meta-evaluation
• The evaluation approach
• Key areas for further research
18

19. Areas for further research:
Theme Primary Data Requirements
East London • Household survey and focus groups with residents
• Private sector leverage
• Stakeholder lessons from park conversion and convergence
Economic • Visitor experience survey
• FDI attracted
• Sustainability of employment outcomes
Sport • Impact of infrastructure investments
• Elite Sport contribution
Community Engagement • Inspire Programme outcomes
• Games Maker survey
Disability and • Perceptions of disability
Sustainability • Additional small scale primary research
Nations and Regions • Pre-Games Training Camps impact
• Local Authority 2012 activities and outcomes
19

20. How you can get involved
• Share knowledge on existing and planned 2012 evaluations and
research
• Share findings of 2012 project level evaluations and research
• Influence surveys and research work planned
• Views and input on methodology and alternative approaches
• Contact: anna.woodham@culture.gsi.gov.uk
20

22. Questions…
22