OECD workshop on measuring the link between public procurement, R&D and innovation. "Quantifying public procurement of R&D in Europe based on administrative data Interim study results" presentation by Lionel Kapff

1. Quantifying public procurement of R&D in Europe based on administrative data Interim study results OECD Workshop Measuring the link between public procurement and innovation Lionel Kapff (lkapff@deloitte.com) Sebastiaan van der Peijl (svanderpeijl@deloitte.com)
Paris, 5th – 6th December 2013

2. © 2013 Deloitte
2
Quantifying public procurement of R&D /// OECD workshop on measuring the link between public procurement and innovation
Contents
Introduction 3 Purpose of the study 4 Methodology 5 Availability of public procurement data across Europe 7 Interim results for selected European countries 9 Concluding remarks 10 Questions & Answers Annexes:
•Contact information;
•Recommended readings.

3. © 2013 Deloitte
3
Introduction
Study commissioned by the European Commission, DG CNECT:
•“Quantifying public procurement of R&D of ICT solutions in Europe” Study consortium:
•t33 + Deloitte Consulting Belgium + Spark Legal Network Measurement of:
•Public procurement of ICT
•Public procurement of R&D
•Public procurement of R&D of ICT solutions Coverage:
•27 EU Member States + Norway + Switzerland
•2008 - 2011 Timeline of the study:
PP
of
R&D
PP
of
ICT
PP of R&D of ICT
1. Methodological analysis
2. Data gathering and analysis
3. Comparative analysis and future methodology
01/2012
04/2012
02/2014
12/2013
Quantifying public procurement of R&D /// OECD workshop on measuring the link between public procurement and innovation
Final Workshop in Brussels in February 2014

4. © 2013 Deloitte
4
Purpose of the study
Objectives of the study:
•Establish statistics on the R&D and ICT procurements made across Europe, including those made at the country-level (and hence not covered by TED) for the first time.
•Set up a methodology for updating the statistics, assuring sustainability and re-use of the work already carried out for the first time data gathering exercise. Policy goals:
•Reliable statistics are the basis for evidence-based policy-making.
•Statistics enable policy-makers to:
•Set targets;
•Benchmark the status of their jurisdiction compared to others; and
•Measure progress of implemented policies.
•The European Commission will make use of the measurements of expenditure patterns in order to design and develop PCP / PPI support policies efficiently and effectively over time.
Quantifying public procurement of R&D /// OECD workshop on measuring the link between public procurement and innovation

5. © 2013 Deloitte
5
Methodology
Funder-based approach:
•Collection of public procurement raw data from administrative data sources (i.e. public procurement notifications and contract award notices) Data gathering from various sources:
•OJEU - TED
•Public procurement authorities at different levels of government (typically: national/regional e-notification platforms for public procurement)
•Private tender alert service providers Computer-based data analysis + manual verification of results Identification of ICT procurements based on CPV codes
•Definition of ICT based on OECD statistical guidelines
•Translation of CPC Rev. 2 list of ICT products into list of relevant CPV codes
Quantifying public procurement of R&D /// OECD workshop on measuring the link between public procurement and innovation

6. © 2013 Deloitte
6
Methodology
Identification of R&D procurements based on keywords: Missing data (e.g. contract values, defence procurement) is estimated based on:
•Extrapolations of existing data
•Additional statistical input and qualitative information (interviews with Ministries of Defence, public procurement authorities and national statistical offices are carried out in all countries)
1. Manual screening of all CPV-73 procurements in TED for compliance with the OECD Frascati Manual definition of R&D
2. Identification of R&D keywords in the Frascati-compliant TED CPV-73 procurements with a textual analysis software and a manual verification + translation of keywords
3. Automatic screening of public procurement raw data (all CPV codes) for R&D procurements based on R&D keywords in titles and descriptions
4. Manual verification of the results based on the OECD Frascati Manual definition of R&D
Quantifying public procurement of R&D /// OECD workshop on measuring the link between public procurement and innovation

7. © 2013 Deloitte
7
Availability of public procurement data across Europe Public portals including under EU-threshold public procurement
Sources: Deloitte, t33, Spark Legal Network; interim results of the “Study on e-Procurement measurement and benchmarking” (EC DG MARKT)
Countries with a mandatory one-stop portal
Belgium, Bulgaria, Cyprus, Denmark, Estonia, Finland, Hungary, Ireland, Latvia, Lithuania, Luxembourg, Netherlands,
Norway, Poland, Portugal, Slovakia, Slovenia
Countries with multiple portals
Czech Republic, Italy, Malta, Netherlands, Rumania, Spain, Sweden, United Kingdom
Countries with a one-stop portal for national authorities
Austria, France, Switzerland
Countries with a voluntary portal
Germany
Countries with a planned portal
Greece
Quantifying public procurement of R&D /// OECD workshop on measuring the link between public procurement and innovation

8. © 2013 Deloitte
8
Availability of public procurement data across Europe Data availability and quality issues
Fragmentation of public procurement data
•Public data is not always centrally available.
•There are more than 300 e-notification platforms scattered across Europe. Incomplete coverage
•Several e-notification platforms have only been recently established.
•Some sensible areas – such as defence – are not (fully) covered.
•National thresholds for publication vary across countries. Data quality issues
•Values of contracts are often not indicated.
•CPV codes are often not used for below-EU-threshold procurements. Data gathering issues
•Batch-downloads of full databases or even searches are often not possible.
•Some public and private data providers are reluctant to cooperate.
Quantifying public procurement of R&D /// OECD workshop on measuring the link between public procurement and innovation

9. © 2013 Deloitte
9
Preliminary interim results for selected European countries R&D public procurement contracts awarded in 2011
Quantifying public procurement of R&D /// OECD workshop on measuring the link between public procurement and innovation
•Total of 1 094 million EUR of R&D procurements identified in 21 countries covered so far.
0-10 million EUR contracting volume
Bulgaria (2.6), Cyprus (3.1), Estonia* (3.2), Ireland (5.2), Norway (8.9), Slovenia (8.2)
21-100 million EUR contracting volume
Austria* (31.0), Belgium (51.0), Czech Republic* (27.0), Greece (22.8), Romania (67.0), Slovakia (36.3), Switzerland (36.8), United Kingdom* (26.0)
11-20 million EUR contracting volume
Finland (10.4), Hungary (13.3), Lithuania (16.9)
>100 million EUR contracting volume
Italy (107.0), France* (170.0), Poland (290.0), Spain (157.0)
* Data currently only includes above EU-threshold procurements.
All results are of preliminary nature only.

10. © 2013 Deloitte
10
Concluding remarks Using administrative data to measure public procurement of R&D
Opportunities
•Administrative data is currently the most detailed, up-to-date and complete data source available on public procurement
•Measures are currently undertaken across Europe to improve the availability, reliability, coverage and comparability of administrative data on public procurement
•E-procurement initiatives
•Open Data initiatives
•Statistical initiatives
Challenges
•R&D public contracts other than those where the benefits accrue exclusively to the contracting authority are exempted from publication obligations under the EU Procurement Directives
•Problems with regard to availability, reliability and international comparability of administrative data on public procurement
•Goodhart’s law: Challenges for target-driven measurement
Quantifying public procurement of R&D /// OECD workshop on measuring the link between public procurement and innovation

11. © 2012 Deloitte
11
Questions and Answers
Quantifying public procurement of R&D /// OECD workshop on measuring the link between public procurement and innovation

12. © 2013 Deloitte
12
Contact information
Do not hesitate to contact us for any input, questions or comments:
Lionel Kapff
Senior Consultant – Deloitte Consulting
Mobile + 32 475 79 24 76
lkapff@deloitte.com
Sebastiaan van der Peijl
Manager – Deloitte Consulting
Mobile + 32 478 80 77 98
svanderpeijl@deloitte.com
Quantifying public procurement of R&D /// OECD workshop on measuring the link between public procurement and innovation

13. © 2013 Deloitte
13
Annex: Recommended readings
Pre-commercial procurement (PCP) and public procurement of innovation (PPI):
•European Commission website on PCP: http://cordis.europa.eu/fp7/ict/pcp/policy_en.html
•European Commission website on PPI: http://ec.europa.eu/enterprise/policies/innovation/policy/lead-market- initiative/public-proc_en.htm
•European Commission (2007): Pre-commercial Procurement: Driving innovation to ensure sustainable high quality public services in Europe, COM(2007) 799 final.
•European Commission (2005): Public Procurement for research and innovation.
•European Parliament (2009): Pre-commercial procurement: driving innovation to ensure sustainable high quality public services in Europe.
•ECORYS (2011): Study on pre-commercial procurement in the field of Security.
•ERAPRISM (2010): Public Procurement for Innovation in Small European Countries.
•Independant Expert Group (2006): Pre-commercial procurement of innovation.
•OECD (2011): Demand-side innovation policies.
•OMC PTP (2009): Exploring Public Procurement as a Strategic Innovation Policy Mix Instrument.
•Ramboll (2008): Opportunities for Public Technology Procurement in the ICT-related sectors in Europe. Availability of public procurement data across Europe:
•Graux & Kronenburg (2012): State of Play: Re-use of Public Procurement Data.
•Cattaneo (2012): The e-Procurement Landscape in the EU, interim results of the “Study on e-Procurement measurement and benchmarking”.
•Siemens & Time.lex (2010): Study on the evaluation of the Action Plan for the implementation of the legal framework for electronic procurement (Phase II). OECD statistical guidelines:
•OECD (2002): Frascati Manual - Proposed Standard Practice for Surveys on Research and Experimental Development.
•OECD (2011): Guide to measuring the Information Society.
Quantifying public procurement of R&D /// OECD workshop on measuring the link between public procurement and innovation

14. © 2013 Deloitte
Deloitte refers to one or more of Deloitte Touche Tohmatsu Limited, a UK private company limited by guarantee, and its network of member firms, each of which is a legally separate and independent entity. Please see www.deloitte.com/about for a detailed description of the legal structure of Deloitte Touche Tohmatsu Limited and its member firms. Deloitte provides audit, tax, consulting, and financial advisory services to public and private clients spanning multiple industries. With a globally connected network of member firms in more than 150 countries, Deloitte brings world-class capabilities and deep local expertise to help clients succeed wherever they operate. Deloitte's approximately 200,000 professionals are committed to becoming the standard of excellence. This publication contains general information only, and none of Deloitte Touche Tohmatsu Limited, Deloitte Global Services Limited, Deloitte Global Services Holdings Limited, the Deloitte Touche Tohmatsu Verein, any of their member firms, or any of the foregoing’s affiliates (collectively the “Deloitte Network”) are, by means of this publication, rendering accounting, business, financial, investment, legal, tax, or other professional advice or services. This publication is not a substitute for such professional advice or services, nor should it be used as a basis for any decision or action that may affect your finances or your business. Before making any decision or taking any action that may affect your finances or your business, you should consult a qualified professional adviser. No entity in the Deloitte Network shall be responsible for any loss whatsoever sustained by any person who relies on this publication.