Assessement Guidance – “Best Value for the Money” Ulrich Norf WW - Public Procurement support CIO Summit , June 2015

1. Ulrich Norf
WW - Public Procurement support
CIO Summit , June 2015
Assessement Guidance –
“Best Value for the Money”

2. AGENDA
• Public sector IT Procurement Market view
• The Position of the European Commission
• Solution for Germany and beyond
• Background
• BKM

3. 3
Market View
•The purchase of goods and services by the public sector accounts for a significant
proportion of public expenditure, and of demand for goods and services in the economy.
In some markets the public sector is likely to be by far the largest buyer, and in a position
to affect competition through its purchasing behaviour. (High Volume, Catalogue,
Shopping basket…)
•Governments increasingly providing commercial services through “e” or “ICT Enabled”
Government driven by a citizen centric approach to expectations and experiences of
private sector and the internet
•Simultaneously there has been greater emphasis and pressure on government efficiency
and transparency in decision making and procurement, as well as a desire for access to
the latest technology and services and reduced cycle time for procurement.
•The result has been an increased emphasis on competitive pricing and with particular
respect to IT related purchases, a large amount of confusion in how to specify the
products needed by government which maximise best public value over the entire
lifetime of the product(s) and not just the up front cost alone.

4. 4
Government Procurement Worldwide
LAR
Some awareness of EU / US
legislation as “Best Practise”
for Govt tenders through ‘Fair
Open Competition Group’ as
well as through NAFTA.
EMEA/EU
The EU member state are using the EU rules for public
procurement specified in 2 directives (2004/18/EC &
2004/17/EC). With respect to IT equipment any tender
over the value of €137,000 for central governments
(€211,000 for non-central government) requires the
contracting authority to publish details on Tenders
Electronic Daily (or TED)
http://ted.publications.eu.int/official/
APAC
In APAC, PRC, Korea, SEA, India, ANZ
have different govt tender processes.
In India Government tenders are
published in newspapers, while in PRC
all tenders are notified to the public
through government websites. With
the exception of ANZ & Singapore,
tender processes are country specific.
There is no specific, detailed legal
guidance
USA
Legal guidance is provided through the U.S.
Federal Acquisition Regulation (FAR). Use of
performance specifications or published
standards are preferred consistent with Office
of Management and Budget (OMB) Circular A-
119. In the USA, any tender over the value of
$25,000 is required to be posted on
FedBizOpps.gov

5. AGENDA
• Public sector IT Procurement Market view
• The Position of the European Commission
• Solution for Germany and beyond
• Background
• BKM

6. EC Directives 2004/17/EC
& 2004/18/EC
6
Technical specifications can be
formulated:
 by reference to product features;
 in terms of performance or functional
requirements; OR
 using a combination of both
Each reference to product features
must be accompanied by the words
“or equivalent“

7. EC Directives 2004/17/EC
& 2004/18/EC
7
No reference to a specific make, origin,
process or brand, unless justified by
the subject-matter of the contract or
the need to describe the contracted
goods, if this is not possible otherwise
 If such a reference is made, the “or
equivalent” wording must be used

8. The Position of the
European Commission
8
Use of brands in the technical specifications for
microprocessors is not justified
 Even if accompanied by the words “or
equivalent”
Use of a minimum clock-rate is discriminatory
Microprocessors may be described using
references such as the type of the microprocessor
and its required performance
 The performance of microprocessors should
be evaluated using appropriate benchmarks
developed by industry consortia or third-party
independent benchmarking firms

9. AGENDA
• Public sector IT Procurement Market view
• The Position of the European Commission
• Solution for Germany and beyond
• Background
• BKM

10. IT Procurement Guidelines for Germany since 2007
hosted by BITKOM and Central Procurement agency
Supported / Awarded by:
OECD
(Organization for Economic Co-operation and
Development)
DIGITALEUROPE
(European Information & Communications
Technology Industry Association in Brussels)
Why was this Website created in 2007?
To support the official tender writes with a guidance as the procurement law, driven by the EU,
has changed and is now requiring benchmarks instead of technical features and brands in all
Member states.
What was and is recommended?
Application based benchmarks from BAPCO has been the first choice,
based on a deep technical analysis
Policy
„Recommendations for procurement of efficient and environmentally friendly
desktop PCs, Notebooks, Server and others IT equipment”
-> developed for the public sector
-> observed by all industries
Developed by:
• German Ministries
• Ministry of the Interior’s Procurement Office
• Ministry of the Environment
• Federal Employment Agency
• Army’s (Federal Office for Information Management and Technology)
• Industry partners (Intel, AMD*, OEMs, LOEMs)
Languages supported: English / French / Spanish / Portuguese / Dutch / German
www.ICT-procurement.com
10

11. Recommendations
11
Creating a standard procedure to
ensure accuracy, transparency and
reproducibility.
Based on Rules of
 Installation Guidance (OS and Driver)
 Do and Don’t
 Benchmarking tool
 Documentation

12. AGENDA
• Public sector IT Procurement Market view
• The Position of the European Commission
• Solution for Germany and beyond
• Background
• BKM

13. 13
Measuring Performance
“Our position is that the only consistent and
reliable measure of performance is the
execution time of real programs, and that all
proposed alternatives to time as the metric or
to real programs as the items measured have
eventually led to misleading claims or even
mistakes in computer design.”
Mainstream Usages
“Perhaps the most important and pervasive
principle of computer design is to focus on the
common case: In making a design trade-off,
favor the frequent case over the infrequent
case. This principle applies when determining
how to spend resources, since the impact of the
improvement is higher if the occurrence is
frequent.” From Computer Architecture: A Quantitative Approach,
Fifth Edition, Hennessy & Patterson

14. 14
Reporting Performance
“The guiding principle of reporting
performance measurements should be
reproducibility – list everything another
experimenter would need to duplicate the
results.”
Multiple Benchmarks
“To overcome the danger of placing too many
eggs in one basket, collections of benchmark
applications, called benchmark suites, are a
popular measure of performance of processors
with a variety of applications. Of course, such
suites are only as good as the constituent
individual benchmarks. Nonetheless, a key
advantage of such suites is that the weakness
of any one benchmark is lessened by the
presence of the other benchmarks.” From Computer Architecture: A Quantitative Approach,
Fifth Edition, Hennessy & Patterson

15. 15
Basic Benchmark
Selection Criteria
1. The benchmark uses real applications or
benchmark applications executing real
workloads, and be based on real-world
scenarios and workflows
2. The benchmark was designed with industry
stakeholder input baked into the
development process, guided by industry
best practices and transparency
From Computer Architecture: A Quantitative Approach,
Fifth Edition, Hennessy & Patterson

16. 16
From Computer Architecture: A Quantitative Approach,
Fifth Edition, Hennessy & Patterson
Define
 What is being measured?
 Identify and reduce all possible variables
Follow the rules
 Stick to the benchmark tool rules and
standards
Be consistent and fair
 Keep things the same and always double
check
 Perform multiple runs for statistical
consistency
 Compare fairly and consistently
Document everything
 Eliminate guesswork and ensure replicability
Best Practice

17. AGENDA
• Public sector IT Procurement Market view
• The Position of the European Commission
• Solution for Germany and beyond
• Background
• BKM

18. 18
From Computer Architecture: A Quantitative Approach,
Fifth Edition, Hennessy & Patterson
Different benchmarks test different
areas of platform performance. For
comprehensive platform
evaluations, it’s best to cover all
three domains of performance.
 Mainstream Usages – use standard OS
libraries, APIs and services
 Web Usages – use web browser
technologies; inherently cross-OS
 Game & GPGPU Usages – use graphics
Benchmark Domains

19. 19
Describing Performance
Performance benchmarks generally are
acceptable way of describing the
performance of a system
Need to choose benchmark(s) that are:
 independent, regulated and widely
recognised;
 relevant;
 up-to-date;
 Reflecting the usage model
Need to set up and follow a rigorous
Methodology
Performance benchmarks do not fit all situations From Computer Architecture: A Quantitative Approach,
Fifth Edition, Hennessy & Patterson

20. Describing Other
Features Unrelated to
Performance
20
• Performance benchmarks tell nothing about
other important CPU & computer system
features unrelated to performance
• Even when benchmarks are used to describe
performance, contracting authorities should:
 Identify the CPU and computer system
features unrelated to performance,
which may help lowering their TCO and
achieve “Best Value for Money”;
AND
 Describe these features in the technical
specifications, in an objective and non-
discriminatory way

21. Placeholder Footer Copy / BU Logo or Name Goes Here