1. SAI Performance Measurement
Framework
Presented by: Yngvild H. Arnesen, INTOSAI Development
Initiative (IDI)
Date: CReCER, Managua, 29-31 October, 2012
INTOSAI Working Group on the Value and Benefits of SAIs
1

2. What is the SAI Performance Measurement
Framework (PMF)?
• A Performance Measurement Framework tailored to Supreme
Audit Institutions (SAIs)
• Intended to give a high level assessment of SAI performance
against established INTOSAI good practice:
– The International Standards of Supreme Audit Institutions (ISSAI)
framework
– The Framework on the Value and Benefits of SAIs
– Other INTOSAI guidance material
• Combination of objective measurement and qualitative
assessment
• Analyzes 7 domains of SAI performance in the country context and
aims at assessing the value and benefits of the SAI to society
(impact)
2

3. Conducting a SAI PMF Assessment
1. Using the SAI PMF will be voluntary for the SAIs – the tool will be a public
good which SAIs can make use of if it helps them achieve their objectives
2. The SAI will decide whether or not to publish results
3 Approaches to Conducting a SAI PMF Assessment:
There are three main approaches to conducting an assessment using the
SAI PMF. The PMF will be equally well suited for all three:
1. SAI Self 2. INTOSAI Peer 3. External
Assessment Review Assessment
• Trained Team • Peer SAI • Professional
from within • INTOSAI PFM Body
the SAI Regional Body • Donor Agency
• Consultant
• External
auditor
SAI Involvement 3

4. Responsibilities and Reporting Lines
INTOSAI Congress (2013)
Approval
Mandate from
INCOSAI XX, 2010
• INTOSAI Capacity Building Committee
• Other Relevant INTOSAI Committees
• INTOSAI-Donor Steering Committee
INTOSAI Working Group on
the Value and Benefits of SAIs
Oversight and Direction
In Consultation with:
WGVBS SAI PMF Task Team SAI PMF Reference Group
Development of SAI Performance Technical Advice, Review and
Measurement Framework Quality Assurance
Coordinator: INTOSAI-Donor Secretariat @ IDI Other Groups:
INTOSAI
INTOSAI-Donor PEFA Secretariat,
Capacity Building
INTOSAI Steering Committee International
INTOSAI-Donor Committee Budget
Capacity Building Members
Steering Committee Members Partnership
Committee
Members
Members
4

5. SAI PMF Timetable
2011, Q3 2011, Q4 2012, Q1 2012, Q2 2012, Q3 2012, Q4 2013, Q1 2013, Q2 2013, Q3 2013, Q4
WGVBS:
Set
Direction
Mapping Existing SAI
Measurement Tools
WGVBS:
Approve
Concept Note for SAI PMF way
forward
Develop SAI PMF Version 2.0
Phase 1 Pilots in 3 SAIs
& Comments from
Stakeholders
WGVBS:
Develop Approve
SAI PMF Exposure
Exposure
Draft
Draft
Phase 2 Piloting and Pre-Consultation
(ongoing into 2014 & 2015)
Exposure
Draft at
INCOSAI
5

6. Why should SAIs use the PMF (1)
• As a step towards ISSAI implementation
– Increased understanding in the INTOSAI community of what good
practice for SAIs entails
– Individual SAIs can benchmark their performance against good
practice – and get an indication of where there is room for
improvement
– They can use the assessment as a basis for more detailed ISSAI
compliance assessments
• To demonstrate progress and value and benefits to society
– The framework will be well suited for measuring progress over
time and demonstrating this to external stakeholders
– Show to stakeholders how the SAI contributes to strengthening
public financial management
6

7. Why should SAIs use the PMF (2)
• Internal performance measurement / annual
reporting
– Help SAIs better manage their own performance
– ISSAI requirement to assess and report on own performance –
lead by example and become a model organization
• Inform and secure support for capacity development
initiatives
– Identify strengths and weaknesses, increase understanding of
reasons for good and weak performance
– Help build a culture for performance improvement
– SAIs can use assessment results to get support from donors and
other external stakeholders
7

8. SAI PMF: Evidence Based
Assessment of Performance
Holistic assessment of Qualitative
the SAI and its assessment
environment – (Performance
including factors which
are not covered by the Report)
indicators
Objective
measurement to
inform qualitative
Measurable
assessment and track indicators
progress over time
8

9. SAI Performance Measurement Framework – Structure
Under the control of the SAI Under the control of the SAI
SAI Environment SAI Capability SAI
Performance Impact
Foundational factors SAI STRATEGY Measured using other
set by the legislative
and/or executive, but C. Strategy for Organizational Development A. SAI assessments. Impact
can not be directly
which can be • 1 indicator Performance attributed to SAI results,
influenced by the SAI
in the long term but the SAI’s likely
SAI CORE BUSINESS • 3 indicators contribution should be
analyzed in the SAI
B. D. Audit Standards and Methodology
performance report.
Independence
Contribution
and Legal • 3-5 indicators to Public
Framework
Financial
• 3 indicators E. Management F. Human G. Communication Management,
& Support Resources and Stakeholder Governance
Structures Management and Tackling
Corruption
• 2 indicators • 3 indicators • 3 indicators
SAI SUPPORT SERVICES
Public Financial Management Environment
Influences the effectiveness of the SAI but not under direct control of the SAI.
Measured through other assessments but referred to explicitly in the narrative SAI performance report.
9

10. SAI Performance Report
- end product from an assessment (30-40 pages?)
Provides a coherent and holistic assessment of SAI performance
• Informed by objective and evidence based performance indicators
• Analyzed within context
• Considered as a whole – not the sum of its indicator scores
Enables SAI heads and senior management to
• Identify barriers to improved performance
• Track performance changes over time
• Promote and communicate the value and benefits of the SAI
Explores how SAI external performance is constrained by
• Country context and PFM environment
• SAI environment
• SAI internal capability 10

11. Potential risks related to having a SAI PMF
Revelations of performance weaknesses
 It will be up to the SAI to decide if results are to be shared
 Openness and commitment to reform likely to encourage external
partners to support capacity development
 May help draw attention to constraints outside the SAI’s control
Possible negative effect of scoring: Use of performance league tables
 Intention is not to aggregate scores to the level of the entire SAI – as all
indicators may not be of equal importance
 Scores partly linked to achievements of SAI’s own mandate
 Narrative report provides context for scoring
 Scoring is already applied in INTOSAI (AFROSAI-E)
 Scoring very useful for measuring progress over time within an SAI
Profusion of Assessment Tools
 Risk that the SAI PMF becomes an additional tool
 Donors and SAIs continue to develop & use other tools
11

12. 6. Further Information
www.idi.no
Mapping of Tools Supreme Audit
for Assessing Institutions
Performance of Performance
Supreme Audit Measurement
Institutions Framework
IDI, May 2012 Concept Note
IDI, May 2012
E-mail: INTOSAI.Donor.Secretariat@idi.no
12