1. Preliminary Findings and Conclusions of the Mid-term Project Evaluation by
Christian Bugnion, Subur Consulting S.L., Team Leader
Dr. Nizam Al-Husainy, National consultant
www.suburconsulting.es
Dhaka, 16 August 2014
AID EFFECTIVENESS PROJECT
Mid-term Evaluation Preliminary Findings
Implemented by the ERD, MoF
With technical and financial support by UNDP
and co-funded by DFID, DANIDA, Australia

2. • Feedback and accountability to stakeholders: Findings, conclusions and
recommendations;
• Inform about programme performance: relevance, efficiency,
performance, effectiveness, ownership, and sustainability;
• Identify good practice and lessons, looking at the achievements to date;
• Make forward-looking recommendations taking into consideration the
remaining two years time-frame of the Project, but also the wider
Development Effectiveness Agenda
Presentation objectives

3. Mix-methods but primarily qualitative analysis based on the following:
• Documentary review of project documents and updates received during the in-
country mission
• Semi-structured individual interviews with (20) key informants from the following
stakeholder groups:
A. GoB (4 ERD, 1 Cabinet, 1 MP, 1 Planning Commission)
B. UNDP (2 ACD, 1 CD, 1 AE specialist)
C. AE Unit (1 NPM, 1 DPM, M&E specialist, ICT Specialist, Aid Specialist) and a group
discussion jointly with UNDP
D. Development Partners: DFID (1), DANIDA (1)
E. Civil society (2), Ernst and Young (1)
• One field visit to interview participants from AE awareness workshop given at
district level (Moulvbazar) by a member of the evaluation team
METHODOLOGY :

4. • Relevance in responding to identified needs
• Efficiency of the project (use of resources)
• Performance of the project including design issues and implementation
• Effectiveness in achieving project results
• Ownership (degree to which ERD is committed to the process)
• Sustainability after the end of the project
EVALUATION CRITERIA:

5. • Sunday 3rd August 2013 : Start review of documents for the MTE and
consultations between the AE team and the national consultant and field
visit
• Sunday 10 August 2014: arrival of Evaluation Team leader and start of
formal key informant interviews, including joint AE team and UNDP joint
meetings.
• Key informant interviews from 10th to 14th August 2014
• 15th August 2014: review and analysis of data and notes and preparation
of the presentation
• 16th August 2014: presentation to primary stakeholders of the preliminary
findings, conclusions and recommendations
• End August 2014: submission of draft evaluation report
Evaluation Agenda

6. Evaluation Preliminary Findings, Conclusions
and Recommendations

7. • The project addresses the issue of Aid Effectiveness, a wider unit
of analysis that cannot be achieved by a limited intervention such
as the AE project in its limited time-frame. However as the AE
project stems from the highest level of ERD authority, it is clearly
responsive to the needs of the government and in line with the
current debate on what is now called Development Effectiveness.
• GoB planning is based on the current 6th 5-year plan (p. 239) and
AE contributes to shaping future planning and partnership
processes (such as JCS 2010-2014)
Relevance

8. • The project has known a number of delays both linked to procurement
and human resource constraints and limitations from both UNDP and
GoB’s side. In addition the year prior to the election more than half a
year of work was lost given political unrest and violent strikes.
• The project is not particularly efficient as delivery rates varied per year
and some activities had to be postponed or cancelled or changed in
light of the context. Late start also means low initial delivery : 47,5% of
expenditures (US$ 1.46 mio or 32.6% of initial budget and 47.5% of
current funds available). BUT delivery ≠ effectiveness!!
Efficiency

9. BUDGET AND DELIVERY

10. Need to understand the context and environment of the Project – a
number of factors led to delay – enabling factor JCS 2010-2014
• 2011 – late recruitment and start of the project delayed to 2012
• Feb. 2012: change of NPD (15 Feb. 2012) – staffing and international staffing
gradually falling in place – TOR for studies/assessment - Holding of awareness
raising activities at Dhaka and Division levels – support to LCG/WG -
Cambodia Aid database identified as model – Development of AIMS
• 213 – part of the year lost strikes and security situation - Visit to VN for aid
policy, capacity assessment of ERD
• 2014 – Beta version of AIMS, final E&Y report ERD capacity assessment, aid
policy outline established, IMC set-up
Performance

11. • Reasonable performance given the constraints – could have been
better, but could also have been worse
• Essential characteristics: low efficiency in part cause no clear road map
to guide the vision of the AE (insufficient coord. and information sharing
between GoB and UNDP on modalities and incremental approach and
theory of change missing in the design) by also by lack of an enabling
environment (e.g. initially low awareness of the AE, purpose of AE, no
clear understanding of AE, no previous preparation within ERD or GoB
except for 5 YP and JCS, so AE new and not streamlined)
• Better reporting on results necessary, low RBM
Performance

12. • Despite average performance, project rating from the key informant interviews is: 4 out of 5
(1 being lowest, 5 being highest) regarding the effectiveness = Good.
• This is based on the following:
A. Awareness within GoB (high-level) of the importance of the AE agenda
B. Development of the AIMS Database (win/win for GoB and DPs) essential to provide
quality data on aid flows
C. On-going process to develop an aid policy (outline ready and objective to finalize the
policy by end of 2014)
D. Revitalisation of the LCG and WGs
E. Participation in international events (feeds national ownership and 5YP and JCS so
multiplier effect)
F. Ownership trough creation of ERD AE Wing and announced co-funding from GoB (leading
to sustainability)
G. Inclusiveness of the consultation process (NGOs) but more can be done (Parliament,
media, etc.)
Effectiveness

13. • Rating Grid (1 lowest to 5 highest)
Effectiveness
E
R
D
2UNDP3 UNDP4ERD5 CS6 DP7 DP8 AE9 GoB12 AE14 CS15 AVG
4 4 4 4 3,5 4 4 4 4 4,5 4 4

14. • The development of an AIMS database is a key achievement. Actually
not yet live on line, but potential for win/win provided GoB and DPs are
committed to its use
• Will contribute to aid transparency and provide credible evidence and
management information regarding aid flows to all authorized
stakeholders
• Slow in its shaping as no clear technical specs set out from the onset
(tweaking of Cambodia’s model based on cost-effectiveness
considerations)
RESULTS PER PROJECT COMPONENT – 1 AIMS

15. • Many awareness raising activities undertaken but wider inclusiveness and
participation required (civil society, NGO, Parliament, South-South Coop)
• Established a capacity assessment ERD wide but will not be able to generate
evidence by the end of the project: at best some recommendations will be
implemented but remains too soon to affect overall ERD aid management
capacity
• Questionable if this is correct entry-point (ERD-wide assessment) rather than
more limited but incremental approach to capacity development (e.g.
selected wings and future AE Wing)
• Other LM capacity development envisaged but not materialised (although CA
are being foreseen over the rest of the project – TOR developed)
COMPONENT 2 : ERD Aid management capacity

16. • Very positive feedback on the dynamics and revitalisation of the
structures and positive appraisal of its performance – of course varies
from WG to WG.
• Unclear follow-up of key findings and recommendations stemming from
WGs
• More monitoring efforts and RBM into facilitating the process, and also
should be under ownership of ERD when project ends (LCG Secretariat
within ERD Wing? Currently all done by AE Unit – substitution?)
COMPONENT 3 : LCG/WG mechanisms

17. • Slow process and back-and-forth movement to the process. Initially unclear
level of commitment to developing an AP – current process now based on VN
experience requires technical support (int. and nat. consultants hired) and
wide consultations with civil society, parliament, private sector, etc.
• Draft outline ready
• Inter-ministerial committee tasked with follow-up
• Again initial lack of reference or preparation for taking up the aid policy
development process (hence visits and examples from other countries)
• Seen as critical to demonstrate ownership and leadership from GoB and
create a coherent framework for DP assistance
COMPONENT 4: AID POLICY

18. • Participation in international events and support of DPs fuels national
ownership
• ERD S and CS and other high-level officials committed to AE agenda and
the development of AE/DE Wing in ERD
• Ownership ≠ Leadership – GoB committed to both but technical support
from DPs necessary (such as UNDP)
• Long-term process that requires a WoG approach beyond the life of the
project
Ownership

19. • Seems guaranteed through ERD AE/DE Wing creation
• Considering also GoB co-funding to the Project (last SC)
• Full operational AIMS (likely by end of project)
• Approval and implementation of the Aid Policy (likely by end of the
project)
• Gradual merging of AEU into ERD AE Wing with hand-over and training
Sustainability

20. 1. The project has still two years to complete its objectives (June 2016)
2. Important milestones have been achieved so far (AIMS, Draft outline of AP,
support to LCG/WG, develop capacity assessment). Project outputs are
nonetheless an essential contribution to the outcomes, but the outcomes are yet
to be appraised.
3.AE agenda (or DE agenda) cannot be accomplished over the short time-frame
with a single small project. The AE project is the initial building block from which
AE (or DE) efforts must be mainstreamed in GoB with the support of DP. Long-
term planning and commitment from both GoB and DPs to this effect is essential
(incremental approach of 15 years or more).
4. Opportunity for synergies and linkages exist within the AE agenda that are
unexploited. Capacity Development should be a harmonized programme for GoB
under the leadership of a DP (UNDP) and inclusive of all DP to work across sectors
CONCLUSIONS

21. 5. Capacity of other LM not yet started, could be envisaged outside of the
project (under a Capacity Development Programme, see point 4.
6. ERD personnel capacity development needs to be given priority with
adequate hand-over from AEU to new ERD wing over the next two years
7. Longer-term support requires a long-term roadmap for how and where
support will be ensured from DPs over the next fifteen years as it is needed
to ensure mainstreaming of leadership and ownership of AE into GoB
8. More attention needed to show results and RBM value – trainings to ERD
CONCLUSIONS

22. • Importance of giving sufficient time to create enabling conditions for any change
process. Given that AE is a long-time process, the creation of a critical mass of
knowledgeable and committed partners is a pre-condition for success (e.g.
importance of awareness raising and dissemination)
• AE requires a specific communication strategy to enable to engage the various
partners and visualize stakeholders’ buy-in and gains.
• Difficulty to report on achievements given untangible nature of some processes.
Hence the importance of a robust RBM in the Project design to inform proper
Monitoring and Evaluation and a proper Results Framework mindful of the hierarchy
of results.
• Importance of leveraging national leadership and ownership through international
exposure and participation in regional initiatives (Post Busan Conference, APD,
etc.) – so capacity to grasp emerging opportunities beyond the prodoc.
Lessons Learnt

23. 1. Continue and consolidate the project achievements, particularly:
▫ Consolidate and roll-out functional AIMS to be used by GoB and all DP including
UNDP in the first phase to show commitment to the process
▫ Continue the process and provision of support to the development of the Aid
Policy and ensure its finalisation
▫ Continue providing support to ERD capacity development and ensure hand-
over, training and coaching of the ERD new DE Wing
▫ Continue to support participation in LCG/WG structure and participation in
international events
2. Capacity development for development effectiveness is not limited to ERD or
specific LM. A harmonized approach amongst DPs is required for an effective
WoG capacity development programme under the aegis of the UNDP. This
global and harmonized CD programme should be inclusive of all CD projects and
initiatives but provide an overall framework for actual results of the CD
process.
RECOMMENDATIONS

24. 3. The value of the inclusion of the Capacity Assessment foreseen amongst
now 5 LM under the AE project is not evident for this project and should be
dropped and included in the wider CD programme mentioned in the
previous point to ensure no duplication of other DP efforts.
4. Develop a fifteen year road-map and strategy for achieving mainstreaming
of AE/DE into WoG with commitment and support from DPs and
incremental approach from ERD to WoG.
5. Plan incremental developments on the basis of achievements and
synchronized with the National FYPs
6. Improve RBM knowledge and capacity through commissioning JOINT
EVALUATIONS of future AE/DE interventions
RECOMMENDATIONS (cont.)

25. 7. Develop a monitoring mechanism for LCG/WG to ensure that key issues are
being addressed (high number of issues do not appear to be followed up or
no evidence of referral). Important issues raised in LCG and WG should lead
to action. The ERD owned LCG Secretariat needs to be properly staffed and
equipped for this function.
8. UNDP and ERD need to develop jointly a comprehensive communication
strategy for AE/DE that can be used for addressing various stakeholders
(media, CS, Parliament) based on a shared understanding of the AE/DE
agenda, including at field level (sub-district).
9. High visibility needs to be given to achievements (when AIMS will be used
and the first analytical report prepared, when the NAP will be officially
adopted, etc.) to contribute to leadership and ownership.
RECOMMENDATIONS (cont.)

26. Questions or comments welcome
Thank you very much!
Christian Bugnion de Moreta, Team Leader
Nizam Al-Husainy, National Consultant
www.suburconsulting.es