Baseline & impact assessments & lessons learnt: UTZ Certified Ghana and Ivory Coast. Ingram wur lei impact assessment utz certified cocoa & lessons 24-06-2013. ICCO conference certification Cameroon

1. Baseline & impact assessments & lessons learnt:
UTZ Certified Ghana and Ivory Coast
Studies commissioned by Solidaridad, UTZ Certified & IDH
ICCO International Workshop On Cocoa Certification
Yaoundé, Cameroon 24 June 2013
Verina Ingram, Yuca Waarts, Lan Ge & Giel Ton

2. Baseline & impact study - Research questions
1. How do UTZ and Solidaridad influence cocoa
farmers and producer groups in terms of
knowledge and practices? what are the results of
those changes on the intended ‘people, planet,
and profit’ outcomes for cocoa farmers in Ghana
and Ivory Coast?
2. Who does the programme reach? To what
extent are these farmers representative of cocoa
farmers nationally?
3. What is the added value of going through the
certification process/being certified for farmers?
How do training and certification influence each
other?
IA 6 UTZ Solidaridad cocoa projects in Ghana

3. Impact logic
 UTZ & Solidaridad activities: training of LBCs, NGOs and
producer organisation on UTZ certification and
management leads to increased capacities on these
subjects
Lead farmers train other farmers
Improved knowledge and implementation
 Increased productivity and quality
 Sustainability outcomes
Training of executives and internal controllers + ICS
 Stronger producer groups
 Better and reliable services to members
 Sustainability outcomes
IA 6 UTZ Solidaridad cocoa projects in Ghana

4. Methodology: Baseline & impact study Ghana
 6 projects in 3 regions in Ghana (Ashanti, Eastern, Western)
 Interviews 385 farmers on production, economic, social &
environmental aspects
 3 phases of participation in projects:
1. farmers just started
2. farmers participating between 6 to 12 months
3. farmers participating >12 months
25% of all farmers interviewed certified in phases 2&3
 Comparison of farmers in 6 projects and 3 ‘control’ groups
same region (non-certified)
 Impact comparing length of participation & livelihood
indicators
 Analysis if other factors influence results
IA 6 UTZ Solidaridad cocoa projects in Ghana

5. Results – a glimpse
Certification + supporting initiatives appear to contribute to
farmers economic, environmental and social benefits:
 The longer farmers participate in the project, the higher
their knowledge levels and the better they implement
good cocoa production practices
 This could be a result of programme activities (but may
also result from bias in sample)
 Other factors also have a positive influence: education
level, number & type of training participated in,
gender and productivity
 region where a project is situated also influences outcomes
- strong regional differences in costs of inputs and labor
IA 6 UTZ Solidaridad cocoa projects in Ghana

6. Results: Representativeness project participants
 Most participants: Male, household head, between 40-60
 Education: between primary and secondary school
 5 family members, 50% = 1st generation migrant
 60% is owner, 40% sharecropper
 Most farmers have <7 acre, production: 2 bags/acre
 Average net income 2011: 2,174 Cedi (= 3.78 USD/day)
 Most rely only on cocoa for earning cash income
 Project participants representative of Ghanaian cocoa
sector apart from membership producer organisation
(very high for project participants)
 No labourers/workers included
 25% farmers had unused farm -
potential for production increases?
IA 6 UTZ Solidaridad cocoa projects in Ghana

7. Results: Added value of certification (process)
 Added value of projects (certification process)
•
•
•
•
Farmers satisfied with training
Better social contacts with other farmers
Knowledge exchange between group members
Communal problems discussed during group meetings
 Room for improvement: service delivery by producer
group/LBC and Internal Control System (ICS) staff
• Information/services about cocoa production
• Feedback from ICS and audits
  indicates focus service delivery to producer group
because high attribution and potentially greater impact
 Farmer participation in interventions too recent to show
how training and certification influence each other.
IA 6 UTZ Solidaridad cocoa projects in Ghana

8. Results: Baseline situation of project farmers
• Knowledge levels: average 38% (between 32% to 42%)
• Implementation of GAPs: average 64% (between 59% to
71%)
• Women have significantly lower scores than men
Suggests that:
• Farmers implement practices way better than their
knowledge level suggests
• Training and capacity building might me more effective
is similar groups of farmers trained together
IA 6 UTZ Solidaridad cocoa projects in Ghana

9. Results: Differences certified & control
Knowledge
Implementation
Farm size
Labour, inputs & planting costs
Productivity
Incomes
Net cocoa income
Gross household income
Cocoa quality
3/6 groups higher
2/6 groups higher
no
no
1/6 groups higher
no
no
2/6 groups higher
no
IA 6 UTZ Solidaridad cocoa projects in Ghana

10. Results: Baseline situation of project farmers
•
Productivity: 2 bags per acre on average, range 0.0212.33 bags per acre
•
Net cocoa income 2011: average 2,174 Cedi annually
(1,087 US$) - outliers influence the average
•
Range net income 2011: 50 – 15,600 Cedi (25 – 7,800
US$)
 wide ranges in production
•
Suggests segmenting farmer groups on basis of
production might be efficient capacity building strategy
IA 6 UTZ Solidaridad cocoa projects in Ghana

11. Results: Attribution challenges
 Myriad and cascading interventions with producer
organisations, e.g. STCP, certification/verification schemes,
trader & government projects etc.
 Attribution of quantitative changes in ultimate outcomes (e.g.
income) of certification projects to one type of support
intervention (e.g. training) is not possible - result of a mix of
factors (e.g. training+market+finance+prices)
 Attribution of quantitative changes in immediate & (some)
intermediate outcomes possible: knowledge levels, adoption
of practices, organisational strengthening, procured volumes
 Detailed information on all interventions needed to allow
changes to be attributed: but such data dispersed, nonuniform and often confidential
IA 6 UTZ Solidaridad cocoa projects in Ghana

12. Results: Conclusion
 Ghana baseline and impact study
indicates certification + supporting
initiatives contribute towards
knowledge and implementation of
GAP
 Relationship between farmers’
participation in projects & cocoa
productivity, farm efficiency and
cocoa income needs investigation in
future M&E
 Future assessment will allow
changes to be attributed to
projects, by comparing the
evolution over time of both project
groups and control groups
 Suggest need to tailor interventions
more to address regional
differences in input & labour costs

13. Methodology: Baseline & impact study Ivory
Coast
 Similar research questions
 Wider sample: cooperatives associated with all 9 certified
traders & traitants, in all cocoa growing areas, all phases of
I
participation in certification
 Sampled taking into account location in agro-ecological zones
to account for production and income differences
 Comparison with a control group non-certified farmers
 Quantitative & qualitative methods + field size
measurements, interviews 822 farmers & 84 cooperative
managers, school teachers, focus groups, local authorities
 More detail gathered on traders projects and context, and
other baseline studies(comparisons/benchmarking)
 Enables analysis of how other & external factors influence
results
IA UTZ, IDH, Solidaridad cocoa UTZ certified farmers in Ivory Coast

14. Logic and pathways to impact.....
High influence
& attribution
Low influence
& contribution
Lessons learnt

15. Ways to increase validity ...............
What we learnt:
Indicators
•
•
Less is better
indicators within sphere of influence (e.g. producer group functioning /services)
Baselines, control groups, counterfactuals and contexts
•
•
Invaluable! If no baseline, realise limits to robustness of ex-ante baselines
Can use control groups in absence of baseline, but maybe difficult to compare in
subsequent assessment
Monitor immediate outcomes
•
Learn for increased performance with immediate outcome indicators that build on
and adjust the intervention logic
Measure intermediate outcomes
•
Evaluate changes in intermediate outcome indicators that are informative for
benchmarking the performance of the intervention
Build a plausible storyline explaining contributions to ultimate
outcomes
•
•
Make use of existing quantitative information to reflect on the impact logic and add
to specifically generated data on indicators
Collect qualitative information that supports and challenges the storyline
Lessons learnt

16. Ways to increase validity ...............
Causality and attribution
 Attribution reduces as approach ultimate outcomes and
impacts: recognize that certification is (just) one of the
factors in wider constellation of factors
 Recognize where interdependencies with other
factors/actors become dominant e.g. market prices,
inflation rates, government policies
 Monitor where there these interdependencies are
managed e.g. coupling data
 Identify where it becomes impossible to claim attribution
e.g. yields
 Provide qualitative stories that explain attribution e.g.
histories of change
Lessons learnt

17. Ways to increase validity ............
Baseline and counterfactual thinking
•
Plausible alternative explanations – external causal factors –accounted for e.g.
history, other interventions, seasonal changes
•
collect comparative information ‘control groups’
•
Random and purposive selection “best and worst cases’’
•
triangulate with similar studies to identify plausible alternative explanations for
observed outcomes
•
Address heterogeneity in outcome patterns
•
purposefully sampled groups & areas, which affect the intervention logic e.g.
big cooperatives, type of traders, soil & climate zones, multiple certification
•
Identify patterns of interventions
●
e.g. different mixes of service provision, inputs, trainers, training modalities
Lessons learnt

18. Cost-effective data collection?
• Data gathered by many individual organizations about
same producer organisations
• Many types of data already existent with different supply
chain actors: e.g. traders, input suppliers, support &
certifcaiton organisations
• How to cost-effectively obtain and share data to conduct
impact assessments?
• Impact is one thing....
• Cost benefit analyses another - insightful tool for decision
making - for a given budget assess the results of mixes of
interventions at farmer household level.
Lessons learnt

19. Our concluding thoughts...
1. Certification + supporting initiatives appear to contribute to
economic, environmental and social benefits - but
attribution difficult and needs a baseline
2. Thus contribution of certification to ultimate outcomes more
pragmatic than net-impact – but net impact of immediate
and intermediate outcomes possible
3. Certification provides a means - not end in itself
4. Time frames needed to bring about and measure social and
economic change more evident after at least 2-3 years
5. Attributing impacts to any one intervention and
organization extremely difficult
6. Indicators need to be pragmatic, scaled and SMART
Lessons learnt

20. Thanks!
Verina Ingram
verina.ingram@wur.nl
Yuca Waarts
yuca.waarts@wur.nl
Lan Ge
Lan.ge@wur.nl
Giel Ton
giel.ton@wur.nl