This document discusses evaluating the impacts of small grants provided to organizations. It presents the intervention logic of how the grants are intended to improve organizational capacities and member outcomes. A mixed-methods research design is proposed to assess these impacts, including comparing key indicators before and after the grants for treatment and comparison groups. Challenges in attribution are addressed, such as other factors influencing outcomes. Quantitative and qualitative tools are presented to measure organizational social capital and capacities. Comparative case-based analysis methods like qualitative comparative analysis are discussed to better understand configurations associated with success.

1. CAPTURING CHANGES IN
ORGANIZATIONAL SOCIAL CAPITAL
AN IMPACT EVALUATION OF A SMALL-GRANT INNOVATION FUND
IDS EVENT “Impact, Learning and Innovation: Towards a Research and
Practice Agenda for the Future” – 26-27 March 2013 – IDS Sussex
Giel Ton

2. WHAT DO WE WANT TO KNOW

3. Intervention logic of innovation grant
APPROVED
BUSINESS PLANS
(PROCESSING
ACTITIVIES)
INVESTMENTS IN
BUSINESS
OPPORTUNITY
ORGANISATIONAL
MANAGEMENT OF
BUSINESS
OPPORTUNITY
BUSINESS PLAN IN
OPERATION
FINANCIAL
SUSTAINABILITY OF
THE ORGANISATION
IMPROVEMENT IN
ORGANISATIONAL
CAPACITIES
IMPROVED ACCESS
TO FINANCIAL
SERVICES
SELFMANAGEMENT
OF THE
ORGANISATION
MOREINCOME TO
MEMBERS
BETTER
NUTRITIONAL
STATUS
BETTER SERVICES
TO MEMBERS
IMPROVED ACCESS
TO NATIONAL AND
REGIONAL
MARKETS
TRAINING ON
BUSINESS
OPPORTUNITY
MORE SENSE OF
BELONGINGNESS
WITH THE
ORGANISATION
INITIAL
ORGANIZATIONAL
CAPACITY
GOOD QUALITY
BUSINESS PLANS
EFFICIENT
QUALITY
CONTROL AND
GRANT
ALLOCATION
SYSTEM

4. APPROVED
BUSINESS PLANS
(PROCESSING
ACTITIVIES)
INVESTMENTS IN
BUSINESS
OPPORTUNITY
ORGANISATIONAL
MANAGEMENT OF
BUSINESS
OPPORTUNITY
BUSINESS PLAN IN
OPERATION
FINANCIAL
SUSTAINABILITY OF
THE ORGANISATION
IMPROVEMENT IN
ORGANISATIONAL
CAPACITIES
IMPROVED ACCESS
TO FINANCIAL
SERVICES
SELFMANAGEMENT
OF THE
ORGANISATION
MOREINCOME TO
MEMBERS
BETTER
NUTRITIONAL
STATUS
BETTER SERVICES
TO MEMBERS
IMPROVED ACCESS
TO NATIONAL AND
REGIONAL
MARKETS
TRAINING ON
BUSINESS
OPPORTUNITY
MORE SENSE OF
BELONGINGNESS
WITH THE
ORGANISATION
INITIAL
ORGANIZATIONAL
CAPACITY
GOOD QUALITY
BUSINESS PLANS
EFFICIENT
QUALITY
CONTROL AND
GRANT
ALLOCATION
SYSTEM
GENERAL EVALUATION QUESTIONS
EXTERNAL EVALUATION OF
BUSINESS PLAN IMPACTS
CHECK KEY
ASSUMPTION
CHECK
IMPACT
CHECK
IMPACT?
TYPOLOGIES
INTERNAL MONITORING OF
GRANT IMPLEMENTATION

5. MIXED-METHODS RESEARCH DESIGN

6. Key impact pathway B:
Business plans improve organisational resilience (org. social capital)
Key impact pathway D:
Grant investments release constraints in access to financial services
Key impact pathway C:
Business plans lead to increased economic performance
INTERVENTION
Key impact pathway A:
Business plans improve capacity to pay organisational costs

7. INTERVENTION

8. Key impact pathway B:
Business plans improve organisational resilience (org. social capital)
Key impact pathway D:
Grant investments release constraints in access to financial services
Key impact pathway C:
Business plans lead to increased economic performance
INTERVENTION
Key impact pathway A:
Business plans improve capacity to pay for collective action
ATTRIBUTION PLAUSIBLE =
NET-EFFECT MEASUREMENT FEASIBLE =
QUASI-EXPERIMENTAL DESIGN
APPROPIATE
CONTRIBUTION STORY = NEED FOR
MONITOIRNG INFORMATION

9. Data collection for building the impact
contribution story
 Intervention
Use operational data for
immediate outcomes
 Ultimate outcomes
Reflect on (existing)
secondary data on poverty
 Intermediate outcomes
Appropriate proxies for
key outcome areas
 Configurations
Process tracing
and monitoring of
interdependencies

10. Impacts of
innovation grants
on collective
marketing groups

11. 1. Core design elements
 Time-series on key indicators:
● Sales: total sales & sales of innovative product
● Patrimony
● Self-funding of recurrent cost
● Access to funds/credit
 Treatment and comparison group
● 30 farmer organizations that are funded
● 20 that are not-funded (though eligible)
● Stratified random selection
 Statistical tests and econometrics outcome variables:
● Growth rates
● Regression with year after implementation as explanatory variable

12. 2. Contribution story and threats to validity
 The grant is assumed to be a vital parts of a ‘package’ of causal factors
that together is sufficient to produce the ultimate outcomes.
• E.g. banks show increasing willingness to negotiate with producer
organisations that have some patrimony
• E.g. through the investment the access to government
procurement (nutritional programmes) is improved but still depend
on parti-political dynamics
 However, it is possible that the grants are not necessary: do you still see
traces of processes (partly) triggered by the intervention
• Most likely, the grant is necessary in some configurations with
some type of organizations/markets, but not in all (‘what works for
whom under what conditions?’)
• Tackling the threats to validity to the claim of contribution: counterfactual
thinking
• Can we explain the realization of the outcomes without making
reference to the grant (‘alternative explanations’)?

13. TOOL FOR MEASURING SOCIAL
CAPTAL

14. Qualitative tool: 38 reports with qualitative
descriptions (approx 15 pp/each)

15. Feeding learning processes

16. Escuela de líderes – Leadership course

17. Question 1 Question 2
INHERENT TENSIONS IN COLLECTIVE
MARKETING THAT NEED ORGANISATIONAL
MECHANISMS TO RESOLVE THEM
This tension
is present in
the activities
we realize
Hardly
present
Never
present
We have managed to
resolve it with
organisational
agreements / internal
regulations
We are
looking for
ways to
resolve it
We don’t
need to
resolve it
1- “Regulating member supply”
Members sometimes protest that the
organisation does not buy all their produce?
2- “Quality assurance systems”
Are there members that try to deliver lower
quality products than is required?
3- “Reduce the need for working
capital”
Members demand cash payment instead of
waiting until the organisations has sold the
product?
4- “Prevention of disloyal behaviour”
Are there members that sell part of their
produce to other buyers though they
promised to sell to the organisation?
5- “Ways to distribute profits”
Do members accept that the organisations
does not distribute all its profits?
6- “Differ benefits and services to
members and non-members”
Is there preferential treatment (e.g. price)
when buying from members compared with
non-members?
7- “Decide on investments and
activities that do not benefit all”
Did the organisation projects or investments
that are only to the benefit of a sub-group of
members of the group?
8- “Delegating and supervising
marketing tasks”
Do members accept that others in the
organisation take decisions on prices of
products sold without prior consult to the
assembly?
9- “Legal responsibility in contracts
and loans”
Do members take responsibility for eventual
fines and sanctions related with sale
contracts or loans that the board negotiates?
10- “Manage political aspirations of
board and staff”
Do members accept that board members or
team staff take party political
responsibilities?
Quantitative
comparative tool
Pregunta 1 Pregunta 2
TENSIONES
INHERENTES A LA
COMERCIALIZACIÓ
N COLECTIVA
Esta
tensión se
presenta
en las
actividades
que
realizamos
como
organizació
n
Se
present
a muy
poco
no se
present
a nunca
hemos logrado
resolverlo con
acuerdos/arregl
os organizativos
estamos
buscand
o la
forma de
resolverl
o
no
necesitamo
s resolverlo
“Regular la
cantidad a acopiar
de los miembros”
“Sistemas de
garantía de
calidad”
“Reducir la
necesidad de
capital de trabajo”
“Prevenir
deslealtad en las
ventas”
“Maneras de
distribuir los
excedentes”
“Diferenciar los
beneficios y
servicios a
miembros y no
miembros”
“Decidir sobre

18. Table 3. Weighing factors used for calculating each tension-containment variable (Tx)
ScoreQ1x
This tension is present
in their activities
Hardly present Never
present
They have managed to resolve 9 6 3
ScoreQ2x They are looking for ways to
resolve it
6 4 2
They don’t need to resolve it 3 2 0
Tx=ScoreQ1 x *ScoreQ2x

19. CASE-BASED COMPARATIVE
ANALYSIS

20. Conventional statistical tricks
Table.. Rotated component matrix on organizational performance indicators
Factor 1:
Economic performance
Factor 2:
Organizational size
Factor 3:
Capital intensity
Patrimony per member (Ln)
-.025 -.006 .979
Active members (Ln)
.189 .874 -.294
Organizational Age (Ln)
.059 .897 .224
Sales (Ln)
.886 .438 -.126
Sales per member (Ln)
.991 -.043 .041
Extraction Method: Principal Component Analysis.
Rotation Method: Varimax with Kaiser Normalization.

22. Set-theoretic tricks: fQCA
 The visual inspection of scatterplots helped to detect and
describe the diversity in the sector of economic farmer
organizations.
 To further deepen the analysis, other case-based comparative
techniques are needed to find groupings of organizations with
distinct scores on each of the tensions that compose our
construct, as certain organizations may perform functions
where not all tensions are relevant and for which the absence
is not indicative of weaker capabilities.
 We expect to explain success with a configurational analysis:
like Qualitative Comparative Analysis (Rihoux and Ragin,
2008; Schneider and Wagemann, 2012), when we have the
‘success-variable’ late 2013.

23. CHALLENGES AHEAD

24.  Practical issues:
● Time-series: confidentiality and ‘contentious
property’
● Quasi-experimental design: attrition of the
comparison group
 Fundamental problems
● ‘Fishing’ versus ‘Iterative process to have a closer
look at cases’
● ‘Success’ is multifacetal: what is the dependant
variable in fQCA?

25. Thanks!
Sietze Vellema
Sietze.vellema@wur.nl
Giel Ton
Giel.ton@wur.nl