Methodological challenges, strengths, and weaknesses of an extensive longitudinal study into children and youth around the world presented by Marta Favara of Young Lives to the CLOSER Longitudinal Methodology Seminar Series 28 July, 2016

1. An extensive longitudinal study into children & youth
around the world
Methodological challenges, strengths and weaknesses
Marta Favara, Young Lives
CLOSER Longitudinal Methodology Series seminar
July 28th , 2016

2. Outline of this presentation
1. Overview of Young Lives data
₋ Young Lives in pills
₋ Sampling design and survey components
₋ Survey questionnaire
₋ Processes in place for designing and implementing the survey questionnaire
2. Nice features of Young Lives data
3. Challenges and risk-mitigation strategies

3. Young Lives in pills
₋ Multi-disciplinary study that aims to:
- improve understanding of childhood poverty and inequalities
- provide evidence to improve policies & practice
⁻ Young Lives components:
Household survey (child, caregiver, younger siblings, children of the YL children,
community representatives); Longitudinal qualitative research; School survey: parallel
to round 2 and 5 of the household survey.
₋ Following nearly 12,000 children in 4 countries: Ethiopia; India (Andhra Pradesh &
Telangana); Peru and Vietnam
₋ Over a 15-year period: first data collected in 2002, with 5 survey rounds
₋ Two age cohorts in each country:
- 2,000 children born in 2000-01
- 1,000 children born in 1994-95
₋ Collaboration:
- Partners in each study country
- Publicly archived survey data (UK Data Archive and listed on the World Bank Micro Data
website) and core-funded by DFID, DGIS, IrishAid

4. 12,000 children in 4 countries over 15 years

5. – Sentinel site sampling; four stages sampling process (region, district/provinces, sentinel
sites, random sampling of children within sites);
– Purposively over-sampled poor areas (40% urban / 60% rural) using different poverty
indicators in each countries
Ethiopia India Peru Vietnam
Sampling design

6. – Demographic information (hh roster), socio economic indicators (wealth index,
food consumption)
– Health information and anthropometrics (YL child, parents, siblings and child of
YL children)
– Education history (all hh members) and cognitive skills (YL child, siblings)
– Subjective wellbeing and psychosocial competencies (YL child, siblings)
– Employment status/history and time use (all hh members)
– Job related skills
– Job and Educational Aspirations/expectations (YL child, parents)
– Expectations about marriage and parenthood (YL child, parents)
– Fertility history
– Marriage/cohabitation history
– Control over assets (intra-household decision making)
– Social norms indicators
– Knowledge on SRH and access to contraceptives
– Sexual behaviours, risky behaviours and criminal activities (Peru)
Information collected

7. Step 1: Design The Survey Questionnaire
Step 2: Tracking and preparing CAPI programme
Step 3: Training and piloting
Step 4: Fieldwork
Step 5: Data cleaning, validations
Step 6: Preliminary analysis and Research
Six steps from design to the field

8. 1. Comprehensive set of information collected at community and household level
(caregiver, YL child, a subsample of (younger) siblings and the children of the YL
children)
2. Longitudinal data covering a period of 15 years from early childhood to adulthood
3. A life-course approach, very relevant for policy design (early childhood, middle
childhood and adolescence)
4. Cross-cohorts, cross-countries comparisons
Nice features

9. – Allows to identify links between earlier circumstances and later outcomes
– Identifying when differences emerge: how persistent particular circumstances are; what shapes later
well-being
– Testing the ‘dynamics’ of well-being (Controlling for time-constant unobservable characteristics)
– Compare two cohorts at the same age (trends, exposure to different policy context)
– Use (panel) siblings data to investigate how household or community circumstances affect child
outcomes at the same age; explore intra-household dynamics; controlling for the influence of past
events and circumstances
– A new generation (Children of YL’s children)
Multi cohorts longitudinal data: Main benefits

10. – Challenge 1. Cohort maintenance
– Challenge 2. Getting comparable measures over time
– Challenge 3. Across countries coordination/comparability
– Challenge 4. Ensure high quality data
– Challenge 5. Data collection methods: switch to CAPI
Multi cohorts longitudinal data: Main Challenges

11. Challenges :
– Some attrition is inevitable
– Cohort is relatively small for a longitudinal study
– Study period is relatively long (three years gap between waves)
Risk mitigating strategies:
– Collecting detailed contact information
– Importance of tracking
₋ Reduces time looking for children when we start the fieldwork
₋ Maintains continuity of social contact and trust between researchers and
families
– Reduce refusal rates as much as possible:
₋ Importance of explaining what we’re doing
₋ Importance of maintaining field teams
₋ Give photos back to families (part of ethics/reciprocity)
₋ Ensure no respondents are over-loaded (by different elements/sub-studies)
₋ Compensations (Losing a day of work has big impact on income)
Challenges: 1. Cohort maintenance & attrition

12. …and we have been quite successful!
YC OC Overall
Ethiopia 2.2% 8.4% 4.3%
India 2.6% 4.3% 3.2%
Peru 6.3% 10.3% 7.3%
Vietnam 2.9% 9.9% 5.3%
Total 3.6% 8.1% 5.0%
ETHIOPIA INDIA
PERU VIETNAM

13. Source: Outes and Dercon, 2008
Non-random attrition
₋ Attriting households (R1-R2) tend to have fewer assets, poorer access to services and
utilities and are less educated (more in Ethiopia and India than Peru and Vietnam)
(Panel A)
₋ These averages hide substantial variation between different types of attriting
households (Panel B)
₋ The presence of non-random attrition does not necessarily imply attrition bias: no
attrition bias found when looking (Ethiopia is an exception)

14. Challenges:
– The questions need to change as the children grow up
– Change in primary respondent/hh head
– Keep as many questions as possible the same across rounds (panel variables)
– Asking the same questions of the YC as we did the OC in earlier rounds (core base
variables)
– Ensure comparability over time (e.g. cognitive tests-- Item Response Theory)
– Keep the order of the survey modules the same over the time
Limitations for comparability:
- Switch from PAPI to CAPI;
- Some changes in the structure of the questionnaire are inevitable
- `Getting stuck’ with the errors of the past to the seek of maintain comparability across
rounds
Challenges: 2. Getting comparable measures over
time

15. Benefits:
– How patterns of relationships are similar/different across countries.
– Understanding why and how specific policies or programmes are effective in one
country.
– Comparative analysis can give greater confidence that evidence found in one country is
applicable to others.
– Learning in relation to methods: trying to develop measures that can be used across
cultures.
Challenges:
– Constructing a questionnaire that suits different national contexts.
– Ethical committee approval and country specific sensitivities.
– Deal with different fieldwork processes.
Risk mitigating strategies :
– Define research priorities and relevant survey questions in each country
– There are also some country variations
– Translation and back translation is key to ensure consistency
– Continuity of country team leaders and fieldworker coordinators.
Challenges: 3. Across countries coordination and comparability

16. Challenges:
– Maintaining increasingly complex survey instruments
– Maintaining strong coordination and liaison between Quant/Qual/ School survey teams
– Participant recall
– Panel conditioning
Risk mitigating strategies:
– Piloting and training are crucial!
₋ Ensure research questions work in the field and are consistent with local situations and children’s ages
₋ Ensure questionnaire are not too long / burdensome
₋ Train teams and learn from practical experience of field work to improve instrument design
₋ Produce accurate instrument manuals and protocols
₋ Uncover ethics issues and give safe space for discussion
₋ Initiate, build and maintain positive team dynamics
₋ Ensure that good data collection systems are in place
– Consistency checks are embedded in CAPI, some information are prefilled, ultimately
some inconsistencies can be solved ex-post
Challenges: 4. Quality of the data

17. • CAPI introduced in R4 – is a different way of doing surveys (e.g. changes dynamic of
interview)
Benefits:
– Eliminate data entry error.
– Know how work is progressing
– Avoid mistakes before they happen
– Ask the right questions (embedded skip pattern)
– Quality improvement (?)
– Reduction in the length of the interview (?)
Challenges:
– Requires more time at the front end (building the programme)
– Fieldworkers to get familiar with a new instruments
– Put in place a data management and transfer systems
– Devolve responsibilities to the in-country data managers (in Peru and Vietnam)
Risk mitigating strategies:
– Extra effort at the front end in programming
– Piloting and testing the application is crucial!
– Training country data managers and fieldworkers on data management and transfer
systems.
Challenges: 5. Introducing CAPI

18. FINDING OUT MORE : www.younglives.org.uk
THANK YOU!

19. Annex

20. Ethiopia
Sampling design (1)
Four stages sampling process:
1. Regions (Amhara, Oromia, SNNPR, Tigray
and Addis Ababa, accounting for 96% of
national population)
2. Woredas (districts) (3-5 districts in each
regions, 20 in total)
3. Kebele (at least 1 for each woredas)
4. 100 young children (born in 2001-02)
and 50 older children (born in 1994-5)
were selected within those sites.
Criteria to select districts:
1. Districts with food deficit profile
2. Districts which capture diversity across
regions and ethnicities in both urban and
rural areas
3. Manageable costs in term of tracking for
the future rounds
Comparing with DHS and WMS 2000: 2000:
Poor hh are over-sampled, but YL covers the
diversity of children in the country including
up to 75% percentile of the Ethiopian
population.

21. India
Sampling design (2)
Four stages sampling process:
1. Regions (Coastal Andhra, Rayalaseema,
and Telangana)
2. Districts
3. 20 sentinel sites (mandal)
4. 100 young children (born in 2001-02)
and 50 older children (born in 1994-5)
were randomly selected within those
sites.
Criteria followed:
1. Uniform distribution across regions
2. One poor and one non-poor district in
each region (based on economic,
human development and infrastructure
indicators)
Comparison to the DHS 1998/9:
YLs hh seem to be slightly wealthier than
the average household in Andhra Pradesh.
Despite these biases YL sample covers the
diversity of children in poor households in
Andhra Pradesh

22. Peru
Sampling design (3)
Sampling process:
1. Sample frame at district level excluding
the top 5% richest district based on
poverty map 2001
2. Districts divided in population groups
ordered by poverty index and randomly
selected to cover rural, urban, peri-
urban coastal, mountain and amazon
areas (random selection proportional to
district population)
3. Within the selected districts a village
was randomly chosen
4. Within each village the street blocks
were counted and randomly numbered
to select the starting point.
Comparison to the DHS 2000:
YL cover the diversity of children and hh in
Peru

23. Vietnam
Sampling design (4)
Four stages sampling process:
1. Regions (5/8 regions, North-East region, Red River
Delta, City, South Central Coast, Mekong Delta.
2. Provinces (5 in total ,1 per region, Lao Cai, Hung
Yen, Da Nang Phu Yen, Ben Tre).
3. Sentinel sites (4 commune per province, 2 poor, 1
average and 1 above-average commune )
4. 100 young children (born in 2001-02) and 50 older
children (born in 1994-5) were selected within
those sites.
Criteria followed (to rank communes):
1. Development of infrastructure,
2. Percentage of poor households in the commune
3. Child malnutrition status.
Comparison to the DHS and VHLSS 2002:
The urban sector is under-represented (in terms of
population and the level of development). YL includes
hh with on average less access to basic services and
slightly poorer than the average in Viet Nam. YL
sample covers the diversity of children in the country.

24. Cognitive skills
Cohort Round 1 (2002) Round 2 (2007) Round 3 (2010) Round 4 (2013) Round 5 (2016)
OC 8 years old 12 years old 15 years old 19 years old 22 years old
Raven's test PPVT PPVT -
Math* Math Math Math
Cloze test
Reading
comprehension
YC 1 year old 5 years old 8 years old 12 years old 15 years old
PPVT PPVT PPVT PPVT
CDA quantitative Math Math Math
Reading
comprehension
Note: *One Item; CDA=Cognitive Development Assessment ; PPVT=Peabody Picture Vocabulary Test; Cloze test=Cloze
test on reading comprehension

25. Soft skills
Cohort Round 1 (2002) Round 2 (2007) Round 3 (2010) Round 4 (2013) Round 5 (2016)
OC 8 years old 12 years old 15 years old 19 years old 22 years old
Agency Agency Agency Agency
Pride Pride Pride Pride
Trust Trust
Inclusion Inclusion
Subjective
wellbeing
Subjective
wellbeing
Subjective
wellbeing
Subjective
wellbeing
Self-esteem Self-esteem
Self-efficacy Self-efficacy
Parent relations Parent relations
Peer relations
Grit
Neuroticism,
Conscientiousness
Job skills
YC 1 year old 5 years old 8 years old 12 years old 15 years old
Agency Agency Agency
Pride Pride Pride
Subjective
wellbeing
Subjective
wellbeing
Subjective
wellbeing
Parent relations Parent relations
Peer relations Peer relations

26. Aspirations and expectations
Cohort Round 1 (2002) Round 2 (2007) Round 3 (2010) Round 4 (2013) Round 5 (2016)
OC 8 years old 12 years old 15 years old 19 years old 22 years old
Aspirations
about Marriage
and Fertility
Aspirations about
Marriage and
Fertility
Educational
aspirations/
expectations
Educational
aspirations/
expectations
Job Aspirations/
Expectations
Job Aspirations/
Expectations
Job Aspirations/
Expectations
YC 1 year old 5 years old 8 years old 12 years old 15 years old
Aspirations about
Marriage and
Fertility
Educational
aspirations/
expectations
Educational
aspirations/
expectations
Job Aspirations
Job Aspirations/
Expectations
Job Aspirations/
Expectations
Subjective earnings
expectations