Graham Brown (Australian Research Centre in Sex, Health and Society) discusses the importance of maintaining a strong evidence base for health promotion.

1. ESAPP Review
A draft report of the
Evidence Synthesis and Application for Policy and Practice project
May 2013
Graham Brown, Kylie Johnston and Jeanne Ellard
Australian Research Centre in Sex, Health and Society
www.latrobe.edu.au/arcshsMelbourne, Australia

2. HIV response in Australia
• Australian HIV response has long recognised that interventions working across
multiple social, political, economic, behavioural and health service
conditions, operating within supportive environments, are more likely to affect
behaviour than those interventions working at one level
• Resurging and emerging epidemics
• Role of antiretroviral (ARV) treatments in preventing HIV transmission
• Continued barriers at a structural level reducing or undermining impact
• Recognising the need to better integrate biomedical, individual, community and
structural approaches for HIV prevention - coined “combination prevention”

3. Evidence gaps
• Shared evidence base is not consistent across strategies or has not been
maintained as the contexts have continued to change.
• Understanding of what works - but less so why, or in what combination.
• This undermines the strength of programs and organisations, and the capacity to
adapt to changing environments with confidence.
• Highlighted within Implementing the UN Declaration Report and Melbourne Declaration

4. identify the areas of HIV prevention where the published evidence of
effectiveness and quality practice is most, modest, or least developed; Section 2.0 of full report
identify the monitoring and evaluation methods used in day to day practice
in community organisations to contribute to that evidence (Australia and
similar epidemics); Section 3.0 of full report
review of capacity-building approaches in Australia and internationally to
increase the quality of evidence being developed in community-based HIV
health promotion; Section 4.0 of full report
develop a draft Monitoring, Evaluation and Learning framework for
community HIV prevention to support building evidence for policy and
practice
Section 5.0 of full report
Develop an draft example application of the Monitoring, Evaluation and
Learning Framework to community based HIV prevention and health
promotion
Section 6.0 of full report

5. Evidence most, modest and least developed
• Review of systematic reviews, economic reviews, narrative reviews and
commentaries on the evidence to guide the prevention of sexual
transmission of HIV in concentrated epidemics (2005+).
• Published evidence from research and practice (reduced to ~130 articles)
• Additional focus - three priority groups identified due to the likely impact of
testing and treating approaches as well as experiencing resurging or emerging
epidemics.
– PLHIV -Gay Men -Priority culturally and linguistically diverse (CALD) communities

6. Generally Implementation evaluation and quality practice indicators with
specific target groups
Example program Evidence on what does
or does not work
Evidence on how it
works (including how to
adapt to context)
Gay men PLHIV Priority CALD
communities in
western countries
Health promotion
Systems
how the interventions
interact and impact
together
Least developed Least developed Least developed Least developed Least developed
Structural Reduction of HIV
stigma, policy reform
Least developed Least developed Least developed Least developed Least developed
Community Mass media, social
media, community
mobilisation
Moderately
developed
Least developed –
varies across modes and
target groups
Moderately
developed
Least developed Least developed
Small Group Structured peer based
workshops
Most developed moderate– varies across
modes and target groups
Moderately
developed
Moderately
developed
Least developed
Individual Peer and professional
counselling
Most developed Most developed Most developed Most developed Moderately
developed
Biomedical
Prevention
Increased testing,
Treatment as
Prevention
Most developed Moderately
developed
Least developed Least developed Least developed
Summary of where published evidence about HIV prevention and health promotion is most, moderately or least developed

7. Recommendations for Improving Evidence Base
Two key interrelated factors:
• Research: Intervention research trials that use a broad range of rigorous designs applied
appropriately to interventions at different levels of health promotion, and investigate what works as
well as why it works and in what context.
• Practice: Stronger implementation research within CBOs with a focus on program theory, quality
practice indicators, and development of sustainable evaluation and quality improvement
approaches that recognise the need to continuously adapt and reorient programs.
Without these reorientations in both research and practice, evidence will =
• Limited to the impact of parallel but unconnected strategies
• Provide little insight to what are the most effective leverage points, and what to change as the situation
evolves.

8. Where MEL&QI is most, modest and least developed
• Rapid review of current practices used in Australia (building on previous work undertaken
by AFAO in 2008)
• Rapid review of evaluation practice in international contexts similar to Australia
(primarily Europe and North America).
• Reviewed the published work, abstracts of key conferences attended by HIV educators in
Australia and internationally and supplement this with other targeted online searches
with organisations. (~reduced to ~100 documents)
• While not a complete audit of all work undertaken - reasonable overview of most key
developments in monitoring and evaluation since 2008 with least intrusion on the
community sector organisations

9. Intervention level Strategies (examples) Process and quality
practice indicators
Impact Indicators Combination
prevention or system
wide synergy indicators
Structural Policy and law reform,
advisory structures,
Moderately developed Least developed Least developed
Community Community engagement
and mobilisation
Moderately developed Least developed Least developed
Online Social Media Least developed Least developed Least developed
Mass media Moderately developed Moderately developed Least developed
Small Group Structured peer based
workshops
Moderately developed Moderately developed Least developed
Individual Peer Counselling Models Most developed Moderately developed Moderately developed
Professional Counselling
models
Most developed Most developed Moderately developed
Summary of where MEL approaches are most, moderately or least developed

10. CBO Capacity Building Initiatives
• In broad terms, most programs aimed to move organisations or sectors through
stages of evaluation capacity
– compliance (fulfilling funding source requirements),
– investment (beyond compliance, evaluation is used to improve programs and is
supported by leadership), and
– advancement (beyond investment, evaluations are increasingly ambitious and contribute
to prevention theory and practice). (Gilliam et al., 2003)
• Full Report gives examples of Australian and International Initiatives
– (incl Acon PEKM)

11. CBO Capacity Building Initiatives
These and other similar initiatives have generally included among their aims
to increase:
• capacity to determine why an intervention works, not just if it works,
• capacity for continuous quality improvement approaches, and
• understanding of, and methods to, identify impact within a combination
prevention or health promotion system,
• documenting and sharing of the knowledge and learning.

12. Monitoring, Evaluation and Learning (MEL) and Quality
Improvement (QI) framework
The framework endeavours to acknowledge:
• the complexity of the evolving health, social and political systems in which
HIV prevention operates;
• the strengths of the partnership response and combination approaches;
and
• the rigour of program logic, program theory, quality improvement and
systems thinking.

13. Priority Community Y
IndividualGroupCommunityStructural
Individual and
clinical focus
services
Targeted Community
development and social
influence
Peer group development
and network focused
projects
Community Targeted
Social marketing
Community venues and
settings based engagement
Organisational and
systemic change
Structural , policy and
social change
Mass Media Social marketing
Population
Health
Outcome
Reducedtransmission
andimpactofHIV
Integrated Combination of Health Promotion Actions and Outputs

14. Priority Community Y
IndividualGroupCommunityStructural
Individual and
clinical focus
services
Targeted Community
development and social
influence
Peer group development
and network focused
projects
Community Targeted
Social marketing
Community venues and
settings based engagement
Organisational and
systemic change
Structural , policy and
social change
Mass Media Social marketing
Population
Health
Outcome
Reducedtransmission
andimpactofHIV
Integrated Combination of Health Promotion Actions and Outputs
Across Priority Communities

15. Integrated Combination of Health Promotion Actions and Outputs
Priority Community Y
Individual and
clinical focus
services
Targeted Community
development and social
influence
Peer group development
and network focused
projects
Community Targeted
Social marketing
Community venues and
settings based engagement
Organisational and
systemic change
Structural , policy and
social change
Mass Media Social marketingIndividualGroupCommunityStructural
Population
Health
Outcome
Reducedtransmission
andimpactofHIV

16. Priority Community Y
Individual and
clinical focus
services
Targeted Community
development and social
influence
Peer group development
and network focused
projects
Community Targeted
Social marketing
Community venues and
settings based engagement
Organisational and
systemic change
Structural , policy and
social change
Mass Media Social marketing
Sector Wide
National Strategy
outcomes
Longer term /
Combined Program
Level Impact
Short Term
/Individual Project
Level Impact
Population
Health
Outcome
IndividualGroupCommunityStructural
Improved relevant
knowledge, attitud
es, skills, and self
efficacy
Enhanced quality
practice indicators
Indicators of
strengthened
community
capacity and
responses
Project level
quality and
impact
indicators
Increased access to
health
services, testing and
treatment
Impact on peer
norms and
experience
Participation of
affected
communities
Increased health
promoting social
norms within priority
communities
Increase in levels of
protective sexual risk
behaviour and
testing in at risk
groups
Strengthened
integration across
health promotion
strategies
Program level
quality and
impact indicators
Increased indicators of
sustained community
responses among
priority populations
Increased sustained
testing and treatment
uptake
Reducedtransmission
andimpactofHIV
Reduced risk
behaviours
Decrease
undiagnosed HIV
Prevention
system level
outcomes (linked
to National HIV
Strategy and Targets)
Improve QoL of
PLWHIV
Increase PLWHIV on
Treatment with UVL
Strengthened systems
in research, evaluation
and workforce
Reduced incidence of
HIV
Integrated Combination of Health Promotion Actions and Outputs
Across Priority Communities

17. Priority Community X Priority Community Y Priority Community Z
IndividualGroupCommunityStructural
Sector Wide
National Strategy
outcomes
Longer term /
Combined Program
Level Impact
Short Term
/Individual Project
Level Impact
Population
Health
Outcome
Organisational and
systemic change
Targeted Community
development and social
influence
Individual
and clinical
focus services
Peer group development
and network focused
projects
Community Targeted
Social marketing
Individual and
clinical focus
services
Structural , policy and
social change
Mass Media Social marketing
Community venues and
settings based engagement
Improved relevant
knowledge,
attitudes, skills,
and self efficacy
Enhanced quality
practice indicators
Indicators of
strengthened
community
capacity and
responses
Project level
quality and
impact
indicators
Increased access to
health
services, testing and
treatment
Impact on peer
norms and
experience
Participation of
affected
communities
Increased health
promoting social
norms within priority
communities
Increase in levels of
protective sexual risk
behaviour and
testing in at risk
groups
Strengthened
integration across
health promotion
strategies
Program level
quality and
impact indicators
Increased indicators of
sustained community
responses among
priority populations
Increased sustained
testing and treatment
uptake
Reducedtransmission
andimpactofHIV
Reduced risk
behaviours
Decrease
undiagnosed HIV
Prevention
system level
outcomes (linked
to National HIV
Strategy and Targets)
Improve QoL of
PLWHIV
Increase PLWHIV on
Treatment with UVL
Strengthened systems
in research, evaluation
and workforce
Reduced incidence of
HIV
Integrated Combination of Health Promotion Actions and Outputs
Across Priority Communities

18. Priority Community X Priority Community Y Priority Community Z
IndividualGroupCommunityStructural
Sector Wide
National Strategy
outcomes
Longer term /
Combined Program
Level Impact
Short Term
/Individual Project
Level Impact
Population
Health
Outcome
..
Organisational and
systemic change
Targeted Community
development and social
influence
Individual
and clinical
focus services
Peer group development
and network focused
projects
Community Targeted
Social marketing
Individual and
clinical focus
services
Individual
and clinical
focus services
Structural , policy and
social change
Mass Media Social marketing
Community venues and
settings based engagement
Improved relevant
knowledge,
attitudes, skills,
and self efficacy
Enhanced quality
practice indicators
Indicators of
strengthened
community
capacity and
responses
Project level
quality and
impact
indicators
Increased access to
health
services, testing and
treatment
Impact on peer
norms and
experience
Participation of
affected
communities
Increased health
promoting social
norms within priority
communities
Increase in levels of
protective sexual risk
behaviour and
testing in at risk
groups
Strengthened
integration across
health promotion
strategies
Program level
quality and
impact indicators
Increased indicators of
sustained community
responses among
priority populations
Increased sustained
testing and treatment
uptake
Reducedtransmission
andimpactofHIV
Reduced risk
behaviours
Decrease
undiagnosed HIV
Prevention
system level
outcomes (linked
to National HIV
Strategy and Targets)
Improve QoL of
PLWHIV
Increase PLWHIV on
Treatment with UVL
Strengthened systems
in research, evaluation
and workforce
Reduced incidence of
HIV
Integrated Combination of Health Promotion Actions and Outputs
Across Priority Communities

19. Priority Community X Priority Community Y Priority Community Z
IndividualGroupCommunityStructural
Sector Wide
National Strategy
outcomes
Longer term /
Combined Program
Level Impact
Short Term
/Individual Project
Level Impact
Population
Health
Outcome
.
.
.
Organisational and
systemic change
Targeted Community
development and social
influence
Individual
and clinical
focus services
Peer group development
and network focused
projects
Community Targeted
Social marketing
Individual and
clinical focus
services
Individual
and clinical
focus services
Structural , policy and
social change
Mass Media Social marketing
Community venues and
settings based engagement
Improved relevant
knowledge, attitud
es, skills, and self
efficacy
Enhanced quality
practice indicators
Indicators of
strengthened
community
capacity and
responses
Project level
quality and
impact
indicators
Increased access to
health services,
testing and
treatment
Impact on peer
norms and
experience
Participation of
affected
communities
Increased health
promoting social
norms within priority
communities
Increase in levels of
protective sexual risk
behaviour and
testing in at risk
groups
Strengthened
integration across
health promotion
strategies
Program level
quality and
impact indicators
Increased indicators of
sustained community
responses among
priority populations
Increased sustained
testing and treatment
uptake
Reducedtransmission
andimpactofHIV
Reduced risk
behaviours
Decrease
undiagnosed HIV
Prevention
system level
outcomes (linked
to National HIV
Strategy and Targets)
Improve QoL of
PLWHIV
Increase PLWHIV on
Treatment with UVL
Strengthened systems
in research, evaluation
and workforce
Reduced incidence of
HIV
Integrated Combination of Health Promotion Actions and Outputs
Across Priority Communities
Inputs/ Resources
Community
Organisations
and advocacy
Clinical and
primary care
Services
(medical and
Counselling)
Advisory
structures, P
olicy, and
resource
allocation*
External
Influences
Social
Determinants
Social Drivers
Community
capacity ,
strength and
participation
Biomedical
testing,
treatment and
prevention
developments
Population
impacts of
testing and
treatments
Social capital
Stigma and
discrimination
Partnership*,
Governance
and
Leadership*
Guiding
Principles
and ethics
Human rights
Research
organisations

20. Inputs/ Resources
Community
Organisations
and advocacy
Clinical and
primary care
Services
(medical and
Counselling)
Advisory
structures,
Policy, and
resource
allocation*
External
Influences
Social
Determinants
Social Drivers
Community
capacity ,
strength and
participation
Biomedical
testing, treatm
ent and
prevention
developments
Population
impacts of
testing and
treatments
Social capital
Stigma and
discrimination
Individual and
interpersonal
theories
Structural and
System theory
Social /
Behavioural
theories
Social and
Epidemiological
Research
Project, Program and
system level evidence
and evaluation
Continuous Quality Improvement, refinement of practice
guidelines and standards, and development of workforce*
Partnership*,
Governance
and
Leadership*
Information Systems* (Monitoring,
Evaluation and Learning)
Priority Community X Priority Community Y Priority Community Z
IndividualGroupCommunityStructural
*Prevention System
Strengthening building
blocks identified by WHO
Sector Wide
National Strategy
outcomes
Longer term /
Combined Program
Level Impact
Short Term
/Individual Project
Level Impact
Population
Health
Outcome
.
.
.
Organisational and
systemic change
Targeted Community
development and social
influence
Individual
and clinical
focus services
Peer group development
and network focused
projects
Community Targeted
Social marketing
Individual and
clinical focus
services
Individual
and clinical
focus services
Structural , policy and
social change
Mass Media Social marketing
Community venues and
settings based engagement
Improved relevant
knowledge, attitud
es, skills, and self
efficacy
Enhanced quality
practice indicators
Indicators of
strengthened
community
capacity and
responses
Project level
quality and
impact
indicators
Increased access to
health
services, testing and
treatment
Impact on peer
norms and
experience
Participation of
affected
communities
Increased health
promoting social
norms within priority
communities
Increase in levels of
protective sexual risk
behaviour and
testing in at risk
groups
Strengthened
integration across
health promotion
strategies
Program level
quality and
impact indicators
Increased indicators of
sustained community
responses among
priority populations
Increased sustained
testing and treatment
uptake
Reducedtransmission
andimpactofHIV
Reduced risk
behaviours
Decrease
undiagnosed HIV
Prevention
system level
outcomes (linked
to National HIV
Strategy and Targets)
Improve QoL of
PLWHIV
Increase PLWHIV on
Treatment with UVL
Strengthened systems
in research, evaluation
and workforce
Reduced incidence of
HIV
Integrated Combination of Health Promotion Actions and Outputs
Across Priority Communities
International
practice and
developments
Guiding
Principles
and ethics
Human rights
Research
organisations

21. Example
inputs and
resources
Example
Project
Example indicators for Project level MEL&QI
(preferably drawn from project’s own project logic)
Example Program Level MEL&QI
(such as range of peer based projects)
Example quality
practice
indicators
Example
Indicators of
Inter-project
quality links
Example
Project
Output
Indicators
Example Project level impact
indicators.
(immediate to 3 month)
Example Program
level quality
indicators
Example Inter-
program quality links
Program level
impact indicators
(3 to 12 month)
Community
organisation
resources
Principles of
peer based
programs
Peer based
staff and
volunteers
Evaluation
from
previous
programs
Small Group
Level Project:
eg- Peer
Group
workshop for
gay men
Quality practice
involvement of target
group in development and
improvement.
Satisfaction measures
Group interaction and
dynamics indicators
Evidence of reciprocal
learning between
participants
Proportion of participants
who complete workshops
Peer referrals /
recommendations
Referrals from
outreach, online
initiatives,
counselling
Discussion or
use of social
marketing
campaign within
workshop
Community
volunteer
engagement
indicators
Number of
workshops
conducted
Average number
of participants
completing
workshops
Alignment of
intended target
group and
activity
participants
A workshop would be focused on only
three or four of a set of project level impact
indicators – depending on the focus of the
workshop. The following is an example of a
set of indicators from which a workshop
may draw:
1. Increase in sexuality related health literacy and support
seeking knowledge.
2. Increase in knowledge and confidence to interact in
diverse and sexualised environments (eg online, SOPV,
etc).
3. Increase in skills and confidence to negotiate sexual
interactions including safe sex practices
4. Increase in confidence to manage HIV disclosure in
sexual and social settings
5. Increase in knowledge and confidence regarding sexual
technique and repertoire
6. Increase in confidence to develop relationships
(intimate and friendship).
7. Increase in indicators of participants influencing their
peers regarding peer program messages
Indicators of participants
influencing their peers in
relation to program
aims
Increased indicators of
sustained community
responses among
priority populations
Indicators of community
level engagement with
strategies
Volunteer recruitment
from peer programs
Strategic links between
peer group project and
community
development projects
Strengthened
integration and
strategic links across
peer based programs
and other promotion
strategies
Referrals to and from
venue outreach, online
initiatives, or
counselling
Increased health
promoting social norms
within priority
communities
Indicators of testing and
treatment uptake
Increase in levels of
protective sexual risk
behaviour and testing
among program
participants
Application of MEL&QI framework to a hypothetical peer group workshop for gay men

22. Project level quality,
monitoring and
evaluation
Program Level quality,
monitoring and
evaluation
Prevention system level
quality, monitoring,
surveillance and evaluation
Project /Service staff Yes Possibly No
Agency/Program Yes Yes Possibly
External evaluators Possibly Yes Possibly
Health Services Data Possibly Yes Yes
Epidemiology and Social
Research Centres /
Department
No Possibly Yes
Guidelines for responsibility for collecting and summarising MEL data

23. Final Comments
This is a draft and at a conceptual level
– Draft summary and full report available for comment
– Presented as a discussion monograph in July
Possibly more than ever our community sector needs to
• look at frameworks and approaches to building and expanding the evidence
base, particularly where it is less developed
• Recognise that the projects and programs will continuously evolve and change
• Understanding the what, why, and in what combination or system of approaches
• Turning this into a useable shared evidence base