Bridgette Masters - NZPsS recipient of the Karahipi Tumuaki President's scholarship

1. Understanding the simplicity &
complexity of evaluating kaupapa
māori programmes
ā i
Bridgette Masters-Awatere
Waikato University
President s
President’s Scholarship Recipient 2008
4 September 2009

2. The SIMPLICITY (description)
• Evaluation has been:
Evolving with humankind
Delivered in a multidisciplinary context
Strongly influenced by the political climate
Engaged culturally from a “one size fits all” approach, which;
• Has a homogenising affect
• Maintains systems & tension
• Ignores challenges in response to power and control issues
• Is responsive to constantly changing political agendas (of the
dominant group)
© THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 4 September 2009 2

3. Research Approach
• Epistemological position of social constructionism
• No claims of objectivity
“If a person is blind to the values and nature of their own beliefs,
they are hardly likely to be in a position to examine the same for
another ethnic group.”
• Applied research
• Anthropology – understanding how Maori maintain their sense of
control and ownership of their programmes when being evaluated
from a paradigm that is different to their own
• Psychology – exploration of how people have experienced and
reconciled such differences among themselves
• Sociology – engaging ways for Maori social organisations,
structures and processes to be maintained and recognised
• Critical approach
© THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 4 September 2009 3

4. Indigenous Theory
• C lt
Culture specific
ifi
“culture-specific approach, epistemology and methodology emerge from
the worldview of the culture in focus rather than from the worldview of the
researcher or the academy.” Shreiber, 2000, p.655
• Non-neutral perspective
“ …because I do not take who I am for granted, I posses a heightened
sense of reflexivity that can make me a better researcher than White
colleagues inexperienced with assessing their social and/or professional
position I surroundings that place them in the role of ‘minority’ and who
often take for granted their privileged status when in the ‘majority’.”
Hendrix, 2002, p.168
• Epistemological orientation
“ We begin to write ourselves as indigenous peoples as if we really were
‘out there’, the ‘other’, with all the baggage that this entails… [Academic
writing] privileges set of texts, views about the history of an idea, what
issues count as significant; and by engaging in the same process
uncritically,
uncritically we too can render indigenous writers invisible or unimportant
while reinforcing the validity of other writers.” Smith, 1999, p.36
© THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 4 September 2009 4

5. What was my research about?
“Can evaluation incorporate cultural values when being
applied to a Kaupapa Maori programme?”
programme?
From participants’ perspectives :
1. How is success determined/defined?
2. What do Maori communities understand evaluation to be?
3. How do evaluators & providers perceive cultural factors to
relate to their work?
l t t th i k?
4. What factors support and/or impede the inclusion of cultural
concepts in an evaluation of a Kaupapa Maori p g
p p p programme?
5. Does what Maori say (providers, evaluators and communities)
align with commentary from international evaluation practice?
© THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 4 September 2009 5

6. The Historical Context - Evaluation
• Linked to human evolution
A simple explanation...
• Formally a young practice
“Evaluation is the systematic
• Multidisciplinary nature assessment of the worth or merit of
some object.”
object.
• Politically driven agendas
A complex explanation...
In the US evaluation became
federally legislated and has been “Evaluation is both simpler and more
complex than their individual
influenced over the years by social
conclusions; simpler at the meta-level,
and political climates of the day more complex i th d t il Th h d
l in the detail... The hard
work in evaluation theory involves
NZ context is similarly aligned to unpacking the way in which
political climate A market-driven
climate. evaluaiton
e al aiton is a per asi e m lti
pervasive multi-
economy initially saw a decline function, multi-role, multi-player
here. But with an increase in By- enterprise: context-dependent here,
Maori-For-Maori funding,
Maori For Maori funding context independent there biased
there,
evaluation grew. here, objective there.”
© THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 4 September 2009 6

7. The Historical Context - Māori
• Indigenous people around the world have to advocate
for the legitimacy & validity of their knowledge because
of the privileging of Western Science.
• The ability of indigenous knowledge is recognised for its
importance to community survival as they are ultimately
the people who depend on such knowledge for their
survival & the ones capable of sustainably conserving it.
• The link between the physical & social envrionment is
evident in Maori literature and provides the basis for
Kaupapa Maori research methodology;
p p gy
• Evolved in response to colonisation processes
• Theory and practice considered equally valid
y y
• Derived from a Matauranga Maori ontology
© THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 4 September 2009 7

8. What the people are saying... Objective 1
• In a political environment where efficiency equals
success,
success which translates as more output for less
resource, the ability to attain predetermined contracted
outputs influenced decisions regarding success factors:
p g g
… the way we define success is different compared to our funders… they’re
still motivated by the numbers coming in, which means we need to be
motivated by the numbers coming in too… It’s the same for our team, we
It s
spread our networks when we have a hui and we might only have 1 person or
5 people, but if we can change the lives and help those people it’s been a
success for us. (kaimahi tumuaki, kaupapa māori provider, R5)
The reporting template is based solely on numbers and I think it’s once or
twice a year that [we get to] submit a narrative report. Some of our contracts
are different, but the onus is on us to back up why the stats are low and all
the th
th other work th t’ gone i t a programme, and th outcomes we’ve h d
k that’s into d the t ’ had
that can’t be measured by a number… (kaimahi, iwi, provider, R2)
© THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 4 September 2009 8

9. What the people are saying... Objective 2
• Understanding of the role evaluation has:
… Having an external person come and check that the outcomes are being
delivered to our service users… [They have a] more objective opinion, and
because they don’t have personal contact… they can come in say what’s
missing in a programme even if that person who’s running it is a nice
person… (kaimahi, māori provider, R5)
...having evaluators come in and they had Maori researchers on their team
who came to evaluate our programme was great for me. Because they
brought their p
g plan and explained how they were g g to evaluate the whole
p y going
programme. After critically examining all the components, they found that the
most successful and least resourced services of the programme were [the
Maori services]… Other people who had been trying to get rid of my role were
stopped in their tracks with that evaluation I saw [the evaluation] as
evaluation.
confirmation and recognition of my work… [The evaluation] also provided
some useful strategies for moving [the service] forward even more. (former
kaimahi, mainstream provider, R2)
, p , )
© THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 4 September 2009 9

10. What the people are saying... Objective 3
• Relation of cultural factors to their work/evaluation:
… we should… embrace their [whanau/community] suspicion, embrace their
fear – even as one of their own. To avoid and ignore them means you think
you know better…(whānau, R6H7)
... [Maori] evaluators have to be fluid in their relationships with either Maori or
non-Maori…
non Maori because [they] have to negotiate their way amongst a range of
non-Maori colleagues who could be informed, or misinformed about Treaty
relationship issues. (evaluator, R6H5)
The level of effort needed to bring [non-Maori] evaluators firstly to the table
and then to engage takes a huge amount of energy. That often has to come
from [Māori] at the cost of energy going towards furthering our own
f [Mā i] t th t f i t d f th i
agendas… (evaluator, H4)
© THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 4 September 2009 10

11. What the people are saying... Objective 4
• Factors that support/impede cultural inclusion in
evaluation:
• Support factors are:
• Partnership relationships
• Consultation and active involvement
• Recognition of our tino rangatiratanga
• Impeding factors are:
• Desire of government/funders to “follow suit” internationally
• Credibility is attributed off shore
• Current political climate / election cycles
© THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 4 September 2009 11

12. What the people are saying... Objective 5
• Congruence with international considerations about
cultural considerations in evaluation:
• Issues of cultural consideration were severely limited,
and i most cases non-existent. While my online
d in t i t t Whil li
examination of various evaluation bodies returned limited
consideration, there did appear to be a g
, pp growingg
consideration of cultural matters amongst societies. The
exceptions were with American and Australia.
• Across all of the evaluation bodies indigenous matters
did not exist. An exception could be made with regards to
one Regional Chapter (Saskatchewan) of the Canadian
Evaluation Society (CES) who had created space for 2
native members or their representatives on its executive.
© THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 4 September 2009 12

13. The COMPLEXITY (enactment)
• How do you engage a multi-level, political process that
satisfies differing needs?
g
• Recognising diversity
• Whose values count?
• Power and control contention
• International context
• Indigenous voice not considered/engaged
• American driven knowledge construction
• Australian considered second leaders (asking NZ/Māori for
direction)
• Māori context
• Ethical approach fundamental
• Autonomy / self determination paramount
© THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 4 September 2009 13

14. WHERE TO FROM HERE?
• Emphasis on & influence from Science with its
preoccupation with methodolatry, validity and
methodolatry validity,
maintaining social compliance (Chamberlain, 2000)
means that the focus is on change at the lower levels of
g
importance; eg. changing processes rather than the
allocation of resources, ownership, power and control
(Tino Rangatiratanga)
Rangatiratanga)…
…paradoxically, while critical researchers locate the powerful in their analyses
of problems, they exclude them from their solutions. The exclusion or
bypassing of the powerful is counterproductive, given critical theorists’ own
claims that they are frequently partially responsible for the problem, through
their direct or indirect control of the economic, political or communicative
practices which sustain it. Unless revolutionary change is advocated or
it
contemplated, social change requires the involvement of the powerful in the
process of education and action designed to serve the critically examined
interests of all (Robinson, 1993, p.236)
© THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 4 September 2009 14

15. REFERENCES
• Guba, E., & Lincoln, Y. (1985). Effective evaluation: Improving the usefulness of evaluation results through
responsive and naturalistic approaches (4th edition ed.). San Francisco: Jossey-Bass Inc.
• Hendrix, K. G. (2002). "Did being black introduce bias into your study?": Attempting to mute the race-related
research of black scholars. The Howard Journal of Communications, 13, 153-171.
, ,
• Henry, E., & Pene, H. (2001). Kaupapa Māori: Locating indigenous ontology, epistemology and methodology in the
academy. Organization: Speaking Out, 8(2), 234-242 .
• Indīgena, F. S., & Kothari, B. (1997). Rights to the Benefits of Research: Compensating Indigenous Peoples for their
Intellectual Contribution. Human Organization, 56(2), 127-137.
g , ( ),
• Keefe, V., Cram, F., Orsmby, C., & Ormsby, W. (1998). Representing the ‘other’ when they’re our relations. Paper
presented at the Cultural Justice and Ethics Symposium: New Zealand Psychological Society Annual Conference,
Wellington.
• Mathison, S. (1993). Rethinking the evaluator role: Partnerships between organizations and evaluators. Paper
presented at the American Evaluation Association Annual Meeting: The promise and practice of evaluation in
organizations that learn, Albany, USA.
• Moewaka Barnes, H. (2003). Maori and evaluation: some issues to consider. In N. Lunt, C. Davidson & K. McKegg
(Eds.), Evaluating Policy and Practice: A New Zealand Reader (pp. 146-151). Auckland: Pearson Education New
Zealand Limited.
Limited
• Neuman, W. (2000). The meanings of methodology. In Social Research Methods (4th ed., pp. 63-88). Boston: Allyn
and Bacon.
• Patton, M. (1997). Utilization-Focused Evaluation: The new century text (3rd ed.). Thousand Oaks, California: Sage.
• Pih
Pihama, L., Cram, F & Walker, S. (2002). C ti methodological space: A lit t
L C F., W lk S (2002) Creating th d l i l literature review of K
i f Kaupapa M i
Maori
research. Canadian Journal of Native Education, 26(1), 30.
• Schreiber, L. (2000). Overcoming methodological elitism: Afrocentrism as a prototypical paradigm for intercultural
research. International Journal of Intercultural Relations, 24, 651-671.
• Smith L T (1999) Decoloni ing methodologies Research and indigeno s peoples London and D nedin Zed
Smith, L. T. (1999). Decolonizing methodologies: indigenous peoples. Dunedin:
Books and University of Otago Press.
© THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 4 September 2009 15

16. The University of Waikato
Private Bag 3105
Hamilton, New Zealand
0800 WAIKATO
www.waikato.ac.nz
© THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 4 September 2009 16