Kate McKegg and Nan Wehipeihana (2010). A practitioners introduction to Developmental Evaluation

Developmental evaluation,[object Object],- A practitioner’s introduction ,[object Object],Kate McKegg and Nan Wehipeihana,[object Object]

Our whakapapa ,[object Object],[object Object]

What is my history?,[object Object]

Workshop Outline,[object Object]

Developmental Evaluation - what is it?,[object Object],Developmental evaluation supports innovation development to guide adaption to emergent and dynamic realities in complex environments. ,[object Object],(Michael Quinn Patton, 2010),[object Object]

Five purposes of developmental evaluation,[object Object]

Five purposes of developmental evaluation,[object Object],Adapted from Patton (2010),[object Object]

Development and innovation in complex situations,[object Object],Intractable issues e.g., poverty, homelessness, environmental degradation, family violence, obesity,[object Object],Uncertainty about options and solutions,[object Object],Political and economic turbulence ,[object Object],Environment characterised by unpredictability and the unexpected emergence of issues and ideas (e.g., rise of social networking),[object Object],Rapid adaption and change occurring across many contexts,[object Object]

DE is informed by complexity and systems thinking (picture sourced from Patton, 2009),[object Object]

Some key complexity concepts,[object Object],Adaption,[object Object],Emergence,[object Object],Non-linearity,[object Object],Uncertainty,[object Object],Interdependence,[object Object]

Exercise One: Handout 1: match each scenario with a complexity concept,[object Object]

Some complexity concepts: Handout 2: practical application,[object Object]

DE - a distinct niche,[object Object],Two distinct niches,[object Object],Pre-formative,[object Object],Dynamic situations,[object Object],Support exploration and innovation before there is a program model (pre-formative),[object Object],Support the ongoing development and adaption of a programme, or innovation in emergent and complex situations (dynamic situations),[object Object]

DE - a distinct niche,[object Object],Different to formative and summative,[object Object],Differs from improvement orientated evaluation (making a programme better),[object Object],It aims to support the ongoing real-time decisions – what to change, expand, close out further develop,[object Object],Important Distinction:,[object Object],Differentiating ongoing strategic thinking and periodic strategic planning ,[object Object],DE as a form of thinking and acting strategically as an innovative intervention unfolds,[object Object]

Increase participation in sport and recreation – ‘by Māori’,[object Object],2007-08 review and programme ‘re-visioning’ ,[object Object],Focus on participation ‘as Māori’ participation,[object Object],Tino rangatiratanga (self determination),[object Object],20 years of Māori development,[object Object],The story of He Oranga Poutama,[object Object]

Case example – He Oranga Poutama and complexity,[object Object],Uncertainty around the new programme concept and direction, not sure what ‘as Māori’ participation would involve,[object Object],New innovative programme and goal,[object Object],Values, principles and vision driven,[object Object],No formal evaluation evidence from other efforts about what to expect, no real data available to measure key concepts and variables,[object Object],Complex environment – contracting economy, political environment, changing organisation,[object Object]

DE – and creative thinking,[object Object],Requires critical AND creative thinking,[object Object],We have to think outside the box,[object Object],Ask questions about what makes sense as we try to connect evaluation to the program development,[object Object]

Creative thinking,[object Object],Draw four straight lines which go through the middle of all of the dots without taking the pencil off the paper,[object Object]

Creative thinking:,[object Object],Thinking outside the boxhttp://www.brainstorming.co.uk/puzzles/ninedotsnj.html,[object Object],Step 2,[object Object],Step 3,[object Object],Start here 1,[object Object],Step 4,[object Object]

When not to use DE,[object Object],Not able or willing to commit the time to actively participate in the evaluation and to build and sustain relational trust – no room for the set and forget evaluation management approach,[object Object],High levels of certainty required – no tolerance for ambiguity,[object Object],Lack of openness to experimentation and reflection – high level of design certainty required,[object Object],No adaptive capacity – no real ability to make changes,[object Object],Unwillingness to ‘fail’ or hear ‘bad news’ – fear of failure context overrides learning focus,[object Object],Poor relationships between management and staff and evaluators – respectful and trusting relationships critical in a context of uncertainty, ambiguity and turbulence,[object Object]

Traditional evaluation is not well suited to all contexts,[object Object]

Exercise 2:Teasing out complexity,[object Object],Handout 3:,[object Object],Read the brief description of Whānau Ora,[object Object],What if anything is complex about Whānau Ora?,[object Object],Share and discuss in small groups,[object Object],Make notes to feedback to the larger group,[object Object]

Roles and relationships of the DE evaluator,[object Object],Evaluator is part of the innovation team,[object Object],Facilitator and learning coach,[object Object],Brings evaluative thinking to the group,[object Object],Support or share innovators’ values and vision,[object Object],Support the feedback process around “what is being developed”,[object Object]

DE - the context and the evaluator,[object Object]

Skills, abilities and dispositions,[object Object],The ability to build and maintain trusting relationship,[object Object],Critical in a context of uncertainty, ambiguity and turbulence,[object Object],High level of facilitation skills, multiple settings, stakeholders and contexts,[object Object],To generate and ‘test’ ideas, to provide feedback, to ask the ‘tough’ questions; highly engaging, to build evaluative capability,[object Object],Deep methodological toolkit,[object Object],Methodological flexibility, eclecticism, and adaptability (Patton, p27, 2010),[object Object],Enquiring,[object Object],Observant and critically reflective,[object Object],Systematic ,[object Object],Attention to data and making data driven decisions,[object Object],Courageous,[object Object],Flexible, adaptable, responsive ‘on the run, on the fly’,[object Object]

What will happen to the piece of wood when the person lets go of it?,[object Object]

Kate McKegg and Nan Wehipeihana (2010). A practitioners introduction to Developmental Evaluation

DE in practice - Inquiry approaches,[object Object],Focus of DE is on what is being developed?,[object Object],What’s emerging?,[object Object],Given what’s emerged, what’s next?,[object Object],The DE evaluator:,[object Object],Inquires into developments,[object Object],Tracks developments,[object Object],Facilitates interpretation of developments so that judgments can be made about the what, how, impact and consequence of developments,[object Object]

DE: Asking questions that matter and matching questions to context,[object Object],Patton (2010) suggests that questions that matter can be thought of as “a tool for working in complex situations” (p227),[object Object],Matching questions to particular situations is a central challenge - situational responsiveness and adaption,[object Object]

6 simple rules (Patton, 2010),[object Object],Connect questions with the ideas, language and frameworks of the people you are working with,[object Object],Less is more. Limit the number of questions within the enquiry framework. 7+ or - 2 is a good rule of thumb,[object Object],Keep the evaluation grounded in whatever basic developmental inquiry framework you and those you’re working with choose to guide your work,[object Object],Distinguish overarching inquiry questions from detailed implementation questions ,[object Object],There are (no) stupid questions,[object Object],Remember that whatever inquiry framework you are using, the focus is on What is being developed?,[object Object]

DE – is values based‘[DE] sits alongside, doesn’t control or dampen the core values of innovation’ (Wehipeihana, cited in Patton, 2010). ,[object Object],Values / processes / relationships become very important,[object Object],‘How’ outcomes / results are achieved is very important,[object Object],Process matters,[object Object],People and relationships matter,[object Object],Where the end is unpredictable and emergent, values and process become anchors,[object Object],‘there will never be enough certain knowledge to guide action’ Wendell Berry cited in Patton, 2010. ,[object Object]

DE – inquiry approaches,[object Object],There is no definitive list of developmental evaluation inquiry approaches - neither should one be constructed,[object Object],Developmental evaluation creatively adapts whatever approaches and methods fit the complexities of the situation. ,[object Object],It is responsive, appropriate, and credible to those you work with, and support opening up new understandings and guiding further development,[object Object],It’s all about persistently asking questions and pursuing credible answers in time to be used. Questioning is the ultimate method. Questions are the quintessential tools. (Patton, 2010),[object Object]

DE – uses a range of interpretive frameworks,[object Object],But applies evaluation logic and thinking in complex situations,[object Object],Enables people to engage in data based, ongoing evaluative sense-making and reasoned argument,[object Object],ensuring the criteria (values) people use to decide what’s ‘good’ and desirable are open to scrutiny and made explicit,[object Object],ensuring the decisions made about development / the ‘forks in the road’ are documented and systematically reflected upon in a critical and open minded way,[object Object]

Our experience: introducing DE to the uninitiated,[object Object],Small steps, getting people grounded, on the same page really matters,[object Object],The most simple of frameworks:,[object Object],What, why, when, how, where and who,[object Object],What is happening and what has changed? ,[object Object],Why?,[object Object],Who has been affected, Who is involved?,[object Object],How did this happen?,[object Object],Where is the situation / programme at now?,[object Object],When do key people need to know what?,[object Object],How do people feel about the change? Why?,[object Object],etc,[object Object]

Or some slightly more evaluative questions:,[object Object],What’s being developed? (WHAT?),[object Object],What sense can we make of emerging issues, evidence, data about this development? (WHAT’S EMERGING?),[object Object],What is the value / importance /significance of what we are doing / achieving? What does it mean to us now and in the future? (SO WHAT?),[object Object],What does this mean for how we should now act? What are our options? And into the future? (NOW WHAT),[object Object]

Applying DE questions,[object Object]

Some useful interpretive frameworks:,[object Object],See Handout 5,[object Object],Appreciative Inquiry,[object Object],Success Case Method,[object Object],Most Significant Change,[object Object],Systems approaches – with an emphasis on perspectives, boundaries and interrelationships,[object Object],Outcome mapping,[object Object],Action research,[object Object]

Example – ‘as Māori’ a developmental journey,[object Object],See Handout 6,[object Object],The process,[object Object],As Māori not prescribed in the provider selection process,[object Object],Series of facilitated discussions ,[object Object],1 and 2 day provider hui (n=4),[object Object],individual providers,[object Object],Ongoing iteration,[object Object],Cycle of feedback loops,[object Object],What’s emerging,[object Object],Shared understanding,[object Object],Co-constructed,[object Object],[object Object],[object Object]

Co-construction of ‘as Māori’,[object Object],Te reo,[object Object],Kanohi ki te kanohi,[object Object],Guided by kawa and tikanga,[object Object],Whānau, hapu, iwi and Māori ,[object Object],Uses Māori institutions e.g. marae, kohanga,[object Object],Whanaungatanga,[object Object],Traditional Māori sports,[object Object],Spirit of us,[object Object],Creating a feeling of belonging and empowerment,[object Object]

What’s the value of ‘as Māori’,[object Object],It’s our own self determination of how we do it, in ways that we want to do it. ,[object Object],To do the things we want to do, in ways that we want to do them,[object Object],Opportunity to portray our uniqueness,[object Object],Effective and better engagement achieved when cultural values and aspects core and/or included,[object Object],Traditional sports are just as valuable - in terms of engaging our communities,[object Object]

Example – ‘as Māori’ a developmental journey,[object Object],An ‘as Māori’ continuum with five dimensions emerging: ,[object Object]

Some final thoughts, tips and questions,[object Object],FAQs we have encountered:,[object Object],How systematic is DE?,[object Object],How does DE differ from Action Research?,[object Object],Isn’t DE just a bunch of stories – when do you get to outcomes? And measurement? And judgment?,[object Object],What questions do you have about DE?,[object Object]

What we’ve learned a long the way,[object Object],It’s a journey with others, this is not for the sole trader, nor for the feint hearted!,[object Object],As an evaluation team, we model the reflection process together,[object Object], This is hard thinking stuff, it doesn’t come easy,[object Object],Great rewards possible,[object Object],Thank goodness we chose and are able to use DE, it’s the right way to go, but it is hard work,[object Object]

Wrap up,[object Object],One thing you learned today,[object Object],One thing you will do as a result of today,[object Object],One thing you will follow up on,[object Object]

References,[object Object],Gamble J., A Developmental Evaluation Primer, J.W. McConnell Family Foundation, 2008.,[object Object],Patton, M. Q. Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use. Guilford Press, June 2010.,[object Object],Patton, M.Q. Utilization-Focused Evaluation, 4th ed., Sage, 2008. ,[object Object],Wehipeihana, N and McKegg K, Developmental Evaluation in an Indigenous Context, Reflections on the journey to date. Presentation to American Evaluation Conference, Orlando, Florida, 2009,[object Object],Westley, Frances; Zimmerman, Brenda; Patton, Michael Q. Getting to Maybe: How the World Is Changed? Random House Canada, 2006,[object Object]

Kate McKegg and Nan Wehipeihana (2010). A practitioners introduction to Developmental Evaluation

Kate McKegg and Nan Wehipeihana (2010). A practitioners introduction to Developmental Evaluation

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente(20)

Destacado

Destacado(7)

Similar a Kate McKegg and Nan Wehipeihana (2010). A practitioners introduction to Developmental Evaluation

Similar a Kate McKegg and Nan Wehipeihana (2010). A practitioners introduction to Developmental Evaluation(20)

Último

Último(20)

Kate McKegg and Nan Wehipeihana (2010). A practitioners introduction to Developmental Evaluation

Notas del editor