This presentation describes Patton's developmental evaluation paradigm, and how the Military Families Learning Network utilizes it for network-wide program evaluation.
1. Developmental Evaluation on the
Ground
Brigitte Scott, Ph.D.
Evaluation and Research Specialist
Military Families Learning Network
2. Military Families Learning Network
Personal Finance
Child Care
Family Advocacy
Network Literacy
Personal Finance
Family Development
Network Literacy
Transition Support
Community Capacity
Building
Nutrition & Wellness
Early Intervention
Special Needs
Life Span Special
Needs
Military Caregiving
4. MQP: DE Defined
“DE supports innovation development to guide
adaptation to emergent and dynamic realities in
complex environments.” —MQP, p. 1
Goal: support project, program, product, and/or
organizational development with timely
feedback
5. MQP: Core DE Question
What is getting developed and what are the
implications of what gets developed?
Complexity theory of change:
Bring people together who are knowledgeable and
committed and they will self-organize, take
action, and work together to create movement,
innovation, and change.
8. Focus on developmentMQP
• NOT the same as continuous improvement
(formative eval or accountability)
– Quality improvement is helping programs meet
standards that have been set.
Development is when people are changing what
they are doing, and the very nature of the
standards are also changing. Standards are adapting
to changing conditions.
10. Complex and dynamic environmentMQP
Complex environment = lack of
central control
What to do to solve problems is
uncertain and there can be conflict
about how to proceed
14. Developmental EvaluatorMQP
• Works collaboratively with innovators to
conceptualize, design, and test new approaches
in a long-term, on-going process of adaptation,
intentional change, and development.
• Ask eval questions
• Apply eval logic
• Gather and report eval data
15. Developmental EvaluatorMQP
Primary Functions:
Elucidate the innovation and adaptation
processes
Track implications and results
Facilitate data-based decision-making
16. “There are no best practices. ‘Best’ is
entirely context-free.”
—MQP, AEA 2014
17. Evaluation for MFLN
• Ongoing development
• Innovation in learning
– Internal practice
– External product
21. Cooperative Agreement
“An opportunity to provide responsive and timely
educational programming through being actively nimble,
flexible, innovative, and creative in true partnership with our
funders as they identify organizational priorities and ask us
to engage in the construction of appropriate and necessary
deliverables that meet the on-time needs of the target
audience. The MFLN and funding partners are seen as true
and honest equals in expertise, providing valued and
accepted feedback bi-directionally. This often results in
surpassing expectations for success in delivery.”
—Kyle Kostelecky, National Program Director, MFLN
26. DE for MFLN: What It Looks Like
Continuous adaptation
27. DE for MFLN: What It Looks Like
Continuous adaptation
Social, political, economic, technological, and demographic
patterns
28. DE for MFLN: What It Looks Like
Continuous adaptation
Social, political, economic, technological, and demographic
patterns
Emergent developments/needs of our target audiences
29. DE for MFLN: What It Looks Like
Continuous adaptation
Social, political, economic, technological, and demographic
patterns
Emergent developments/needs of our target audiences
Cooperative agreement environment
30. DE for MFLN: What It Looks Like
Continuous adaptation
Social, political, economic, technological, and demographic
patterns
Emergent developments/needs of our target audiences
Cooperative agreement environment
eXtension
31. DE for MFLN: What It Looks Like
Continuous adaptation
Social, political, economic, technological, and demographic
patterns
Emergent developments/needs of our target audiences
Cooperative agreement environment
eXtension
DoD
32. DE for MFLN: What It Looks Like
(not this. . . .)
33. DE for MFLN: What It Looks Like
(but this. . . .)
35. DE for MFLN: What It Looks Like
Inductive
Chaotic and messy
36. DE for MFLN: What It Looks Like
Inductive
Chaotic and messy
Embrace forks in the road
37. DE for MFLN: What It Looks Like
Inductive
Chaotic and messy
Embrace forks in the road
Developmental moments
Track what’s going on
Understand what’s going on
Adapt to what’s going on
38. DE for MFLN: What It Looks Like
Everyone is an
evaluator
39. Complex development situations are ones in
Inspired by Jeff Conklin,
cognexus.org
which this…
Time
39
Michael Quinn Patton
AEA eStudy webinar 2014
44. DE Successes for MFLN
• Very responsive to our partners’ programmatic
requests and innovations
45. DE Successes for MFLN
• Very responsive to our partners’ programmatic
requests and innovations
• Doubling in size in 2015 with competitive award
– New CAs and new opportunities for innovation in
programming
46. DE Successes for MFLN
• Very responsive to our partners’ programmatic
requests and innovations
• Doubling in size in 2015 with competitive award
– New CAs and new opportunities for innovation in
programming
• Programming
47. DE Successes for MFLN
• Very responsive to our partners’ programmatic
requests and innovations
• Doubling in size in 2015 with competitive award
– New CAs and new opportunities for innovation in
programming
• Programming
• “Walk the talk”: model use of social media,
collaborative learning, personal learning
networks
49. DE Challenges for MFLN
• Innovation is challenging, slow, and . . . contentious
50. DE Challenges for MFLN
• Innovation is challenging, slow, and . . . contentious
• Moving from program improvement mode to redesign
based on evaluation findings
51. DE Challenges for MFLN
• Innovation is challenging, slow, and . . . contentious
• Moving from program improvement mode to redesign
based on evaluation findings
• IRB challenges
52. DE Challenges for MFLN
• Innovation is challenging, slow, and . . . contentious
• Moving from program improvement mode to redesign
based on evaluation findings
• IRB challenges
• “In your face” evaluation
53. DE Challenges for MFLN
• Innovation is challenging, slow, and . . . contentious
• Moving from program improvement mode to redesign
based on evaluation findings
• IRB challenges
• “In your face” evaluation
• Daily, evaluative thinking
54. DE Challenges for MFLN
• Innovation is challenging, slow, and . . . contentious
• Moving from program improvement mode to redesign
based on evaluation findings
• IRB challenges
• “In your face” evaluation
• Daily, evaluative thinking
• Platform delivery limitations
55. DE Challenges for MFLN
• Innovation is challenging, slow, and . . . contentious
• Moving from program improvement mode to redesign
based on evaluation findings
• IRB challenges
• “In your face” evaluation
• Daily, evaluative thinking
• Platform delivery limitations
• Military culture
56. DE Challenges for MFLN
• Innovation is challenging, slow, and . . . contentious
• Moving from program improvement mode to redesign
based on evaluation findings
• IRB challenges
• “In your face” evaluation
• Daily, evaluative thinking
• Platform delivery limitations
• Military culture
• Doubling in size in January 2015
58. DE Features for MFLN
• Reports
• Leadership
• Reflective discussion groups
• Transparency
• Collaboration
60. Patton, Michael Quinn. (2014).
“Intermediate Developmental Evaluation.”
AEA eStudy Webinar.
Patton, Michael Quinn. (2011). Developmental
Evaluation: Applying Complexity Concepts to
Enhance Innovation and Use. Guilford: New
York.
61. Brigitte Scott
@4ed_eval
brigit2@vt.edu
www.blogs.extension.org/militaryfamilies
Icons made by Freepik:
http://www.flaticon.com. Flaticon is
licensed by Creative Commons 3.0.
Notas del editor
DE is not ongoing formative eval
DE supports and documents development
DE is not ongoing formative eval
DE supports and documents development
Complex environment = lack of central control; what to do to solve problems is uncertain and there can be conflict about how to proceed
Focus on development, which is not the same as continuous improvement (formative eval) or accountability
Improvement vs. development
Quality improvement is helping programs meet standards that have been set.
Developmental evaluation is when people are changing what they are doing and the very nature of the standards are also changing. Standards are adapting to changing conditions
DE: standards themselves and what it means to meet them and how to talk about them and how to apply them is being developed and changed in relationship to a changing world; very different from having a predetermined set of quality criteria that a program has to meet.
Focus on development, which is not the same as continuous improvement (formative eval) or accountability
Improvement vs. development
Quality improvement is helping programs meet standards that have been set.
Developmental evaluation is when people are changing what they are doing and the very nature of the standards are also changing. Standards are adapting to changing conditions
DE: standards themselves and what it means to meet them and how to talk about them and how to apply them is being developed and changed in relationship to a changing world; very different from having a predetermined set of quality criteria that a program has to meet.
DE becomes part of intervention
Developmental evaluation in the context of MFLN is an evaluation paradigm that facilitates program development and innovation through continuous adaptation to the dynamic and changing MFLN cooperative agreement environment.
Our programming doesn’t intend to necessarily become a fixed, standardized model. But it does help us identify effective principles that inform ongoing development
Cooperative agreement:
NIFA, DoD, and UIUC
Different than traditional grant, which is when you articulate your research goals and deliverables before receiving your funding; instead, we receive funding based on a loosely defined set of criteria (plans of work)
An award similar to a grant, but I which the sponsor’s staff may be actively involved in proposal preparation, and anticipates having substantial involvement in research activities once the award has been made.
Can you tell me if the PF VLE was a result of that responsiveness? I'm looking for examples of how our plans of work can change during the year based on DoD conversations and request. I can't remember, but I thought the VLE was an example of that--not planned for 2014 but instead requested after the year started. Yes, this is a great example. DoD thought is was coming at some point - we had talked about it in the past. They felt that it was time, as it met a quickly evolving need from their end. They asked and stressed how important it would be to accomplish this at this time and we went back to our PF team and they were open, excited, and offered a plan for how they could do it within their budget (not adding but rearranging and deleting other deliverables). After PF’s new plan was discussed with DoD, all were happy to move forward. Another example of how this cooperative agreement has worked from our end (we instigated a suggested change) was when I began to hear more and more about how military caregiving was an increasing concern from those Anne and I talked with when we were at conferences (especially military conferences). We were not hearing this from MC&FP and so I did some homework on my end, began to have these conversations with Betsy and Eddy, and suggested that we should consider expanding in this area. They took this idea back to their offices and partners and then came back to us with an approval to move forward. We did this within our 2010 award at the end when we needed to re-capture funds and construct a re-budget. We had about $40K to play with and conducted a 5 month pilot. MC&FP liked it that it was fully funded for 2012 (and of course has been ever since with great success). So it goes both ways, they ask and we deliver - we suggest and they agree. It’s been great. There are other examples if you need more, just let me know.
eXtension: emergent and dynamic organization both tied to and simultaneously oriented away from Cooperative Extension.
Monthly reports
Quarterly reports
Annual reports
Weekly leadership team meetings
Webinar evaluation reports
Quarter webinar reports
Social Media Specialists meetings
Net Lit support meetings
.
We
Real time innovation is challenging
--We’re very good at innovating and request of our partners; not as nimble for us to innovate on our own (still top down in that way);
--PIs bought out for only a small portion of time; innovation and change is slow, may not have everyone’s buy-in
Challenge: We were really in program implementation/improvement mode for the first four years. How do you introduce the possibility that you are no longer in program improvement mode but in fact that program needs to be redesigned? How do you prepare the people you’re working with for the possibility that you are no longer in program improvement mode but rather that you need to begin adapting and changing to findings of formative evaluation?
Difficult to make changes quickly to eval and research
--DoD looking for all kinds of information, but IRB is a challenge (18 month IRB process for working with contractors; hard to be responsive and provide real-time feedback)
Evaluation is constantly “in your face”
--Getting everyone to think evaluatively is challenging
Platform limitations for reaching people on bases;
--Limits innovation
Military culture (private, formal);
Doubling in size in 2015—lots of work to get done and most days are about just getting through the day
Real time innovation is challenging
--We’re very good at innovating and request of our partners; not as nimble for us to innovate on our own (still top down in that way);
--PIs bought out for only a small portion of time; innovation and change is slow, may not have everyone’s buy-in
Challenge: We were really in program implementation/improvement mode for the first four years. How do you introduce the possibility that you are no longer in program improvement mode but in fact that program needs to be redesigned? How do you prepare the people you’re working with for the possibility that you are no longer in program improvement mode but rather that you need to begin adapting and changing to findings of formative evaluation?
Difficult to make changes quickly to eval and research
--DoD looking for all kinds of information, but IRB is a challenge (18 month IRB process for working with contractors; hard to be responsive and provide real-time feedback)
Evaluation is constantly “in your face”
--Getting everyone to think evaluatively is challenging
Platform limitations for reaching people on bases;
--Limits innovation
Military culture (private, formal);
Doubling in size in 2015—lots of work to get done and most days are about just getting through the day
Real time innovation is challenging
--We’re very good at innovating and request of our partners; not as nimble for us to innovate on our own (still top down in that way);
--PIs bought out for only a small portion of time; innovation and change is slow, may not have everyone’s buy-in
Challenge: We were really in program implementation/improvement mode for the first four years. How do you introduce the possibility that you are no longer in program improvement mode but in fact that program needs to be redesigned? How do you prepare the people you’re working with for the possibility that you are no longer in program improvement mode but rather that you need to begin adapting and changing to findings of formative evaluation?
Difficult to make changes quickly to eval and research
--DoD looking for all kinds of information, but IRB is a challenge (18 month IRB process for working with contractors; hard to be responsive and provide real-time feedback)
Evaluation is constantly “in your face”
--Getting everyone to think evaluatively is challenging
Platform limitations for reaching people on bases;
--Limits innovation
Military culture (private, formal);
Doubling in size in 2015—lots of work to get done and most days are about just getting through the day
Real time innovation is challenging
--We’re very good at innovating and request of our partners; not as nimble for us to innovate on our own (still top down in that way);
--PIs bought out for only a small portion of time; innovation and change is slow, may not have everyone’s buy-in
Challenge: We were really in program implementation/improvement mode for the first four years. How do you introduce the possibility that you are no longer in program improvement mode but in fact that program needs to be redesigned? How do you prepare the people you’re working with for the possibility that you are no longer in program improvement mode but rather that you need to begin adapting and changing to findings of formative evaluation?
Difficult to make changes quickly to eval and research
--DoD looking for all kinds of information, but IRB is a challenge (18 month IRB process for working with contractors; hard to be responsive and provide real-time feedback)
Evaluation is constantly “in your face”
--Getting everyone to think evaluatively is challenging
Platform limitations for reaching people on bases;
--Limits innovation
Military culture (private, formal);
Doubling in size in 2015—lots of work to get done and most days are about just getting through the day
Real time innovation is challenging
--We’re very good at innovating and request of our partners; not as nimble for us to innovate on our own (still top down in that way);
--PIs bought out for only a small portion of time; innovation and change is slow, may not have everyone’s buy-in
Challenge: We were really in program implementation/improvement mode for the first four years. How do you introduce the possibility that you are no longer in program improvement mode but in fact that program needs to be redesigned? How do you prepare the people you’re working with for the possibility that you are no longer in program improvement mode but rather that you need to begin adapting and changing to findings of formative evaluation?
Difficult to make changes quickly to eval and research
--DoD looking for all kinds of information, but IRB is a challenge (18 month IRB process for working with contractors; hard to be responsive and provide real-time feedback)
Evaluation is constantly “in your face”
--Getting everyone to think evaluatively is challenging
Platform limitations for reaching people on bases;
--Limits innovation
Military culture (private, formal);
Doubling in size in 2015—lots of work to get done and most days are about just getting through the day
Real time innovation is challenging
--We’re very good at innovating and request of our partners; not as nimble for us to innovate on our own (still top down in that way);
--PIs bought out for only a small portion of time; innovation and change is slow, may not have everyone’s buy-in
Challenge: We were really in program implementation/improvement mode for the first four years. How do you introduce the possibility that you are no longer in program improvement mode but in fact that program needs to be redesigned? How do you prepare the people you’re working with for the possibility that you are no longer in program improvement mode but rather that you need to begin adapting and changing to findings of formative evaluation?
Difficult to make changes quickly to eval and research
--DoD looking for all kinds of information, but IRB is a challenge (18 month IRB process for working with contractors; hard to be responsive and provide real-time feedback)
Evaluation is constantly “in your face”
--Getting everyone to think evaluatively is challenging
Platform limitations for reaching people on bases;
--Limits innovation
Military culture (private, formal);
Doubling in size in 2015—lots of work to get done and most days are about just getting through the day
Real time innovation is challenging
--We’re very good at innovating and request of our partners; not as nimble for us to innovate on our own (still top down in that way);
--PIs bought out for only a small portion of time; innovation and change is slow, may not have everyone’s buy-in
Challenge: We were really in program implementation/improvement mode for the first four years. How do you introduce the possibility that you are no longer in program improvement mode but in fact that program needs to be redesigned? How do you prepare the people you’re working with for the possibility that you are no longer in program improvement mode but rather that you need to begin adapting and changing to findings of formative evaluation?
Difficult to make changes quickly to eval and research
--DoD looking for all kinds of information, but IRB is a challenge (18 month IRB process for working with contractors; hard to be responsive and provide real-time feedback)
Evaluation is constantly “in your face”
--Getting everyone to think evaluatively is challenging
Platform limitations for reaching people on bases;
--Limits innovation
Military culture (private, formal);
Doubling in size in 2015—lots of work to get done and most days are about just getting through the day
Real time innovation is challenging
--We’re very good at innovating and request of our partners; not as nimble for us to innovate on our own (still top down in that way);
--PIs bought out for only a small portion of time; innovation and change is slow, may not have everyone’s buy-in
Challenge: We were really in program implementation/improvement mode for the first four years. How do you introduce the possibility that you are no longer in program improvement mode but in fact that program needs to be redesigned? How do you prepare the people you’re working with for the possibility that you are no longer in program improvement mode but rather that you need to begin adapting and changing to findings of formative evaluation?
Difficult to make changes quickly to eval and research
--DoD looking for all kinds of information, but IRB is a challenge (18 month IRB process for working with contractors; hard to be responsive and provide real-time feedback)
Evaluation is constantly “in your face”
--Getting everyone to think evaluatively is challenging
Platform limitations for reaching people on bases;
--Limits innovation
Military culture (private, formal);
Doubling in size in 2015—lots of work to get done and most days are about just getting through the day