SlideShare una empresa de Scribd logo
1 de 61
Descargar para leer sin conexión
Developmental Evaluation on the 
Ground 
Brigitte Scott, Ph.D. 
Evaluation and Research Specialist 
Military Families Learning Network
Military Families Learning Network 
Personal Finance 
Child Care 
Family Advocacy 
Network Literacy 
Personal Finance 
Family Development 
Network Literacy 
Transition Support 
Community Capacity 
Building 
Nutrition & Wellness 
Early Intervention 
Special Needs 
Life Span Special 
Needs 
Military Caregiving
Developmental Evaluation on the Ground
MQP: DE Defined 
“DE supports innovation development to guide 
adaptation to emergent and dynamic realities in 
complex environments.” —MQP, p. 1 
Goal: support project, program, product, and/or 
organizational development with timely 
feedback
MQP: Core DE Question 
What is getting developed and what are the 
implications of what gets developed? 
Complexity theory of change: 
Bring people together who are knowledgeable and 
committed and they will self-organize, take 
action, and work together to create movement, 
innovation, and change.
MQP: Key Characteristics of DE
Focus on development
Focus on developmentMQP 
• NOT the same as continuous improvement 
(formative eval or accountability) 
– Quality improvement is helping programs meet 
standards that have been set. 
Development is when people are changing what 
they are doing, and the very nature of the 
standards are also changing. Standards are adapting 
to changing conditions.
Complex and dynamic environment
Complex and dynamic environmentMQP 
Complex environment = lack of 
central control 
What to do to solve problems is 
uncertain and there can be conflict 
about how to proceed
Real time feedback
DE becomes part of intervention
Developmental Evaluator
Developmental EvaluatorMQP 
• Works collaboratively with innovators to 
conceptualize, design, and test new approaches 
in a long-term, on-going process of adaptation, 
intentional change, and development. 
• Ask eval questions 
• Apply eval logic 
• Gather and report eval data
Developmental EvaluatorMQP 
Primary Functions: 
Elucidate the innovation and adaptation 
processes 
Track implications and results 
Facilitate data-based decision-making
“There are no best practices. ‘Best’ is 
entirely context-free.” 
—MQP, AEA 2014
Evaluation for MFLN 
• Ongoing development 
• Innovation in learning 
– Internal practice 
– External product
DE for MFLN: Why It Fits
Program development
Cooperative agreement environment 
+ 
eXtension Initiative
Cooperative Agreement 
“An opportunity to provide responsive and timely 
educational programming through being actively nimble, 
flexible, innovative, and creative in true partnership with our 
funders as they identify organizational priorities and ask us 
to engage in the construction of appropriate and necessary 
deliverables that meet the on-time needs of the target 
audience. The MFLN and funding partners are seen as true 
and honest equals in expertise, providing valued and 
accepted feedback bi-directionally. This often results in 
surpassing expectations for success in delivery.” 
—Kyle Kostelecky, National Program Director, MFLN
Constant evaluative feedback
Constant evaluative feedback and 
discussion 
• Monthly reports (internal, DoD) 
• Quarterly reports (NIFA) 
• Annual reports (internal, DoD, NIFA) 
• Webinar evaluation reports 
• Quarterly webinar reports 
• Weekly leadership team meetings 
• Social Media Specialists meetings 
• Net Lit support meetings
Evaluation used for decision-making
Evaluator is on leadership team
DE for MFLN: What It Looks Like 
Continuous adaptation
DE for MFLN: What It Looks Like 
Continuous adaptation 
Social, political, economic, technological, and demographic 
patterns
DE for MFLN: What It Looks Like 
Continuous adaptation 
Social, political, economic, technological, and demographic 
patterns 
Emergent developments/needs of our target audiences
DE for MFLN: What It Looks Like 
Continuous adaptation 
Social, political, economic, technological, and demographic 
patterns 
Emergent developments/needs of our target audiences 
Cooperative agreement environment
DE for MFLN: What It Looks Like 
Continuous adaptation 
Social, political, economic, technological, and demographic 
patterns 
Emergent developments/needs of our target audiences 
Cooperative agreement environment 
eXtension
DE for MFLN: What It Looks Like 
Continuous adaptation 
Social, political, economic, technological, and demographic 
patterns 
Emergent developments/needs of our target audiences 
Cooperative agreement environment 
eXtension 
DoD
DE for MFLN: What It Looks Like 
(not this. . . .)
DE for MFLN: What It Looks Like 
(but this. . . .)
DE for MFLN: What It Looks Like 
Inductive
DE for MFLN: What It Looks Like 
Inductive 
Chaotic and messy
DE for MFLN: What It Looks Like 
Inductive 
Chaotic and messy 
Embrace forks in the road
DE for MFLN: What It Looks Like 
Inductive 
Chaotic and messy 
Embrace forks in the road 
Developmental moments 
Track what’s going on 
Understand what’s going on 
Adapt to what’s going on
DE for MFLN: What It Looks Like 
Everyone is an 
evaluator
Complex development situations are ones in 
Inspired by Jeff Conklin, 
cognexus.org 
which this… 
Time 
39 
Michael Quinn Patton 
AEA eStudy webinar 2014
Michael Quinn Patton 
AEA eStudy webinar 2014 
40 
And this…
Turns out to be this… 
Time 
41 
Michael Quinn Patton 
AEA eStudy webinar 2014
Michael Quinn Patton 
AEA eStudy webinar 2014 
42 
…looks like this
DE Successes for MFLN
DE Successes for MFLN 
• Very responsive to our partners’ programmatic 
requests and innovations
DE Successes for MFLN 
• Very responsive to our partners’ programmatic 
requests and innovations 
• Doubling in size in 2015 with competitive award 
– New CAs and new opportunities for innovation in 
programming
DE Successes for MFLN 
• Very responsive to our partners’ programmatic 
requests and innovations 
• Doubling in size in 2015 with competitive award 
– New CAs and new opportunities for innovation in 
programming 
• Programming
DE Successes for MFLN 
• Very responsive to our partners’ programmatic 
requests and innovations 
• Doubling in size in 2015 with competitive award 
– New CAs and new opportunities for innovation in 
programming 
• Programming 
• “Walk the talk”: model use of social media, 
collaborative learning, personal learning 
networks
DE Challenges for MFLN
DE Challenges for MFLN 
• Innovation is challenging, slow, and . . . contentious
DE Challenges for MFLN 
• Innovation is challenging, slow, and . . . contentious 
• Moving from program improvement mode to redesign 
based on evaluation findings
DE Challenges for MFLN 
• Innovation is challenging, slow, and . . . contentious 
• Moving from program improvement mode to redesign 
based on evaluation findings 
• IRB challenges
DE Challenges for MFLN 
• Innovation is challenging, slow, and . . . contentious 
• Moving from program improvement mode to redesign 
based on evaluation findings 
• IRB challenges 
• “In your face” evaluation
DE Challenges for MFLN 
• Innovation is challenging, slow, and . . . contentious 
• Moving from program improvement mode to redesign 
based on evaluation findings 
• IRB challenges 
• “In your face” evaluation 
• Daily, evaluative thinking
DE Challenges for MFLN 
• Innovation is challenging, slow, and . . . contentious 
• Moving from program improvement mode to redesign 
based on evaluation findings 
• IRB challenges 
• “In your face” evaluation 
• Daily, evaluative thinking 
• Platform delivery limitations
DE Challenges for MFLN 
• Innovation is challenging, slow, and . . . contentious 
• Moving from program improvement mode to redesign 
based on evaluation findings 
• IRB challenges 
• “In your face” evaluation 
• Daily, evaluative thinking 
• Platform delivery limitations 
• Military culture
DE Challenges for MFLN 
• Innovation is challenging, slow, and . . . contentious 
• Moving from program improvement mode to redesign 
based on evaluation findings 
• IRB challenges 
• “In your face” evaluation 
• Daily, evaluative thinking 
• Platform delivery limitations 
• Military culture 
• Doubling in size in January 2015
Developmental Evaluation on the Ground
DE Features for MFLN 
• Reports 
• Leadership 
• Reflective discussion groups 
• Transparency 
• Collaboration
DE Features for MFLN
Patton, Michael Quinn. (2014). 
“Intermediate Developmental Evaluation.” 
AEA eStudy Webinar. 
Patton, Michael Quinn. (2011). Developmental 
Evaluation: Applying Complexity Concepts to 
Enhance Innovation and Use. Guilford: New 
York.
Brigitte Scott 
@4ed_eval 
brigit2@vt.edu 
www.blogs.extension.org/militaryfamilies 
Icons made by Freepik: 
http://www.flaticon.com. Flaticon is 
licensed by Creative Commons 3.0.

Más contenido relacionado

Destacado

Test of hypothesis (z)
Test of hypothesis (z)Test of hypothesis (z)
Test of hypothesis (z)Marlon Gomez
 
Interval estimation for proportions
Interval estimation for proportionsInterval estimation for proportions
Interval estimation for proportionsAditya Mahagaonkar
 
Demographic data by dr najeeb memon
Demographic data by dr najeeb memonDemographic data by dr najeeb memon
Demographic data by dr najeeb memonmuhammed najeeb
 
Population data sources
Population  data  sourcesPopulation  data  sources
Population data sourcesRezaul Karim
 
Sources of demographic data
Sources of demographic dataSources of demographic data
Sources of demographic dataTarique Azmi
 
Theory of estimation
Theory of estimationTheory of estimation
Theory of estimationTech_MX
 
Vital statistics and demography
Vital statistics  and demographyVital statistics  and demography
Vital statistics and demographywrigveda
 
Population and demography
Population and demographyPopulation and demography
Population and demographyEunice Macapia
 
Test of hypothesis
Test of hypothesisTest of hypothesis
Test of hypothesisvikramlawand
 
Hypothesis testing ppt final
Hypothesis testing ppt finalHypothesis testing ppt final
Hypothesis testing ppt finalpiyushdhaker
 
#SPW13 - Educational Data Mining: Empowering young innovators - María Begoña ...
#SPW13 - Educational Data Mining: Empowering young innovators - María Begoña ...#SPW13 - Educational Data Mining: Empowering young innovators - María Begoña ...
#SPW13 - Educational Data Mining: Empowering young innovators - María Begoña ...Brussels, Belgium
 
Hypothesis testing; z test, t-test. f-test
Hypothesis testing; z test, t-test. f-testHypothesis testing; z test, t-test. f-test
Hypothesis testing; z test, t-test. f-testShakehand with Life
 
2015 Upload Campaigns Calendar - SlideShare
2015 Upload Campaigns Calendar - SlideShare2015 Upload Campaigns Calendar - SlideShare
2015 Upload Campaigns Calendar - SlideShareSlideShare
 
What to Upload to SlideShare
What to Upload to SlideShareWhat to Upload to SlideShare
What to Upload to SlideShareSlideShare
 

Destacado (20)

Test of hypothesis (z)
Test of hypothesis (z)Test of hypothesis (z)
Test of hypothesis (z)
 
Interval estimation for proportions
Interval estimation for proportionsInterval estimation for proportions
Interval estimation for proportions
 
Demographic data by dr najeeb memon
Demographic data by dr najeeb memonDemographic data by dr najeeb memon
Demographic data by dr najeeb memon
 
Population data sources
Population  data  sourcesPopulation  data  sources
Population data sources
 
Sources of demographic data
Sources of demographic dataSources of demographic data
Sources of demographic data
 
Sources of demographic data
Sources of demographic dataSources of demographic data
Sources of demographic data
 
Theory of estimation
Theory of estimationTheory of estimation
Theory of estimation
 
Vital statistics
Vital statisticsVital statistics
Vital statistics
 
Vital statistics and demography
Vital statistics  and demographyVital statistics  and demography
Vital statistics and demography
 
Population and demography
Population and demographyPopulation and demography
Population and demography
 
Demography ppt
Demography pptDemography ppt
Demography ppt
 
Test of hypothesis
Test of hypothesisTest of hypothesis
Test of hypothesis
 
Hypothesis testing ppt final
Hypothesis testing ppt finalHypothesis testing ppt final
Hypothesis testing ppt final
 
#SPW13 - Educational Data Mining: Empowering young innovators - María Begoña ...
#SPW13 - Educational Data Mining: Empowering young innovators - María Begoña ...#SPW13 - Educational Data Mining: Empowering young innovators - María Begoña ...
#SPW13 - Educational Data Mining: Empowering young innovators - María Begoña ...
 
Ambient intelligence
Ambient intelligenceAmbient intelligence
Ambient intelligence
 
Mutations powerpoint
Mutations powerpointMutations powerpoint
Mutations powerpoint
 
Hypothesis testing; z test, t-test. f-test
Hypothesis testing; z test, t-test. f-testHypothesis testing; z test, t-test. f-test
Hypothesis testing; z test, t-test. f-test
 
Hypothesis Testing
Hypothesis TestingHypothesis Testing
Hypothesis Testing
 
2015 Upload Campaigns Calendar - SlideShare
2015 Upload Campaigns Calendar - SlideShare2015 Upload Campaigns Calendar - SlideShare
2015 Upload Campaigns Calendar - SlideShare
 
What to Upload to SlideShare
What to Upload to SlideShareWhat to Upload to SlideShare
What to Upload to SlideShare
 

Similar a Developmental Evaluation on the Ground

5 Barrier to Effective Employee Training Programs and How to Crush Them | Web...
5 Barrier to Effective Employee Training Programs and How to Crush Them | Web...5 Barrier to Effective Employee Training Programs and How to Crush Them | Web...
5 Barrier to Effective Employee Training Programs and How to Crush Them | Web...BizLibrary
 
Monitoring and evaluation to improve fundraising bids
Monitoring and evaluation to improve fundraising bidsMonitoring and evaluation to improve fundraising bids
Monitoring and evaluation to improve fundraising bidsNatalie Blackburn
 
Organizing for Results - How to Build an Effective Marketing Function in an I...
Organizing for Results - How to Build an Effective Marketing Function in an I...Organizing for Results - How to Build an Effective Marketing Function in an I...
Organizing for Results - How to Build an Effective Marketing Function in an I...Jeffrey Rich
 
What do 10,000 development professionals know about improving project outcomes?
What do 10,000 development professionals know about improving project outcomes?What do 10,000 development professionals know about improving project outcomes?
What do 10,000 development professionals know about improving project outcomes?Chris Proulx
 
Measuring Performance: Linking Money To Mission
Measuring Performance: Linking Money To MissionMeasuring Performance: Linking Money To Mission
Measuring Performance: Linking Money To Mission4Good.org
 
Strategic Planning Workshop for Refugee Women's Network
Strategic Planning Workshop for Refugee Women's NetworkStrategic Planning Workshop for Refugee Women's Network
Strategic Planning Workshop for Refugee Women's NetworkEileen Ansley, PMP, SSBB
 
3 Conversations For Designing a Self-Sustaining Talent Management Process
3 Conversations For Designing a Self-Sustaining Talent Management Process 3 Conversations For Designing a Self-Sustaining Talent Management Process
3 Conversations For Designing a Self-Sustaining Talent Management Process David Lipsky, PhD
 
5 Barriers to Effective Employee Training Programs - Webinar 08.21.14
5 Barriers to Effective Employee Training Programs - Webinar 08.21.145 Barriers to Effective Employee Training Programs - Webinar 08.21.14
5 Barriers to Effective Employee Training Programs - Webinar 08.21.14BizLibrary
 
Forging Successful Learning Centers: Critical Considerations and Evidence-Bas...
Forging Successful Learning Centers: Critical Considerations and Evidence-Bas...Forging Successful Learning Centers: Critical Considerations and Evidence-Bas...
Forging Successful Learning Centers: Critical Considerations and Evidence-Bas...Lisa D'Adamo-Weinstein
 
Nonprofit Turn-Arounds: The Road to Recovery When Your Organization Is in the...
Nonprofit Turn-Arounds: The Road to Recovery When Your Organization Is in the...Nonprofit Turn-Arounds: The Road to Recovery When Your Organization Is in the...
Nonprofit Turn-Arounds: The Road to Recovery When Your Organization Is in the...Bloomerang
 
Project Learning Ppt
Project Learning PptProject Learning Ppt
Project Learning PptMelymartini
 
DDL Programme Meeting Oct12
DDL Programme Meeting Oct12DDL Programme Meeting Oct12
DDL Programme Meeting Oct12Helen Beetham
 
THINKING ABOUT CSR IN PRACTICE: thoughts, tools and examples – Lecture to McG...
THINKING ABOUT CSR IN PRACTICE: thoughts, tools and examples – Lecture to McG...THINKING ABOUT CSR IN PRACTICE: thoughts, tools and examples – Lecture to McG...
THINKING ABOUT CSR IN PRACTICE: thoughts, tools and examples – Lecture to McG...Wayne Dunn
 
【SYSU】MoC Star Leader for MarComm app
【SYSU】MoC Star Leader for MarComm app【SYSU】MoC Star Leader for MarComm app
【SYSU】MoC Star Leader for MarComm appSupa lun
 
State of Recruitment: 2015 CEHD Graduate Programs
State of Recruitment: 2015 CEHD Graduate ProgramsState of Recruitment: 2015 CEHD Graduate Programs
State of Recruitment: 2015 CEHD Graduate ProgramsBahati Banks-Cox
 
LMI Total Leader Sales Presentation May 09
LMI Total Leader Sales Presentation May 09LMI Total Leader Sales Presentation May 09
LMI Total Leader Sales Presentation May 09bwunder
 
Meeting the Needs of the Modern Learner
Meeting the Needs of the Modern LearnerMeeting the Needs of the Modern Learner
Meeting the Needs of the Modern LearnerAxonify
 

Similar a Developmental Evaluation on the Ground (20)

5 Barrier to Effective Employee Training Programs and How to Crush Them | Web...
5 Barrier to Effective Employee Training Programs and How to Crush Them | Web...5 Barrier to Effective Employee Training Programs and How to Crush Them | Web...
5 Barrier to Effective Employee Training Programs and How to Crush Them | Web...
 
Program planning
Program planningProgram planning
Program planning
 
Monitoring and evaluation to improve fundraising bids
Monitoring and evaluation to improve fundraising bidsMonitoring and evaluation to improve fundraising bids
Monitoring and evaluation to improve fundraising bids
 
Organizing for Results - How to Build an Effective Marketing Function in an I...
Organizing for Results - How to Build an Effective Marketing Function in an I...Organizing for Results - How to Build an Effective Marketing Function in an I...
Organizing for Results - How to Build an Effective Marketing Function in an I...
 
What do 10,000 development professionals know about improving project outcomes?
What do 10,000 development professionals know about improving project outcomes?What do 10,000 development professionals know about improving project outcomes?
What do 10,000 development professionals know about improving project outcomes?
 
Measuring Performance: Linking Money To Mission
Measuring Performance: Linking Money To MissionMeasuring Performance: Linking Money To Mission
Measuring Performance: Linking Money To Mission
 
Strategic Planning Workshop for Refugee Women's Network
Strategic Planning Workshop for Refugee Women's NetworkStrategic Planning Workshop for Refugee Women's Network
Strategic Planning Workshop for Refugee Women's Network
 
3 Conversations For Designing a Self-Sustaining Talent Management Process
3 Conversations For Designing a Self-Sustaining Talent Management Process 3 Conversations For Designing a Self-Sustaining Talent Management Process
3 Conversations For Designing a Self-Sustaining Talent Management Process
 
5 Barriers to Effective Employee Training Programs - Webinar 08.21.14
5 Barriers to Effective Employee Training Programs - Webinar 08.21.145 Barriers to Effective Employee Training Programs - Webinar 08.21.14
5 Barriers to Effective Employee Training Programs - Webinar 08.21.14
 
Forging Successful Learning Centers: Critical Considerations and Evidence-Bas...
Forging Successful Learning Centers: Critical Considerations and Evidence-Bas...Forging Successful Learning Centers: Critical Considerations and Evidence-Bas...
Forging Successful Learning Centers: Critical Considerations and Evidence-Bas...
 
Nonprofit Turn-Arounds: The Road to Recovery When Your Organization Is in the...
Nonprofit Turn-Arounds: The Road to Recovery When Your Organization Is in the...Nonprofit Turn-Arounds: The Road to Recovery When Your Organization Is in the...
Nonprofit Turn-Arounds: The Road to Recovery When Your Organization Is in the...
 
Project Learning Ppt
Project Learning PptProject Learning Ppt
Project Learning Ppt
 
DDL Programme Meeting Oct12
DDL Programme Meeting Oct12DDL Programme Meeting Oct12
DDL Programme Meeting Oct12
 
The Value of Infrastructure
The Value of InfrastructureThe Value of Infrastructure
The Value of Infrastructure
 
THINKING ABOUT CSR IN PRACTICE: thoughts, tools and examples – Lecture to McG...
THINKING ABOUT CSR IN PRACTICE: thoughts, tools and examples – Lecture to McG...THINKING ABOUT CSR IN PRACTICE: thoughts, tools and examples – Lecture to McG...
THINKING ABOUT CSR IN PRACTICE: thoughts, tools and examples – Lecture to McG...
 
Proven Ways to Measure the Impact of Training
Proven Ways to Measure the Impact of TrainingProven Ways to Measure the Impact of Training
Proven Ways to Measure the Impact of Training
 
【SYSU】MoC Star Leader for MarComm app
【SYSU】MoC Star Leader for MarComm app【SYSU】MoC Star Leader for MarComm app
【SYSU】MoC Star Leader for MarComm app
 
State of Recruitment: 2015 CEHD Graduate Programs
State of Recruitment: 2015 CEHD Graduate ProgramsState of Recruitment: 2015 CEHD Graduate Programs
State of Recruitment: 2015 CEHD Graduate Programs
 
LMI Total Leader Sales Presentation May 09
LMI Total Leader Sales Presentation May 09LMI Total Leader Sales Presentation May 09
LMI Total Leader Sales Presentation May 09
 
Meeting the Needs of the Modern Learner
Meeting the Needs of the Modern LearnerMeeting the Needs of the Modern Learner
Meeting the Needs of the Modern Learner
 

Último

Children who live with Grandparents are really lucky
Children who live with Grandparents are really luckyChildren who live with Grandparents are really lucky
Children who live with Grandparents are really luckySERUDS INDIA
 
War in Ukraine and problematics of the Ukrainian refugees in USA
War in Ukraine and problematics of the Ukrainian refugees in USAWar in Ukraine and problematics of the Ukrainian refugees in USA
War in Ukraine and problematics of the Ukrainian refugees in USAival6
 
PPT Item # 2 -- Announcements Powerpoint
PPT Item # 2 -- Announcements PowerpointPPT Item # 2 -- Announcements Powerpoint
PPT Item # 2 -- Announcements Powerpointahcitycouncil
 
2024: The FAR, Federal Acquisition Regulations - Part 16
2024: The FAR, Federal Acquisition Regulations - Part 162024: The FAR, Federal Acquisition Regulations - Part 16
2024: The FAR, Federal Acquisition Regulations - Part 16JSchaus & Associates
 
Hub Design Inspiration Graphics for inspiration
Hub Design Inspiration Graphics for inspirationHub Design Inspiration Graphics for inspiration
Hub Design Inspiration Graphics for inspirationStephen Abram
 
What Teenagers Have to Say about the Grandparents
What Teenagers Have to Say about the GrandparentsWhat Teenagers Have to Say about the Grandparents
What Teenagers Have to Say about the GrandparentsSERUDS INDIA
 
Water for Prosperity and peace - United Nations World Water Development Repo...
Water for Prosperity and peace -  United Nations World Water Development Repo...Water for Prosperity and peace -  United Nations World Water Development Repo...
Water for Prosperity and peace - United Nations World Water Development Repo...Christina Parmionova
 
SPONSORED CONTENT - MyGovWatch - RFP Cliches Debunked: What Government Buyers...
SPONSORED CONTENT - MyGovWatch - RFP Cliches Debunked: What Government Buyers...SPONSORED CONTENT - MyGovWatch - RFP Cliches Debunked: What Government Buyers...
SPONSORED CONTENT - MyGovWatch - RFP Cliches Debunked: What Government Buyers...JSchaus & Associates
 
Water can create peace or spark conflict.
Water can create peace or spark conflict.Water can create peace or spark conflict.
Water can create peace or spark conflict.Christina Parmionova
 
Sensitivity Training for 2023 BSKE.pptx
Sensitivity Training for  2023 BSKE.pptxSensitivity Training for  2023 BSKE.pptx
Sensitivity Training for 2023 BSKE.pptxAllidaacLuap
 
World Happiness Report 2024- Full Report
World Happiness Report 2024- Full ReportWorld Happiness Report 2024- Full Report
World Happiness Report 2024- Full ReportEnergy for One World
 
2024: The FAR, Federal Acquisition Regulations - Part 17
2024: The FAR, Federal Acquisition Regulations - Part 172024: The FAR, Federal Acquisition Regulations - Part 17
2024: The FAR, Federal Acquisition Regulations - Part 17JSchaus & Associates
 
O Conselho Estadual de Cultura e o Incentivo à Cultura no RS: relato de expe...
O Conselho Estadual de Cultura e o Incentivo à Cultura no RS:  relato de expe...O Conselho Estadual de Cultura e o Incentivo à Cultura no RS:  relato de expe...
O Conselho Estadual de Cultura e o Incentivo à Cultura no RS: relato de expe...Alvaro Santi
 
For World Water Day 2024, we promote the vital link between water and peace.
For World Water Day 2024, we promote the vital link between water and peace.For World Water Day 2024, we promote the vital link between water and peace.
For World Water Day 2024, we promote the vital link between water and peace.Christina Parmionova
 
Managing Planning and Development of Citie- 26-2-24.docx
Managing Planning and  Development of  Citie-  26-2-24.docxManaging Planning and  Development of  Citie-  26-2-24.docx
Managing Planning and Development of Citie- 26-2-24.docxJIT KUMAR GUPTA
 
Best charity ideas parents give their children’s
Best charity ideas parents give their children’sBest charity ideas parents give their children’s
Best charity ideas parents give their children’sSERUDS INDIA
 
National Women's Month Celebration for PENRO Quezon
National Women's Month Celebration for PENRO QuezonNational Women's Month Celebration for PENRO Quezon
National Women's Month Celebration for PENRO QuezonAryaCapale
 
OECD Webinar - ESG to deliver well-being in resource-rich regions: the role o...
OECD Webinar - ESG to deliver well-being in resource-rich regions: the role o...OECD Webinar - ESG to deliver well-being in resource-rich regions: the role o...
OECD Webinar - ESG to deliver well-being in resource-rich regions: the role o...OECDregions
 
Living life to the full: How can we make our longer lives healthier, happier ...
Living life to the full: How can we make our longer lives healthier, happier ...Living life to the full: How can we make our longer lives healthier, happier ...
Living life to the full: How can we make our longer lives healthier, happier ...ResolutionFoundation
 
Leveraging Water for Peace - World Water Day 2024
Leveraging Water for Peace - World Water Day 2024Leveraging Water for Peace - World Water Day 2024
Leveraging Water for Peace - World Water Day 2024Christina Parmionova
 

Último (20)

Children who live with Grandparents are really lucky
Children who live with Grandparents are really luckyChildren who live with Grandparents are really lucky
Children who live with Grandparents are really lucky
 
War in Ukraine and problematics of the Ukrainian refugees in USA
War in Ukraine and problematics of the Ukrainian refugees in USAWar in Ukraine and problematics of the Ukrainian refugees in USA
War in Ukraine and problematics of the Ukrainian refugees in USA
 
PPT Item # 2 -- Announcements Powerpoint
PPT Item # 2 -- Announcements PowerpointPPT Item # 2 -- Announcements Powerpoint
PPT Item # 2 -- Announcements Powerpoint
 
2024: The FAR, Federal Acquisition Regulations - Part 16
2024: The FAR, Federal Acquisition Regulations - Part 162024: The FAR, Federal Acquisition Regulations - Part 16
2024: The FAR, Federal Acquisition Regulations - Part 16
 
Hub Design Inspiration Graphics for inspiration
Hub Design Inspiration Graphics for inspirationHub Design Inspiration Graphics for inspiration
Hub Design Inspiration Graphics for inspiration
 
What Teenagers Have to Say about the Grandparents
What Teenagers Have to Say about the GrandparentsWhat Teenagers Have to Say about the Grandparents
What Teenagers Have to Say about the Grandparents
 
Water for Prosperity and peace - United Nations World Water Development Repo...
Water for Prosperity and peace -  United Nations World Water Development Repo...Water for Prosperity and peace -  United Nations World Water Development Repo...
Water for Prosperity and peace - United Nations World Water Development Repo...
 
SPONSORED CONTENT - MyGovWatch - RFP Cliches Debunked: What Government Buyers...
SPONSORED CONTENT - MyGovWatch - RFP Cliches Debunked: What Government Buyers...SPONSORED CONTENT - MyGovWatch - RFP Cliches Debunked: What Government Buyers...
SPONSORED CONTENT - MyGovWatch - RFP Cliches Debunked: What Government Buyers...
 
Water can create peace or spark conflict.
Water can create peace or spark conflict.Water can create peace or spark conflict.
Water can create peace or spark conflict.
 
Sensitivity Training for 2023 BSKE.pptx
Sensitivity Training for  2023 BSKE.pptxSensitivity Training for  2023 BSKE.pptx
Sensitivity Training for 2023 BSKE.pptx
 
World Happiness Report 2024- Full Report
World Happiness Report 2024- Full ReportWorld Happiness Report 2024- Full Report
World Happiness Report 2024- Full Report
 
2024: The FAR, Federal Acquisition Regulations - Part 17
2024: The FAR, Federal Acquisition Regulations - Part 172024: The FAR, Federal Acquisition Regulations - Part 17
2024: The FAR, Federal Acquisition Regulations - Part 17
 
O Conselho Estadual de Cultura e o Incentivo à Cultura no RS: relato de expe...
O Conselho Estadual de Cultura e o Incentivo à Cultura no RS:  relato de expe...O Conselho Estadual de Cultura e o Incentivo à Cultura no RS:  relato de expe...
O Conselho Estadual de Cultura e o Incentivo à Cultura no RS: relato de expe...
 
For World Water Day 2024, we promote the vital link between water and peace.
For World Water Day 2024, we promote the vital link between water and peace.For World Water Day 2024, we promote the vital link between water and peace.
For World Water Day 2024, we promote the vital link between water and peace.
 
Managing Planning and Development of Citie- 26-2-24.docx
Managing Planning and  Development of  Citie-  26-2-24.docxManaging Planning and  Development of  Citie-  26-2-24.docx
Managing Planning and Development of Citie- 26-2-24.docx
 
Best charity ideas parents give their children’s
Best charity ideas parents give their children’sBest charity ideas parents give their children’s
Best charity ideas parents give their children’s
 
National Women's Month Celebration for PENRO Quezon
National Women's Month Celebration for PENRO QuezonNational Women's Month Celebration for PENRO Quezon
National Women's Month Celebration for PENRO Quezon
 
OECD Webinar - ESG to deliver well-being in resource-rich regions: the role o...
OECD Webinar - ESG to deliver well-being in resource-rich regions: the role o...OECD Webinar - ESG to deliver well-being in resource-rich regions: the role o...
OECD Webinar - ESG to deliver well-being in resource-rich regions: the role o...
 
Living life to the full: How can we make our longer lives healthier, happier ...
Living life to the full: How can we make our longer lives healthier, happier ...Living life to the full: How can we make our longer lives healthier, happier ...
Living life to the full: How can we make our longer lives healthier, happier ...
 
Leveraging Water for Peace - World Water Day 2024
Leveraging Water for Peace - World Water Day 2024Leveraging Water for Peace - World Water Day 2024
Leveraging Water for Peace - World Water Day 2024
 

Developmental Evaluation on the Ground

  • 1. Developmental Evaluation on the Ground Brigitte Scott, Ph.D. Evaluation and Research Specialist Military Families Learning Network
  • 2. Military Families Learning Network Personal Finance Child Care Family Advocacy Network Literacy Personal Finance Family Development Network Literacy Transition Support Community Capacity Building Nutrition & Wellness Early Intervention Special Needs Life Span Special Needs Military Caregiving
  • 4. MQP: DE Defined “DE supports innovation development to guide adaptation to emergent and dynamic realities in complex environments.” —MQP, p. 1 Goal: support project, program, product, and/or organizational development with timely feedback
  • 5. MQP: Core DE Question What is getting developed and what are the implications of what gets developed? Complexity theory of change: Bring people together who are knowledgeable and committed and they will self-organize, take action, and work together to create movement, innovation, and change.
  • 8. Focus on developmentMQP • NOT the same as continuous improvement (formative eval or accountability) – Quality improvement is helping programs meet standards that have been set. Development is when people are changing what they are doing, and the very nature of the standards are also changing. Standards are adapting to changing conditions.
  • 9. Complex and dynamic environment
  • 10. Complex and dynamic environmentMQP Complex environment = lack of central control What to do to solve problems is uncertain and there can be conflict about how to proceed
  • 12. DE becomes part of intervention
  • 14. Developmental EvaluatorMQP • Works collaboratively with innovators to conceptualize, design, and test new approaches in a long-term, on-going process of adaptation, intentional change, and development. • Ask eval questions • Apply eval logic • Gather and report eval data
  • 15. Developmental EvaluatorMQP Primary Functions: Elucidate the innovation and adaptation processes Track implications and results Facilitate data-based decision-making
  • 16. “There are no best practices. ‘Best’ is entirely context-free.” —MQP, AEA 2014
  • 17. Evaluation for MFLN • Ongoing development • Innovation in learning – Internal practice – External product
  • 18. DE for MFLN: Why It Fits
  • 20. Cooperative agreement environment + eXtension Initiative
  • 21. Cooperative Agreement “An opportunity to provide responsive and timely educational programming through being actively nimble, flexible, innovative, and creative in true partnership with our funders as they identify organizational priorities and ask us to engage in the construction of appropriate and necessary deliverables that meet the on-time needs of the target audience. The MFLN and funding partners are seen as true and honest equals in expertise, providing valued and accepted feedback bi-directionally. This often results in surpassing expectations for success in delivery.” —Kyle Kostelecky, National Program Director, MFLN
  • 23. Constant evaluative feedback and discussion • Monthly reports (internal, DoD) • Quarterly reports (NIFA) • Annual reports (internal, DoD, NIFA) • Webinar evaluation reports • Quarterly webinar reports • Weekly leadership team meetings • Social Media Specialists meetings • Net Lit support meetings
  • 24. Evaluation used for decision-making
  • 25. Evaluator is on leadership team
  • 26. DE for MFLN: What It Looks Like Continuous adaptation
  • 27. DE for MFLN: What It Looks Like Continuous adaptation Social, political, economic, technological, and demographic patterns
  • 28. DE for MFLN: What It Looks Like Continuous adaptation Social, political, economic, technological, and demographic patterns Emergent developments/needs of our target audiences
  • 29. DE for MFLN: What It Looks Like Continuous adaptation Social, political, economic, technological, and demographic patterns Emergent developments/needs of our target audiences Cooperative agreement environment
  • 30. DE for MFLN: What It Looks Like Continuous adaptation Social, political, economic, technological, and demographic patterns Emergent developments/needs of our target audiences Cooperative agreement environment eXtension
  • 31. DE for MFLN: What It Looks Like Continuous adaptation Social, political, economic, technological, and demographic patterns Emergent developments/needs of our target audiences Cooperative agreement environment eXtension DoD
  • 32. DE for MFLN: What It Looks Like (not this. . . .)
  • 33. DE for MFLN: What It Looks Like (but this. . . .)
  • 34. DE for MFLN: What It Looks Like Inductive
  • 35. DE for MFLN: What It Looks Like Inductive Chaotic and messy
  • 36. DE for MFLN: What It Looks Like Inductive Chaotic and messy Embrace forks in the road
  • 37. DE for MFLN: What It Looks Like Inductive Chaotic and messy Embrace forks in the road Developmental moments Track what’s going on Understand what’s going on Adapt to what’s going on
  • 38. DE for MFLN: What It Looks Like Everyone is an evaluator
  • 39. Complex development situations are ones in Inspired by Jeff Conklin, cognexus.org which this… Time 39 Michael Quinn Patton AEA eStudy webinar 2014
  • 40. Michael Quinn Patton AEA eStudy webinar 2014 40 And this…
  • 41. Turns out to be this… Time 41 Michael Quinn Patton AEA eStudy webinar 2014
  • 42. Michael Quinn Patton AEA eStudy webinar 2014 42 …looks like this
  • 44. DE Successes for MFLN • Very responsive to our partners’ programmatic requests and innovations
  • 45. DE Successes for MFLN • Very responsive to our partners’ programmatic requests and innovations • Doubling in size in 2015 with competitive award – New CAs and new opportunities for innovation in programming
  • 46. DE Successes for MFLN • Very responsive to our partners’ programmatic requests and innovations • Doubling in size in 2015 with competitive award – New CAs and new opportunities for innovation in programming • Programming
  • 47. DE Successes for MFLN • Very responsive to our partners’ programmatic requests and innovations • Doubling in size in 2015 with competitive award – New CAs and new opportunities for innovation in programming • Programming • “Walk the talk”: model use of social media, collaborative learning, personal learning networks
  • 49. DE Challenges for MFLN • Innovation is challenging, slow, and . . . contentious
  • 50. DE Challenges for MFLN • Innovation is challenging, slow, and . . . contentious • Moving from program improvement mode to redesign based on evaluation findings
  • 51. DE Challenges for MFLN • Innovation is challenging, slow, and . . . contentious • Moving from program improvement mode to redesign based on evaluation findings • IRB challenges
  • 52. DE Challenges for MFLN • Innovation is challenging, slow, and . . . contentious • Moving from program improvement mode to redesign based on evaluation findings • IRB challenges • “In your face” evaluation
  • 53. DE Challenges for MFLN • Innovation is challenging, slow, and . . . contentious • Moving from program improvement mode to redesign based on evaluation findings • IRB challenges • “In your face” evaluation • Daily, evaluative thinking
  • 54. DE Challenges for MFLN • Innovation is challenging, slow, and . . . contentious • Moving from program improvement mode to redesign based on evaluation findings • IRB challenges • “In your face” evaluation • Daily, evaluative thinking • Platform delivery limitations
  • 55. DE Challenges for MFLN • Innovation is challenging, slow, and . . . contentious • Moving from program improvement mode to redesign based on evaluation findings • IRB challenges • “In your face” evaluation • Daily, evaluative thinking • Platform delivery limitations • Military culture
  • 56. DE Challenges for MFLN • Innovation is challenging, slow, and . . . contentious • Moving from program improvement mode to redesign based on evaluation findings • IRB challenges • “In your face” evaluation • Daily, evaluative thinking • Platform delivery limitations • Military culture • Doubling in size in January 2015
  • 58. DE Features for MFLN • Reports • Leadership • Reflective discussion groups • Transparency • Collaboration
  • 60. Patton, Michael Quinn. (2014). “Intermediate Developmental Evaluation.” AEA eStudy Webinar. Patton, Michael Quinn. (2011). Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use. Guilford: New York.
  • 61. Brigitte Scott @4ed_eval brigit2@vt.edu www.blogs.extension.org/militaryfamilies Icons made by Freepik: http://www.flaticon.com. Flaticon is licensed by Creative Commons 3.0.

Notas del editor

  1. DE is not ongoing formative eval DE supports and documents development
  2. DE is not ongoing formative eval DE supports and documents development Complex environment = lack of central control; what to do to solve problems is uncertain and there can be conflict about how to proceed
  3. Focus on development, which is not the same as continuous improvement (formative eval) or accountability Improvement vs. development Quality improvement is helping programs meet standards that have been set. Developmental evaluation is when people are changing what they are doing and the very nature of the standards are also changing. Standards are adapting to changing conditions DE: standards themselves and what it means to meet them and how to talk about them and how to apply them is being developed and changed in relationship to a changing world; very different from having a predetermined set of quality criteria that a program has to meet.
  4. Focus on development, which is not the same as continuous improvement (formative eval) or accountability Improvement vs. development Quality improvement is helping programs meet standards that have been set. Developmental evaluation is when people are changing what they are doing and the very nature of the standards are also changing. Standards are adapting to changing conditions DE: standards themselves and what it means to meet them and how to talk about them and how to apply them is being developed and changed in relationship to a changing world; very different from having a predetermined set of quality criteria that a program has to meet.
  5. DE becomes part of intervention
  6. Developmental evaluation in the context of MFLN is an evaluation paradigm that facilitates program development and innovation through continuous adaptation to the dynamic and changing MFLN cooperative agreement environment. Our programming doesn’t intend to necessarily become a fixed, standardized model. But it does help us identify effective principles that inform ongoing development
  7. Cooperative agreement: NIFA, DoD, and UIUC Different than traditional grant, which is when you articulate your research goals and deliverables before receiving your funding; instead, we receive funding based on a loosely defined set of criteria (plans of work) An award similar to a grant, but I which the sponsor’s staff may be actively involved in proposal preparation, and anticipates having substantial involvement in research activities once the award has been made. Can you tell me if the PF VLE was a result of that responsiveness? I'm looking for examples of how our plans of work can change during the year based on DoD conversations and request. I can't remember, but I thought the VLE was an example of that--not planned for 2014 but instead requested after the year started.  Yes, this is a great example.  DoD thought is was coming at some point - we had talked about it in the past.  They felt that it was time, as it met a quickly evolving need from their end.  They asked and stressed how important it would be to accomplish this at this time and we went back to our PF team and they were open, excited, and offered a plan for how they could do it within their budget (not adding but rearranging and deleting other deliverables).  After PF’s new plan was discussed with DoD, all were happy to move forward.  Another example of how this cooperative agreement has worked from our end (we instigated a suggested change) was when I began to hear more and more about how military caregiving was an increasing concern from those Anne and I talked with when we were at conferences (especially military conferences).  We were not hearing this from MC&FP and so I did some homework on my end, began to have these conversations with Betsy and Eddy, and suggested that we should consider expanding in this area.  They took this idea back to their offices and partners and then came back to us with an approval to move forward.  We did this within our 2010 award at the end when we needed to re-capture funds and construct a re-budget.  We had about $40K to play with and conducted a 5 month pilot.  MC&FP liked it that it was fully funded for 2012 (and of course has been ever since with great success).  So it goes both ways, they ask and we deliver - we suggest and they agree.  It’s been great.  There are other examples if you need more, just let me know. eXtension: emergent and dynamic organization both tied to and simultaneously oriented away from Cooperative Extension.
  8. Monthly reports Quarterly reports Annual reports Weekly leadership team meetings Webinar evaluation reports Quarter webinar reports Social Media Specialists meetings Net Lit support meetings
  9. .
  10. We
  11. Real time innovation is challenging --We’re very good at innovating and request of our partners; not as nimble for us to innovate on our own (still top down in that way); --PIs bought out for only a small portion of time; innovation and change is slow, may not have everyone’s buy-in Challenge: We were really in program implementation/improvement mode for the first four years. How do you introduce the possibility that you are no longer in program improvement mode but in fact that program needs to be redesigned? How do you prepare the people you’re working with for the possibility that you are no longer in program improvement mode but rather that you need to begin adapting and changing to findings of formative evaluation? Difficult to make changes quickly to eval and research --DoD looking for all kinds of information, but IRB is a challenge (18 month IRB process for working with contractors; hard to be responsive and provide real-time feedback) Evaluation is constantly “in your face” --Getting everyone to think evaluatively is challenging Platform limitations for reaching people on bases; --Limits innovation Military culture (private, formal); Doubling in size in 2015—lots of work to get done and most days are about just getting through the day
  12. Real time innovation is challenging --We’re very good at innovating and request of our partners; not as nimble for us to innovate on our own (still top down in that way); --PIs bought out for only a small portion of time; innovation and change is slow, may not have everyone’s buy-in Challenge: We were really in program implementation/improvement mode for the first four years. How do you introduce the possibility that you are no longer in program improvement mode but in fact that program needs to be redesigned? How do you prepare the people you’re working with for the possibility that you are no longer in program improvement mode but rather that you need to begin adapting and changing to findings of formative evaluation? Difficult to make changes quickly to eval and research --DoD looking for all kinds of information, but IRB is a challenge (18 month IRB process for working with contractors; hard to be responsive and provide real-time feedback) Evaluation is constantly “in your face” --Getting everyone to think evaluatively is challenging Platform limitations for reaching people on bases; --Limits innovation Military culture (private, formal); Doubling in size in 2015—lots of work to get done and most days are about just getting through the day
  13. Real time innovation is challenging --We’re very good at innovating and request of our partners; not as nimble for us to innovate on our own (still top down in that way); --PIs bought out for only a small portion of time; innovation and change is slow, may not have everyone’s buy-in Challenge: We were really in program implementation/improvement mode for the first four years. How do you introduce the possibility that you are no longer in program improvement mode but in fact that program needs to be redesigned? How do you prepare the people you’re working with for the possibility that you are no longer in program improvement mode but rather that you need to begin adapting and changing to findings of formative evaluation? Difficult to make changes quickly to eval and research --DoD looking for all kinds of information, but IRB is a challenge (18 month IRB process for working with contractors; hard to be responsive and provide real-time feedback) Evaluation is constantly “in your face” --Getting everyone to think evaluatively is challenging Platform limitations for reaching people on bases; --Limits innovation Military culture (private, formal); Doubling in size in 2015—lots of work to get done and most days are about just getting through the day
  14. Real time innovation is challenging --We’re very good at innovating and request of our partners; not as nimble for us to innovate on our own (still top down in that way); --PIs bought out for only a small portion of time; innovation and change is slow, may not have everyone’s buy-in Challenge: We were really in program implementation/improvement mode for the first four years. How do you introduce the possibility that you are no longer in program improvement mode but in fact that program needs to be redesigned? How do you prepare the people you’re working with for the possibility that you are no longer in program improvement mode but rather that you need to begin adapting and changing to findings of formative evaluation? Difficult to make changes quickly to eval and research --DoD looking for all kinds of information, but IRB is a challenge (18 month IRB process for working with contractors; hard to be responsive and provide real-time feedback) Evaluation is constantly “in your face” --Getting everyone to think evaluatively is challenging Platform limitations for reaching people on bases; --Limits innovation Military culture (private, formal); Doubling in size in 2015—lots of work to get done and most days are about just getting through the day
  15. Real time innovation is challenging --We’re very good at innovating and request of our partners; not as nimble for us to innovate on our own (still top down in that way); --PIs bought out for only a small portion of time; innovation and change is slow, may not have everyone’s buy-in Challenge: We were really in program implementation/improvement mode for the first four years. How do you introduce the possibility that you are no longer in program improvement mode but in fact that program needs to be redesigned? How do you prepare the people you’re working with for the possibility that you are no longer in program improvement mode but rather that you need to begin adapting and changing to findings of formative evaluation? Difficult to make changes quickly to eval and research --DoD looking for all kinds of information, but IRB is a challenge (18 month IRB process for working with contractors; hard to be responsive and provide real-time feedback) Evaluation is constantly “in your face” --Getting everyone to think evaluatively is challenging Platform limitations for reaching people on bases; --Limits innovation Military culture (private, formal); Doubling in size in 2015—lots of work to get done and most days are about just getting through the day
  16. Real time innovation is challenging --We’re very good at innovating and request of our partners; not as nimble for us to innovate on our own (still top down in that way); --PIs bought out for only a small portion of time; innovation and change is slow, may not have everyone’s buy-in Challenge: We were really in program implementation/improvement mode for the first four years. How do you introduce the possibility that you are no longer in program improvement mode but in fact that program needs to be redesigned? How do you prepare the people you’re working with for the possibility that you are no longer in program improvement mode but rather that you need to begin adapting and changing to findings of formative evaluation? Difficult to make changes quickly to eval and research --DoD looking for all kinds of information, but IRB is a challenge (18 month IRB process for working with contractors; hard to be responsive and provide real-time feedback) Evaluation is constantly “in your face” --Getting everyone to think evaluatively is challenging Platform limitations for reaching people on bases; --Limits innovation Military culture (private, formal); Doubling in size in 2015—lots of work to get done and most days are about just getting through the day
  17. Real time innovation is challenging --We’re very good at innovating and request of our partners; not as nimble for us to innovate on our own (still top down in that way); --PIs bought out for only a small portion of time; innovation and change is slow, may not have everyone’s buy-in Challenge: We were really in program implementation/improvement mode for the first four years. How do you introduce the possibility that you are no longer in program improvement mode but in fact that program needs to be redesigned? How do you prepare the people you’re working with for the possibility that you are no longer in program improvement mode but rather that you need to begin adapting and changing to findings of formative evaluation? Difficult to make changes quickly to eval and research --DoD looking for all kinds of information, but IRB is a challenge (18 month IRB process for working with contractors; hard to be responsive and provide real-time feedback) Evaluation is constantly “in your face” --Getting everyone to think evaluatively is challenging Platform limitations for reaching people on bases; --Limits innovation Military culture (private, formal); Doubling in size in 2015—lots of work to get done and most days are about just getting through the day
  18. Real time innovation is challenging --We’re very good at innovating and request of our partners; not as nimble for us to innovate on our own (still top down in that way); --PIs bought out for only a small portion of time; innovation and change is slow, may not have everyone’s buy-in Challenge: We were really in program implementation/improvement mode for the first four years. How do you introduce the possibility that you are no longer in program improvement mode but in fact that program needs to be redesigned? How do you prepare the people you’re working with for the possibility that you are no longer in program improvement mode but rather that you need to begin adapting and changing to findings of formative evaluation? Difficult to make changes quickly to eval and research --DoD looking for all kinds of information, but IRB is a challenge (18 month IRB process for working with contractors; hard to be responsive and provide real-time feedback) Evaluation is constantly “in your face” --Getting everyone to think evaluatively is challenging Platform limitations for reaching people on bases; --Limits innovation Military culture (private, formal); Doubling in size in 2015—lots of work to get done and most days are about just getting through the day