Evaluation Principles: Theory-Based, Utilization-Focused, Participatory. Find out more in this presentation about three approaches to evaluation.

1. Evaluation principles: Theory-based Utilization focused Participatory Washington Mental Health Transformation Evaluation Task Group

2. A call for Theory-based evaluation “ The theory-driven approach is essential to tracking the many elements of the program [or initiative], and assuring that the results identified in the evaluation are firmly connected to the program’s activities. Tracking all aspects of the system makes it more plausible that the results are due to program activities. . . .and that the results generalize to other programs of the same type” -- Carol H. Weiss

7. “ Framing” a Theory of Change As described by Hernandez & Hodges, 2001 Population Frame Consider the issues and strengths Strategies Frame Consider principles and components Desired Outcomes Frame Consider Short- and long-term outcomes

8. A sample framework As described by Hernandez & Hodges, 2001, 2003 Issues Lack of appropriate services Location of services Lack of community members as service providers Lack of knowledge of residents’ perceptions of services available Lack of parental involvement in agency decision making Strategies Develop a core group of community members Develop and administer community needs survey Develop volunteer parent network Involve parents in agency decision-making Outcomes Increased parent representation in agency decision-making (list of meetings and results of parent participation) Increase in appropriate services (services resulting from needs) Increased accessibility (location, hours, transportation)

9. Baltimore’s After-School Strategy Theory of change Increased program quality Increased program slots After- School Programs Adopting research-based Standards Leveraging public and private funds Capacity-building and TA Accountability, monitoring, and evaluation Increased participation rates Skills, competencies, and attitudes that lead to long-term improvements in academic and non-academic outcomes

10. Strategy: School system reform to address mental health issues of kindergartners with EBD Children with EBD in school-based pre-kindergarten programs receive appropriate treatment. Child outcomes System outcomes Teachers in school-based, pre-kindergarten programs are able to recognize signs of emotional disharmony. Develop an agreement for mental health providers to train teachers in school-based, pre-kindergarten programs on recognizing the signs of emotional disharmony . Psychiatric specialists from outside mental health programs are on staff at school-based, pre-kindergarten programs. Convene meetings between the school system and public, private, and nonprofit providers of mental health services for young children. Develop an agreement for mental health providers to place psychiatric staff in school-based, pre-kindergarten programs on a part-time basis. Initiative Activities Kindergartners with EBD participate appropriately in classroom activities. Sample Partial Logic Model #1 for a Success By 6 ® Initiative Develop an agreement for mental health providers to accept referrals and treat children with EBD from school-based, pre-kindergarten programs. With parental consent, school psychologists and outside psychiatric specialists assess children that show signs of emotional dysharmony. Services are available in quality mental health programs for pre-school children with EBD. Quality mental health providers make administrative arrangements so that more slots are available for pre-school children with EBD. With parental consent, school psychologists and outside psychiatric specialists refer Children with EBD to quality mental health providers. Teachers in school-based, pre-kindergarten programs refer children that show signs of emotional disharmony to staff. Mental health providers accept referrals and treat Children with EBD from school-based, pre-kindergarten programs. Write grants, solicit corporate sponsors, and advocate in the legislature for funding to provide more slots in quality mental health programs for pre-school children with EBD.