TTIPEC: Monitoring and Evaluation (Session 2)
•
2 recomendaciones•1,147 vistas
Powerpoint on monitoring and evaluation for sessions 2 of the TTIPEC Ghana workshop
Recomendados
Recomendados
Más contenido relacionado
La actualidad más candente
La actualidad más candente (20)
Destacado
Similar a TTIPEC: Monitoring and Evaluation (Session 2)
Similar a TTIPEC: Monitoring and Evaluation (Session 2) (20)
Más de Research to Action
Más de Research to Action (20)
Último
Último (20)
TTIPEC: Monitoring and Evaluation (Session 2)
- 1. Monitoring & Evaluation IN PRACTICE TTI PEC Anglophone Africa Regional Peer Learning Workshop Coconut Grove Regency Hotel | Accra, Ghana Wednesday, July 16th, 2014 ! ! Mo Adefeso-OlatejuRebecca Oluwatoyin Doherty
- 2. Linking Monitoring, Evaluation, & Measurement SHORT VIDEOS Importance of monitoring & evaluation https://www.youtube.com/watch?v=2mBcb-AQ8xQ 4:00minutes - end ! Indicators https://www.youtube.com/watch?v=SZqOKW2_jNU ! Indicators: Basic Concepts https://www.youtube.com/watch?v=qmrDNuJzRv4
- 4. 4 BASIC M&E POINTS MUST BE FOLLOWED Who is our population? What data is to be collected from them? Or about them? How will the data captured enhance communications? And also fit within the rest of the M&E analysis? How will the data be analyzed? Who will be doing the analysis? What do we know about our baseline data? How will the process be managed? And by whom? An organizational assessment is needed…in order to establish a process for M&E! Source: https://sites.google.com/site/newmediacommunications2010/mobiles-and-m-e/2-case-studies
- 5. ASSESSMENT OF READINESS In order for the M&E Function to succeed, it is necessary to meet two major conditions: ! ▪ Organizational readiness – Is there a clear M&E framework with targets? – Is the planning process evidence-based? – Is the TT focused on achieving results (leadership, communications) ! ▪ Technical readiness - Do the staff have the relevant experience and skills? WORKSHEET
- 6. What do we monitor? Inputs. Activities. RESULTS. WORKSHEET
- 7. Results chain Resources Processes Within control of the organization Partial Org. Control No Org. Control Sphere of Control: Advocacy, Tactics, Inputs, Activities, Outputs Sphere of Influence: Changes in behavior, relationships, attitudes, agenda Sphere of Concern: Policy & Political Context WORKSHEET WORKSHEET
- 8. Results chain – Example WORKSHEET
- 9. Indicators, baseline and targets (& Planning) WORKSHEET
- 10. Strategies & approaches for identifying evidence of change • Signposts – output log, web analytics, media tracking, citation tracking – discourse analysis, social network analysis • Outcome (or influence) log – record of interactions with decision-makers, anecdotes shared by stakeholders (e.g. with the director) • Feedback mechanisms – stakeholder surveys – interviews – focus group discussions WORKSHEET
- 11. Integrated Learning Processes in M&E - Illustration jan marfeb apr sep novoct decmay julyjune aug Quarterly report Half-yearly & yearly report WORKSHEET
- 13. WEBSITES: www.raguide.org www.resultsaccountability.com BOOK ORDERS: www.trafford.com www.amazon.com Results Accountability Decision-making and Budgeting Fiscal Policy Studies Institute Santa Fe, New Mexico
- 14. “If you do what you always did, you will get what you always got.” Kenneth W. Jenkins President, Yonkers NY NAACP TO THINK OR NOT TO THINK ABOUT M&E? FOOD FOR THOUGHT…
- 15. The first step in performance accountability is to DRAW A FENCE Around something that has ORGANIZATIONAL OR FUNCTIONAL IDENTITY The Whole Organization Division A Division B Unit Division C Function Unit 1 WORKSHEET
- 16. HOW DO WE THINK ABOUT WHO, WHAT & WHY WE MEASURE?
- 17. Different Kinds of Progress 1. Data a. Population indicators Actual turned curves: movement for the better away from the baseline. b. Program performance measures: customer progress and better service: How much did we do? How well did we do it? Is anyone better off? 2. Accomplishments: Positive activities, not included above. 3. Stories behind the statistics that show how individuals are better off.
- 18. What’s Next? A Basic Action Plan for Results Accountability TRACK 1: POPULATION ACCOUNTABILITY ● Establish results ● Establish indicators, baselines and charts on the wall ● Create an indicators report card ● Set tables (action groups) to turn curves TRACK 2: PERFORMANCE ACCOUNTABILITY ● Performance measures, and charts on the wall for programs, agencies and service systems ● Use 7 Questions supervisor by supervisor and program by program in management, budgeting and strategic planning
- 20. If I can…. Measure, Track, Improve & Show Performance Education Performance Accountability What’s in it for me? Then…. here’s what’s in it for me: ! Improvement = more money ! Success at the State’s Accountability Game ! Student and Parent Satisfaction ! Tool vs. fingers crossed ! Knowing where your students are between achievement tests ! Knowing that you’ve made a difference
- 21. WORKSHEET
- 22. WORKSHEET
- 23. WORKSHEET
- 24. Turn the Curve Exercise: Program Performance 5 min: Starting Points - timekeeper and reporter - identify a program to work on - two hats (yours plus partner’s) 10 min: Performance measure baseline - choose 1 measure to work on – from the lower right quadrant - forecast – OK or not OK? 15 min: Story behind the baseline - causes/forces at work - information & research agenda part 1 - causes 15 min: What works? (What would it take?) - what could work to do better? - each partners contribution - no-cost / low-cost ideas - information & research agenda part 2 – what works 10 min: Report Convert notes to one page Two pointers to action WORKSHEET
- 25. Program: _______________ Performance Measure (Lay definition) Performance Measure Baseline Story behind the baseline --------------------------- --------------------------- (List as many as needed) Partners --------------------------- --------------------------- (List as many as needed) Three Best Ideas – What Works 1. --------------------------- 2. --------------------------- 3. ---------No-cost / low-cost ONE PAGE Turn the Curve Report: Performance Sharp Edges 4. --------- Off the Wall 4. --------- Off the Wall WORKSHEET
- 26. 2 customers 1 measure 1 measure Discuss how we’re doing 2 partners 2 what works ideas (1 nc/lc) Discuss how to implement 20 Minute Exercise 40 WORKSHEET
- 27. 1 population 1 result 1 indicator Discuss how we’re doing 2 partners 2 what works ideas (1 nc/lc) Discuss how to implement 20 Minute Exercise 1. What population are we concerned about? (e.g. U.S. population) 2. What condition(s) do we want for this population? (Americans are free of cancer.) 3. How could we measure these conditions? ( (e.g. cancer rates) 4. How are we doing on the most important of these measures? (baseline history) 5. Who are the partners with a role to play in doing better? 6. What works – what would it take – to do better? (What is our role?) 7. What do we propose to do? POPULATION ACCOUNTABILITY WORKSHEET
- 28. Results Based Grantmaking Exercise A. Result _________________________ B. Indicators 1. _______________ 2. _______________ 3. _______________ For each: Getting Better Getting Worse or About the Same? C. Story behind the baselines (Why better or worse?) __________ __________ __________ __________ D. Partners with a role to play __________ __________ __________ __________ E. What would it take? (5 most important strategies) 1. _____________ 4. ______________ 2. _____________ 5. ______________ 3. _____________ F. Our Roles (3 best ideas) 1. __________________ 3. _________________ 2. __________________ Off the wall:. _________ 10 min: 10 min: 10 min: 15 min: 10 min: WORKSHEET
- 29. Performance Measurement: How to Develop Them Sectoral Examples See Also: raguide.org Questions 3.10 and 3.11
- 30. ● # of people served ● % participants who got jobs ● staff turnover rate ● # participants who got jobs ●% of children reading at grade level ● cost per unit of service ● # applications processed ● % patients who fully recover What Kind of PERFORMANCE MEASURE? Upper Left Lower Right Upper Right Lower Left Lower Right Upper Left Upper Left Lower Right QUANTITY ! ! How much did we do? QUALITY ! ! How well did we do? E F O R T E F E C T Is anyone better off? WORKSHEET
- 31. How much did we do? Professional Development How well did we do it? Is anyone better off? Quantity Quality EffectEffort Customers # teachers served # students # parents Activities # training sessions # hours of training # study groups # TA contacts # mentoring relationships Common measures % teachers certified in subjects they teach % customers satisfied with how service is delivered % staff with good morale Unit cost per training unit (HL #3) Activity specific measures % positive evaluations on training % attendance in study groups % of faculty participating in study groups % mentoring pairs who complete mo. rept. % mentees satisfied with match # # ! # # ! # ! # # # % teachers satisfied that prof dev helped them be a better teacher (DDA #2) % teacher retention rate - total - “quality” teachers (DDA #1) % mentees who report relationship helped build their skills % teachers who demonstrate new skills - by observation (HL #2) - by self report % students in “trained” classrooms with improved test scores (HL #1) % students with reduced behavior problems % students with good school attendance % teachers who report better use of time SAMPLE WORKSHEET
- 32. How much did we do? Professional Development How well did we do it? Is anyone better off? Quantity Quality EffectEffort Number of students & teachers served ! Number of workshops Unit cost ! % of teachers with PD plan # teachers who used training (by self rept or observation) ! # Teachers retained ! # Achievement in classroom with trained teachers % teachers who used training (by self rept or observation) Teacher retention rate ! Achievement rates in classroom with trained teachers
- 33. How much did we do? Environment: Water Quality How well did we do it? Is anyone better off? Quantity Quality EffectEffort Number of stream stations monitored Average sites per monitor per month Number miles of healthy streams Percent miles of healthy streams
- 34. How much did we do? Commerce Tourism How well did we do it? Is anyone better off? Quantity Quality EffectEffort Number of books sent to interested consumers Cost per inquiry delivered Number of tourists Number tourist businesses Tourism market share Growth in tourist industry
- 35. How much did we do? Personnel Department How well did we do it? Is anyone better off? Quantity Quality EffectEffort Number of Applicants Processed Average Recruitment Period Workforce New Hires Workforce Turnover Rate (non-promotions) Customer Satisfaction
- 36. How much did we do? Every Program (Financial Performance Measures) How well did we do it? Is anyone better off? Quantity Quality EffectEffort Cost (total) Unit Cost Benefit Value (total) Cost-Benefit (ratio)
- 37. How much did we do? Corrections How well did we do it? Is anyone better off? Quantity Quality EffectEffort # Inmates Rate of overcrowding # Recidivism % Recidivism
- 38. How much did we do? Blank How well did we do it? Is anyone better off? Quantity Quality EffectEffort X X X X
- 39. PLANNING ALONG M&E OBJECTIV ES PERFOR MANCE INDICAT ORS INPUTS ACTIVIT IES OUTPUTS OUTCOM ES IMPACT TIMELINE RESPONSI BILITY WORKSHEET
- 40. More Resources ONLINE RESOURCES http://resultsaccountability.com/ ! http://researchtoaction.org ! http://www.dgmt-growingconfidence.co.za/ ! YOUTUBE TRAINING VIDEOS ! https://www.youtube.com/watch?v=MUSzePBTpAY ! https://www.youtube.com/watch?v=2mBcb-AQ8xQ ! https://www.youtube.com/watch?v=SZqOKW2_jNU ! https://www.youtube.com/watch?v=qmrDNuJzRv4
- 41. More Resources M&E Online Tools ! ALT METRICS http://www.researchtoaction.org/2013/09/how-to-harness-the-power-of-altmetrics-a-researchers-guide/ http://www.researchtoaction.org/2013/08/altmetrics-and-the-global-south-increasing-research-visibility/ ! USING GOOGLE TO TRACK AND IMPROVE YOUR RESEARCH IMPACT http://www.researchtoaction.org/2013/08/using-google-to-track-and-improve-your-research-impact/ http://opendocs.ids.ac.uk/opendocs/handle/123456789/2878#.U8VcXl7yDwJ ! GOOGLE ANALYTICS http://www.researchtoaction.org/2013/11/key-indicators-for-google-analytics/ http://www.google.com/analytics/ ! HOW TO GET STARTED WITH GOOGLE ANALYTICS http://mashable.com/2011/05/23/how-to-use-google-analytics/ ! M&E in Theory and Action ! MONITORING & EVALUATION: SOME TOOLS, METHODS & APPROACHES http://gametlibrary.worldbank.org/FILES/215_M&E%20Guidelines%20on%20Indicators%20and%20Data%20Sources.pdf ! THEORY OF CHANGE VS. LOG FRAMES http://www.researchtoaction.org/2012/05/theory-of-change-vs-logframes/ http://www.knowledgebrokersforum.org/wiki/item/theory-of-change-vs-logframes http://mande4mfi.wordpress.com/category/theory-of-change/ ! CASE STUDIES ON M&E https://sites.google.com/site/newmediacommunications2010/mobiles-and-m-e/2-case-studies ! EXAMPLES OF PROGRAM OUTCOMES www.whatworks.org EVALUATION PLANS http://nonprofitanswerguide.org/faq/evaluation/what-is-an-evaluation-plan-and-why-create-one/
- 43. For further inquiries, please contact: Dr. Modupe Adefeso-Olateju, mo.olateju@tepcentre.com Rebecca O. Doherty, doherty.rebecca@gmail.com THANK YOU