Analysis of Multiple Pilots for ICT-supported Lifelong Competence Development, Davinia Hernández-Leo, davinia.hernandez@upf.edu, TENCompetence Winter School 2009, 1-6 February Innsbruck, Austria

1. Analysis of Multiple Pilots for ICT-supported Lifelong Competence Development Davinia Hernández-Leo [email_address] Winter School 2009, 1-6 February 2009 Innsbruck, Austria http://www.tencompetence.org/node/167

3. Overview Experiences and motivation Research questions Reasons for doing research / possible outcomes Literature review Context of the research /conceptual framework Methods to answer research questions

5. Overview, experimentation in ICT Experiences and motivation Research questions Reasons for doing research / possible outcomes Context of the research /conceptual framework Experimental validation methods Literature review Experimental Methods Observational Historical Controlled

7. Example (Zelkowitz, Wallace, 1998)

11. Multi-pilot evaluation (cycle 1) Comp. dev. for social inclusion Agora LLL in the film industry Digital Cinema Incr. capacity of w.m. professionals UNESCO-IHE Water Management Continuing learning of teachers ICT Teacher Learning ISSUE: See if the TENCompetence concept is feasible and capable of being exploited in a useful manner Cycle 1 LLL in the film industry 1)LD system 2)PCM1.0 Experiment, two groups. Questionnaires Continuing learning of teachers 1) PCM1.0 Experiment, two groups. Questionnaires Better job (performance), prof. setting, incl. workplace, certificate, CoP blended Better job (performance), informal setting, incl. workplace, distance

13. Multi-pilot evaluation (cycle 2) PCM2.0 PDP, LD system, TENTube Web container Better job (performance), share; informal setting, incl. workplace Continuing learning of teacher s PCM2.0 (PDP, LearnWeb2) Better job (performance), prof. setting, incl. workplace, certificate,CoP blended CoP for special education CD Special Education Bulgaria Comp. dev. for social inclusion Agora LLL in the film industry Digital Cinema Incr. capacity of w.m. professionals UNESCO-IHE Water Management LLL in the film industry Comp. dev. for social inclusion Incr. capacity of w.m. professionals Continuing education of teachers ICT Teacher Learning

18. Overview, experimentation in ICT Experiences and motivation Research questions Reasons for doing research / possible outcomes Context of the research /conceptual framework Experimental validation methods Literature review Experimental Methods Data collection instruments Data analysis Observational Historical Controlled Questionnaires Interviews Log files … Qualitative Quantitative

27. Additional slides to be used during the session

31. Data sources for the evaluation of the Agora pilot Records of opinions and observations of what was being perceived in Àgora once the pilot had finished (collected by Àgora staff) Observations post-pilot Qualitative descriptions of the context characteristics in which the pilot is framed Description of the Àgora context TENCompetence server logs, analysis of 512 sessions (a session is considered one usage period of a user from login to logout) Log files Qualitative: experts’ opinions two weeks before the end of the pilot Focus group with experts Qualitative: participants’ opinions two weeks before the end of the pilot Focus group with participants Record of observations (technical issues, about the activities, interactions with experts and other participants, behaviour, other incidents, etc.) The observations were done by 6 different experts (Àgora staff, UPF researchers) Observations during the pilot Quantitative participant characteristics, expectations and evaluation. Questionnaires before (pre-test) and after (post-test) the pilot experience Type of data Data source

32. Agora pilot PDP LD system

40. Data sources for the evaluation of the WM pilot Learners’ outcomes as described by the pilot implementers Final learners’ outcomes Qualitative descriptions of the context characteristics in which the pilot is framed Context TENCompetence server logs, analysis of 4095 sessions (a session is considered one usage period of a user from login to logout) Log files Record of observations by pilot implementers (technical incidents, about the activities, etc.) Observations during the pilot Quantitative participant characteristics, expectations and evaluation Questionnaires before (pre-test) and after (post-test) the pilot experience Type of data Data source

41. Water Management pilot

42. Main results of the WM pilot - Participants were homogeneous : highly educated , relatively young, and for most of them the motivation to join the pilot was job-related (study for a new function or job or improve their current job level, keep up to date within my existing function or job…) - Heterogeneous : from 47 different countries, different number of years of experience in the profession - The participants also recognized that they learnt with respect to the various competence types - Participants used shared blogs for communication purposes and rated it as (very) useful - The majority of the participants were very active in the pilot and more than a half of them were able to complete it 1. The TENC infrastructure can be used in LLL situation involving professionals in the WM area from all over the World with different job-related experience Partial results Findings