Keynote Young Researcher's Workshop at ITK07, Finland Abstract: Online settings provide non-reactive ways to observe learners' activities. While the learning itself is not directly visible for example in a logfile or within the history of a Wiki, the growing expressiveness in online learning environments takes us a step nearer to witness and analyse learning processes. In this presentation four questions will be discussed: (1) What is the relation between possible user activities in recent E-Learning environments and actual learning processes? (2) What data is available to observe the users' activities? (3) What data analysis methods are fitting to model these activities? (4) What research designs do we need to get more insight into learning processes? As an example, some ideas, experiences and results from an ongoing research project about online user participation and motivation will be presented and discussed, focussing on a multimethod approach.

1. Online Learning Processes –
What Users Do and Why it is Important
Karsten D. Wolf
Didactical Design of Interactive Learning Environments
interaktiivinen tekniikka koulutuksessa 2007
Wednesday, 18.04.2007
cc by Karsten D. Wolf 2007
1

2. Karsten D. Wolf
Faculty educational science
Didactical design of
interactive learning environments
2

3. www.everlearn.eu
User Generated Content
Didactical Prototyping
Collaborative Learning
cc by Karsten D. Wolf 2007
3

4. 4

5. roadmap
1. Why educational research is hard
2. How can we produce evidence in educational research?
3. Multimethod approaches in educational research
4. Alternative methods: Intra-individual analysis of micro process timeseries
5. Educational technology as a research frontier in learning science
6. Some applied results
© Karsten D. Wolf 2007
5

6. soft sciences are hard
cc by Karsten D. Wolf 2007
6

7. cc by Karsten D. Wolf 2007
7

8. btw: this is not a real sim.
rocks do not decelerate before hitting the ground!
cc by Karsten D. Wolf 2007
8

9. what if rocks would behave?
cc by Karsten D. Wolf 2007
9

10. cc by Karsten D. Wolf 2007
10

11. covariates hell
pic by Don Gato onWolf 2007
cc by Karsten D. Flickr
11

12. 3 strategies
control them
live with them
r&d approach
Fischer/Waibel/Wecker (2006):
Nutzenorientierte Grundlagenforschung im Bildungsbereich
Argumente einer internationalen Diskussion
cc by Karsten D. Wolf 2007
12

13. experimental research
design based research
engineering approaches
cc by Karsten D. Wolf 2007
13

14. cc by Karsten D. Wolf 2007
14

15. 75%
cc by Karsten D. Wolf 2007
15

16. Educational Researcher, Vol. 32, No. 9, 3–14
Educational Researcher, Vol. 33, No. 2, 3–11
cc by Karsten D. Wolf 2007
16

17. Problems with quasi-experiments
n
o
s
p
im ox
Sd
a
r
a
p
Steyer 2007 - www.causal-effects.de cc by Karsten D. Wolf 2007
17

18. Problems with quasi-experiments
Steyer 2007 - www.causal-effects.de cc by Karsten D. Wolf 2007
18

19. Problems with quasi-experiments
Steyer 2007 - www.causal-effects.de cc by Karsten D. Wolf 2007
19

20. Random Assignment & Propensity Score Matching
• Random Assignment
• is the population large enough?
• can the assignment truly random?
• doesn‘t it feel like a lottery?
• Propensity Score Matching
• need to collect covariates
• matching at an individual level between treatment and control
© Karsten D. Wolf 2007
20

21. Multimethod approaches: assessment methods
• self-reports (behaviors, psychological contructs, forms of behaviors)
• informant assessment
• ecological momentary assessment
• web based / computer based methods
• ability tests
• implicit methods (performance measures)
• sequential observational methods
© Karsten D. Wolf 2007
21

22. Multimethod approaches:assessment methods
• quantitative text analysis
• multilevel analysis: physiological & biochemical measures
• brain imaging
• experimental methods
• nonreactive methods
© Karsten D. Wolf 2007
22

23. Experiments vs. Assessments
Why is Explanandum?
If (A1 and A2) then Explanandum (general law, explanans)
A1 (antecedent condition 1)
A2 (antecedent condition 2)
Therefore: E (explanandum)
Experiments aim to develop and test general laws
Assessments aim to show that the antecedent conditions are met
Erdfelder & Musch 2006
© Karsten D. Wolf 2007
23

24. Level of nonreactivity
Low High
Type 0 Type 1 Type2 Type 3 Type 4 Type 5
Setting initially designed
yes yes yes yes yes no
or selected for research
Participants are aware
likely likely likely likely no no
of the research setting
Participants are aware of the
likely likely likely no no no
research question
Participants are aware of the
likely likely no no no no
research hypothesis
Participants are aware of the
likely no no no no no
measures‘ manipulability
Participative Bogus pipeline Personality Cover story Lost letter Analysis of
Examples expert interview technique questionaire experiments technique archival data
Fritsche and Linneweber 2006
© Karsten D. Wolf 2007
24

25. Methods for analyzing multimethod data
• Multitrait-Multimethod Matrix (Campbell and Fiske 1959)
• categorical variables analysis for asessing multimethod association
• multimethod item response theory
• multilevel analysis
• structural equation models
• longitudinal methods
© Karsten D. Wolf 2007
25

26. cc by Karsten D. Wolf 2007
26

27. 27

28. 28

29. 29

30. 30

31. 31

32. 32

33. I feel taken seriously (EG1 TN1 SoLe III)
33

34. 1 2 3 4 5 6 7 8 9 10 11 12 13 14
34

35. Taken seriously Cognitive Load Participation
I feel good I understand I am interested
35

36. I am interested for19 students in EG1 SoLe III
36

37. I feel taken seriously (EG1 TN1 SoLe III)
37

38. Trend
I feel taken seriously (EG1 TN1 SoLe III)
Linear Trend
38

39. Trend
I feel taken seriously (EG1 TN1 SoLe III)
3rd Polynom
39

40. Trend
I feel taken seriously (EG1 TN1 SoLe III)
Spline λ=10.000
40

41. Trend
I feel taken seriously (EG1 TN1 SoLe III)
Spline λ=100.000
41

42. Decomposition
I feel taken seriously (EG1 TN1 SoLe III)
Trend Microprocesses
42

43. ARMA-Analysis
43

44. Model parameters
R2 Φ
AIC BIC
Φ1 = 0,307
AR(1) 1201,8 1208,0 0,094
Φ1 = 0,238
AR(2) 1190,2 1200,9 0,139
Φ2 = 0,223
Φ1 = 0,237
Φ2 = 0,222
AR(3) 1192,1 1206,5 0,139
Φ3 = 0,007
Φ1 = 0,236
Φ2 = 0,241
Φ4 =
AR(5) -0,077
1194,2 1215,8 0,145
Φ5 = -0,018
Φ3 = 0,030
44

45. AR(2)-Model
45

46. Multivariate Analyis:
Cross Correlations
0,120
I I 0,122
FG
P
-0.143
0.146
CL
t-3 t-2 t-1 t
46

47. Multivariate Models:
GMDH
I 0,097
FG
P
-0.15
0.13
CL
t-2 t-1 t
47

48. Fuzzy Modelling (Zadeh)
“As the complexity of a system increases, our
ability to make precise and significant
statement about its behaviour diminishe until
a threshold is reached … the closer one look
at a real-world problem the fuzzier becomes
its solution” (Kosko, 94)
48

49. Fuzzification
1
0,5
0
Negative_X ZERO_X Positive_X
49

50. Fuzzy Rule Induction
IF
TakenSeriously (t-1)
NOT_Participation (t-1)
& NOT_Participation (t-2)
& TakenSeriously (t-2)
OR OR
NOT_Participation (t-1) NOT_Understand (t-1)
& CognitiveLoad (t-2) & CognitiveLoad (t-2)
THEN FeelGood TN1 (prewhitened)
50

51. Advantages?
Better communication?
Different interrelationships at different
levels? (Thresholds)
non-linear models?
basis for building hypotheses?
51

52. disadvantages
time consuming
lot of rules
no selection criteria
no „test of significance“
52

53. Online learning
as a research frontier?
cc by Karsten D. Wolf 2007
53

54. cc by Karsten D. Wolf 2007
54

55. cc by Karsten D. Wolf 2007
55

56. how to measure this?
what is happening here?
cc by Karsten D. Wolf 2007
56

57. ???
cc by Karsten D. Wolf 2007
57

58. ???
Learning accompanying actions
Browse
Search
Write
…
cc by Karsten D. Wolf 2007
58

59. ???
creation
cc by Karsten D. Wolf 2007
59

60. ???
construction
cc by Karsten D. Wolf 2007
60

61. ???
communication
cc by Karsten D. Wolf 2007
61

62. ???
cooperation
cc by Karsten D. Wolf 2007
62

63. ???
collaboration
cc by Karsten D. Wolf 2007
63

64. Some results from a prior online seminar
ZfE Cover
© Prof. Dr. Karsten D. Wolf 2007
64

65. Activity
# forum # created
# chat msg
entries words
female 46.2 7.0 2301
n=23 (57.92) (7.44) (363.5)
male 24.7 3.0 1919
n=13 (23.43) (3.08) (419.0)
1-tail significant test, n=36
© Prof. Dr. Karsten D. Wolf 2007
65

66. Activity and Post-Interest
# forum # created
# chat msg
entries words
Post- .11 .19 .34*
interest (.258) (.135) (.022)
1-tail significant test, n=36
© Prof. Dr. Karsten D. Wolf 2007
66

67. Pre-Interest and Activity
# forum # created
# chat msg
entries words
.30* .28 .36*
Pre-interest
(.043) (.053) (.019)
1-tail significant test, n=36
© Prof. Dr. Karsten D. Wolf 2007
67

68. Activity and Post-Interest
# forum # created
# chat msg
fact
entries words
by p ors ou
re-in .34* t
t(.022)
eres
Post- .11 .19
t!
interest (.258) (.135)
1-tail significant test, n=36
© Prof. Dr. Karsten D. Wolf 2007
68

69. Activity and Social Relatedness
# forum # created
# chat msg
entries words
social -.08 .30* .31*
relatedness (.319) (.036) (.035)
1-tail significant test, n=36
© Prof. Dr. Karsten D. Wolf 2007
69

70. Pre-Knowledge and Activity
# forum # created
# chat msg
entries words
pre-
.34* .47** .54***
problem
(.021) (.002) (<.001)
solving
1-tail significant test, n=36
© Prof. Dr. Karsten D. Wolf 2007
70

71. Activity and Knowledge
# forum # created
# chat msg
entries words
post-
.03 .28* .43**
problem
(.425) (.048) (.005)
solving
1-tail significant test, n=36
© Prof. Dr. Karsten D. Wolf 2007
71

72. Which means: constructivism is right after all!
© Karsten D. Wolf 2007
72

73. © Prof. Dr. Karsten D. Wolf 2005
http://www.karsten-d-wolf.de/
wolf@uni-bremen.de
73