1.  WEKA: A Machine
Machine Learning with Learning Toolkit
WEKA  The Explorer
• Classification and
Regression
• Clustering
Bernhard Pfahringer • Association Rules
(based on material by Eibe Frank, Mark • Attribute Selection
Hall, and Peter Reutemann) • Data Visualization
Department of Computer Science,  The Experimenter
University of Waikato, New Zealand  The Knowledge
Flow GUI
 Other Utilities
 Conclusions

2. WEKA: the bird
The Weka or woodhen (Gallirallus australis) is an
endemic bird of New Zealand. (Source: WikiPedia)
Copyright: Martin Kramer (mkramer@wxs.nl)
12/05/12 University of Waikato 2

3. WEKA: the software
 Machine learning/data mining software written in
Java (distributed under the GNU Public License)
 Used for research, education, and applications
 Complements “Data Mining” by Witten & Frank
 Main features:
 Comprehensive set of data pre-processing tools,
learning algorithms and evaluation methods
 Graphical user interfaces (incl. data visualization)
 Environment for comparing learning algorithms
12/05/12 University of Waikato 3

4. History

Project funded by the NZ government since 1993
 Develop state-of-the art workbench of data mining tools
 Explore fielded applications
 Develop new fundamental methods
12/05/12 University of Waikato 4

5. History (2)
 Late 1992 - funding was applied for by Ian Witten
 1993 - development of the interface and infrastructure
 WEKA acronym coined by Geoff Holmes
 WEKA’s file format “ARFF” was created by Andrew Donkin
ARFF was rumored to stand for Andrew’s Ridiculous File Format
 Sometime in 1994 - first internal release of WEKA
 TCL/TK user interface + learning algorithms written mostly in C
 Very much beta software
 Changes for the b1 release included (among others):
“Ambiguous and Unsupported menu commands removed.”
“Crashing processes handled (in most cases :-)”
 October 1996 - first public release: WEKA 2.1
12/05/12 University of Waikato 5

6. History (3)
 July 1997 - WEKA 2.2
 Schemes: 1R, T2, K*, M5, M5Class, IB1-4, FOIL, PEBLS, support
for C5
 Included a facility (based on Unix makefiles) for configuring and
running large scale experiments
 Early 1997 - decision was made to rewrite WEKA in Java
 Originated from code written by Eibe Frank for his PhD
 Originally codenamed JAWS (JAva Weka System)
JA
 May 1998 - WEKA 2.3
 Last release of the TCL/TK-based system
 Mid 1999 - WEKA 3 (100% Java) released
 Version to complement the Data Mining book
 Development version (including GUI)
12/05/12 University of Waikato 6

7. The GUI back then…
TCL/TK interface of Weka 2.1
12/05/12 University of Waikato 7

8. WEKA: versions
 There are several versions of WEKA:
 WEKA 3.4: “book version” compatible with
description in data mining book
 WEKA 3.5.5: “development version” with lots of
improvements
 This talk is based on a nightly snapshot of WEKA
3.5.5 (12-Feb-2007)
12/05/12 University of Waikato 8

9. WEKA only deals with “flat” files
@relation heart-disease-simplified
@attribute age numeric
@attribute sex { female, male}
@attribute chest_pain_type { typ_angina, asympt, non_anginal, atyp_angina}
@attribute cholesterol numeric
@attribute exercise_induced_angina { no, yes}
@attribute class { present, not_present}
@data
63,male,typ_angina,233,no,not_present
67,male,asympt,286,yes,present
67,male,asympt,229,yes,present
38,female,non_anginal,?,no,not_present
...
12/05/12 University of Waikato 9

10. WEKA only deals with “flat” files
@relation heart-disease-simplified
@attribute age numeric
@attribute sex { female, male}
@attribute chest_pain_type { typ_angina, asympt, non_anginal, atyp_angina}
@attribute cholesterol numeric
@attribute exercise_induced_angina { no, yes}
@attribute class { present, not_present}
@data
63,male,typ_angina,233,no,not_present
67,male,asympt,286,yes,present
67,male,asympt,229,yes,present
38,female,non_anginal,?,no,not_present
...
12/05/12 University of Waikato 10

11. java weka.gui.GUIChooser
12/05/12 University of Waikato 11

12. 12/05/12 University of Waikato 12

13. 12/05/12 University of Waikato 13

14. java -jar weka.jar
12/05/12 University of Waikato 14

15. Explorer: pre-processing the data
 Data can be imported from a file in various
formats: ARFF, CSV, C4.5, binary
 Data can also be read from a URL or from an SQL
database (using JDBC)
 Pre-processing tools in WEKA are called “filters”
 WEKA contains filters for:
 Discretization, normalization, resampling, attribute
selection, transforming and combining attributes, …
12/05/12 University of Waikato 15

16. 12/05/12 University of Waikato 16

17. 12/05/12 University of Waikato 17

18. 12/05/12 University of Waikato 18

19. 12/05/12 University of Waikato 19

20. 12/05/12 University of Waikato 20

21. 12/05/12 University of Waikato 21

22. 12/05/12 University of Waikato 22

23. 12/05/12 University of Waikato 23

24. 12/05/12 University of Waikato 24

25. 12/05/12 University of Waikato 25

26. 12/05/12 University of Waikato 26

27. 12/05/12 University of Waikato 27

28. 12/05/12 University of Waikato 28

29. 12/05/12 University of Waikato 29

30. 12/05/12 University of Waikato 30

31. 12/05/12 University of Waikato 31

32. 12/05/12 University of Waikato 32

33. 12/05/12 University of Waikato 33

34. 12/05/12 University of Waikato 34

35. 12/05/12 University of Waikato 35

36. 12/05/12 University of Waikato 36

37. Explorer: building “classifiers”
 Classifiers in WEKA are models for predicting
nominal or numeric quantities
 Implemented learning schemes include:
 Decision trees and lists, instance-based classifiers,
support vector machines, multi-layer perceptrons,
logistic regression, Bayes’ nets, …
 “Meta”-classifiers include:
 Bagging, boosting, stacking, error-correcting output
codes, locally weighted learning, …
12/05/12 University of Waikato 37

38. 12/05/12 University of Waikato 38

39. 12/05/12 University of Waikato 39

40. 12/05/12 University of Waikato 40

41. 12/05/12 University of Waikato 41

42. 12/05/12 University of Waikato 42

43. 12/05/12 University of Waikato 43

44. 12/05/12 University of Waikato 44

45. 12/05/12 University of Waikato 45

46. 12/05/12 University of Waikato 46

47. 12/05/12 University of Waikato 47

48. 12/05/12 University of Waikato 48

49. 12/05/12 University of Waikato 49

50. 12/05/12 University of Waikato 50

51. 12/05/12 University of Waikato 51

52. 12/05/12 University of Waikato 52

53. 12/05/12 University of Waikato 53

54. 12/05/12 University of Waikato 54

55. 12/05/12 University of Waikato 55

56. 12/05/12 University of Waikato 56

57. 12/05/12 University of Waikato 57

58. 12/05/12 University of Waikato 58

59. 12/05/12 University of Waikato 59

60. 12/05/12 University of Waikato 60

61. 12/05/12 University of Waikato 61

62. 12/05/12 University of Waikato 62

63. 12/05/12 University of Waikato 63

64. 12/05/12 University of Waikato 64

65. 12/05/12 University of Waikato 65

66. 12/05/12 University of Waikato 66

67. 12/05/12 University of Waikato 67

68. 12/05/12 University of Waikato 68

69. 12/05/12 University of Waikato 69

70. 12/05/12 University of Waikato 70

71. 12/05/12 University of Waikato 71

72. 12/05/12
QuickTime™ and a T IFF (LZW) decompressor are needed to see this picture. University of Waikato 72

73. 12/05/12
QuickTime™ and a T IFF (LZW) decompressor are needed to see this picture. University of Waikato 73

74. 12/05/12 University of Waikato 74

75. 12/05/12 University of Waikato 75

76. 12/05/12 University of Waikato 76

77. 12/05/12 University of Waikato 77

78. Qui c k Ti m e ™ a nd a TIFF (LZW) d ec om pre s s o r a re n e ed e d to s e e th i s p i c tu re .
12/05/12 University of Waikato 78

79. 12/05/12 University of Waikato 79

80. 12/05/12 University of Waikato 80

81. 12/05/12 University of Waikato 81

82. 12/05/12 University of Waikato 82

83. 12/05/12 University of Waikato 83

84. 12/05/12 University of Waikato 84

85. 12/05/12 University of Waikato 85

86. 12/05/12 University of Waikato 86

87. 12/05/12 University of Waikato 87

88. 12/05/12 University of Waikato 88

89. 12/05/12 University of Waikato 89

90. 12/05/12 University of Waikato 90

91. 12/05/12 University of Waikato 91

92. 12/05/12 University of Waikato 92

93. Explorer: clustering data
 WEKA contains “clusterers” for finding groups of
similar instances in a dataset
 Some implemented schemes are:
 k-Means, EM, Cobweb, X-means, FarthestFirst
 Clusters can be visualized and compared to “true”
clusters (if given)
 Evaluation based on loglikelihood if clustering
scheme produces a probability distribution
12/05/12 University of Waikato 93

94. 12/05/12 University of Waikato 94

95. 12/05/12 University of Waikato 95

96. 12/05/12 University of Waikato 96

97. 12/05/12 University of Waikato 97

98. 12/05/12 University of Waikato 98

99. 12/05/12 University of Waikato 99

100. 12/05/12 University of Waikato 100

101. 12/05/12 University of Waikato 101

102. 12/05/12 University of Waikato 102

103. Explorer: finding associations
 WEKA contains the Apriori algorithm (among
others) for learning association rules
 Works only with discrete data
 Can identify statistical dependencies between
groups of attributes:
 milk, butter ⇒ bread, eggs (with confidence 0.9 and
support 2000)
 Apriori can compute all rules that have a given
minimum support and exceed a given confidence
12/05/12 University of Waikato 103

104. 12/05/12 University of Waikato 104

105. 12/05/12 University of Waikato 105

106. 12/05/12 University of Waikato 106

107. Explorer: attribute selection
 Panel that can be used to investigate which
(subsets of) attributes are the most predictive ones
 Attribute selection methods contain two parts:
 A search method: best-first, forward selection,
random, exhaustive, genetic algorithm, ranking
 An evaluation method: correlation-based, wrapper,
information gain, chi-squared, …
 Very flexible: WEKA allows (almost) arbitrary
combinations of these two
12/05/12 University of Waikato 107

108. 12/05/12 University of Waikato 108

109. 12/05/12 University of Waikato 109

110. 12/05/12 University of Waikato 110

111. 12/05/12 University of Waikato 111

112. 12/05/12 University of Waikato 112

113. 12/05/12 University of Waikato 113

114. Explorer: data visualization
 Visualization very useful in practice: e.g. helps to
determine difficulty of the learning problem
 WEKA can visualize single attributes (1-d) and
pairs of attributes (2-d)
 To do: rotating 3-d visualizations (Xgobi-style)
 Color-coded class values
 “Jitter” option to deal with nominal attributes (and
to detect “hidden” data points)
 “Zoom-in” function
12/05/12 University of Waikato 114

115. 12/05/12 University of Waikato 115

116. 12/05/12 University of Waikato 116

117. 12/05/12 University of Waikato 117

118. 12/05/12 University of Waikato 118

119. 12/05/12 University of Waikato 119

120. 12/05/12 University of Waikato 120

121. 12/05/12 University of Waikato 121

122. 12/05/12 University of Waikato 122

123. 12/05/12 University of Waikato 123

124. 12/05/12 University of Waikato 124

125. Performing experiments
 Experimenter makes it easy to compare the
performance of different learning schemes
 For classification and regression problems
 Results can be written into file or database
 Evaluation options: cross-validation, learning
curve, hold-out
 Can also iterate over different parameter settings
 Significance-testing built in!
12/05/12 University of Waikato 125

126. 12/05/12 University of Waikato 126

127. 12/05/12 University of Waikato 127

128. 12/05/12 University of Waikato 128

129. 12/05/12 University of Waikato 129

130. 12/05/12 University of Waikato 130

131. 12/05/12 University of Waikato 131

132. 12/05/12 University of Waikato 132

133. 12/05/12 University of Waikato 133

134. 12/05/12 University of Waikato 134

135. 12/05/12 University of Waikato 135

136. The Knowledge Flow GUI
 Java-Beans-based interface for setting up and
running machine learning experiments
 Data sources, classifiers, etc. are beans and can
be connected graphically
 Data “flows” through components: e.g.,
“data source” -> “filter” -> “classifier” -> “evaluator”
 Layouts can be saved and loaded again later
 cf. Clementine ™
12/05/12 University of Waikato 136

137. 12/05/12 University of Waikato 137

138. 12/05/12 University of Waikato 138

139. 12/05/12 University of Waikato 139

140. 12/05/12 University of Waikato 140

141. 12/05/12 University of Waikato 141

142. 12/05/12 University of Waikato 142

143. 12/05/12 University of Waikato 143

144. 12/05/12 University of Waikato 144

145. 12/05/12 University of Waikato 145

146. 12/05/12 University of Waikato 146

147. 12/05/12 University of Waikato 147

148. 12/05/12 University of Waikato 148

149. 12/05/12 University of Waikato 149

150. 12/05/12 University of Waikato 150

151. 12/05/12 University of Waikato 151

152. Sourceforge.net – Downloads
12/05/12 University of Waikato 152

153. Sourceforge.net – Web Traffic
WekaDoc Wiki introduced – 12/2005
WekaWiki launched – 05/2005
12/05/12 University of Waikato 153

154. Projects based on WEKA
 45 projects currently (30/01/07) listed on the WekaWiki
 Incorporate/wrap WEKA
 GRB Tool Shed - a tool to aid gamma ray burst research
 YALE - facility for large scale ML experiments
 GATE - NLP workbench with a WEKA interface
 Judge - document clustering and classification
 RWeka - an R interface to Weka
 Extend/modify WEKA
 BioWeka - extension library for knowledge discovery in biology
 WekaMetal - meta learning extension to WEKA
 Weka-Parallel - parallel processing for W EKA
 Grid Weka - grid computing using WEKA
 Weka-CG - computational genetics tool library
12/05/12 University of Waikato 154

155. WEKA and PENTAHO
 Pentaho – The leader in Open Source Business
Intelligence (BI)
 September 2006 – Pentaho acquires the Weka project
(exclusive license and SF.net page)
 Weka will be used/integrated as data mining component in
their BI suite
 Weka will be still available as GPL open source software
 Most likely to evolve 2 editions:
 Community edition
 BI oriented edition
12/05/12 University of Waikato 155

156. Limitations of WEKA
 Traditional algorithms need to have all data in
main memory
 ==> big datasets are an issue
 Solution:
 Incremental schemes
 Stream algorithms
MOA “Massive Online Analysis”
(not only a flightless bird, but also extinct!)
12/05/12 University of Waikato 156

157. Conclusion: try it yourself!
 WEKA is available at
http://www.cs.waikato.ac.nz/ml/weka
 Also has a list of projects based on WEKA
 (probably incomplete list of) WEKA contributors:
Abdelaziz Mahoui, Alexander K. Seewald, Ashraf M. Kibriya, Bernhard
Pfahringer, Brent Martin, Peter Flach, Eibe Frank, Gabi Schmidberger, Ian H.
Witten, J. Lindgren, Janice Boughton, Jason Wells, Len Trigg, Lucio de
Souza Coelho, Malcolm Ware, Mark Hall, Remco Bouckaert, Richard Kirkby,
Shane Butler, Shane Legg, Stuart Inglis, Sylvain Roy, Tony Voyle, Xin Xu,
Yong Wang, Zhihai Wang
12/05/12 University of Waikato 157