1. Machine Learning and Inductive Inference Hendrik Blockeel 2001-2002

11. Example: given molecules that are active against some disease, find out what is common in them; this is probably the reason for their activity.

21. Overview of design choices type of training experience games against self games against expert table of good moves determine type of target function determine representation determine learning algorithm … … … … ready! Board   Board  Move linear function of 6 features … gradient descent

52. + G S Current versionspace contains all rectangles covering S and covered by G, e.g. h = <2-5,2-3> h S = {<3-5,2-3>} G = {<1-6, 1-3>}

56. Equivalence between inductive and deductive systems inductive system training examples new instance deductive system training examples new instance inductive bias result (by proof) result (by inductive leap)

71. Clustering tree abundance(Tubifex sp.,5) ? T = 0.357111 pH = -0.496808 cond = 1.23151 O2 = -1.09279 O2sat = -1.04837 CO2 = 0.893152 hard = 0.988909 NO2 = 0.54731 NO3 = 0.426773 NH4 = 1.11263 PO4 = 0.875459 Cl = 0.86275 SiO2 = 0.997237 KMnO4 = 1.29711 K2Cr2O7 = 0.97025 BOD = 0.67012 abundance(Sphaerotilus natans,5) ? yes no T = 0.0129737 pH = -0.536434 cond = 0.914569 O2 = -0.810187 O2sat = -0.848571 CO2 = 0.443103 hard = 0.806137 NO2 = 0.4151 NO3 = -0.0847706 NH4 = 0.536927 PO4 = 0.442398 Cl = 0.668979 SiO2 = 0.291415 KMnO4 = 1.08462 K2Cr2O7 = 0.850733 BOD = 0.651707 yes no abundance( ...) <- &quot;standardized&quot; values (how many standard deviations above mean)

83. Overfitting: example + + + + + + + - - - - - - - - - - - - - - + - - - - - area with probably wrong predictions

88. accuracy on training data accuracy on unseen data size of tree accuracy effect of pruning

91. Rules from trees: example Outlook Humidity Wind No Yes No Yes Yes Sunny Overcast Rainy High Normal Strong Weak if Outlook = Sunny and Humidity = High then No if Outlook = Sunny and Humidity = Normal then Yes …

93. Pruning rules: example A false true B true true false true false if A=true then true if A=false and B=true then true if A=false and B=false then false Tree representing A  B Rules represent A  (  A  B) A  B

97. Heuristics compared Good heuristics are strictly concave

98. Why concave functions? E E 1 E 2 p p 2 p 1 Assume node with size n , entropy E and proportion of positives p is split into 2 nodes with n 1 , E 1 , p 1 and n 2 , E 2 p 2 . We have p = (n 1 /n)p 1 + (n 2 /n) p 2 and the new average entropy E’ = (n 1 /n)E 1 +(n 2 /n)E 2 is therefore found by linear interpolation between ( p 1 ,E 1 ) and ( p 2 ,E 2 ) at p . Gain = difference in height between ( p, E ) and ( p,E’ ). (n 1 /n)E 1 +(n 2 /n)E 2 Gain

100. Generic TDIDT algorithm function TDIDT( E : set of examples) returns tree; T' := grow_tree( E ); T := prune ( T' ); return T ; function grow_tree( E : set of examples) returns tree; T := generate_tests ( E ); t := best_test ( T , E ); P := partition induced on E by t ; if stop_criterion ( E , P ) then return leaf( info ( E )) else for all E j in P: t j := grow_tree( E j ); return node( t , {( j,t j )}; 

105. Clustering tree abundance(Tubifex sp.,5) ? T = 0.357111 pH = -0.496808 cond = 1.23151 O2 = -1.09279 O2sat = -1.04837 CO2 = 0.893152 hard = 0.988909 NO2 = 0.54731 NO3 = 0.426773 NH4 = 1.11263 PO4 = 0.875459 Cl = 0.86275 SiO2 = 0.997237 KMnO4 = 1.29711 K2Cr2O7 = 0.97025 BOD = 0.67012 abundance(Sphaerotilus natans,5) ? yes no T = 0.0129737 pH = -0.536434 cond = 0.914569 O2 = -0.810187 O2sat = -0.848571 CO2 = 0.443103 hard = 0.806137 NO2 = 0.4151 NO3 = -0.0847706 NH4 = 0.536927 PO4 = 0.442398 Cl = 0.668979 SiO2 = 0.291415 KMnO4 = 1.08462 K2Cr2O7 = 0.850733 BOD = 0.651707 yes no abundance( ...) <- &quot;standardized&quot; values (how many standard deviations above mean)