Multi-criteria Decision Analysis for Customization of Estimation by Analogy Method
1. Multi-criteria Decision Analysis for Customization of Estimation by Analogy Method AQUA + Jingzhou Li Guenther Ruhe University of Calgary, Canada PROMISE’08, May 13, 2008
2.
3. /14 Prediction accuracy distribution 1. Proposed EBA method AQUA + —Architecture Data set for AQUA + AQUA Existing EBA Predicting Phase2 Effort estimates Objects under estimation Learning Phase1
4.
5. 2. Decision-centric process model of EBA /14 Processed Historical Data Raw Historical Data D8. Determining closest analogs D2. Dealing with missing values D1. Impact analysis of missing values D7. Retrieving analogs Objects Under Estimation Effort Estimates D9. Analogy adaptation D11. Comparing EBA methods in general D10. Choosing evaluation criteria D6. Determining similarity measures D3. Object selection D5. Attribute weighting & selection D4. Discretization of attributes
6. /14 EBA ( DB ) = C ( D , DB , Ch ) Data set type 1 Data set type 2 Data set type k …… Classification according to characteristics of the data sets S i.j for D i ? 3. Customization of EBA — why? D = { D 1 , D 2 , …, D 11 }, D i = { S i.j | solution alternatives of task D i } DB: a historical data set for EBA Ch: a set of characteristics describing DB Customization 1 Customization 2 Customization k
7. /14 New Data Set Which heuristic should be used? 4. Customization of EBA — how? Empirical knowledge gained from empirical studies.
8.
9.
10. /14 7. Data sets Data Sets #Objects #Attributes %Missing Values %Non-Quantitative Attributes Source USP05-RQ 121 14 2.54 71 Li et al., 2005 USP05-FT 76 14 6.8 71 Li et al., 2005 ISBSG04-2 158 24 27.24 63 ISBSG, 2004 Kem87 15 5 0 40 Kemerer et al., 1987 Mends03 34 6 0 0 Mendes et al., 2003 Desh89 81 10 0.006 20 Shepperd et al., 1997
11. /14 8. Decision Analysis Using ELECTRE Outranking graph and analysis data for Desh89 (an example) Heuristic MMRE Pred(0.25) H0 0.62 0.44 H1 0.61 0.44 H3 0.6 0.42 H4 0.59 0.42 CfsSubset (Cfs) 0.52 0.4 Wrapper (Wp) 0.66 0.43
Slide 11: 4 Heuristics as a header for the lower Introduce a simplified formulae exlaing how the coefficients were calculated. - Be prepared for the following questions: (1) Are there alternatives to using RSA to determine the importance of the attributes? (2) What is the overall effort of the method(s) (3) Wham means the name AQUA? (4) When do you recommend apply the method? (and when better not?) (5) Needs the learning be done after each new prediction (data point)??