1. A Quantitative Study of the Contribution of Earned Value Management to Project Success on External Projects Under Contract Robert A. Marshall Ph.D. Thesis Defense August 20, 2007
2. Candidate’s appreciation of the committee Philippe Ruiz, Ph.D., Committee Chairman and Professor of Statistics Christophe Bredillet, Ph.D., Dean’s Representative and Program Director Frank Anbari, Ph.D., Professor of Project Management George Washington University Wayne Abba, MPA, EVM Subject Matter Expert
3. Candidate’s appreciation of the committee cont. Russell Archibald, Ph.D. PMI/APM/IPMA Fellow Darren Dalcher, Ph.D. Professor of Project Management Middlesex University Lynn Crawford, DBA Professor of Project Management Univ. of Technology Sydney
4. Three part outline of the presentation PART A Background, framework, results and contribution PART B Technical aspects of research and data PART C Summary and directions for future research
7. The domain is in need of more research generally “ The area of contracts is one aspect of project management that is in definite need of new ideas and tools.” (Abu-Hijleh and Ibbs, 1989) Only 2.4% of project management conference papers presented in the US and abroad from 1996-1998 were classified as procurement (Zobel & Wearne, 2000) Only 4% of project management research articles printed in English between 1960-1999 were classified as procurement (Kloppenborg & Opfer, 2002)
8.
9. EVM is actively promoted by major project management organizations Project Management Institute (PMI) Association of Project Management (APM) Association for the Advancement of Cost Engineering (AACE) Project Management Association of Japan (PMAJ)
10. Accumulation of evidence indicates a low level of EVM use Less than 1% of projects (Fleming and Koppelman, 1996) “ If such a favorable view (merits of EVM) is so strongly stated, why then is EVM methodology still not used widely as suggested by existing literature? (Kim, 2000) High potential; low usage (Besner and Hobbs, 2006)
17. Variable No. 1: Principles of EVM (ANSI/EIA-748) a. Plan all work scope for the project to completion; b. Decompose the scope into finite pieces that can be assigned to a responsible person or organization to control; c. Integrate program work scope, schedule, and cost objective into a performance measurement baseline plan against which accomplishments may be measured. Control changes to the baseline; d. Use actual costs incurred and recorded in accomplishing the work performed;
18. Variable No. 1: Principles of EVM (ANSI/EIA-748) cont. e. Objectively assess accomplishments at the work performance level; f. Analyze significant variances from the plan, forecast impacts, and prepare an estimate at completion based on performance to date and work to be performed; g. Use EVMS information in the company's management processes.
19. Variable No. 2: Work Breakdown Structure (WBS) with respect to Contract Formation Items a. Scope development (wbs1) b. Risk assessment (wbs2) c. Should cost-estimates (wbs3) d. Schedule planning (wbs4) e. Payment planning (wbs5) f. Evaluating bids and negotiating with bidders (wbs6) g. Ensuring fairness and equity to contract (wbs7)
20. Variable No. 2: “S” Curves with respect to a single Contract Administration item a. Monitoring work (scurve)
21. Variable No. 2: EVM Performance metrics with respect to multiple Contract Administration items a. Controlling schedule (metric1) b. Controlling scope (metric2) c. Controlling cost (metric3) d. Evaluating and processing change orders (metric4) e. Evaluating and processing payment requests (metric5)
22. Variable No. 2: EVM Performance metrics with respect to multiple Contract Administration items cont. f. Evaluation and analyzing delays (metric6) g. Evaluating and analyzing claims (metric7) h. Acceptance of completed work (metric8) i. Contract close-out (release of claims) (metric9) j. Post-project audits (metric10)
23. Variable No. 3: Project Success (Shenhar et al., 1997) a. Meeting Schedule and Budget Goals - how well the project satisfied the resource constraints of schedule and budget. b. Benefit to Customer – how well the project satisfied the functional requirements and technical specifications of the organization that benefit from the project’s realization.
24. Variable No. 3: Project Success (Shenhar et al., 1997) cont. c. Commercial Success. How well the project satisfies the desired business results (includes return on investment for performing organization; or enhancing the profits of the benefiting organization). d. Preparing for the Future - The contribution the project made to the performing organization in developing new business opportunities; technologies or increasing core competencies.
27. Research hypotheses based on low EVM usage and belief that a bias may exist – H3, H4 and H5 H3: There will be no significant difference between Fixed-price and Cost-plus contracts on the contributions of EVM’s Work breakdown Structure during Contract Formation H4: There will be no significant difference between Fixed-price and Cost-plus contracts on the contribution of EVM’s "S” Curve during Contract Administration H5: There will be no significant difference between Fixed-price and Cost-plus contracts on the use of EVM’s Performance Metrics during Contract Administration
28. Findings: H1 Results offered support for H1. Linear correlation-regression showed a moderately strong relationship between the independent variable Principles of EVM and dependent variable Project Success ( r = .474; β = .474, t = 6.443, n =145). The results were significant setting p at p < .05 ( a = .000). The independent variable explains 22.47% of the variation in the dependent variable (R² = .2247). As predicted, Principles of EVM are significant positive predictors of Project Success
30. Findings: H2 Results did not offer support for H2. Contrary to the prediction, a difference exists between fixed-price and cost-plus contracted projects with respect to the contribution of Principles of EVM to Project Success , measured separately. The results favor fixed-price contracted projects. Linear correlation-regression showed a moderately strong positive relationship between the independent variable Principles of EVM and the dependent variable Project Success for the cost-plus group ( r = .405; β = .405, t = 4.432, n =102); and a moderately strong and relatively higher positive relationship for the fixed-price group ( r = .585; β = .585, t = 4.622, n =43). The independent variable explains 16.40% of the variation in the dependent variable in cost-plus contracted projects; and 34.22% in fixed-price contracted projects. The results are statistically significant setting p at p < .05 ( a = .000), separately
31. Findings: H3 Results offered support for H3, with exceptions. As predicted, there are no significant differences between fixed-price and cost-plus contracted projects on the contributions of EVM’s Work Breakdown Structure to Contract Formation items with the exception of wbs4 (Schedule Planning) and wbs5 (Payment Planning) (Wilks’ λ = .936, χ² = 9.389, df2, p = .009)(eigenvalue = .068; canonical correlation = .253). Five items were removed leaving a single canonical root with two discriminant coefficients (wbs4 and wbs5). No significant differences between fixed-price and cost-plus contracts were observed with respect to the contribution of EVM’s WBS to Contract Formation items for wbs1, 2, 3, 6, and 7
32. Two canonical coefficients distinguish fixed-price from cost-plus contracted projects with respect to the use of the WBS The contribution of wbs5 (Payment Planning) was relatively greater in FP as compared to CP contracted projects The contribution of wbs4 (Schedule planning) was relatively greater in CP as compared to FP contracted projects
33. Findings: H4 Results offered support for H4. As predicted, there is no significant difference between fixed-price and cost-plus contracted projects on the contribution of EVM’s "S” Curve to Contract Administration items (single item included in H5 analysis)
34. Findings: H5 Results offered support for H5, with exceptions. The results show no significant difference between fixed-price and cost-plus contracted projects on the contributions of EVM’s Metrics to Contract Administration (Wilks’ λ = .943, χ² = 8.434, df1, p = .004) (eigenvalue = .061; canonical correlation = .240), with the exception of metric5 ( Evaluating and Processing Payment Requests ). A total of ten items were removed during the analysis leaving a single canonical root with one discriminant coefficient (metric5). No significant differences between fixed-price and cost-plus contracts were observed with respect to the contribution of EVM’s Metrics to Contract Administration items for scurve, metric1, 2, 3, 4, 6, 7, 8, 9, and 10
35. A single canonical coefficient distinguishes fixed-price from cost-plus contracted projects with respect to the use of the S-curve and EVM performance metrics The contribution of metric5 ( Evaluating and processing payment requests ) was relatively greater in FP as compared to CP contracted projects
36. Multiple implications stem from this research EVM directly and measurable contributes to project success EVM is dynamic (not a checklist) EVM alone does not explain project success Both CP and FP contract types benefit from EVM EVM optimized in FP contracts
37. Multiple implications stem from this research cont. EVM planning is as important as EVM control Emphasize WBS schedule planning in CP projects Emphasize WBS payment planning in FP projects Emphasize metrics in processing payment requests in FP projects
38. This research increases the body of knowledge regarding project procurement management Five publications to date have stemmed from this research which have reached a global audience
39. Related publishing's Marshall, Robert A. (2005). “The Case for Earned Value Management with Fixed Price Contracts.” C ontract Management Magazine. Marshall, Robert A. (2006). “The contribution of earned value management to project success on contracted efforts: A quantitative statistics approach within the population of experienced practitioners.” Proceedings of the International Research Conference on Organizing by Projects (IRNOP7). Marshall, Robert A. (Editor, 2nd Ed.). (2007). Professional Practice Guide to Earned Value. Association for the Advancement of Cost Engineering International. AACE: Morgantown, WV.
40. Related publishing's cont. Marshall, Robert A. (2007). “The Principles of Earned Value Management as Predictor Variables of Project Success.” Transactions o f the 51st Annual International Meeting of the Association for the Advancement of Cost Engineering (AACE). Morgantown, WV. Marshall, Robert A. (2007). “The contribution of earned value management to project success on contracted efforts: A quantitative statistics approach within the population of experienced practitioners.” Journal of Contract Management. (September Issue)
41. This research offers nine specific contributions The unique (and repeatable) testing of the ANSI-748 principles of EVM as predictor variables of project success The unique inference that fixed-price contracted projects benefit as much as, if not more, than cost-plus contracts when applying EVM Unique quantitative approach with respect to the contribution of the principles of EVM to project success which solidify beliefs through methodological triangulation
42. This research offers nine specific contributions cont. The unique inference that the contribution of the mechanics of EVM (WBS, S-curve and metrics) do not significantly differ between fixed-price contracted projects and cost-plus contracted projects (with three exceptions noted below) The unique inference that the contribution of the WBS to payment planning is greater in fixed-price contracted projects as compared to cost-plus contracts The unique inference that the contribution of the WBS in schedule planning is greater in cost-plus contracted projects as compared to fixed-price contracts
43. This research offers nine specific contributions cont. The unique inference that the contribution of the EVM metrics to evaluating and processing payment requests is relatively greater in fixed-price contracted projects as compared to cost-plus contracted projects The mechanics of EVM contribute as much to project planning (WBS) as they do to control (S-curve and metrics)
44. The results are timely and relevant EVM shown to have high potential; low usage (Besner and Hobbs, 2006) Schedule management techniques are more important to high-uncertainty projects (Shenhar, et, 2002) “ Schedule risk is often under appreciated for its contribution to driving contract performance and cost overruns” (U.S. DoD EVM Implementation Guide, 2005) Budget and scope management techniques more important with projects with low uncertainty (Shenhar, et al, 2002)
46. Technical aspects of research and data Pre-survey tasks Research approach and methodology determination Survey instrument development Data collection strategy determination and effort Post-survey tasks Survey instrument testing Sample testing for social desirability of group responses Data testing for reliability and assumptions Data treatments Statistical analysis
47. Research approach selected was quantitative Appropriate for testing hypotheses Lends itself to relatively short duration studies Uses known measurement and analytic techniques Needed representative sample available Compliments existing research and strengthens knowledge through methodological triangulation
48. Three primary statistical methods were used Pearson’s product-moment correlation - r Bivariate linear regression – β Canonical discriminant analysis - Wilks’ λ
49. Survey instrument developed for ease of use and analysis Thirty-eight (38) questions (plus MC-10 battery) Likert-type semantic differential scaling Interval level scale for continuous data Available on-line (July 6, 2005 - March 10, 2006) Approx. 7-10 minutes to take
50. Data collection strategy included attributes of random and purposive sampling Deliberate sampling frame included four subgroups with requisite experience DAU EVM Community PMI CPM APM EVM SIG AACE EVM SIG Approximate sampling frame size of 2500 Participants solicited via email without bias on two separate occasions
51. Survey data attributes 148 of 256 respondents completed survey - 57.8% 145 of potentially 2500 samples used - 5.8% Three outlier data points (residual values > 3 σ removed) Less than one percent missing values ( μ - imputation) Residuals displayed normal homoscedasticity and normal linear relationship to expected value
52. Survey data predominantly public projects, defense and information technology Industry 3 2.1 2.1 2 1.4 3.4 19 13.1 16.6 67 46.2 62.8 16 11.0 73.8 10 6.9 80.7 15 10.3 91.0 7 4.8 95.9 6 4.1 100.0 145 100.0 Description Heavy construction Light construction (residential and commercial) Government/public works Military/defense Information technology Professional services (banking, consulting, management, etc.) Aerospace Software development Other Total Frequency Percent Cum. Percent
53. Independent samples t-test of MC-10 social desirability data indicating no significant different in scores about CR and FP groups Independent Samples Test 2.597 .109 .520 143 .604 .20565 .39546 -.57606 .98736 .487 68.9 .628 .20565 .42217 -.63658 1.048 Assumptions Equal variances assumed Equal variances not assumed Dependent variable Contract type F Sig. Levene's test for equality of variances t df Sig. (2-tailed) M Difference SE Difference Lower Upper 95% CI of the Difference t-test for equality of means
65. Directions for future research Qualitative or quantitative study as they relate to EVM or are otherwise affected by the core project management processes of initiating or closing The depth to which scope should be defined. The question yet to be answered regarding EVM is whether an optimal level of WBS exists. Second, how the utility curve for varying degrees of WBS decomposition might look
66. Directions for future research cont. Greater stratification of contract types as categorical variables with respect to the principles of EVM and project success Exploration of variables and dimensions not covered by this study Predictive ability of EVM principles for the specific achievement of project financial objectives
