5. State of practice:
Techniques for architecture evaluation
Experience 83%
Prototyping 70%
Scenarios 54%
Checklists 40%
Simulation 17%
Questionaires 16%
Metrics 15%
Others 11%
Math. models 5%
[Source: AliBabar & Gorton 2009, survey with 88 architects]
5
6. Research Questions
• RQ1:
Scenario-based methods
& sustainability evaluation?
• RQ2:
What kind of architecture-level metrics?
• RQ3:
Implications for industry & research?
6
7. Search
• 14 conferences, 12 journals, 10 books
• Included only methods/metrics
from renowned venues
• Excluded dormant or unvalidated
methods/metrics
7
8. Research Questions
• RQ1:
Scenario-based methods
& sustainability evaluation?
• RQ2:
What kind of architecture-level metrics?
• RQ3:
Implications for industry & research?
8
12. ATAM ALMA
Change
Sustainability evaluation
scenario spec.
Ripple effect
analysis
Variation point
analysis
Architecture
improvement
Tradeoff
analysis
Existing
artifacts/tools
Return on
Investment
12
13. ATAM ALMA
Change
Sustainability evaluation
Quality attribute Top-down, bottom-up,
scenario spec. scenario template no template
Ripple effect Manual, based on Manual, based on
analysis experience experience
Variation point n/a n/a
analysis
Architecture Modifiability Tactics, n/a
improvement Styles
Tradeoff Utility tree with n/a
analysis stakeholder preferences
Existing No support No support
artifacts/tools
Return on Cost: 30-70 person days Cost: unknown
Investment Benefit: not quantified Benefit: not quantified
13
14. Findings RQ1: Scenarios
• ATAM: more features
• ALMA: more specific for sustainability
• Manual ripple effect analysis
• No reverse engineering
• No tool integration
• Return on Investment unknown
• In practice: use best of both methods
14
15. Research Questions
• RQ1:
Scenario-based methods
& sustainability evaluation?
• RQ2:
What kind of architecture-level metrics?
• RQ3:
Implications for industry & research?
15
23. Findings RQ2
• >40 architecture-level
source code metrics in literature
• Measures for architecture
and modularization quality
• Success factors for application:
– Mix of metrics
– Combination with
class-level / process metrics
• Limitations:
– Empirical validation
– Tool support
23
24. Research Questions
• RQ1:
Scenario-based methods
& sustainability evaluation?
• RQ2:
What kind of architecture-level metrics?
• RQ3:
Implications for industry & research?
24
25. Findings RQ3: Implications
• More empirical research
• Method integration
• Effective formal models
• Codify experiences
• Explore other
approaches (simulation?)
25
26. Conclusions
• Sustainability Evaluation of Architectures
– change scenarios:
basic support for sustainability criteria
– architectural code metrics:
measure modularization, lack empirical evidence
• Research opportunities
– combined methods
– empirical research
26