PEnDAR webinar 1 with notes
•Descargar como PPTX, PDF•
1 recomendación•2,515 vistas
First webinar from the PEnDAR project, outlining the distributed system design challenge, and begin to investigate the application of advanced system engineering techniques for all phases of system design to ensure that critical cost and performance targets can be met. The goal is to make expensive performance failures and cost overruns a thing of the past!
Recomendados
Recomendados
Más contenido relacionado
La actualidad más candente
La actualidad más candente (16)
Destacado
Similar a PEnDAR webinar 1 with notes
Similar a PEnDAR webinar 1 with notes (20)
Último
Último (20)
PEnDAR webinar 1 with notes
- 1. © Predictable Network Solutions Ltd 2017 www.pnsol.com PEnDAR Focus Group Performance Ensurance by Design, Assuring Requirements First Webinar
- 2. © Predictable Network Solutions Ltd 2017 www.pnsol.com Click to edit Master title style 2 Goals • Consider how to enable Validation & Verification of cost and performance • For distributed and hierarchical systems • via sophisticated but easy-to-use tools • Supporting both initial and ongoing incremental development • Provide early visibility of cost/performance hazards • Avoiding costly failures • Maximising the chances of successful in-budget delivery of acceptable end-user outcomes Partners Supported by: PEnDAR project
- 3. © Predictable Network Solutions Ltd 2017 www.pnsol.com Click to edit Master title style 3 Our offer to you Learn what we know about the laws of the distributed world: • Where the realm of the possible lies; • How to push the envelope in a way that benefits you most; • How to innovate without getting your fingers burnt. Get an insight into the future: • Understand more about achieving sustainability and viability; • See where competition may lie in the future. Our ‘ask’ Help us to find out: • In which markets/sectors/application domains will the benefits outweigh the costs? • Which are the major benefits? • Which are just ‘nice to have’? • How could this methodology be consumed? • Tools? • Services? Role of the focus group
- 4. © Predictable Network Solutions Ltd 2017 www.pnsol.com The system delivery challenge
- 5. © Predictable Network Solutions Ltd 2017© Predictable Network Solutions Ltd 2017 www.pnsol.com 5 System delivery challenge System Requirements Time pressure Complexity Cost/resource constraints Forces hierarchical decomposition Forces parallel development Change during development Subcontracted subsystems Use of COTs componentsVague/contradictory Leave ‘hard’ problems to the end Existing asset re-use constraints
- 6. © Predictable Network Solutions Ltd 2017 www.pnsol.com Click to edit Master title style 6 How to specify and verify subsystem requirements/acce ptance criteria Functional outcomes Performance outcomes Resource constraints MEET Risks MANAGE Peak Loads Integration testing Rework MINIMISE Post-deployment scaling System delivery challenge
- 7. © Predictable Network Solutions Ltd 2017© Predictable Network Solutions Ltd 2017 www.pnsol.com 7 Constrained performance hazards Unconstrained performance hazards Distributed Centralised Dedicated Shared Resources Structure Traditional avionics Cloud apps Traditional mainframe Standalone microcontrollers Modern avionics Client-server systems Established approaches running out of steam?
- 8. © Predictable Network Solutions Ltd 2017 www.pnsol.com Dimensions of the challenge
- 9. © Predictable Network Solutions Ltd 2017© Predictable Network Solutions Ltd 2017 www.pnsol.com 9 Outcomes Delivered Resources Consumed Variability Exception /failure Externally created Mitigation System impact Propagation Scale Schedulability Capacity Distance Number Time Space Density
- 10. © Predictable Network Solutions Ltd 2017© Predictable Network Solutions Ltd 2017 www.pnsol.com 10 Outcomes Delivered Resources Consumed Variability Exception /failure Scale
- 11. © Predictable Network Solutions Ltd 2017© Predictable Network Solutions Ltd 2017 www.pnsol.com 11 Outcomes Delivered Resources Consumed Variability Exception /failure Scale Distance Number Time Space Schedulability Capacity Density
- 12. © Predictable Network Solutions Ltd 2017© Predictable Network Solutions Ltd 2017 www.pnsol.com 12 Outcomes Delivered Resources Consumed Variability Exception /failure Mitigation Propagation Scale Schedulability Capacity Distance Number Time Space Density
- 13. © Predictable Network Solutions Ltd 2017© Predictable Network Solutions Ltd 2017 www.pnsol.com 13 Outcomes Delivered Resources Consumed Variability Exception /failure Externally created Mitigation Propagation Scale Schedulability Capacity Distance Number Time Space Density
- 14. © Predictable Network Solutions Ltd 2017© Predictable Network Solutions Ltd 2017 www.pnsol.com 14 Outcomes Delivered Resources Consumed Variability Exception /failure Externally created Mitigation System impact Propagation Scale Schedulability Capacity Distance Number Time Space Density
- 15. © Predictable Network Solutions Ltd 2017© Predictable Network Solutions Ltd 2017 www.pnsol.com 15 Outcomes Delivered Resources Consumed Variability Exception /failure Externally created Mitigation System impact Propagation Scale Schedulability Capacity Distance Number Time Space Density
- 16. © Predictable Network Solutions Ltd 2017 www.pnsol.com Practical methodology
- 17. © Predictable Network Solutions Ltd 2017 www.pnsol.com Click to edit Master title style 17 Support stages of the SDLC • Design • Feasibility analysis • Hierarchical decomposition • Subsystem acceptance criteria • Verification • Checking delivery of quantified outcomes • Evaluating resource usage • Re-verification during system lifetime • Validation • Quantification of performance criteria • Checking coverage and consistency Quantify hazards • Failure to meet outcome requirements • Physical constraints • Schedulability constraints • Supply chain constraints • Failure to meet resource constraints • Scaling • Correlations Interaction with System Development Life Cycle
- 18. © Predictable Network Solutions Ltd 2017 18 www.pnsol.com Quantifying intent • The key challenge is to establish quantified intentions • For outcomes/resources/costs • Then a variety of mathematical techniques can be applied • queuing theory • large deviation theory • ∆Q algebra • This is “only rocket science” • Not brain surgery!
- 19. © Predictable Network Solutions Ltd 2017 19 www.pnsol.com POLL Imagine two technical scenarios: • In the first, a client asks for “a replacement for its phone system”. • In the second, a client asks for “a VoIP system with an average MOS of 4.2, the ability to take 30 concurrent calls, a failover system for when it hits capacity (e.g. voicemail/recorded message), first priority for emergency calls and second priority for current calls”. If the first scenario is 1 and the second is 5, where on a scale of 1 to 5 would you place the requirements you generally encounter in your field?
- 20. © Predictable Network Solutions Ltd 2017 20 www.pnsol.com 20 Quantifying timeliness Outcome requirement: Suppose we could get the marketing department to specify how long it’s acceptable to wait for the first frame of a video: • 50% of responses within 3 seconds • 95% of responses within 10 seconds • 99.9% of responses within 15s • 0.1% failure rate
- 21. © Predictable Network Solutions Ltd 2017 www.pnsol.com Click to edit Master title style 21 • Suppose the black line shows the delivered CDF • From measurement, simulation or analysis • This is everywhere above and to the left of the requirement curve • This means that the timeliness requirement is satisfied • If not, there is a performance hazard Meeting a timeliness requirement
- 22. © Predictable Network Solutions Ltd 2017 www.pnsol.com Questionnaire analysis Who are you and what do you think?
- 23. © Predictable Network Solutions Ltd 2017© Predictable Network Solutions Ltd 2017 www.pnsol.com 23 0 1 2 3 4 5 Telecomms / Infrastructure Telecomms / Services Cloud Services media production infrastructure Research & Innovation App Development / Design IoT Security Operations Aerospace middleware App Development / Testing 0 1 2 3 4 Management System Design / Architect Capability development / Research Marketing / Sales 0 1 2 3 4 5 6 7 8 Requirements Definition System / Product Design System / Product Development System / Product Integration and/or Testing Deployment Operations and Maintainance Performance Monitoring and Usage Analytics What field do you work in? What is your role/what level do you work at? Which parts of the product lifecycle are you involved in?
- 24. © Predictable Network Solutions Ltd 2017© Predictable Network Solutions Ltd 2017 www.pnsol.com 24 Where do system/product requirements come from? 0 1 2 3 4 1 2 3 4 5 Precise and unambiguous? 0 1 2 3 4 1 2 3 4 5 Quantifiable and quantified? 0 1 2 3 4 5 6 They come directly from the client/customer They are derived in collaboration with the client/customer Generated internally using experience / engineering judgement Derived from standards / regulations or other definitive external sources Developed using internal funding and experience of anticipated customer needs
- 25. © Predictable Network Solutions Ltd 2017© Predictable Network Solutions Ltd 2017 www.pnsol.com 25 How / when does your organisation determine whether requirements have been / will be met? 0 1 2 3 4 5 6 7 Redesign / rework at our client/customer's cost Carry on, leaving the issue to be resolved later Attempt to change the requirements Redesign / rework at my organisation's cost What is the typical response to the discovery that requirements will not be / have not been met? 0 1 2 3 Through customer feedback after deployment/delivery, e.g. by waiting for complaints or a lack of them There are test and integration steps whose completion is determined informally, e.g. by demonstration There is a test and integration methodology with formal sign-off procedures that are purely internal There is a test and integration methodology with formal sign-off procedures involving the client/customer
- 26. © Predictable Network Solutions Ltd 2017 26 www.pnsol.com Importance of performance and resources For the systems / products you produce / operate, how critical is performance to success? 4.4 To what extent is your role accountable for aspects of performance that are critical for success? 3.2 To what extent do you, in your role, have influence/control over how resources are shared? 2.8 Given that resources are shared, other parts of the system / other systems may consume resources on which you rely: to what extent is this a concern? 4.0 Given that resources are shared, your system / product may consume resources on which others rely: to what extent is this a concern? 3.8 My role is primarily concerned with the performance impact on the individual users / uses My role is concerned with both the overall performance and the impact on individual users / uses System performance is not a concern of my role
- 27. © Predictable Network Solutions Ltd 2017 www.pnsol.com Summing up
- 28. © Predictable Network Solutions Ltd 2017 www.pnsol.com Click to edit Master title style 28 Benefits • Avoid infeasible developments • 'fail early’ • prune blind alleys • Address scalability early • including real-world constraints • avoid 'heavy tail’ • See the whole risk landscape • not just the 'first problem’ • Be able to write a safety case • Even for shared-resource distributed systems Costs • Have to ‘quantify intent’ • may meet resistance • Effort in adopting new tools and techniques • Upfront work needed before “real” development starts • may not fit expectations/metrics of ‘progress’ Expected benefits and costs of this approach
- 29. © Predictable Network Solutions Ltd 2017 www.pnsol.com Thank you!
Notas del editor
- PEnDAR - Performance ENsurance by Design, Analysing Requirements TSB REFERENCE: 132304
- We starting this project because we are seeing cost/performance hazards becoming visible late in the development process – too late to save some projects! We see this as a multi-$B problem worldwide. What we are investigating is how to enable verification and validation of cost and performance as opposed to simply functional aspects, and to do this for distributed and hierarchical systems, supporting both initial development and ongoing maintenance and incremental development. We aim to provide early visibility of system cost/performance hazards to avoid costly failures and maximise the chances of successful in-budget delivery of acceptable end-user outcomes. This project is a feasibility study of how techniques that exist to achieve this can be brought into more general use in the industry. The project partners are: Predictable Network Solutions, with a long history of addressing performance issues in large-scale distributed systems Test and Verification Solution, who have expertise in test and verification, particularly for safety-critical and automotive applications Vodafone Group, in the role of a system-of-systems integrator, aiming to deliver innovative services, often across a wide geographical area The project is supported by InnovateUK.
- The focus group is an opportunity to articulate some of what we know about the boundaries of the possible in distributed computing, and to get insight into achieving viability and sustainability in large and complex and developments We aim to find out where and how these techniques can be applied, and what the major benefits can be are, and whether a tool or service solution will be the best fit. What are the constraints on applicability of new approaches: established procedures, market inertia etc. As yet there’s nothing to sell!
- What is the core challenge in delivering distributed shared-resources systems that deliver satisfactory performance at reasonable cost? How can they be feasible and sustainable? How can the tail risks be constrained? I.e. problems that occur infrequently and so may not manifest until after the system has been deployed. Managing constraints on system developers: cosmic, coming from the laws of physics ludic, arising from the way systems are constructed and how various ‘games’ of chance get played out ecological, arising from the ecosystem of suppliers and vendors and what is actually available to work with.
- System requirements are often vague and/or contradictory, and change during (and after) development. Complexity forces hierarchical decomposition of the problem, creating boundaries that may hinder optimal development, including commercial boundaries with third-party suppliers Time pressure forces parallel development that may not fit naturally with the hierarchy, and encourages leaving tricky issues for later, when they will tend to cause re-work and overruns, and leave tail-risks. Cost and resource constraints force sharing of resources, both within the system and with other systems (for example when network infrastructure or computing resources are shared), and may also require re-use of existing assets (own or third-party) that may not be ideal, in particular their behaviour may be inadequately quantified. Everyone knows this – not everyone deals with it smoothly.
- How can we decompose a top-level requirement into requirements on subsystems so that we can have confidence that meeting all the lower-level requirements will satisfy the top-level one? For functional outcomes (‘doing the right thing’) there are various ways of dealing with this. Performance outcomes means delivering an outcome within a time bound (‘doing the right thing in the right time’), not just performing them at a particular rate. We want to minimise the amount of integration testing we have to do and re-work, particularly when these become a bug-fixing cycle. At the same time we want to manage risks, especially those to do with moving from a lab/pilot phase to a wider deployment, where loading and scaling factors may be problematic.
- There are established approaches, but these may be running out of steam. Let’s consider two of the dimensions of this problem: (1) To what extent are resources are dedicated or shared (either within the system or even with unrelated systems)? (2) How far is the system (and its resources) distributed? Examples range from standalone microcontrollers with their own (if small) dedicated resources to virtualized cloud apps (note IoT links these two domains!). Traditional avionic systems are distributed but consist of dedicated hardware units connected by dedicated communication links, whereas the shift is to use software modules sharing processing platforms, connected by links shared between different functions. From the bottom left to the top right, performance hazards go from well-managed to very unconstrained. Think about your own experiences – where do your systems of concern fit on this diagram? Virtualisation is driving us to more shared resources; cost constraints are forcing use of pre-packed services located remotely.
- We’re now going to run through some of the technical dimensions of this challenge
- This captures what we have learnt about system delivery problems over the last decade. There’s a lot here so we’re going to break it down!
- They key task with shared-resource systems is to find a way to quantify and manage the performance/resource tradeoff. Quantifying and managing the performance/resource tradeoff (yellow centre) is specific to each particular system; the issues around it can de dealt with by applying generic techniques. Analysis of the central problem is complemented by a synthesis of other techniques. The three key aspects to consider are: Scale – how are the resource/performance trades affected by the scale of the system? Exception/failure – how are these managed, given that they become inevitable in a shared, distributed system Variability – how variable are the resources and the demand for outcomes?
- Scale has two dimensions: Space – either in terms of physical distance, affecting transmission times, or in terms of numbers of users/demands on the system, which together create a notion of ‘density’ that can drive the economics of the solution. Time – on long timescales the question is one of capacity, on short ones of schedulability.
- Exception and failure are specifically not a question of ‘coding errors’ or hardware faults (although those are a factor) but more one of temporary shortage of resources, resulting, for example, in the loss of a packet or a deadline being missed. Two approaches to handling this are mitigation (re-transmitting a packet, for example) or propagation (packet loss resulting in a failed transfer), requiring handing at a higher layer. These interact, and the optimal approach will depend on the frequency and severity of the failures and the costs of handling them in different ways.
- Variability applies both to resources and to load, and its key aspect is correlation: Positively correlated, e.g. by TV advert breaks Negatively correlated, e.g. use of one part of the system precludes simultaneous use of another Uncorrelated, basically a random effect. Correlations can be externally generated or be a result of the operation of the system
- We need to consider both the impact on individual outcomes and the impact on the ability of the rest of the system to deliver collective outcomes.
- Once the core is understood, the rest is manageable with the right tools.
- Need to support stages in the SDLC. In Design: Feasibility: can you deliver the outcomes with sufficient timeliness with acceptable use of resources Hierarchical decomposition Acceptance criteria Verification requires checking quantified outcomes, in a way that is ‘cheap’ enough to re-apply during the system lifetime.
- Rocket science used to be something only world superpowers could do – now you only need to be a billionaire! It’s well enough understood to be reproducible, and is just (complex) engineering. Brain surgery requires experience, skill and gut feel – not easy to teach! Outcomes are hard to quantify.
- 20
- Any CDF whose curve is always to the left and above this one represents an outcome that is “acceptable”. If the black line crosses the blue line we have a performance hazard.
- Response were scored from 1 - 5
- Looking at a more formal approach to managing cost/performance hazards – do the benefits and costs of this balance out? There’s a push to use standard commodity infrastructure for safety/mission critical purposes – saves a lot of costs but also introduces risk. Need to be able to make a safety case! Virtualisation is coming in everywhere – what are the risks? Case studies done inside the project show that getting intentions to be quantified can be hard; however explaining that allowing for some possibility of delay or failure can dramatically reduce the delivery costs may encourage engagement. Even functional verification can be considered ‘too expensive’.
- Two further webinars will give more details.