Upgradability: A Hybrid Quality Problem

The Quality
Attribute of
Upgradability
Len Bass with
Hiroshi Wada, Ingo Weber, Liming Zhu,
Ross Jeffery

NICTA Copyright 2012 From imagination to impact

About NICTA

National ICT Australia

• Federal and state funded research
company established in 2002
• Largest ICT research resource in
Australia
• National impact is an important
success metric
• ~700 staff/students working in 5 labs
across major capital cities
• 7 university partners NICTA technology is
• Providing R&D services, knowledge in over 1 billion mobile
transfer to Australian (and global) ICT phones
industry

2
NICTA Copyright 2012 From imagination to impact

Consider the follow sequence.
• You have prepared an upgrade to an existing large
enterprise system
– You have coded it
– You have tested it
– It is ready!!
• Alternatively, the IT department (or you) get a package
from a third party – a vendor or open source – that has
been coded and tested.
• What happens then?

NICTA Copyright 2012 From imagination to impact 3

Consider the follow sequence.
• You have prepared an upgrade to an existing large
enterprise system
– You have coded it
– You have tested it
– It is ready!!
• Alternatively, the IT department (or you) get a package
from a third party – a vendor or open source – that has
been coded and tested.
• What happens then?
– ~10% of the time the upgrade will fail.


This is the upgradability problem
• How do we make upgrading a system less
problematic?
• Talk outline
– Characteristics of the upgrade problem
– FMEA analysis
• Possible causes of failure
• Failure prevention, detection, and recovery
– Relation to existing product and process quality work


Upgrades to enterprise systems are a very
common occurrence
Upgrade frequency of some common systems

Application Average release interval
Facebook (platform) < 7 days
Google Docs <50 days
Media Wiki 21 days
Joomla 30 days

This frequency would suggest it is important to get
the upgrades correct


Unfortunately, Upgrades Fail Often
• 4.6-10 component failures each month in three
large-scale Internet services. Mostly during
regular maintenance
• Average and maximum failure rates from a
survey of systems administrators are 8.6% and
50%.
• Some claim that user visible failures from
upgrade outweigh user visible failures from
software errors.


Why is this?
• Installation is complicated.
– Installation guides for SAS 9.3 Intelligence, IBM i, Oracle 11g for
Linux are ~250 pages each
– Apache description of addresses and ports (one out of 16
descriptions) has following elements:
• Choosing and specifying ports for the server to listen to
• IPv4 and IPv6
• Protocols
• Virtual Hosts
– The number of configuration options that must be set can be
large
• Hadoop has 206 options
• Hbase has 64
– Many dependencies are not visible until execution


Provides Research Agenda
• Indeed, the surprise is not that upgrades fail
8.6% of the time but that they are successful
91.4% of the time.

• Rich area for research.


What kind of problem is this - product?
• ISO 25010 provides
– A quality in use model composed of five
characteristics (some of which are further subdivided
into subcharacteristics) that relate to the outcome of
interaction when a product is used in a particular
context of use.
– I.e. is upgradability a quality of the system being
upgraded?
• The answer is yes.


What kind of problem is this – process?
• ITIL (Information Technology Infrastructure Library)
– Change Management aims to ensure that
standardised methods and procedures are used for
efficient handling of all changes.
• SPICE – ISO 15504
– process assessment provides the means of
characterizing the current practice within an
organizational unit in terms of the capability of the
selected processes.
• Is upgradability of quality of the process used to manage
information technology?
• The answer is yes.


Upgradability is a hybrid quality problem
• A hybrid quality problem is one in which
improvement involves both product and process
and in which the product has process
awareness.
• Many product centered conferences
– Dependability
– Security
–…
• Some process centered conferences
– Software Process Improvement
– SPICE
– SPEG
–…

Hybrid quality improvement is not well
served by the academic community
• Hybrid quality improvement – as we shall see – involves
close interaction between product, process and tools to
support the process.
• Venues that should emphasize this interaction include
– Profes (Product focused Software Development and
Process Improvement)
– ASQ (Conference on Quality and Improvement)
• Yet an examination of the CFPs and proceedings for
these conferences shows a distinction between process
activities and product characteristics
• We will present the results of a FMEA (Failure Mode and
Effects Analysis) style analysis for upgradability and then
return to the hybrid quality issue

FMEA
• Failure Modes and Effect Analysis is an
inductive failure analysis for analysis of failure
modes.
• FMEA involves describing
– Potential failure modes
– The severity and likelihood of these failures.
• We will focus on the first portion and generate
the potential failure modes as well as potential
prevention, detection, and recovery from these
failures.
• I.e. we are performing an FMEA style
analysis, not an FMEA, per se.

Scenario for Upgradability
• We are concerned with the following scenario
– Version N+1 of an enterprise system is available for
deployment.
• Version N+1 can be deployed by developers
• Version N+1 can be deployed by the Information Technology
Department (The Release Manager if there is one).
– Version N+1 is completely coded and tested by its
developers.
• Measures can include
– Downtime
– Resources (hardware or personnel) required to
perform the upgrade
– Number of failed attempts to install upgrade

Fundamental goals during upgrade
• The literature identifies four fundamental goals
while upgrade is occurring.
– Efficiently manage resources
– Completely and correctly specify configurations
– Manage multiple versions to avoid problems with
version mismatch.
– Maintain consistency of persistent data.
• Failures are caused by the violation of one of
these fundamental goals.
– Our FMEA analysis will look at potential causes for
violations of one of these goals.


Activities during an upgrade of a system
• Make the upgrade available.
• Prepare the environment. Ensure that there are
sufficient resources available for installation and
that assumed software is available.
• Configuration
• Deployment
• Activation


Organization of next portion of the
presentation
• For each activity
˗ Potential fault (a fault is a failure in waiting)
˗ Prevention of the fault
˗ Detection of the fault
˗ Correction of the fault

• Research opportunity
• Blank cell
• Cell with only partial coverage


Make Upgrade available
Fault possibility Prevention Detection Recovery
Element omitted/included Manifest Recreate
incorrectly in installing Bill of lading distribution
software

System corrupted during Hash code, Retransmit
movement checksum

Source of distribution from Digital
an untrusted site signature

Forgotten/misplaced Separate secret
credentials Independent
channel for new
credentials

Credential verifier Codify
unavailable acceptable
credentials in
distribution

Prepare environment
Incorrect versions of support Include version Encode hash of
libraries number in APIs
specification
Utilize services
to announce
incompatibilities
Multiple versions of support Include version
libraries simultaneously required number in name
Libraries expose
version numbers
Linkers version
aware
Insufficient resources Rolling Upgrade

Schema modification on Convert data to
database new schema
prior to upgrade


Configuration

Missing parameter Parameter
database
Parameter built
into tool
Static analysis
of code

Incorrectly specified Abstract Check
parameter specification syntax
Validate
against a
specification

Inconsistent Constraint
parameters checker

Deployment

Insufficient resources Pre-allocate
during
preparation
Rolling
upgrade

Inconsistent hardware Verify during
preparation

Operator error Undo
mechanism


Activation

Discovered hidden Monitoring Recovery
dependency block
Multiple Separation Version
simultaneous Dynamic aware code
versions Software and data
Update
Automatic
translation of
data when
old schema is
used
Version
aware load
balancer


Our activities in this space so far (green
cells)
• Mixed version race condition solution
• Operator undo


What is the “mixed version race condition”
• Common practice when pushing an upgrade to a
large number of servers is to perform the
upgrades one (or several) servers at a time
• This means that version N+1 (the new version)
will be available on some servers and version N
(the old version) will be available on other
servers.
• Suppose version N+1 has functionality not
available in version N


Now consider the following sequence
1. A client (browser) issues a request that is
routed by the load balancer to an instance of
version N+1
2. Version N+1 sends JavaScript assuming new
functionality back to the client.
3. Client sends an AJAX request that utilizes new
functionality and the load balancer routes it to
an instance of version N.
4. Error because version N does not have the new
functionality.


Mixed Version Race Condition
Client (browser) Server
1 Start
rolling
upgrade
2
Initial request
HTTP reply with New
embedded JavaScript 3 Version

4 AJAX callback
Old
5 Version

X ERROR


What does the solution involve?
1. Label communication between instances and
the client with version information
2. Modify load balancer so that messages are
routed to an appropriate version
3. Modify load balancer so that messages are
balanced across all child instances.


Why is this a hard problem?
• Large installations have multiple distributed load balancers that must
be kept in synch. I.e. some load balancers may know about new
version and some may not
• Not enough to put version number in message
– Suppose second request goes to a load balancer that does not yet know about
version N+1.
• Must keep messages balanced so that all servers handle roughly
the same number of requests.

/service /service

/service/vN /service/vN+1 /service/vN

server server server server server server


Operator undo
• After perofmring an operation in AWS, may want
to go back to original state – i.e. Undo the
operation
• Not always that straight-forward:
– Attaching volume is no problem while the instance is
running, detaching might be problematic
– Creating / changing auto-scaling rules has effect on
number of running instances
• Cannot terminate additional instances, as the rule would
create new ones!
– Deleted / terminated / released resources are gone!


Undo for System Operators
Administrator

begin- do
do
do rollback
transaction

+ commit
+ pseudo-delete


Approach
Administrator

begin- do
do
do rollback
transaction

Sense cloud Sense cloud
resources states resources states

Undo System


Approach
Administrator

begin- do
do
do rollback
transaction


Goal
Goal Initial
Initial
state
state state
state

Undo System


Approach
Administrator

begin- do
do
do rollback
transaction


Goal
Goal Initial
Initial Set of
Set of
state
state state
state actions
actions

Execute Generate code Plan

Undo System


Upgradability as a process&product quality
• Architecture of the system being upgraded can
affect the process of installation
– Suppose the system checks for version information
from dependent libraries. Then the process must
encompass descriptions of what to do if an error
condition occurs.
• Process of upgrade can affect the architecture of
the product.
– Suppose the process is supported by a tool that
checks the health of the installation of version N+1.
Then the system must make visible the information
used by this tool.


Summary
• Upgrade is an important problem
– Upgrade failures affect user satisfaction
– Upgrade failures happen frequently
• Upgrade involves the interaction of product and
process quality issues.
– Communities are focussed on improving the quality of
the process or the product. Not the joint
process/product quality.
• Multiple opportunities for research exist.


Q&A

Thank You!

Research study opportunities in dependable cloud computing:
• Software Architecture
• Data Management
• Performance Engineering
• Autonomic Computing

To find out more, send your CV and undergraduate details to
students@nicta.com.au

Upgradability: A Hybrid Quality Problem

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Upgradability: A Hybrid Quality Problem

Similar to Upgradability: A Hybrid Quality Problem (20)

More from Len Bass

More from Len Bass (20)

Recently uploaded

Recently uploaded (20)

Upgradability: A Hybrid Quality Problem