1. 1
Strategic approach to ascertain accurate decisions
after unplanned service outage in Telecom operations
Deep Kamal Singh
27 September 2013
2. 2
Introduction
• Changes in System are Inevitable
• Uncertainty, Variations, Deviations are Un Avoidable
• Prime objective of Business – Profit & Cost Control
• Factors that control cost of variation/deviation
– Number of interdependent systems/services disrupted
– Knowledge Gap between cross functional domains
– Time spent in reaching a decision
• Exponentially proportional to Stake holders involved
• Directly proportional to number of services/systems
disrupted
• Need of the hour, Business Need
Strategic approach to ascertain accurate decisions after unplanned service outage in Telecom operations
3. 3
• Organizations are facing a complex and
risky environment
–A “do more with less”
–“Tight” economic climate
–Expanding global priorities
–Necessity to enable innovation
The Strategic Approach
Identify All
Services
Identify
Revenue of
every
service
Define
Threshold
Form Crisis
Management
Team
Strategic approach to ascertain accurate decisions after unplanned service outage in Telecom operations
Identify All
Impacted
Services
Estimate
outage &
Calculate
Revenue
Consolidate
Options
Decide
• Delay
• GO Live
• Fall Back
Planning
Execution - Handling Crisis
Organizations are facing a
complex and risky environment
– A “do more with less”
– “Tight” economic climate
– Expanding global
priorities
– Necessity to enable
innovation
4. 4Presentation Title
Knowledge Area
Processes
Initiating
Process
Group
Planning Process Group Executing Process Group
Monitoring &Controlling
Process Group
Closing
Process Group
Project Integration
Management
4.1 Develop
Project
Charter
4.2 Develop Project Management Plan
4.3 Direct and Manage Project
Execution
4.4 Monitor and Control
Project Work
4.6 Close
Project or
Phase
4.5 Perform Integrated
Change Control
Project Scope
Management
5.1 Plan Scope Management 5.5 Validate Scope
5.2 Collect Requirements 5.6 Control Scope
5.3 Define Scope
5.4 Create WBS
Project Time
Management
6.1 Plan Schedule Management
6.7 Control Schedule
6.2 Define Activities
6.3 Sequence Activities
6.4 Estimate Activity Resources
6.5 Estimate Activity Durations
6.6 Develop Schedule
Project Cost
Management
7.1 Plan Cost Management
7.4 Control Costs7.2 Estimate Costs
7.3 Determine Budget
Project Quality
Management
8.1 Plan Quality Management 8.2 Perform Quality Assurance 8.3 Perform Quality Control
Project Human
Resource
Management
9.1 Plan Human Resource
Management
9.2 Acquire Project Team
9.3 Develop Project Team
9.4 Manage Project Team
Project
Communications
Management
10.1 Plan Communications
Management
10.2 Manage Communications
10.3 Control
Communications
Project Risk
Management
11.1 Plan Risk Management
11.6 Monitor and Control
Risks
11.2 Identify Risks
11.3 Perform Qualitative Risk Analysis
11.4 Perform Quantitative Risk
Analysis
11.5 Plan Risk Responses
Project
Procurement
Management
12.1 Plan Procurement Management 12.2 Conduct Procurements 12.3 Control Procurements
12.4 Close
Procurements
Project Stakeholder
Management
13.1 Identify
Stakeholders
13.2 Plan Stakeholder Management
13.3 Manage Stakeholder
Engagement
13.4 Control Stakeholder
Engagement
5. 5
Agenda
Introduction
Activities and Phases
Being prepared for uncertainty
Handling uncertainty
Theory in practice
In Conclusion
Q & A
Strategic approach to ascertain accurate decisions after unplanned service outage in Telecom operations
6. 6
Activities and Phases
• Complexity Assessment is first step to prepare for
uncertainty
• Concise definition of Decision makers for every stages
• Knowing Phases of activity – Strategic approach to take
next step under unplanned outage/failure depends upon
the phase
– 3 phases of any activity in simplistic view
• Preparation
• Execution
• Operation
Strategic approach to ascertain accurate decisions after unplanned service outage in Telecom operations
7. 7
Activity Complexity
1 2 3 4 5 6 7
Complexity 1 4 9 16 25 36 49
Teams 1 2 3 4 5 6 7
0
10
20
30
40
50
60
Complexity
Activity Complexity Vs. Systems Involved
Strategic approach to ascertain accurate decisions after unplanned service outage in Telecom operations
• Complexity of activity
– Exponentially proportional to
number of Inter-dependent
systems involved
– Directly proportional to
number of cross functional
teams involved
– Directly proportional to
Allocated duration versus
tasks list ratio
8. 8
Impact of failure during Activity phases
Presentation Title
Planning Phase
• Delayed Activity timelines
• Budget overhead
• Resources re-alignment
• Impact on cost to business
•Man hours invested in
preparation, development, planning, testing
etc.
•Business loss due to delayed launch
•Delay overhead cost on logistics
•Cascading delay on other planned activities
Execution Phase
• Increased service downtime Sudden revenue
loss
• Impact on cost to business
•Direct revenue loss
•Customer dissatisfaction Unpredictable
future damages
•Increased calls to customer care and support
systems
•Resource working in extended hours
Unpredictable future issues.
•If Fallback is decided Total investment
sunk.
Operation Phase
• Live Service downtime Sudden revenue loss
• Impact on cost to business
•Direct revenue loss
•Customer dissatisfaction Unpredictable
future damages
•Increased calls to customer care and support
systems
•Unplanned additional Man hour investment
•Resource working in extended hours
Unpredictable future issues.
9. 9
Agenda
Introduction
Activities and Phases
Being prepared for uncertainty
Handling uncertainty
Theory in practice
In Conclusion
Strategic approach to ascertain accurate decisions after unplanned service outage in Telecom operations
10. 10
Activity Planning
• The present day Approach
– Preparation, Review and approval of detailed step wise
execution plan
• Estimation of planned outage duration.
– Business teams analyzes and calculate total cost of activity
to business
• Including cost of planned outages, consequential
costs, logistics, HR overhead and so on.
– Every step of activity is executed, monitored and controlled if
everything goes „as per plan‟
• Shortcomings – We can not predict what may go wrong and
thus we can not be 100% ready for unplanned outages
Strategic approach to ascertain accurate decisions after unplanned service outage in Telecom operations
11. 11
Planning for uncertainty
Being prepared for something that can not be planned – During
Preparation Phase
• Identification of all system and services
• Evaluation of revenue earnings of each services
• Formation of Crisis management team
• Unanimous view of Project managers, Activity owners, Team
leads and Business teams over
– Revenue potential of each service
– All factors that impact business for each service
• Documentation of the unanimous view – “ Service Impact
Matrix”
Strategic approach to ascertain accurate decisions after unplanned service outage in Telecom operations
12. 12
Preparing Service impact matrix
• “Service impact matrix” - single document of reference to assess
impact to business during outage
– List all services which are going to be affected due to the
planned activity
– Include subscriber/user/consumer base of every service
– Include time based distribution of revenue earnings of each
service,
– Include direct and indirect cost to business for each service if
failure is encountered at any time duration
• Service Impact Matrix must be finalized, reviewed and approved by
all stake holders.
– Saves time to estimate cost of outage to business during crisis
– Prevents lengthy review/approval process
Strategic approach to ascertain accurate decisions after unplanned service outage in Telecom operations
13. 13
Example Service impact matrix
SNO Service Name
Subscriber
Base
(In Mn)
Hourly Revenue Potential [in 100K Rs.] Indirect cost to business
H01 H02 H03 H04 H… H21 H22 H23 H24
Customer
Care
Overhead
Dependent
Services
impact
Customer
Satisfaction
Impact
1 Voice Calls 14.34 …
1.1 Local Home network 14.34 …
1.2 Local cross network 14.34 …
1.3 National Long Distance 09.60 …
1.4 International Long Distance 00.12 …
2 SMS 14.34 …
2.1 Local 02.21 …
2.2 National 02.21 …
2.3 International 00.01 …
3 Data Usage 01.90 …
4 Paper Recharges 08.14 …
5 ETOPUP 11.89 …
6 USSD Services 14.34 …
6.1 Subscriber Info 14.34 …
6.2 Subscription management 04.30 …
6.3 VAS Services 01.80 …
7 Subscriber Life Cycle 14.34 …
7.1 Subscriber Activation 0.03 …
7.2 Subscriber Churn 00.01 …
7.3 Service management 00.01 …
Strategic approach to ascertain accurate decisions after unplanned service outage in Telecom operations
14. 14
Agenda
Introduction
Activities and Phases
Being prepared for uncertainty
Handling uncertainty
Theory in practice
In Conclusion
Q & A
Strategic approach to ascertain accurate decisions after unplanned service outage in Telecom operations
15. 15
Identifying affected services and subscriber
base
In case of any unexpected outage we need to find out total impact
to business, thus
• First and foremost, get update from all teams and consolidate
list of services that are affected.
• Classify level of disruption for each impacted service
– Intermittent, Partial or Total
• Estimate outage duration and dependency factors for the
impacted service
• Consolidated outage stats for each impacted service in a single
view for comparison and correlation.
Strategic approach to ascertain accurate decisions after unplanned service outage in Telecom operations
Considering the revenue impacting factors associated with the activity, below
equation can be deduced
If
R(S1), R(S2)…..R(Sn) = Avg. revenue per user per minute of Service1,2,…n
O(S1),O(S2)…..O(Sn) = Total estimated outage duration of Service1,2,…n
Sb(S1),Sb(S2)….Sb(Sn) = Estimated percentage of affected subscriber base
during outage of Service1, 2,….n
Su(S1), Su(S2)…..Su(Sn) = Total users of Service1, 2, ….n
Then
Total loss to business during service outage =
∑ [ R(Sn) x O(Sn) x Sb(Sn) x Su(Sn) / 100 ]
16. 16
Deciding next course of action
• Identify impacted services
• Calculate Revenue Impact
– Consolidate outage stats of all affected
services
– Correlate available data with corresponding
details in Service Impact Matrix
• Assess Variable and Dependency factors
• Mathematically deduce all possible next steps
to formulate “The Decision Matrix”
Strategic approach to ascertain accurate decisions after unplanned service outage in Telecom operations
17. 17
Formulating Decision matrix
• The Decision matrix
– Mapping of Options vs. Risk vs. Revenue Impact
vs. Variance factor
– Crisis Management team to prepare
• Cross functional managers and tech leads to tabulate
domain specific next step of actions
• Categorization of options
– Track 1 :: Go-Live
– Track 2 :: Delay
– Track 3 :: Fall back
Strategic approach to ascertain accurate decisions after unplanned service outage in Telecom operations
18. 18
Finalizing Decisions
• Ignore – impossible, unattainable, high risk options
– Rule out options which are
• least possible,
• have highest risk,
• have highest number of variable factors
• Final time estimation
– Get „Time to implement‟ from respective teams
– Readiness Confirmation – technically and logistically
• Final out put of crisis management team:-
Decision matrix with final concluded set of options and details of
‘time to implement’ presented to apex decision makers
Strategic approach to ascertain accurate decisions after unplanned service outage in Telecom operations
19. 19
Agenda
Introduction
Activities and Phases
Being prepared for uncertainty
Handling uncertainty
Theory in practice
In Conclusion
Q & A
Strategic approach to ascertain accurate decisions after unplanned service outage in Telecom operations
20. 20
Case study : Activity Details
• Billing System upgrade in 14 telecom circles of a leading telecom operator in
India
• The activity requires complete outage of billing system for 8 hours, Since
billing system stays unavailable, all connected services (as listed) also face
subsequent downtime:
– Voice calls – Local / National Long distance / International long distance
– SMS
– Data Browsing
– Real-time data charging
– Recharges – Voucher recharge and E-topup
– USSD
– Unified Subscriber life cycle management – Activation, Churn, daily jobs, and other offline
process.
– Business reporting – MIS
• The complex activity involves changes at many functional systems
– High level of coordination and control required between Cross functional
technical teams.
– Due to numerous Inter-dependent systems, outage of one system or
service may result in consequential outage of other services also
Strategic approach to ascertain accurate decisions after unplanned service outage in Telecom operations
21. 21
Case study 1
• Subscriber : 5.4
• Activity: Core billing system upgrade
• Issue during execution: ETOPUP not working with new version
– Important to note here that multiple rounds of testing on test-bed and rehearsals were
conducted which showed no sign of issues earlier.
• Technical teams started investigating and fix the issue ASAP.
• Business team kept waiting for updates from Technical team to decide next step.
• All possible next actions were greatly debated without reaching a final conclusion
– ETOPUP : DELAY
• Given estimated time to fix the issue was 2 Hours
– Revenue assurance team : FALL BACK
• Cancelling complete activity and reinstating old version billing system
– Billing system team : GO LIVE
• leaving ETOPUP team to fix “their” issue separately
• Business team: absence of conclusive data No decision was taken
• Finally after 95 minutes of extended downtime, ETOPUP issue was resolved and then
upgraded version of billing system was made live.
Strategic approach to ascertain accurate decisions after unplanned service outage in Telecom operations
22. 22
Evolution of strategic Approach
• Learning from failure – What took time to reach a decision?
– Lack of information to take a decision
• “What to do next” is Biggest Time eater
– Lack of predefined delay threshold for critical milestones
– Lack of designated managers to manage crisis
• Biased Decision Making is biggest Hindrance to consensus
• Absence of a dedicated Crisis management team.
– Lack of clarity on activity owners, team leaders, business owners and
decision makers
• Dedicated team lead/managers always suggest biased options which
favor their own team
– Lack of methodical approach to ensure quick decision making
The learning drawn from events of site 1 became foundation for evolution
of strategic approach which is discussed in this presentation.
Strategic approach to ascertain accurate decisions after unplanned service outage in Telecom operations
23. 23
Case study 2
• Subscriber Base : 9.28 million
• Activity Details: Same activity as Case study 1
• The Issue: Once new version of billing system was made live for test traffic
– It was identified that some of call handling units were not functioning
• When call routed on those units voice calls, video calls, SMS, USSD
were failing
– Other units were functioning without any issues
• Crisis management team swung in action and as a first step it was deduced
that 20% of units were not functioning
– This meant Voice calls, SMS and USSD requests were impacted by 20%.
Important to note that the Crisis management team was doing all coordination
and fact gathering with cross functional technical and business teams, the
Technical teams were dedicatedly focused to resolve the issue.
Strategic approach to ascertain accurate decisions after unplanned service outage in Telecom operations
Continued…
24. 24
Case study 2
• Total Impact identified Service impact matrix referred
Assessment
– Revenue loss when Billing system is made live with 20% failures in 3 service
– Cost of delaying the launch – Which meant 100% subscriber base will not be able
to use ANY service until the issue is fixed
– Cost of fallback was also compared with above two costs
• 3 Tracks Options (Delay, Go Live, Fallback ) consolidated
– Presented to Business owners and Function heads
• Decision Taken
– Within the duration of 45 minutes allocated for system testing before launch –
Business team successfully reached a decision to remove those 20% units from
network (so call will not land on them) and make the system Live as per schedule
Even with unexpected variations encountered during critical launch
schedule, the strategic approach helped reaching a right decision
methodically
Strategic approach to ascertain accurate decisions after unplanned service outage in Telecom operations
25. 25
Agenda
Introduction
Activities and Phases
Being prepared for uncertainty
Handling uncertainty
Theory in practice
In Conclusion
Q & A
Strategic approach to ascertain accurate decisions after unplanned service outage in Telecom operations
26. 26
Identify impacted
services
Identify impacted
subscriber/user base
• Estimate probable
service outage
duration for each
impacted service
Calculate total
revenue impact
against each
impacted service
Evaluate
indirect, additional
and future cost to
business due to
service outages
Formulate list of
possible next steps
or possible options.
Identify variable
(uncertain) factors
and Risk associated
with each concluded
option.
Prepare Option vs.
Risk vs. Revenue
Impact vs. Variance
factor matrix having
each of 3 tracks
• Track 1 - Go-Live
• Track 2 – Delay
• Track 3 – Fall back
From every track
rule out options
which are
• Least possible
• Have highest risk,
• Have high variable
factors.
Evaluate time to
implement for every
option.
Present Decision makers with Tracks
containing concluded set of options and
details of „time to implement‟ for each
option.
Conclusion
27. 27
Thank You
Deep Kamal Singh
Project Manager – BSS APAC
Comverse Network Sys.
Deep.Singh@comverse.com
+91-9971003585
Shay Shargal, PMP
CEO, RBS Projects
Shay@rbsprojects.co.il
+972-52-23-11-11-0
www.rbsprojects.co.il
Past President – PMI Israel Chapter
Notas del editor
Since its inception, technology has advanced many folds, and by every passing decade we grow larger, smarter, and efficient than we were ever beforeThe advancement of technology brings CHANGE. It brings change in terms of business growth, adaptability, innovation, cost control andthere are endless upgrades, tune ups and additions that keep on happening in the network system, for same reasons there are regular set of activities planned and executed, ranging from a small node level change to a complete system upgrade or replacement.After more than 50 years and four generations of mobile technology, subject matter experts have fashioned standard operating procedures of all activities that are executed in the ecosystem, however system failures still happen leaving no operator untouched by unforeseen outages impacting services of end subscriber.
Introduction Changes are inevitable trade off betweenTechnical and Business teams Decision taking time Vs impact to business graph Need of the hour??Activities and Phases Activities – complexity of activity increases exponentially with number of interdependent systems (Stake holders) involved Phases of activity – Preparation – Execution - OperationBeing prepared for uncertainty Identifying all services which will be impacted due to activity Service revenue impact matrix – Service, Subscriber Base, Time SlotForming strategic approach Understanding ‘approaches’ (tracks) after every deviation Preparing Decision MatrixTheory in practiceIn Conclusion
Complexity of activityActivity planning team must assess complexity of the activity to prepare for uncertainty accordinglyDecision makers for all stages must be concisely predefinedPhases of activity – Strategic approach to take next step under unplanned outage/failure depends largely upon phase of activityIn simplistic view there are 3 phases of any activityPreparation ExecutionOperation
Introduction Changes are inevitable trade off betweenTechnical and Business teams Decision taking time Vs impact to business graph Need of the hour??Activities and Phases Activities – complexity of activity increases exponentially with number of interdependent systems (Stake holders) involved Phases of activity – Preparation – Execution - OperationBeing prepared for uncertainty Identifying all services which will be impacted due to activity Service revenue impact matrix – Service, Subscriber Base, Time SlotForming strategic approach Understanding ‘approaches’ (tracks) after every deviation Preparing Decision MatrixTheory in practiceIn Conclusion
Being prepared for something that can not be planned – During Preparation PhaseIdentify all services which are getting affected during activity executionWe can not predict which service may go down during activity, but certainly We can mark down all services that are getting affected during activity.Identify revenue earnings of each services for planned duration of executionWe can not predict at what time unexpected outage may be encountered, but certainlyWe can evaluate duration based revenue impact of every affected service, so we know in advance which service will cost more to business if at some point we have to chose between more than one.A group of specialized members must be formed to assist and enable Decision makers take well informed and right decision in least amount of time during crisisIt is important that members of these group do not borne any bias or inclination for any service or system.Project managers, Activity owners, Team leads and Business teams must have same view over revenue potential of all services, and all factors that can reduce time for taking decision during crisis – this single view should be documented in “Service Impact Matrix”“Service impact matrix” is single document of reference to assess impact to business revenue in case an unexpected outage is encountered, this document mustList all services which are going to be affected due to the planned activityShow subscriber/user/consumer base of every serviceShow time based distribution of revenue earnings from each service, time intervals can be hourly, bi-hourly or any other slot as deemed suitable by business teamShow direct and indirect cost to business for each service if failure is encountered at any time durationDuring planning/preparation phase of activity, Service Impact Matrix must be finalized, reviewed and approved by all stake holders.
Introduction Changes are inevitable trade off betweenTechnical and Business teams Decision taking time Vs impact to business graph Need of the hour??Activities and Phases Activities – complexity of activity increases exponentially with number of interdependent systems (Stake holders) involved Phases of activity – Preparation – Execution - OperationBeing prepared for uncertainty Identifying all services which will be impacted due to activity Service revenue impact matrix – Service, Subscriber Base, Time SlotForming strategic approach Understanding ‘approaches’ (tracks) after every deviation Preparing Decision MatrixTheory in practiceIn Conclusion
Revenue Impact CalculationDuring outage once stats of all impacted services are consolidated, Business owners must correlate the available data of impacted services with their corresponding details in Service Impact MatrixSince data included in Service Impact matrix is already well reviewed and approved by all stakeholders, it saves lot of time to exactly find out cost of outage to business with every passing minute.Variable and Dependency factorsDuring crisis expected outage duration given by technical teams are always indicative and can not be considered hard and binding – decision makers must keep the variable & dependency factors also in perspective and prepare set of possible next steps accordingly.With reference of the Service impact matrix and impact stats Business owners can mathematically deduce best next stepWhether to wait until impacted service get fixed, and so face outage of all other services until the impacted service is up and running, i.e. DELAYWhether to leave the impacted service as it is and thus carry on with activity and make all other services live as per schedule, i.e. GO AHEADOr to completely roll back the activity and thus make the system live and functioning as it was before, i.e. FALL BACK
Its prerogative of crisis management team to work out and prepare set of possible next actions and categorize them into the three tracksThe output of this exercise is Decision MatrixDecision matrix is essentially a mapping of Options vs. Risk vs. Revenue Impact vs. Variance factor Cross functional managers and tech leads should prepare the tabulated listing of all options with clear information about the revenue loss, risks and variable factors associated with each possible next actionThe options in the matrix should be categorized in three tracksTrack 1 – Go-LiveThis means activity will be carried on as planned, despite having unexpected impacts in one or more services This track ensures that main activity gets completed within time and service outage duration of all proper functioning services stays under approved limits.Track 2 – DelayHold all sub-activities until the unexpected problem at one or more services gets fixedThis track ensures that all services will be live once the change is implemented completely, however it inherently subjects business to risk of bigger revenue loss if the affected services are having low revenue earning potential and/or longer time is taken to fix the issue.Track 3 – Fall backCompletely call off the activity, This will ensure that all services function the same way as it were functioning before implementation of change, However fallback of the change directly means all efforts and cost invested in the activity becomes void, in addition separate cost of doing the same activity again in future should also be accounted for.
Ignore – impossible, unattainable, high risk itemsFrom every track, options which are either least possible, or have highest risk, or have highest number of variable factors must be ruled out.For every option listed in the tracks, time needed by each team to implement the option must be documentedEvery team should confirm its readiness (technically and logistically) to go ahead with any of the option well in time.Out put of crisis management team:-Decision matrix with final concluded set of options and details of ‘time to implement’ should be presented to apex decision makers
Introduction Changes are inevitable trade off betweenTechnical and Business teams Decision taking time Vs impact to business graph Need of the hour??Activities and Phases Activities – complexity of activity increases exponentially with number of interdependent systems (Stake holders) involved Phases of activity – Preparation – Execution - OperationBeing prepared for uncertainty Identifying all services which will be impacted due to activity Service revenue impact matrix – Service, Subscriber Base, Time SlotForming strategic approach Understanding ‘approaches’ (tracks) after every deviation Preparing Decision MatrixTheory in practiceIn Conclusion
In Q3 2010, A leading telecom operator in India planned nationwide upgrade of its core prepaid billing system (BSS), this activity was to be executed in 14 telecom circles across India, this is a complex activity involving hardware additions and retrofits, complete software upgrade, During planning process phase of this project the group of project managers and functional heads jointly began to estimate the risks involved in each implementation and further to prepare mitigation approach, as a result of many joint sessions over same apparition the concept and idea was born which became prime subject documented in this paper.The Operator had 14 sites running on old billing system and thus same upgrade was planned for all the sites. As on all sites there are different teams managing the same services, a great zeal of planning, coordination, testing and control was required at each site to execute the activity and thus every step of activity was properly documented and was reviewed and approved by all stakeholders. Even with great level of planning and coordination during activity execution in starting few sites various deviation were observed and which impacted cost of activity to business and imparted sudden revenue losses, With every implementations technical teams concluded all necessary learning to ensure same problem are not faced again in next implantation, and it was observed by project managers and functional heads that a reliable approach is required to ensure critical decisions are taken in least possible time
Core billing system upgrade activity was completed as per schedule projected in timelines, however when new version of billing system was brought up for testing it was found that ETOPUP is not able to connect whereas all other services were able to connect and perform testing at their end, technical teams started working out to get to the cause and find solution. Minute by minute the time allocated for testing before go-live was getting reduced for ETOPUP service, side by side testing was in progress from all other teams.As the time allocated for testing reached completion, testing status from all teams was shared with business whereas problem with ETOPUP service was still not found, at this stage business owner of ETOPUP service was advocating for roll back of complete activity, Billing system team was adamant that since other services are working with new version, there is no fault at their end thus roll back will not be done, to decide next course of action core business team was not having visibility over the technical details and it was also not known that how long it may take to fix the issue. Between these discussions no one was actually taking ownership to either give confirmation to go Live or call off the activity and roll back, it is important to realize that outage period of all services Voice calls, SMS, VAS, USSD et al was gradually increasing and by every passing minute revenue loss to business kept rising. After 95 minutes of extended downtime issue was identified and fixed and all services were made live with new version of billing system.
Learning from the complex implementation at first site were analyzed in great details, and as result of several process defining session a commonly applicable reference was evolved as a general methodological approach to ensure right decision is taken by apex decision makers in least possible time during unexpected outage in telecom ecosystemLack of information to take a decisionLack of predefined delay threshold for critical milestonesLack of designated managers to manage crisisLack of clarity on activity owners, team leaders, business owners and decision makersLack of methodical approach to ensure quick decision makingThese learning drawn from events of site 1 became foundation for evolution of strategic approach which is discussed in this presentation.
Introduction Changes are inevitable trade off betweenTechnical and Business teams Decision taking time Vs impact to business graph Need of the hour??Activities and Phases Activities – complexity of activity increases exponentially with number of interdependent systems (Stake holders) involved Phases of activity – Preparation – Execution - OperationBeing prepared for uncertainty Identifying all services which will be impacted due to activity Service revenue impact matrix – Service, Subscriber Base, Time SlotForming strategic approach Understanding ‘approaches’ (tracks) after every deviation Preparing Decision MatrixTheory in practiceIn Conclusion
