An analysis of the supply chain risk

An Analysis of the Supply Chain Risks of Industrial Control Systems
by
Stephen D. Mahnke
A Capstone Project Submitted to the Faculty of
Utica College
August 2019
in Partial Fulfillment of the Requirements for the Degree of
Master of Science in Cybersecurity

ii
© Copyright 2019 by Stephen D. Mahnke
All Rights Reserved

iii
Abstract
The purpose of this research project is to demonstrate the increasing vulnerability of
aging industrial control systems (ICS) as they approach obsolescence and to identify some
mitigation methods which will reduce the risk of compromise. These systems are responsible for
the operation and control of physical devices such as valves, turbine speed, train switching,
centrifuges, and chemical mixing operations. Industrial control systems play an important role in
many of the industries which are considered Critical Infrastructure, the United States defines
Critical Infrastructure as an industry which is “…considered so vital to the United States that
their incapacitation or destruction would have a debilitating effect on security, national economic
security, national public health or safety, or any combination thereof” (Department of Homeland
Security, n.d., para. 1). Due to the complexity and expense, there is a tendency for these systems
to remain functional for decades, digitalization of these systems further complicates their
vulnerability to being compromised as they do not receive regular patching and updates.
Industrial control systems are designed to function reliably and without fail for as long as
possible. As the systems move towards obsolescence, maintenance and repair become more
difficult as replacement components become unavailable, software more difficult to obtain, and
necessary skills for technicians is lost. As a result of these vulnerabilities, recovery from even a
minor maintenance issue can be very difficult. Recognition of this eventuality, by both the
vendor and the system owner before this is a problem can provide agreeable methods for all
parties to keep ahead of the total obsolescence of a system. Additionally, mitigation of the risks
of compromise is available through planned supply chain processes. Keywords: Cybersecurity,
Dr. Michael Sanchez, supply chain, obsolescence, vulnerability, hardware, software, human
competence.

iv
Table of Contents
Statement of the Problem.................................................................................................... 1
Supply Chain Vulnerability......................................................................................... 4
Purpose of the Study.................................................................................................... 7
Research Questions...................................................................................................... 7
Literature Review ............................................................................................................... 9
Obsolescence ............................................................................................................... 9
Documentation........................................................................................................... 10
Hardware Risks.......................................................................................................... 11
Software Risks........................................................................................................... 13
Human Competency Risks ........................................................................................ 14
Supply Chain Risk Mitigation ................................................................................... 15
Lifecycle .................................................................................................................... 16
Inventory Plan............................................................................................................ 18
Hardware Strategy ..................................................................................................... 19
Software Strategy....................................................................................................... 20
Licensing.................................................................................................................... 21
Disaster Recovery Plan.............................................................................................. 22
Summary.................................................................................................................... 24
Discussion of the Findings................................................................................................ 26
The Supply Chain Risks ............................................................................................ 26
Obtaining Known Good Software and Hardware...................................................... 28
Loss of Human Competence...................................................................................... 30
Collective Best Practices ........................................................................................... 32
Conclusion ........................................................................................................................ 35
References......................................................................................................................... 39

1
Statement of the Problem
Some of the what we consider to be Critical Infrastructure may be controlled, or have
their processes regulated by, some type of an Industrial Control System (ICS) (Mahmood, 2017).
Each country has its own definition of what types of sectors are a component which meets the
definition of, and can be classified as, Critical Infrastructure (Rouse, 2016). The Department of
Homeland Security in the United States has separated out a total of 16 sectors which are
“…considered so vital to the United States that their incapacitation or destruction would have a
debilitating effect on security, national economic security, national public health or safety, or any
combination thereof” (Department of Homeland Security, n.d., para. 1). The list of Critical
Infrastructure sectors in the United State is comprised of Chemical, Commercial Facilities,
Communications, Critical Manufacturing, Dams, Defense Industrial Base, Emergency Services,
Energy, Financial Services, Food and Agriculture, Government Facilities, Healthcare and Public
Health, Information Technology, Nuclear Reactors, Materials and Waste, Transportation
Systems and Water and Wastewater Systems (Department of Homeland Security, n.d.).
Industrial Control Systems, or Operational Technology (OT), within these Critical
Infrastructure sectors can include Supervisory Control and Data Acquisition (SCADA) and
Distributed Control Systems (DCS) and are often found to be used in the daily routine of the core
business (FireEye, n.d). Some uses of these types of Industrial Control Systems involve the
control of railroad switching automation, SCADA system health monitoring for industrial
equipment such as transformers and air compressors, Programmable Logic Controllers (PLC)
and input/output components for electrical, mechanical, hydraulic and pneumatic systems
(Stouffer, Pillitteri , Lightman , Abrams , & Hahn, 2015). Historically, ICS components have
been based off analog components and communications that were not easily compatible with

2
digital computer networking, but the tendency has begun to include the embedding of digital
processing elements, allowing the systems to share connections, providing information and
allowing control functionality to computer networks (Stouffer, et al., 2015).
Roughly ninety percent of the nation’s Critical Infrastructure is privately owned and
operated (Stouffer, et al., 2015) Due to the high complexity and associated costs of design and
installation, Industrial Control Systems tend to have a very long lifecycle, many of them are 10-
20 years old, some are even up to 50, comprised of multivendor products, outdated technology,
and are very difficult to replace or modify (Mahmood, 2017). The complexity of these systems
also tends to mean that they may never be patched for software updates of vulnerabilities,
downtime is discouraged because no profit is made while the process is not running, there may
be no recovery or backup options, or it’s possible that a system cannot be shut down due to some
other reason (Stouffer, et al., 2015) NIST goes on to explain that unlike IT systems, updates to
an Industrial Control System need to be thoroughly tested by the vendors and the owners, ideally
using laboratory mockups. Updating ICS systems could also require recertification of the whole
system in order to comply with industrial or federal regulations (Mahmood, 2017). Babar
Mahmood (2017), Assistant Vice President of IT Security and Risk Management for MUFG
Financial Services, discusses that there are a lot of factors that contribute to the disdain for
updating and patching Industrial Control Systems, costs, downtime and a risk of required
recertification add to a general lack of the requisite skill sets between the IT and OT workforce.
There are additional movements within Industrial Control System environments to
purposefully connect these systems to the business network so that they can be monitored in real
time using traditional IT security practices, this has the potential for making these systems easier
to manage and administer, and thus more productive and profitable (Vijayan, 2018). Network

3
connectivity could even go so far as to provide for remote off-site access which could save time
for a technician, allowing them to quickly access the system in order to resolve a problem
without requiring travel, which has an additional benefit of reducing downtime and lost profits
(Vijayan, 2018). Jaikumar Vijayan explained that often there are many people who can access
and manipulate the control system, including process vendors and engineers, add this collectively
to operations, IT personnel, supervisors and leadership, and the potential exists for a significant
amount of people that could have the potential of gaining network access. Connection of these
systems to the business networks and possibly directly to the Internet increases the vulnerability
potential simply by providing connectivity outside of the process environment, possibly from
outside of the internal organizational security controls (Palmer, 2019).
Due to the high complexity and costs of design and installation, there is a tendency for
companies to leave Industrial Control Systems installed for a very long time, which has the
potential to lead to some problems specifically related to the supply chain (Livingston, Sanborn,
Slaughter, & Zonneveld, 2019). Obsolescence introduces a potential for a vulnerability pathway
which has not been looked at to the level of detail that it should (SENTRYO, 2017).
Historically, Industrial Control Systems were not designed with modern security functionality as
a design element, these systems were intended to be operated either in an analog environment, or
an air-gapped network with no connections to anything else outside of the system (Perelman,
2017). The modernization of these systems introduces more digitalized componentry into the
makeup of the design, often creating a system that resembles an Information Technology (IT)
system more than an Operational Technology System (Livingston, et al., 2019). The growing
popularity of connecting these systems to business networks and the Internet provides for
potential access to the systems which could be detrimental to their functionality (Vijayan, 2018).

4
Supply Chain Vulnerability
As the installed ICS equipment becomes older, obtaining hardware from the vendor could
become increasingly impossible as they phase out of one generation and move into another
(McCrea, 2018). As industrial control systems owners begin to have trouble obtaining spare
parts from the vendor, a customer may need to resort to seeking procurement options from
alternative, less trustworthy, third party suppliers or auction sites (McCrea, 2018). For a time,
obtaining unopened, new old stock could be an attainable option, eventually though used
inventory with a questionable past from third party suppliers or auction sites will become the
only viable option (International Association of Oil & Gas Producers, 2016). The International
Association of Oil & Gas Producers (IOGP) states that the vendor is going to recommend that
any customer who is facing this should update their system, but if an ICS has been running
without fail for decades, the difficulty of obtaining the occasional part is going to necessitate
acceptance of a reactive recovery strategy instead of beginning with the modernization of an
entire control system (International Association of Oil & Gas Producers, 2016). IOGP further
explained that the expense, complexity, downtime, engineering and loss of revenue are not worth
the outcome, so system owners and operators choose to do nothing until the need arises
(International Association of Oil & Gas Producers, 2016).
Obtainment of known good software is another factor related to an aging Industrial
Control System (Boyens, Paulsen, Moorthy , & Bartol, 2015). In the event of a disaster,
rebuilding of a computer, or replacement of a controller, PLC or other digital component could
require the reinstallation and configuration of vendor software, which may also be difficult to
obtain (Lee, Assante, & Conway, 2016). Eventually, failing to have a local copy, obtaining the
software could fall to looking to less than desirable websites, or outside sources (Boyens, et al.,

5
2015). Vendors will eventually also stop supplying and supporting aging software, and they will
typically encourage their customers to update to a later version, which may then require that
computers or hardware also be upgraded (Boyens, et al., 2015).
Technical skills and expertise for the systems is the final supply chain vulnerability to be
addressed in this paper. These lost skills can affect both the system owner, as well as the system
vendor (International Association of Oil & Gas Producers, 2016). As these ICS components
become older, there will be less retained knowledge, and through a lack of training opportunities,
competent technicians will eventually become harder to find, especially as the vendors
understandably begin to train their personnel to work on the newer model or next generation
options (International Association of Oil & Gas Producers, 2016). IOGP discussed how many
factors could lead to a shortage of skilled worker availability, a lack of a sustainable training
plan, employees transitioning to a new company or workforce reductions due to retirement, and
even forgetfulness due to lack of interaction with running systems. Eventually, outsourced
maintenance companies may begin to fill a need in attempting to support these older systems,
who may likely not have the detailed technical expertise necessary to properly and safely ensure
the integrity of the ICS is understood (Department of Homeland Security Office of Cybersecurity
and Communications National Cybersecurity and Communications Integration Center, 2016).
In all cases, the eventual outcome is likely to lead to forcing an ICS owner to source a
service from a middleman, who may be a potentially questionable entity through intentional or
unintentional actions (McCrea, 2018). The level of business acumen of the outside organization
may not meet or exceed the ICS owner organizations expectations, and the cybersecurity
controls, awareness and processes could also be less than that of the ICS owner (Boyens,
Paulsen, Bartol, Shankles, & Moorthy, 2012). The work practices of the outside organization

6
could additionally mean that the provided equipment hardware is not properly tested to OEM
specifications before being sold to a customer (Boyens, et al., 2012). It is also very possible that
the software which is available for provision is not as the original vendor intended it to be and
could contain malicious code, and possibly could be modified or corrupt in some manner
(Boyens, et al., 2012). Obtaining support services using technicians from outside of the vendor,
or even outside of the industry, who are possibly only qualified due to their willingness to
attempt to work on the systems, could have the potential to provide a pathway for poor decision
making, bad work control processes and could have an undesirable consequence, either through
purposeful or accidental means (Knapp & Langill, 2015). Most industrial control systems are
proprietary, and the requisite and specialized skills are very likely not available outside of the in-
house training provided to vendor personnel (Knapp & Langill, 2015).
Obtaining and utilizing these questionable products and services could lead to additional
unpredictable, difficult to diagnose failures, possibly immediate or erratic instability of the ICS
(Boyens, et al., 2015). A compromise in the integrity of an industrial control system could have
the ability to affect the safety of the employees, plant or community (Nuclear Regulatory
Commission, 2010). Evaluation and ultimate rebuilding of a compromised industrial control
system could be very difficult, especially if there are no disaster recovery procedures available
(Nuclear Regulatory Commission, 2010). Having a virus propagate through an industrial control
system may require at a minimum a complete software reinstallation on all computer equipment
(Lee, et al., 2016). If the compromise had the ability to provide malicious firmware to installed
control hardware, reinstallation of the firmware, or possibly replacement of the equipment could
be a requirement to ensure that the integrity of the system is as it was intended (Lee, et al., 2016).

7
The reputation of the vendor or industrial control system owner may also be affected by
any negative outcome to the integrity of a safely operating process control network (Stouffer, et
al., 2015). Having the right ingredients, a compromise could lead to the entire industry having
their reputation tarnished, at least for a time. The supply chain presents risks to industrial control
system operations that have potentially not been fully evaluated and could lead to catastrophic
consequences. Possible sources of compromise could be delivered intentionally or accidentally,
may come from a business competitor, or a highly funded nation state as an act of cyber warfare
(Stouffer, et al., 2015). All these vulnerabilities may be able to affect an industrial control
system through the supply chain by providing counterfeit parts, modified or corrupt software, or
by the introduction of careless, untrained, possibly malicious actors into the role of technicians.
The cascading effects could ultimately result in a requirement to consider the entire industrial
control system replacement or modernization.
Purpose of the Study
The purpose of this study is to analyze the supply chain risks which are posed to an
industrial control system from external sources. The goal of the study is to determine whether
the supply chain poses a great risk to ICS owners, providing supporting evidence of the findings
through the review of governmental regulatory guidance, industry reports and subject matter
expert evaluations. This research should additionally uncover potential solutions, procedures or
processes which may be beneficial to help limit the effect of or contribute to the remediation of
supply chain risks to an ICS owner.
Research Questions
Industrial Control Systems tend to be installed for a very long time, much longer than
traditional IT systems, and due to the requirement for them to have very little downtime,

8
patching and vulnerability management may be severely delayed or nonexistent. As these
systems get older, the hardware, software, and requisite technician skills become increasingly
difficult to find. The following questions will be addressed as an outcome of this research:
Q1. What are the Supply Chain risks to Industrial Control Systems?
Q2. How can the issue of obtaining known good replacement parts and
software be made more secure?
Q3. How can the problem of obtaining known good vendor support be
ensured?

9
Literature Review
The potential risks posed by supply chain vulnerabilities to Industrial Systems increase in
magnitude as the system gets older (International Association of Oil & Gas Producers, 2016).
The International Association of Oil and Gas Producers (IAOGP) (2016) explained that many of
these industrial systems, which are designed to remain in place, without being upgraded, for a
total lifecycle of 25 years, are often in continuous use for up to and exceeding 50 years. These
systems were originally not built with connectivity and security in mind, as such things did not
exist at the time. Upgrades, retrofits and replacement can be difficult financially and
operationally for an organization, especially while a system is still profitably running error free
(Mahmood, 2017).
Obsolescence
According to IAOGP, the impact of obsolescence can be felt in several ways. One of
these is upon equipment failure and subsequent discovery that replacements for the failed
components are no longer available from approved sources (International Association of Oil &
Gas Producers, 2016). Another possible impact results from the end of software provisioning
and production, ultimately resulting in an end of vendor support (International Association of Oil
& Gas Producers, 2016). Finally, through a loss of human competence, as the upkeep of training
programs become unsustainable, familiarity and technical skills will fade and are lost
(International Association of Oil & Gas Producers, 2016). Ultimately, a system becomes so out-
of-date that the vendor comes to a point of being unable to guarantee the availability of spare
parts, the ability to repair or recover a system, or even to provide any technical support of the
product (International Association of Oil & Gas Producers, 2016).

10
Recognizing that there is a potential future vulnerability, the risks should be mitigated
ahead of time through the development of recognized processes and procedures. Obsolescence is
a problem facing many systems, going even beyond the confines of industrial control systems,
but the comparison differs in the fact that industrial control systems are designed for a much
longer lifecycle, and are very often responsible for Critical Infrastructure related systems which
could have a direct effect on safety, security and emergency preparedness (SSEP), the shorter
expected lifecycle of alternate systems induces more frequent upgrades (Nuclear Regulatory
Commission, 2010). References to these types of non-ICS systems will only be made as
appropriate to support the main topic of research. The goal of this literature review is to discover
and compare mitigating methods which could potentially lesson the attack surface area of an
obsolete system, specifically industrial control systems. Going beyond the expected lifecycle of
an Industrial Control System, at the time of design, inception and installation, security was not
even a consideration for these systems, rather high reliability was the primary focus (Obregon,
2015).
Documentation
The research for this analysis was procured using the resources of the Utica College
Library, the scholar.google.com search engine, and governmental regulations and records,
reports, whitepapers and system expert analysis. The search parameter terms include: Industrial
Control System, Supply Chain, Risk Management, obsolescence vulnerabilities, and other related
combinations were used to narrow down some of the information and to gather research
resources. Much of the information can be found to be spread out through the analysis of
governmental resources such as the Department of Homeland Security (DOHS), Nuclear
Regulatory Commission (NRC), Nuclear Energy Institute (NEI), North American Electric

11
Reliability Corporation (NERC), Federal Energy Regulatory Commission (FERC) and the
National Institute of Standards and Technology (NIST). An attempt was made to focus on peer
reviewed articles and reports to assist in the research of the topic.
Hardware Risks
As any system ages, obtaining known good replacement parts becomes increasingly
difficult, this can be relatable to anything from cars to washing machine parts. Some of the
various problems regarding replacement hardware can be found through the insertion of
counterfeit components and unauthorized non-OEM reproductions. Parts with an unknown,
untraceable history could contain flaws through having been tampered with either on purpose, or
inadvertently. There are several examples of different types of tampering, including the insertion
of corrupt or maliciously installed software or firmware, as well as being subject to poor
manufacturing and development practices in third party or aftermarket vendors through poor
reverse engineering (Boyens, et al., 2015). Possible examples of tampering could also be the
inclusion of additional unwanted components into hardware, such as the addition of unwanted
wireless or cellular communications capability, GPS tracking devices, or the inclusion of
computer chips which may be capable of exploiting existing vulnerabilities, and allow for the
possible introduction of new, undiscovered vulnerabilities (Boyens, et al., 2015). Hardware
manipulation can be accomplished through the addition of unexplainable components such as
what happened with Super Micro Servers which contained the addition of a tiny microchip the
size of a grain of rice that wasn’t part of the main boards’ original design, this additional
microchip is suspected of providing for a backdoor, hardware level access for China into the
computers (Robertson & Riley, 2018).

12
Management of hardware obsolescence issues for industrial control systems needs to be
an ongoing process, requiring continued monitoring, risk assessment, reevaluations and
reprioritization of developed strategies to adjust the overall plan, this is especially true for
Critical Infrastructure (International Association of Oil & Gas Producers, 2016). Newer
hardware has the potential to not be compatible with older versions of software, and if the older
model of hardware is no longer available, development of plan should be considered to address
any eventuality before an event occurs (International Association of Oil & Gas Producers, 2016).
Some options for this can be included in parts replenishment paths through reverse engineering
and requesting another vendor make like for like components, provide repair of damaged
components, or planning an upgrade path for a partial modification (International Association of
Oil & Gas Producers, 2016). Modifications to industrial control systems could be worked out
through the vendor in order to limit the risk to the complexities of the system, or even through
another vendor who would provide modification options for an existing system.
Disposal of hardware may also contain risks, providing the opportunity for useful details
about the company to find its way onto an auction site, and possibly into the hands of a bad actor
(Radvanovsky, 2014). Robert Radvanovsky purchased third party ICS components with the
intention of connecting them directly to the Internet for a study he did in Operation
RUGGEDTRAX. Ensuring that devices are wiped clean or reset to factory defaults should be
part of the recycling process (CISA, 2018). CISA, a subset of the Department of Homeland
Security standing for Cyber+ Infrastructure, also recommended that sanitization of devices
include deletion of data through formatting or degaussing of the hard drive with a strong magnet,
removal or memory cards for physical destruction or overwriting data with random bits (CISA,
2018).

13
Software Risks
The vulnerabilities which are posed by software to an industrial control system could be
an ongoing, substantial risk to an organization, able to be compromised at any point in time
during its lifecycle (Polydys & Wisseman, 2009). Vendor software is susceptible to quite a
number of pathways for being vulnerable, from their own use of third party or open source
software in their offered systems, their provided updates and patches, or even through the end of
life disposal process of various media, hands on manipulation can allow vulnerabilities to be
discovered for systems that are still in use by other system owners (Ashford, 2017). Often, due
to the complexities of Industrial Control Systems, components are placed into service with the
expectation that they will be maintained and managed by outside vendors who would provide
assistance for any complicated maintenance or upgrades (Obregon, 2015). To simplify the
management and tracking of login credentials, industrial control systems are often installed with
their default configurations and settings in place, this would make it easier for the vendor to train
their technicians, and remove the complexities of tracking customer-specific passwords and
access parameters (Obregon, 2015).
Software obsolescence issues are not always considered in the same way as hardware
obsolescence issues (International Association of Oil & Gas Producers, 2016). Yet these
vulnerabilities can pose just as great a risk, if not a greater risk, to intellectual property theft,
business operations, safety and services, effecting an organizations reputation and trust (Polydys
& Wisseman, 2009). As an industrial control system ages, vendors tend to stop supporting their
software, for example such as when Microsoft stopped supporting Windows XP, forcing users to
migrate to Windows 7 (Microsoft, 2019). The ability to migrate software to newer operating
systems often becomes impossible, this alone could be a primary factor forcing owners to retain

14
outdated hardware and resist an upgrade (Rennie, 2017). Additionally, the fact that older
industrial control system technology traditionally was developed using specialized vendor
specific communication protocols, and were intended to act as standalone system, detached from
standardized IT systems, could lead to an eventual upgrade option becoming confined to a major
system overhaul (Obregon, 2015).
Human Competency Risks
A slow degradation of human competency and skills retention pose another risk to an
aging industrial control system. As a vendor continuously improves their offerings, introducing
next generation models, employee training would tend to focus more on the currently available
product line, with less emphasis and need to focus on the older options (International Association
of Oil & Gas Producers, 2016). Loss of available expertise for older systems may also be
attributed to simple employee attrition, an estimated 50 percent of skilled labor will retire in the
next 10 years, more will take positions with other companies, or change jobs or roles in their
current company through advancement and lateral movement (Rennie, 2017). Personnel
replacement, documentation availability and lack of expertise of current employees are an
eventual result of obsolescence, and moving into newer technologies forces vendors to train their
technicians to install and work on those newer systems as a primary function of their jobs
(International Association of Oil & Gas Producers, 2016).
The availability of system specific knowledge on both the end user and supplier side
becomes increasingly scarce the older an industrial control system becomes (International
Association of Oil & Gas Producers, 2016). As a vendor begins to lose the human competence
required to adequately provide support for a solution, an industrial control system operator
should begin to determine methods to mitigate the loss of the skills required to maintain their

15
system (International Association of Oil & Gas Producers, 2016). Options for this could be to
either bring the necessary training and support in house, providing for their own maintenance, or
to upgrade the control system in some manner that is agreeable to both the operator and the
vendor (International Association of Oil & Gas Producers, 2016). As Operational Technology
comes more into alignment with Information Technology, some of the high level skills may
become transferrable, but the overall philosophy is completely different between the two types of
systems, so they cannot be treated entirely the same (Mahmood, 2017).
Supply Chain Risk Mitigation
The idea of Supply Chain Risk Management (SCRM) looks toward mitigation of the
complexities associated with a globally distributed supply chain (Boyens, et al., 2015).
Development of processes, procedures, and controls for the purpose of empowering an industrial
control system owner to limit the risks of exposure to obsolescence issues is the goal of this type
of management plan (Boyens, et al., 2015). The target audience for having a Supply Chain Risk
Mitigation strategy goes beyond ICS owners, and actually is applicable to all Information and
Communication Technology (ICT) (Boyens, et al., 2015). Some of the foundational practices
which should be employed build on existing standardization and the maturity of other processes,
such as having baseline security controls, disaster recovery plans and programs, establishment of
multiple procurement relationships, sources and delivery routes and identification and
classification of “high impact” systems (Boyens, et al., 2015).
Ideally, obtaining components directly from an OEM or through authorized resellers
would be a preferable method, along with the use of anti-counterfeit policies, a method of
ensuring authenticity and a process to ensure tamper resistant and tamper-evident packaging
(Boyens, et al., 2015). Obtaining out of production components should come with the written

16
assurances of the provider’s ability to verify the integrity, security and quality of the provided
equipment, whether coming from the OEM, authorized or unauthorized resellers, or through a
reproduction (Boyens, et al., 2015). Tamper evident packaging during warehousing and
shipping, along with trackable shipping practices should be employed during the procurement
process as well to mitigate the potential for in route tampering (United States Nuclear Regulatory
Commission, 2019).
Lifecycle
Each organization must decide on a lifecycle philosophy for their industrial control
system, this potentially could require revisiting as often as needed. These philosophies could be
summarized into three different categories: run to failure, preventative maintenance only, or
perform scheduled upgrades and retrofits (Gaiaschi, 2015). Most organizations would likely
benefit from using a combination of these strategies, dependent on the risks, availability of parts
and other factors, possibly varying between system, subsystem or even component (Gaiaschi,
2015). Development of a lifecycle plan is an opportunity to get in front of an issue of
obsolescence before it is susceptible to a catastrophic failure, and can be scheduled so as to affect
only one system or train at a time, minimizing the impact to an organization through planning as
a result of unexpected, unprepared for downtime (Gaiaschi, 2015).
Run to failure. The idea of running a system until it breaks does not address
maintenance concerns until an actual component failure occurs (Gaiaschi, 2015). Pietro Gaiaschi
(2015) explains that the short-term results of this philosophy is lessened reoccurring maintenance
costs, but the effect is a higher long-term cost. Pietro Gaiaschi (2015) also explained that care
should be taken regarding choosing this strategy, consideration should include the functional
basis and outcome of the component failure, safety should play heavily into the decision-making

17
process. This maintenance philosophy will additionally require having an adequate spare parts
inventory on hand or available (Gaiaschi, 2015).
Preventative maintenance only. Preventative maintenance strategies involve some sort
of periodicity regarding inspection, testing, repair or replacement of system components
(Gaiaschi, 2015). Pietro Gaiaschi (2015) describes that this preventative maintenance strategy
has some added benefits such as reducing the number of spares which are required to be kept in
inventory, reduction of the costs and frequency of emergent repairs, and has a possibility of
reducing insurance premiums. Preventative maintenance does have its own disadvantages as
well, the potential for human error could have a negative effect on a running control system, due
to this, performance of preventative maintenance is best contained to scheduled system outages
(Gaiaschi, 2015).
Perform scheduled upgrades and retrofits. Eventually, all industrial control systems
owners are likely to be faced with the need to upgrade or retrofit their systems (Gaiaschi, 2015).
One of the methods of managing the risks associated with obsolescence involves a long-term
lifecycle plan, possibly extending out to 10 years or more (Gaiaschi, 2015). This plan could be
designed for minimum impact to an overall system, possibly aligning with scheduled outages on
one train or system, with the ultimate result over the ten-year plan being complete system
upgrade, all the while planning beyond (Gaiaschi, 2015). Pietro Gaiaschi (2015) explained that
upgrading in this manner allows for better planning, learning from lessons, more efficiently
utilizing resources, and often vendors will provide programs and incentives which could
maximize this approach. Development of a lifecycle plan could potentially occur through a
cognitive agreement between the control system owner and the vendor regarding some type of
modernization agreement (Rockwell Automation, 2019). Rockwell Automation provided some

18
options for upgrading including the following, all at once, in phases, per system or train, and
with their help if needed to identify and classify the most critical component or biggest risks to
pursue first. They discussed that upgrade options could be based on simple hardware or software
modifications or extend to complex system reconfigurations in order to optimize performance
and reliability long term. Reaching out to alternate, non-OEM vendors for recommendations and
quotes should be a part of any upgrade path decision.
Inventory Plan
During the design phase of an industrial control system obsolescence strategy, some
forethought must be placed into the amount of inventory to keep on hand (Boyens, et al., 2015).
Inventory philosophies can be centralized or per location, and depending on the criticality of the
component, the inventory practice could be a blend of the two strategies (Boyens, et al., 2015).
Inventory management can also be affected by the preventative maintenance strategy which is
decided upon by an organization, having more parts on hand may be necessary for a run to
failure philosophy, where regular preventative maintenance is not performed to ensure that
component failure is delayed (Gaiaschi, 2015). Another option for the management of hardware
obsolescence could be the development of a non-vendor relationship providing a pathway for
having the failed components repaired, rebuilt, reconditioned and recertified (Gaiaschi, 2015).
This would be especially important for severely out of production equipment, or in cases where
the vendor is no longer in business (Gaiaschi, 2015).
Development of an inventory plan to mitigate an event requiring an emergent obtainment
of a replacement component could save a substantial amount of money through expedited
shipping means alone. Replacements for hard to obtain inventory should be kept on hand, with
up to date timelines for procurement being available for other key components (Stouffer, et al.,

19
2015). Identification of key components can also be communicated to the vendor for assistance
in various strategies to incorporate into an inventory management solution, such as Last-Time
Buy offers, substitutions, or to begin the planning process for a proposal of a recommended
upgrade path strategy (International Association of Oil & Gas Producers, 2016).
Hardware Strategy
Obtaining known good hardware, OEM specification certified, unmodified and
guaranteed to be free of defects, can become very difficult for a control system owner,
exponentially so as the system ages and is subjected to the eventual issues of obsolescence
(Boyens, Paulsen, Moorthy , & Bartol, 2015). Validation of the integrity of the hardware
becomes more difficult as trusted distribution path relationships begin to fall away (Boyens, et
al., 2015). Forward thinking would be beneficial towards alleviation of this threat path through
the establishment of trusted distribution paths before an emergent event (Nuclear Energy
Institute, 2010). Validation of the overall security posture of reseller vendors and ensuring their
understanding of the requirements of shipping agreements such as tamper evident packaging
could be worked upon early on (Nuclear Energy Institute, 2010). Development of a relationship
with an outside vendor to provide known good, tested and verified hardware, recertified and reset
to factory specifications is an optional outcome for a planned strategic recovery path (ABB,
n.d.).
Management of the risk of hardware obsolescence may be possible through the use of
vendor buyback programs where vendors would buy back and refurbish components which were
decommissioned due to upgrades from any of their customers, allowing for more available
inventory, and subsequently extending the life of the equipment for other system owners
(Gaiaschi, 2015). A side benefit to this would be to help offset the availability of these

20
components to bad actors who could reverse engineer vulnerabilities for them or discover
potentially useful attack details as exposed through obtaining eBay parts such as were discovered
for Project RUGGEDTRAX (Radvanovsky, 2014). Robert Radvanovsky (2014) discussed in his
whitepaper that many of the equipment he purchased from auction sites contained former
customer details, such as name, address, phone number, network address and range and
hostname, along with the firmware version number, this information can all be very beneficial to
an adversarial attack strategy.
Software Strategy
Mitigation of the vulnerabilities pertaining to software could reside in obtaining known
good versions of software directly from the vendor upon installation or upgrades (Polydys &
Wisseman, 2009). Having a process to digitally store original versions of this software long term
allows for future use of software which doesn’t require later emergent obtainment through
questionable means, and should be a part of a disaster recovery plan (Ready.gov, n.d.). A secure,
air-gapped, central software repository, backed up in many locations, would be useful for storing
required software after it has been validated to be safe for use, which could provide for
cryptographic verification which would also ensure that it hasn’t been tampered with while in
storage (North American Transmission Forum, 2017). Cryptographic verification is a method to
verify that software is unmodified by providing a digital fingerprint, otherwise known as a
cryptographic hash value, of the intended software, any modification to any part would change
the cryptographic hash value (North American Transmission Forum, 2017). NERC recommends
that obtaining a cryptographic hash value from the vendor be provided using a different, alternate
method as obtainment of the software, ensuring that a compromised delivery path would not
intercept and ensure both matched.

21
Vendors can also provide their software utilizing a Self-Signed Digital Certificate
(Symantec, 2019). Symantec explained that the code signing of software provides a trusted
digital signature to content alongside of a cryptographic hash. A user downloading a software
would compare the hash used to sign the digital certificate against the hash upon download, as
well as validating the integrity of the root digital signature (Symantec, 2019).
NEI recommends that as a part of a defense in depth strategy, software be tested on non-
production systems before implementation (Nuclear Energy Institute, 2010). Often the patch
management program for industrial control systems means that nothing is done unless additional
functionality is required, or to address a stability issue (Department of Homeland Security,
2008). Maintaining a comprehensive inventory of operating systems, software, firmware, along
with the version details, coinciding with knowing the up to date vulnerabilities for all of the
software in use, should be understood by the system owners, along with the ongoing
development of a protection strategy (Ashford, 2017). Barring an ongoing patch management
process, some options for protections could include physical security or logical access control
systems, whitelisting, monitoring traffic and system hardening (Obregon, 2015).
Licensing
Vendor software often comes with license files, serial numbers or activation codes
(Evora, 2018). Allan Evora, a leading expert in control systems integration and President of
Affinity Energy recommended that license files, serial numbers or activation codes should not be
forgotten about, and care must be taken to ensure that these files and codes are handled in the
same manner as the software. Sometimes, vendor interaction is required for reinstallation of
software after a hardware failure (Templeton, 2013).

22
There is also the possibility that vendors could use a method of hardware-locked
licensing, where a software installation routine would examine the hardware configuration of a
host computer to develop an alpha-numeric code, essentially a digital fingerprint based on the
currently installed components (Templeton, 2013). This could mean that replacement of any
hardware would result in a different alpha-numeric code, this new unlock code would then need
to be provided to a vendor for a new activation code (Templeton, 2013). The hardware locking
of software requires interaction with a system developer during installation, and the older the
software becomes, the more difficult it will be for a vendor to be able to provide the support
necessary to provide for an activation code, this could be discussed with a vendor in advance of a
catastrophic event, possibly with agreement to remove the required hardware lock upon ending
support (Templeton, 2013).
Disaster Recovery Plan
For an Industrial Control System, failure is not an option, any downtime is seen as an
unacceptable outcome directly affecting profitability and the bottom line (Stouffer, et al., 2015).
Any sort of anomalous behaviors in an industrial control system, including failure, are likely
difficult to troubleshoot and recover from, and without having a backup or a plan, could actually
be irrecoverable without a modification or upgrade (Stouffer, et al., 2015). For industrial control
systems, the focus tends to be on the high availability philosophy which requires that the system
remain running indefinitely (Mahmood, 2017). Redundancy in industrial control systems is one
way to ensure continued run time of a system, failure of one component would automatically fail
over to a backup component (Lynch, n.d.).
Having a disaster recovery plans for any foreseeable event is recommended for industrial
control system owners, this could be system specific and involve multiple plans (Nuclear Energy

23
Institute, 2010). Being in a position to quickly react to, and recover from a failure of any kind
should therefore be part of a disaster recovery plan, and could include procedures to identify,
patch, clean or replace components or systems during a forced outage (Nuclear Energy Institute,
2010). A disaster recovery plan can be useful for restoration following physical disasters,
hardware failures, unsuccessful patching or software upgrades, as well as a severe cyber-or
physical attack (Department of Homeland Security, 2008).
Having a well-maintained backup and archive plan should be kept current, validated for
functionality at some planned frequency, kept for some predetermined retention period, and have
multiple storage locations and methods (Department of Homeland Security, 2008). Data can be
lost at any moment through theft, corruption, failed hardware or human error, development of a
backup plan should begin with what data is important, and a process for retrieving or replacing it
(Ready.gov, n.d.). Organizations also need to determine what an acceptable frequency for
backups is, regarding an Industrial Control System, it may only really be necessary as changes to
the software are implemented, this may not really be desirable though, possibly the recovery may
involve reinstallation steps for recovery of an older, captured hard disk image, and subsequent
installation of patching and configuration changes until the system is brought current (Evora,
2018).
Allan Evora recommends that the creation of disk images, as opposed to application
backups is the preferable method (Evora, 2018). Allan Evora discussed in his article how
application backups require the possible purchasing of a new computer, reload of the operating
system, setup of environmental variables, application of necessary Windows updates and
reloading application software and patches, very complicated, cumbersome and requiring many
steps. Disk images provide an exact replica of a system at a specific moment in time, these can

24
be copied and kept in different formats at different locations, shortening the restoration process
potential exponentially (Evora, 2018). Allan recommended a method for testing the recovery
process could be as simple as cloning a hard drive, removing the installed one and replacing it
with the clone to see that it works.
Summary
The fact that industrial control systems are designed to run continually for such a long
period of time effectively guarantees that the ability to maintain them would eventually begin to
become difficult (Mahmood, 2017). Today’s standards seem to dictate that an expected and
predictable lifespan is designed into technology, the eventual failure is accepted as normal
(Kaufmann, 2012). Industrial control systems being designed to run beyond the vendors’ ability
to provide for the continuation of them places owners in an awkward position, to replace an
entire system which is still completely functional and profitable or begin to stockpile
questionable and untested components (Gaiaschi, 2015). Recognition of the risks of the pathway
of vulnerabilities through the supply chain should bring about development of mitigating
strategies to remove the potential attack surface before an event occurs (Boyens, et al., 2015).
Industrial Control Systems exist within many of our Critical Infrastructures (Mahmood,
2017). The fact that Critical Infrastructure is defined as “so vital to the United States that their
incapacitation or destruction would have a debilitating effect on security, national economic
security, national public health or safety,” (Department of Homeland Security, n.d., para. 1),
should bring about a heightened sense of awareness with regards to the application of every
protective method possible (Department of Homeland Security, n.d.). Preemptively addressing
obvious problem areas before they become an emergent issue prepares industrial control system

25
owners to quickly respond to and recover from a system compromise (Nuclear Energy Institute,
2010).

26
Discussion of the Findings
This research project reviewed quite a few different examples of sources on this topic
including reports, white papers and regulatory requirements and recommendations. One of the
primary vulnerabilities and attack vectors to an aging industrial control system come from the
supply chain (Livingston, et al., 2019). The underlying design basis of these systems essentially
revolves around their being very reliable, extremely rugged, and capable of lasting for many
years, often decades (International Association of Oil & Gas Producers, 2016). Some of the
drawbacks to this would be that the hardware components, software, and technical competence
become harder to obtain as a system phases into obsolescence (McCrea, 2018). As a result, as
the equipment becomes more and more unobtainable, the quality of the supply chain sources
become less and less trustworthy, which creates an increasing vulnerability in the procurement
process (McCrea, 2018).
There appear to be three primary groups of vulnerabilities which a system owner can be
affected with through a breakdown in the trustworthiness of the supply chain (International
Association of Oil & Gas Producers, 2016). These three risks can be narrowly summarized into
hardware, software and human competence, and the severity becomes more likely and prevalent
the older a system becomes, eventually leading to complete obsolescence, with the highest level
of risk (International Association of Oil & Gas Producers, 2016).
The Supply Chain Risks
Due to the expectation that an Industrial Control System will last for such a long time,
eventual obsolescence of its technology is inevitable and predictable, which follows suit with any
system, the older it gets, the harder it becomes to support it (International Association of Oil &
Gas Producers, 2016). Often, the obsolescence of an industrial control system means not only

27
that it will become more difficult to manage, repair and maintain the equipment, eventually it
will become entirely impossible (International Association of Oil & Gas Producers, 2016). All
these lifecycle qualities can be directly attributed to the vendor supported supply chain and
subsequent procurement processes (Livingston, et al., 2019). The ultimate probability is that an
industrial control system can fall victim to these supply chain risks, and that the risks are
increased exponentially as a system gets older, which could occur though malicious or accidental
means (Livingston, et al., 2019).
The research which was conducted in this research project primarily focused on three
high level vulnerabilities, obtaining known good hardware, quality OEM software, and the
difficulty of maintaining a level of human competence (International Association of Oil & Gas
Producers, 2016). Each of these focus areas tended to degrade alongside of the obsolescence of
the installed equipment (International Association of Oil & Gas Producers, 2016). As with any
technology, developing new, more efficient offerings is the primary goal of any vendor in
competition, which leads to a research, production and support lifecycle for any product, at
which time they phase into the next offering (McCrea, 2018).
As an industrial control system becomes obsolete, hardware can become increasingly rare
(Boyens, et al., 2015). With only less trustworthy inventory to choose from, and the
unpredictable risk of counterfeit reproductions, any available inventory is susceptible to
tampering or poor manufacturing practices and failures in quality of workmanship, which can
introduce unknown, unplanned for weaknesses into the reliability of the control system, these
can be caused intentionally or accidentally (Boyens, et al., 2015). Coinciding with this, as a
system ages, clean OEM software also becomes much more difficult to locate and obtain,
vendors will phase out, and eventually stop supporting or providing older versions of software,

28
which will lead to owners looking elsewhere to obtain necessary copies, generally while they are
in the middle of an emergency, at which time they will very possibly be forced to settle with
obtaining a version from a less reputable source on the Internet (Boyens, et al., 2015). The third
and final risk addressed in this discussion involves the loss of technical human competence
(Rennie, 2017). As a system becomes more obsolete, vendor technicians are typically trained to
be proficient in their newer systems, engineers will retire, documentation is lost, and ultimately
the skills necessary to operate, maintain and repair these very complicated systems fade away
(Rennie, 2017).
Obtaining Known Good Software and Hardware
Some of the tangible risks of operating an industrial control system are that components
will fail, as they will with any system from washing machines to cars. Resulting from the failure
of these devices, coupled with the age of the system, is the ever-increasing difficulty of repair
(Gaiaschi, 2015). For a system owner, a disaster recovery plan needs to be a part of all critical
systems, having a plan to recover and restore a system before an event occurs allows for a
streamlined response, potentially down to a component level (Boyens, et al., 2015). Even with a
disaster recovery plan in place, eventually the vendor will move on and no longer provide any
support for systems as part of their life cycle plan, which would include provision of hardware,
their proprietary software, or technical support, whether remote or on site (Boyens, et al., 2015).
Hardware obsolescence leads to an eventual impossibility of obtaining any OEM physical
components that are known to be good, with a traceable history, certified and guaranteed to be
free of defects and material workmanship (Boyens, et al., 2015). A mature industrial control
system owner should be forward thinking and begin to establish component level plans to
mitigate or limit the risks of being vulnerable to questionable distribution paths (Nuclear Energy

29
Institute, 2010). Without an OEM option, the opportunity could exist for the development of a
relationship with an outside vendor who is capable enough to provide guaranteed equipment that
is tested and verified to comply with OEM factory specifications (ABB, n.d.). There is also the
possibility to source a vendor who can potentially repair or reproduce through reverse
engineering required unavailable equipment (ABB, n.d.).
Vendors could additionally themselves be cognizant of their own issues of obsolescence
and proactively offer to provide extended support opportunities (Gaiaschi, 2015). These
opportunities could involve the vendors themselves accepting the return of failed components for
repair, or buyback of functional equipment either used or from inventory from customer
upgrades and offering customers the ability to do a “last time buy” before completely cutting ties
with prior generation systems (International Association of Oil & Gas Producers, 2016). Having
these returned components available would provide for a larger inventory of OEM tested and
verified equipment, and would subsequently extend the life of the equipment for owners who
may not be ready for an upgrade plan discussion (Gaiaschi, 2015). Keeping these systems in the
hands of the original equipment manufacturer (OEM) also has an added benefit of potentially
limiting the access to a bad actor looking to reverse engineer a vulnerability (Radvanovsky,
2014).
Software obsolescence is another significant vulnerability posed to an industrial control
system (Boyens, et al., 2015). This could take the form of the original operating systems,
applications, patches or even firmware residing on the hardware (Lee, et al., 2016). Without
having a local back up copy, a customer who lacks the ability to obtain support from a vendor is
going to be required to find another means to procure the software, likely at a less trustworthy
website, possibly with malicious code included in it (Boyens, et al., 2015). A customer, as part

30
of a disaster recovery plan, should ensure that they have copies of the necessary software
versions and also include any documentation for all the restorative processes for their equipment,
including license keys, instructions, configuration or setup details (Templeton, 2013).
Loss of Human Competence
The human skills which are required in order to operate, maintain and repair an industrial
control system will also naturally fade away through system obsolescence (International
Association of Oil & Gas Producers, 2016). Loss of these skills can occur through multiple
means, such as normal employee attrition such as retiring, promotions and finding another job,
through the vendors no longer offering training on dated infrastructure, and forgetting through
lack of interaction (Rennie, 2017). Eventually, along with knowledge, even documentation,
becomes lost, mismanaged or in poor condition, which could happen for both the vendor and the
control system owner, people who understand the systems may also succumb to the issue of
obsolescence as they learn new skills, old skills are forgotten (International Association of Oil &
Gas Producers, 2016). At this point, a system owner may be forced to seek support through
vendors with no experience or training on the systems, which is typically only offered from the
OEM vendor, support which may simply be offered only through a willingness to try, often at a
willingness to take a risk, and with a hope to make a profit, possibly ending with a catastrophic
outcome (International Association of Oil & Gas Producers, 2016).
An industrial control system owner is faced with failures in human competence from both
within and without their organization. Involvement in the installation of an industrial control
system as part of a plant modification provides for understanding, expertise and skills for the
employer, in house (Rennie, 2017). The planning and design process helps to train people who
have the capacity to understand the functionality and requirements of the system, how it works,

31
how to maintain it and test the functionality of the individual components through being involved
in the decision making (Rennie, 2017). The problem for most companies, within 10 years 50
percent of the workforce will retire, taking those learned skills and knowledge with them
(Rennie, 2017). Significantly more will be lost to promotions or leaving the company, retaining
highly qualified employees becomes impossible, determination of a method to retain the skills
long term should be addressed by an industrial control system owner, failure to do so will hasten
the eventuality of being susceptible to issues of obsolescence (International Association of Oil &
Gas Producers, 2016).
Without any option for vendor provided training or support, industrial control system
owners have an opportunity and an obligation to consider bringing these training functions in
house as a benefit to themselves (International Association of Oil & Gas Producers, 2016).
Working with the ICS vendor for these sorts of options early on could develop agreeable
mitigating plans, beneficial to both, for prolonging this aspect of the obsolescence issue (Boyens,
et al., 2015). The idea of a Supply Chain Risk Management (SCRM) plan would work towards
mitigation of these threat vectors well in advance by identifying safe methods to procure
necessary equipment beyond the vendors end of support deadline (Boyens, et al., 2015). Another
of the philosophies which could be worked out through a vendor relationship would be a plan out
of, or avoiding obsolescence through a limited, prolonged support agreement between the owner
and vendor (Rockwell Automation, 2019). Knowing that the eventuality of obsolescence is
going to affect an owner’s systems, planning for the occurrence and working out a strategy with
a vendor is in their own best interest (Gaiaschi, 2015).

32
Collective Best Practices
These vulnerabilities all seem to have some options for mitigation through a good vendor
relationship and through the development of an agreeable plan of action such as a long-term
lifecycle plan (Gaiaschi, 2015). The Supply Chain Risk Management plan can also preemptively
identify, assess and determine ways to mitigate the risks of the global supply chain, at an early
stage through bringing in procurement relationships before a system failure leads to an emergent
repair (Boyens, et al., 2015). Many of these best practices are available through review of
several of the various documented federal regulations for the various industries and their white
papers (Nuclear Energy Institute, 2010). Often, common sense, specific mitigation strategies
may reside in alternate industrial regulations which are not directly associated with that of the
control system owner.
Having a healthy relationship between the owner and the vendor can work both ways,
and to the benefit of everyone, including other system owners (Gaiaschi, 2015). Recognizing the
complexities of industrial control systems, and the effect that catastrophic failure could have on
the safety of the plant, employees, and the general public should provide insight into encouraging
the development of an upgrade strategy for any system facing obsolescence (Gaiaschi, 2015). If
system owners classify the importance of their installed equipment, components and systems,
they could approach the vendors to help them with the design and development of a systematic,
affordable upgrade strategy (Rockwell Automation, 2019). Vendors could also help system
owners by assistance in arranging for a trusted alternate source for providing certifiable
equipment, possibly providing manufacturing specifications (ABB, n.d.).
Obtaining hardware should involve procurement through a supplier who can prove that
their security posture is able to guarantee the quality of the products they are offering (Boyens, et

33
al., 2015). This security posture should then be able to confidently allow for provision of written
verification and a guarantee as to the quality of the component (Boyens, et al., 2015). This
documented posture should be a resultant expectation from either an OEM vendor, an authorized
reseller, aftermarket producer, or even a procurement specialist arranged by the system owner
(Boyens, et al., 2015). Mitigation of the risks posed by the supply chain vulnerability should
also exist in transit, and should rely on an agreed upon shipping strategy, which should include a
trackable shipping method, and tamper resistant, tamper evident packaging at a minimum
(Boyens, et al., 2015).
Software can be very difficult to find in the original format once the vendor stops
supporting it (Lee, et al., 2016). Having a process to store the correct software versions and any
supporting documentation in place ahead of an emergent disaster recovery situation is the
recommended best practice (Ready.gov, n.d.). Obtaining known good, digitally signed software
from a vendor or supplier would be the ideally recommended best practice (Symantec, 2019).
Alternately use of a cryptographic hash code should require that they separately provide, using
separate means, a cryptographic hash validation, which is essentially a digital fingerprint for a
file, in order to ensure that the software is as they intended, without corruption or malicious
insertion of other code (North American Transmission Forum, 2017). The recommendation is
that the software and the cryptographic hash code would be shipped using two completely
different transmission processes, such as email, web download or mailing of physical media, and
phone or email of the cryptographic hash code (North American Transmission Forum, 2017).
Human competence, losing the skills required to maintain a system due to obsolescence
reasons is also an avoidable outcome, prior planning and forecasting can easily identify this
eventuality, and through working with the vendor a strategy that works for both can be

34
developed to mitigate the consequences of a complete lack of talent availability in the industry
(International Association of Oil & Gas Producers, 2016). Vendors typically own all of the
proprietary specialized training for their systems, and they tend to keep it in house, with the
eventuality of obsolescence, a system owner may be able to work with a vendor to possibly
provide the training program to them, so they can bring it in house, in order to train their
incoming technical employees (International Association of Oil & Gas Producers, 2016). The
loss of skills and human competence are just a natural part of the vendors need to develop ever
newer technology in order to remain competitive, so a system owner will eventually be required
to support these losses, or work with an industrial control system vendor to develop an
affordable, agreeable strategy to ensure that their systems remain somewhat current (ABB, n.d.).
Recognition of the current control system condition, development of a maintenance
strategy, knowledge of an upgrade path option, planning out obsolescence and developing
relationships with secure supply chain sources seem to incur maturity of a system owner
(Boyens, et al., 2015). Knowing which key components are most at risk and which components
are most difficult to obtain could lead to preliminarily developing an inventory process for
having a specific quantity of parts on hand ahead of time, with known replenishment lead times
being a consideration (Gaiaschi, 2015). The operators and owners of industrial control systems
hold the keys for many of the most important industries which are considered Critical
Infrastructure, and they have a responsibility to ensure that they are maintainable, supportable,
safe, reliable, and that anomalous events can be easily responded to (Department of Homeland
Security, n.d.).

35
Conclusion
The purpose of this research project is to demonstrate the vulnerabilities to industrial
control systems that are available through the supply chain (Livingston, et al., 2019). Due to the
reliability and longevity that is designed into these highly complex industrial control systems,
obsolescence becomes a serious threat vector, in various forms, specifically hardware, software,
and human competence (International Association of Oil & Gas Producers, 2016). Much of what
the world would consider as Critical Infrastructure has some facets comprised of varying degrees
of industrial control equipment (Mahmood, 2017). The definition of Critical Infrastructure
according to the Department of Homeland Security in the United States is a sector which is
“…considered so vital to the United States that their incapacitation or destruction would have a
debilitating effect on security, national economic security, national public health or safety, or any
combination thereof” (Department of Homeland Security, n.d., para. 1). This would indicate a
serious responsibility to ensure for the safe and reliable operation of these systems.
Compromise of these systems can potentially have catastrophic, possibly unsolvable
consequences if their operation is interrupted for any reason (Nuclear Regulatory Commission,
2010). Components of industrial control systems design often exist to ensure the safety of the
plant, it’s employees or the surrounding community, it may ensure some type of security, or have
features which provide for emergency preparedness (Nuclear Regulatory Commission, 2010).
Lack of preparedness for obsolescence eventualities by both the vendor and the system owner
may cause an event to be a difficult obstacle to overcome, through obtaining recovery
equipment, software or support, and may even negatively effect the reputation of the involved
parties, possibly the industry (Stouffer, et al., 2015).

36
Obsolescence truly effects everything, eventually all things fail and become difficult to
impossible to obtain, maintenance strategies fail, even knowledge and technical skills are lost.
Industrial control systems are comprised of very proprietary components, most of them are likely
only produced by the original equipment manufacturer, and upon their end of life, support and
availability will come to an abrupt end (International Association of Oil & Gas Producers, 2016).
Once an industrial control system is in this end of support phase, spare parts inventory dwindles
to nothing, known good software becomes more difficult to obtain, and human technical ability
moves on, fading into different careers, retirement, or lack of use (International Association of
Oil & Gas Producers, 2016). Eventually an unprepared system owner will face this supply chain
vulnerability head on with no option but to utilize a questionable procurement recovery path
(McCrea, 2018). Hardware may not be properly tested or manufactured, software could harbor
unknown threats, and well-intentioned organizations could offer technical services for which
they are not qualified (Boyens, et al., 2012).
The supply chain vulnerability should be a recognizable risk that both a system owner
and vendor should be very cognizant of mitigating, making plans early on, ideally beginning
with acquisition of the equipment at installation (Rockwell Automation, 2019). A customer
should be aware going into the initial system purchase what the expected timeline is for a vendor
production timeframe and support cycle, with the vendor providing regular notifications
recommending routine options for avoidance of any eventual obsolescence issues (Boyens, et al.,
2015). System owners would be wise to have a disaster recovery plan in place for their systems,
they should know what the most critically important parts and components are, what their
maintenance philosophy is going to be on each component, and preemptively develop an
inventory strategy (Nuclear Regulatory Commission, 2010). Knowledge of these would lead

37
them then into a Supply Chain Risk Management plan where the customer would work out ahead
of time the details of what methodologies are going to go into obtaining their unobtainable needs,
either directly through the OEM, their resellers, or alternately using aftermarket sources (Boyens,
et al., 2015).
With about ninety percent of the Critical Infrastructure being privately owned and
operated, the importance of these systems to a country requires that their owners understand the
threats, have mitigating countermeasures planned and in place, while strategic upgrade paths
should be understood (Stouffer, et al., 2015). The safe operation, maintenance and upkeep of
industrial control systems requires that active planning go into potential future failures (Boyens,
et al., 2015). Understanding the consequences of any and all possible failure potentials ahead of
time allows for proper planning through the concise development of processes, procedures and
controls (Boyens, et al., 2015). Federal regulations currently have the potential of requiring
recertification of an industrial control system modification or upgrade, causing delays, added
stress, financial burden and uncertainty to a system owner (Mahmood, 2017). This likely results
in the ultimate outcome of discouraging system upgrades and avoidance of obsolescence
(Mahmood, 2017).
Obsolescence avoidance is a topic that requires constant vigilance, failure of an industrial
control system will likely have a debilitating effect as defined by the Homeland Security
description of Critical Infrastructure (Department of Homeland Security, n.d., para. 1). Federal
regulations should encourage thoughtful, planned technical modifications to industrial control
systems and pave the way for painless evolution, possibly being involved in the planning
processes to streamline recertification. Having prepared supply chain controls in place should
not only apply to issues of aging infrastructure, utilization of things like tamper resistant

38
packaging, seals, digitally signed software and cryptographic hash codes for software would be
just as applicable to new installations as for procurement of existing equipment supplies (Nuclear
Regulatory Commission, 2010). There are many recommended guidelines already in place, they
are although, not all in one place, this is where development of an overall security strategy which
best befits an organization comes in, providing the tools required to withstand the inevitable
catastrophic, sudden failures brought about by obsolescence (Gaiaschi, 2015).

39
References
ABB. (n.d.). Refurbished Parts Service for distributed control systems. Retrieved from Asea
Brown Boveri LTD.: https://new.abb.com/control-systems/service/offerings/spares-and-
consumables/refurbished-parts-service
Ashford, W. (2017, April 13). Six key security weaknesses in industrial systems. Retrieved from
Computer Weekly: https://www.computerweekly.com/news/450416794/Six-key-
security-weaknesses-in-industrial-systems
Boyens, J., Paulsen, C., Bartol, N., Shankles, S. A., & Moorthy, R. (2012, October).
nvlpubs.nist.gov. Retrieved from Notional Supply Chain Risk Management Practices for
Federal Information Systems: https://nvlpubs.nist.gov/nistpubs/ir/2012/NIST.IR.7622.pdf
Boyens, J., Paulsen, C., Moorthy , R., & Bartol, N. (2015, April). NIST Special Publication 800-
161: Supply Chain Risk Management Practices for Federal Information Systems and
Organizations. Retrieved from nvlpubs.nist.gov:
https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-161.pdf
CISA. (2018, October 30). Security Tip (ST18-005) Proper Disposal of Electronic Devices.
Retrieved from US-CERT: https://www.us-cert.gov/ncas/tips/ST18-005
Department of Homeland Security. (2008, December). Recommended Practice for Patch
Management of Control Systems. Retrieved from ICS-CERT: https://ics-cert.us-
cert.gov/sites/default/files/recommended_practices/RP_Patch_Management_S508C.pdf
Department of Homeland Security. (n.d.). Critical Infrastructure Sectors. Retrieved from
Department of Homeland Security: https://www.dhs.gov/cisa/critical-infrastructure-
sectors
Department of Homeland Security Office of Cybersecurity and Communications National
Cybersecurity and Communications Integration Center. (2016, September).
Recommended Practice: Improving Industrial Control System Cybersecurity with
Defense-in-Depth Strategies. Retrieved from Homeport.uscg.mil:
https://homeport.uscg.mil/Lists/Content/Attachments/1557/NCCIC_ICS-
CERT_Defense_in_Depth_September_2016.pdf
Evora, A. (2018, February 20). Disaster Recovery: Preparing for SCADA Computer Failure.
Retrieved from Affinity Energy: https://www.affinityenergy.com/disaster-recovery-
preparing-scada-computer-failure/
FireEye. (n.d). Industrial Control Systems and Critical Infrastructure. Retrieved from
ThreatProtectWorks: https://www.threatprotectworks.com/Solutions-for-Industrial.asp
Gaiaschi, P. (2015, August). Control system obsolescence. Retrieved from Digital Refining:
https://www.digitalrefining.com/article_1001151.pdf

40
International Association of Oil & Gas Producers. (2016). Obsolescence and life cycle
management for automation systems Recommended practice. Retrieved from
http://www.energysafetycanada.com/files/pdf/process_safety/551_pd.pdf
Kaufmann, D. (2012, September 11). 'Designed to fail' electronics a global problem. Retrieved
from DW: https://www.dw.com/en/designed-to-fail-electronics-a-global-problem/a-
16369155
Knapp, E. D., & Langill, J. T. (2015). Hacking Industrial Control Systems. Retrieved from
ScienceDirect.com: https://www.sciencedirect.com/topics/computer-science/industrial-
control-system
Lee, R. M., Assante, M. J., & Conway, T. (2016, March 18). TLP: White, Analysis of the Cyber
Attack on the Ukrainian Power Grid, Defense Use Case. Retrieved from nerc.com:
https://www.nerc.com/pa/CI/ESISAC/Documents/E-
ISAC_SANS_Ukraine_DUC_18Mar2016.pdf
Livingston, S., Sanborn, S., Slaughter, A., & Zonneveld, P. (2019, January 31). Managing cyber
risk in the electric power sector: Emerging threats to supply chain and industrial control
systems. Retrieved from Deloitte Insights:
https://www2.deloitte.com/insights/us/en/industry/power-and-utilities/cyber-risk-electric-
power-sector.html
Lynch, G. A. (n.d.). PLC Redundancy. Retrieved from ICS Engineering Inc.:
http://www.icsenggroup.com/plc-redundancy.shtml
Mahmood, B. (2017, December 27). The State of Security in Industrial Control Systems.
Retrieved from The State of Security: https://www.tripwire.com/state-of-security/ics-
security/state-security-industrial-control-systems/
McCrea, B. (2018, October 03). SUPPLY CHAIN: 5 Ways to Address Component Obsolescence.
Retrieved from SourceToday.com: https://www.sourcetoday.com/supply-chain/5-ways-
address-component-obsolescence
Microsoft. (2019, June 7). Windows XP support has ended. Retrieved from Microsoft Support:
https://support.microsoft.com/en-us/help/14223/windows-xp-end-of-support
North American Transmission Forum. (2017, November 6). Software Integrity & Authenticity.
Retrieved from NERC:
https://www.nerc.com/pa/comp/guidance/EROEndorsedImplementationGuidance/CIP-
010-3%20R1.6%20Software%20Integrity%20and%20Authenticity.pdf
Nuclear Energy Institute. (2010, April). Cyber Security Plan for Nuclear Power Reactors.
Retrieved from NRC.gov: https://www.nrc.gov/docs/ML1011/ML101180437.pdf
Nuclear Regulatory Commission. (2010, January). Regulatory Guide 5.71: CYBER SECURITY
PROGRAMS FOR NUCLEAR FACILITIES . Retrieved from nrc.gov:
https://scp.nrc.gov/slo/regguide571.pdf

41
Obregon, L. (2015, September 23). Secure Architecture for Industrial Control Systems.
Retrieved from SANS Reading Room: https://www.sans.org/reading-
room/whitepapers/ICS/secure-architecture-industrial-control-systems-36327
Palmer, D. (2019, March 27). Half of industrial control system networks have faced
cyberattacks, say security researchers. Retrieved from ZDNet:
https://www.zdnet.com/article/half-of-industrial-control-system-networks-have-faced-
cyber-attacks-say-security-researchers/
Perelman, B. (2017, May 16). ICS Environments: Insecure by Design. Retrieved from Security
Week: https://www.securityweek.com/ics-environments-insecure-design
Polydys, M. L., & Wisseman, S. (2009, February). Software Assurance in Acquisition:
Mitigating Risks to the Enterprise. Retrieved from Defense Technical Information
Center: https://apps.dtic.mil/dtic/tr/fulltext/u2/a495389.pdf
Radvanovsky, R. (2014, October 21). Project RUGGEDTRAX Preliminary Findings. Retrieved
from LinkedIn SlideShare: https://www.slideshare.net/BobRadvanovsky/ruggedtrax-
findings21oct2014prelim
Ready.gov. (n.d.). IT Disaster Recovery Plan. Retrieved from Ready.gov:
https://www.ready.gov/business/implementation/IT
Rennie, I. (2017, June 28). Obsolescence Management of Software . Retrieved from Asset
Guardian: https://www.assetguardian.com/obsolescence-management-of-software-
components/
Robertson, J., & Riley, M. (2018, October 4). The Big Hack: How China Used a Tiny Chip to
Infiltrate U.S. Companies. Retrieved from Bloomberg Buisinessweek:
https://www.bloomberg.com/news/features/2018-10-04/the-big-hack-how-china-used-a-
tiny-chip-to-infiltrate-america-s-top-companies
Rockwell Automation. (2019). MODERNIZATION: Staying Competetive in a Global Market.
Retrieved from Rockwell Automation:
https://www.rockwellautomation.com/en_NA/capabilities/industrial-maintenance-
support/overview.page?pagetitle=Modernization&docid=4577964e48bcb918747a354e7f
92c738
Rouse, M. (2016, March). critical infrastructure. Retrieved from WhatIs.com:
https://whatis.techtarget.com/definition/critical-infrastructure
Schwab, W., & Poujol, M. (2018, June). The State of Industrial. Retrieved from Kaspersky Labs:
https://ics.kaspersky.com/media/2018-Kaspersky-ICS-Whitepaper.pdf
SENTRYO. (2017, January 17). Why are ICS Vulnerable? Retrieved from sentryo.net:
https://www.sentryo.net/why-are-ics-vulnerable/

42
Stouffer, K., Pillitteri , V., Lightman , S., Abrams , M., & Hahn, A. (2015, May). Guide to
Industrial Control Systems (ICS) Security. Retrieved from NIST:
https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-82r2.pdf
Symantec. (2019). How Code Signing Works. Retrieved from Digicert:
https://www.websecurity.symantec.com/security-topics/how-code-signing-works
Templeton, T. (2013, November 8). A Brief History of Software Protection and Licensing.
Retrieved from Templeton Interactive: http://www.templeton-interactive.com/blog/a-
brief-history-of-software-protection-and-licensing/
United States Nuclear Regulatory Commission. (2019, May 30). PART 73—PHYSICAL
PROTECTION OF PLANTS AND MATERIALS. Retrieved from U.S. NRC:
https://www.nrc.gov/reading-rm/doc-collections/cfr/part073/full-text.html#part073-0054
Vijayan, J. (2018, March 19). 8 questions to ask about your industrial control systems security.
Retrieved from CSO Online: https://www.csoonline.com/article/3262641/8-questions-to-
ask-about-your-industrial-control-systems-security.html

An analysis of the supply chain risk

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (17)

Similar a An analysis of the supply chain risk

Similar a An analysis of the supply chain risk (20)

Último

Último (20)

An analysis of the supply chain risk