3. 1
SSH User Key Remediation
Executive Summary
Sloppy management of authentication keys for SSH, an encryption protocol used for
automation in IT systems, risks catastrophic IT failure in banks, government and industry. Most
organizations have no process for managing, removing, and changing access-granting keys.
This violates SOX, FISMA, PCI, and HIPAA, all which require proper control of access to servers
and proper termination of access.
A cyber weapon or virus using SSH keys to spread could destroy IT infrastructure and online
data of an enterprise or government agency within minutes. Similar attacks were first used by
the Morris worm, which took down the Internet in 1988.
IT staff can use poorly managed SSH keys to create permanent backdoors to production
servers, bypassing all normal privileged access management and session auditing systems.
Risky insiders include anyone who has ever had access to a server, including former system
administrators, consultants, and anyone with access to backups or decommissioned hardware.
The problem has been largely overlooked by IT security auditors, because the it is highly
technical and hidden inside the system administrator domain in a distributed environment where
no single person has had full visibility of the problem.
Every organization should audit their SSH keys and other automated access mechanisms
(including Kerberos). Furthermore, they should set up and enforce policies for managing
automated access and remedy existing risks or face an existential threat when viruses using
SSH keys to spread inside an organization emerge. Remediation includes removing unused
keys, enforcing controlled key setup processes, automating key setups to reduce number
of administrators who can set up keys (and to save costs), changing all keys regularly, and
controlling what can be done with each key.
This white paper focuses on SSH user key remediation as a process which all organizations
utilizing SSH should be aware of and consider implementing. It will outline a basic process and
set of tools which can be utilized to identify the existing trust relationships in your environment,
bring legacy keys under control, and automate the creation, deployment, rotation and removal of
keys.
4. 2
SSH User Key Remediation
Virus and Cyber Weapon Threat
A cyber weapon or virus can be engineered to use SSH keys to spread to nearly all servers
within an organization once it has penetrated one server. This ruins layered defense, and
experience has shown that viruses frequently manage to penetrate individual servers in
an organization. A cyber weapon would likely use multiple attack vectors to penetrate an
organization, and could use SSH keys (or other trust relations such as Kerberos) to spread
throughout the server organization’s server infrastructure in minutes after penetrating the first
server.
In 1988 the Morris Worm utilized trust relationships for automated access based on what was
called “rhosts” authentication. In nearly all modern systems this has been replaced by trust
relationships based on SSH user keys. A recent IBM X-Force study found most attacks against
Linux/Unix servers utilize SSH. Many computer forensics experts are aware of cases were SSH
keys have been used to spread an attack from one server to another.
Experience has shown that most organizations have on the average 8 to 100+ SSH keys
configured granting access to each Unix/Linux server. Some keys also grant high-level
administrative access. The mesh of key-based access is so dense that it is highly like that an
attack can spread to nearly all servers in an organization, especially if the virus also utilizes other
attack vectors to escalate privileges to “root” (high-level administrator) after penetrating a server.
Implementing SSH keys as an attack vector in a virus is quite easy, requiring only a few hundred
lines of code. There is no doubt this will be exploited, soon. Once inside a server, a virus may
open a backdoor, steal, alter or corrupt data, subvert encryption systems or databases or
outright destroy the server and any data (including backup media) accessible to the server.
In the worst case, a virus or cyber weapon using multiple attack vectors could spread globally,
Internet-wide, enterprise-wide in a matter of minutes. Combined with destruction technologies
it could wipe out on-line data, hot/warm stand-by disaster recovery systems and backups
accessible via tape robots and render most servers in enterprises inoperable by wiping their
operating systems, storage (including network drives), and reprogramming flash memories,
device firmware, and configuration memories.
The worst case could mean life with no functioning banks, retail chains unable to process
payments or supply inventory, logistics companies unable to operate effectively, most
government agencies down, gas stations without gas with refineries having shut down and
logistics being very inefficient, no or limited telecommunications and significant parts of power
generation and distribution networks down for weeks or even months.
5. 3
SSH User Key Remediation
Exposure to Substantial Insider Risks
Improper management of SSH user keys opens a substantial insider risk to the primary
information artery of many large enterprises. After studying the SSH user key management
operations of a substantial number of the world’s largest enterprises numerous concerning
trends have appeared. The following trends appear to be common:
• Insufficient (sometimes completely missing) controls for who can enable permanent keybased access to a system account (also called functional or process accounts) on a
production server
• Keys are stored and SSH servers configured in a manner that permits anyone logging into
a user account to configure new authorized keys for login, effectively granting permanent
access even if original access was only limited to a particular administrative
task
• No documentation what each key is used for and why it was created
• No systematic removal of keys when they are no longer needed (because organizations
do not know what each key is used for, and removing needed keys could cause critical
applications to fail)
• No systematic rotation (change) of SSH keys. Keys may have been copied by any
administrator having had access to an account holding the private key. They are small
enough to be even written on the back of one’s hand, or may leak from backups or
inadequately wiped decommissioned hardware - and continue to provide access until the
authorized key is removed or keys are changed
• No visibility as to how many SSH keys are granting access to how many servers
• No visibility as to who can access what servers using key-based access
• Normal privileged access management / auditing systems can be bypassed using keybased access, opening an effectively uncontrolled, unaudited backdoor into production
systems.
The insider risks are substantial and involve in some cases hundreds of system administrators,
people who can access to backups, people with access to decommissioned hardware, and
include personnel and contractors who may have had access to systems in the past in addition
to current employees in relevant roles.
Most Enterprises Are Non-Compliant with Mandatory
Security Regulations
Most enterprises, government agencies, and health care providers appear to be non-compliant
with mandatory security regulations and laws, as well as internal security policies (including in
some cases policies mandated by their customers), because they do not properly control, audit,
and terminate key-based access to their IT systems and data.
6. 4
SSH User Key Remediation
All major security regulations - SOX (Sarbanes-Oxley for public companies), FISMA (US
government), HIPAA (Health information), PCI (Credit card processing) - as well as NIST
standards and equivalent international standards - require controlling access to servers and
proper termination of access. These standards are being violated by improper management
of SSH keys that allows permanent backdoors to be left on critical servers, bypassing normal
controls for privileged access.
Key-based access is extremely widespread, and it is the best practice for automated and
scripted access to servers. While as safe as anything when properly managed, it turns out
nearly all organizations have been sloppy in managing key-based access. Similar problems also
abound with other automated access mechanisms, including Kerberos and certificate-based
access.
The situation is not a result of any vulnerabilities or flaws in the SSH protocol itself or in the most
commonly used implementations of the protocol (including commercial Tectia SSH and the open
source OpenSSH products).
Rather, it is a result of years of lack of clear guidelines or policies relating to SSH keys, lack of
understanding of the scope and implications of the problem, insufficient time and resources
to dig into the issue to gain understanding or develop solutions, earlier lack of good tools and
guidelines for solving key management issues, and perhaps a general reluctance of auditors
to flag issues for which they don’t know effective solutions exist. In addition to this, identity
and access management as a field has focused primarily on interactive user access with little
consideration of automated access. The fact of the matter seems to be actually that there are
significantly more permanent trust relationships for automated access than interactive user
accounts in many environments.
The problem has probably gone under radar because it is deeply technical and obscure, in the
domain of system administrators. Each system administrator typically only sees a small corner
of the IT environment, and does not have a full picture. Administrators and their managers are
often so busy that while they may recognize that there is a problem, they simply have not had
time to analyze the scope or possible implications of the problem.
Finally, to date, many IT security auditors do not have SSH user key management on their
checklist in relation to its role in identity access governance, and the issue has not been
sufficiently highlighted in implementation and auditing guidelines for SOX, PCI, FISMA,
or HIPAA connecting access control to authentication methods. Even many CISOs (Chief
Information Security Officers) are only vaguely aware of the problem, and many CIOs and IT
risk management professionals have never heard of it. Yet at the same time it represents an
existential risk for any major bank or enterprise and a systemic risk for the banking system or a
developed country.
7. 5
SSH User Key Remediation
SSH Key Remediation as a Process
Before the problem can be remedied, its existence must be recognized. Since remediation
often involves a substantial amount of effort and affects operational processes in IT, obtaining
funding, approvals and urgency for change, a remediation project may require the co-operation
of IT security managers, IT risk management (including IT auditors), IT/Security architects, IT
operations managers, and even higher-level general management (especially since in many
cases a remediation project should be started without waiting for the next budget cycle).
The problem generally involves all Unix/Linux servers, many Windows servers (especially
those using SSH/SFTP for file transfers or scripting, which is increasingly common and
recommended), IBM mainframes (commonly using SSH/SFTP based file transfers and remote
command execution), and even Mac servers in heterogeneous environments. Thus several
teams within IT operations may need to be involved in the remediation project and proper
support and endorsement within the organization is needed.
Remediation of the problem involves several steps:
• Discovering all existing trust-relationships (who can access what). This requires discovering
of all user keys that are authorized for login (“authorized keys”) and all private keys (“identity
keys”). This must be done for all users on all servers (and preferably also desktops), and
involves several special issues.
• Monitoring the environment in order to determine which keys are actually used and where
the keys are used from (one or multiple sources) and removing keys that are no longer in
use
• Enforcing proper processes for all key setups and other key operations by relocating
all authorized keys to a root-owned location and changing SSH configuration
accordingly, so that only the key manager (or root) can remove or install new authorized
keys, and detecting unauthorized key operations occurring outside the key manager (e.g.
keys set up by a root user) and generating alerts about such key activities
• Automating key setups and key removals, eliminating manual work, human errors, and
reducing the number of administrators who need to have sufficient privileges to do key
setups from possibly several hundred to essentially nobody (though root can do it, but
such setups are detected, and the few administrators with high-level access to key manager
can also cause it to create new keys)
• Rotating keys, i.e., changing every authorized key (and corresponding identity keys)
regularly, so that any compromised (copied) keys cease to work and proper termination of
access can be ensured
• Controlling where each key can be used from and what commands can be executed
using the key.
8. 6
SSH User Key Remediation
A further risk mitigation technique is to define certain internal boundaries within the organization,
and strictly control what key-based trust relationships can cross which boundaries and in which
direction, and enforce strict IP address restrictions and “forced command” restrictions at least for
authorized keys involving trust relationships crossing such boundaries.
It should be understood that while a key manager can reliably discover all authorized keys on
the servers that it has access to, it is not possible to guarantee finding all copies of private keys.
Some copies of private keys could be on desktops and laptops that are not scanned, on off-line
USB memory sticks, backups, powered off machines or even printed on paper for later manual
entry. Thus one can never say with absolute certainty who can currently log into an account
with an authorized key. Only after rotating all keys can one be sure that previously copied keys
cannot be used.
Conclusion
Nearly all Fortune 500 companies and major government agencies appear to be non-compliant
in terms of how SSH user keys are managed in relation to access control and authentication.
In general the major security issues which organizations consider on a day to day basis are
primarily focused in relation to virus risks, hacking risks, and rogue employee risks. Nonetheless,
the identity access governance around automated accounts directly correlates to these obvious
risks in the most simple terms of cause and effect. Without the proper management of SSH
user keys and proper remediation of legacy environments, organizations face significant
increased risks to the above mentioned threat vectors, even existential threats.
This is however not an issue unique to large enterprises. Organizations of all sizes face
challenges in managing SSH user keys. In fact, most of the next 10,000 largest companies in
the world are also exposed to the same issues. On the other side of the coin, even customers
with less than 20 servers in smaller security-sensitive organizations can benefit from automated
systematic key management practices. It all depends on the risk tolerance in terms of managing
the access control of the critical servers.
In summary, while SSH is considered the gold-standard for data-in-transit security, the current
threat landscape requires organizations to rethink how they are managing access to their SSH
networks. The SSH protocol has done a superb job in protecting data-at-transit, but effective
management and oversight of the access to the SSH environment is now becoming a major
concern. With Universal SSH Key Manager, organizations of all types and sizes can quickly
and easily take control of their secure shell environment, remediate the usage of their SSH user
keys, and meet today’s modern security challenges while fulfilling or exceeding compliance
mandates such as SOX, PCI, FISMA, and HIPAA.
To get started with SSH key remediation in your environment, please visit:
www.ssh.com/remediation or contact sales@ssh.com.
