Topic: Analysis of the SolarWinds exploit
Axel Kloth, Founder and CTO of Axiado, shares his insights on the so-called SolarWinds hack in 2020. This discussion will cover:
- How it happened
- Why it is dangerous
- What we can and should expect in the aftermath
- Why it is so difficult to remove and prove that it is gone
The presentation is followed by a Q&A session with event attendees.
2. What is the SolarWinds Hack?
• In the press, you’ll see phrases such as “supply chain attack”.
• While that is a correct classification, it does not really explain how it
came about and what it is, and neither does it tell us what the
severity of the hack is.
• Let’s analyze it (and I’ll try to keep it as simple as I can).
3. SolarWinds – Background and Tools
• SolarWinds Inc. is an American company that develops software for
businesses to help manage their networks, systems, and information
technology infrastructure. (from Wikipedia)
• SolarWinds has an impressive list of tools available, some for free,
that help system and network administrators with operation,
administration, maintenance and provisioning (OAM&P) of networks
and servers.
• OAM&P tools have become a must-have since data centers grew to
thousands, if not hundreds of thousands of servers, in each one of
them.
• Manual OAM&P had become impossible a decade ago.
4. SolarWinds Toolset Overview
• Network Automation
• Performance Monitoring
• Bandwidth Analysis
• Manage Configs
• IP Address Management
• Switch Port Management
• VoIP Monitoring
• Troubleshooting Tools
5. SolarWinds Toolset Overview
• The toolset is intended to help system and network administrators in
deploying infrastructure and troubleshooting elements of the
infrastructure.
• By the very nature of these tools, probing and testing is crucial.
• The same applies to updating elements – firmware and software.
• That’s where the trouble begins.
6. Trouble in Paradise
• The OAM&P tools mostly run on Linux.
• They are deployed from a set of servers that manage the users’ or the
tenants’ servers, routers and switches as well as firewalls and other
infrastructure.
• These tools must have high levels of privilege to work properly.
• For any kind of firmware updates, they must have access to the
administrators’ username (usually “root” or “admin”) and the
passwords.
• For industry-standard x86-64 servers, the usual path to update
firmware is through the Baseboard Management Controller.
7. Baseboard Management Controller
• The Baseboard Management Controller (or in short BMC) is
effectively a PC inside the server to support server administration.
• All BMCs in use today are based on ARM processor cores and run
Linux or variants thereof.
• They have full access to the server and its components.
• They can update the servers’ and their own firmware.
8. BMC Communication
• BMC communication is facilitated through an additional network port
or a virtual port over a server’s network connection.
• There is no innate or inherent support for any kind of hardware that
improves access security to the BMC.
• The communication between an OAM&P control center (often called
an NMC for Network Management Center) and the data center is
assumed to be through an IPSec Virtual Private Network (VPN)
tunnel, with a set of firewalls on both ends.
• BMCs do not support security functions in hardware for above
reasons – the communication is assumed to be secure.
9. Attack Scenario
• Use the OAM&P Software to penetrate the BMCs in target data
centers.
• Once the BMC is penetrated and under the attackers’ control, use the
BMC features to install malicious firmware for the servers and their
BMCs to the degree needed.
• The server’s new firmware allows anything you want from exfiltrating
username/passwords to outright theft of data from the servers and
thus the data center.
• You can even disable logging functions on the server, and since most
firewalls are standard x86-64-based servers, disable the firewalls.
10. How do you get into the OAM&P SW?
• SolarWinds defended itself well against all kinds of attacks.
• They were not breached directly.
• The attackers were smart enough to understand that an attack against
SolarWinds directly would be detected.
• Instead, they opted for an indirect attack.
11. Building Tools
• Tools are built using internal (usually protected, version-controlled
and signed) and external repositories.
• It is not uncommon to see that a tool is built from thousands of
different pieces of source code.
• The build scripts for these tools are usually protected, version-
controlled and signed as well. Breaching those is difficult.
• Once an attacker knows which uncontrolled, non-versioned and
unsigned pieces of source code or DLLs (pre-compiled source code)
are included in the build, an attack path forms.
12. Compromising a Repository File
• An attacker can pose as a legitimate developer of source code on any
project, in any repository.
• Pick an important file that – when included in the build – has access
to crucial data.
• Compromise this file, compile it and make it a DLL or any other non-
reviewable piece of software, and contribute it to the repository.
• On the next build of tools using the file you’ll be included (and have
access to whatever you want).
13. Compromised Build
• Once the compromised file is included in the repository, and the tool
build has taken place, you have your desired backdoor into the tools,
and therefore into any infrastructure that uses this toolset.
• At this point in time you can log into the NMCs servers to access any
client and tenant that is administered out of this NMC.
• With that access you can compromise any server, router, switch and
firewall in your target network.
• If done right, it leaves no traces.
14. Implications
• Even well-protected targets can be compromised, even when and if
they log all server accesses.
• You have access to anything you want.
• Your target might never find out it was compromised.
• Since your target does not know it was compromised, it might leak
data permanently.
• How do you clean up if you don’t know that you were compromised?
15. Conclusion
• This is one of the worst imaginable hacks ever.
• In essence, everyone must consider itself compromised and review its
entire infrastructure.
• Even well-protected targets with good security practices might be
compromised and not know it.
• There are already multiple variants out there.
• Raindrop, Teardrop, Sunspot, and Sunburst are known and are
confirmed deployed.
