Recon 2017: By Kirill Nesterov, Alexander Tlyapov
Digital Substation is an essential part of every electrical network. It is also a base ground for modern Smart Grid technologies. More than 4000 of IEC 61850 compatible substations operated in Europe, 20 000+ worldwide, each of the comprising communication and flow of gigawatts of electrical current between large power plants (thermoelectrical, hydroelectrical or even nuclear) and their respective consumers. Such consumers include cities, industrial objects and power plants themselves. During this talk we will focus on security analysis results of key Digital Substation component - Relay Protection Terminals. Protective relays are devices for detection of electrical faults. When such fault is detected relay device designed to trip a circuit breaker. Without them problems like over-current, over-voltage, reverse power flow, over-frequency, and under-frequency can lead to colorful and impressive pictures of giant electric arcs accompanied by bunch of sparks with total blackouts as a result.
Nowadays protective relays became digital devices with network access through which operators can access different services like self-testing, statistics, logs and others. More of it, electrical lines are also combined with fiber-optic lines for communications. Electrical part of such lines need minimal traffic, but protection against surges. So such lines can be leased to different organizations, exposing great target for attacker. All of services inside such networks are available through different industrial protocols like IEC 61850 (MMS, GOOSE), IEC104 and Modbus, a not very industrial protocols HTTP, FTP, SSH and everybody’s favorite proprietary protocols. We will show how to dig very deep inside Relay Protection Terminal and how to abuse numerous weaknesses and vulnerabilities inside.
5. Substation in a nutshell
• Primary system devices
• Circuit breakers, Disconnect and grounding switches, Power transformers,
Instrument transformers, Generators
• Secondary system devices
• Protection, Reclosers, Annunciators, Meters, sensors, Fault recorders, Control
switches and interfaces
• Computers are here!
https://en.wikipedia.org/wiki/Electrical_substation
6. Substation in a nutshell
• Primary system devices
• Circuit breakers, Disconnect and grounding switches, Power transformers,
Instrument transformers, Generators
• Secondary system devices
• Protection, Reclosers, Annunciators, Meters, sensors, Fault recorders, Control
switches and interfaces
• Computers are here!
https://en.wikipedia.org/wiki/Electrical_substation
8. Substation in a nutshell
• Everything is in IEC 61850
• Set of protocols (GOOSE, MMS, SV, etc.)
• Dafaq Substation Configuration Language (SCL)
• Digital Protective Relay (also IEDs)
• Network bacchanalia
• Interconnections with substation
• System operator, Billing, Transmission support
• Ethernet, Power Line Communications (PLC)
9. Security of substations
• IEC 61850
• tldr; No security
• Exploiting the GOOSE Protocol: A Practical Attack on Cyber-infrastructure by Juan Hoyos,
Mark Dehus, Timthy X Brown
• Poisoned GOOSE: Exploiting the GOOSE Protocol
http://crpit.com/confpapers/CRPITV149Kush.pdf
• IEC 62351
• tldr; use No security via SSH tunnel
• Set of words to encapsulate everything from IEC 61850 in encryption
• Haha, you know, distribution owners update and vendors provide updates
10. Antiviolence reminder:
transformers and geoshmalitics
• Santa Barbabararaba is in another universe
• We are not electrical engineers and that is not the point of the talk
• Yes, we heard that transformer is not like Optimus Prime
• They just didn’t saw them transforming
• While colors don’t match, Eleron gas source is planet Cybertron!
• If you want bash us for electrical misanything - just call your therapist
?
11. Generic Relay Terminal Internals
• PowerPC (MPC860)
• RTOS
• Protocols
• IEC61850 (MMSLite)
• Proprietary protocol for updates
• Optional Web
• Poor debug facilities
• Today’s menu
• En salada la Switzerland, Germany, France, USA
13. The software is divided into two main parts:
• Common firmware (bootloader, RTOS pSOS+ code, …)
• Modules that implement additional protocols (IEC61850, DNP3, Modbus, …)
The firmware is available as a file with the extension ".PCK" included with the application
for the installation - FIRMWAREUPDATE.EXE
PCK File is a container with .KON files, xml with update options and soon
Target device – SIPROTEC 7SJ64x
14. Contains records with file description
struct PCK_file_record
{
char Name[252];
DWORD CRC;
DWORD Size;
}
PCK file format
15. Files in 7SJ64X_04.93.01.PCK
Code stored in KON files. One PCK file may contain KON files for different CPU. In this example we
have bootloader variants for CCPU and 384K.
PCK file format
16. KON file is set of tagged records with different types. Structure of the record header:
struct KON_section_header
{
char SectionTypeName[4];
DWORD size;
}
struct KON_file_header
{
char Signature[4];
KON_section_header Sections[];
}
In the present case we had the following types of records:
• “HEAD” (char code_type[4]; DWORD minaddr; DWORD maxaddr; DWORD entry_point ; DWORD xorcks)
• “INFO” (char unit[8]; char device[8]; char version[15]; BYTE number)
• "TITL“ (char title[] )
• "DATC“ (DWORD datc_start_addr; DWORD datc_size; DWORD crc)
• "DATA“ (DWORD start_addr; DWORD datca_size; DWORD crc)
• "ENDE“
• https://github.com/rigmar/Recon2017/tree/master/SIPROTEC
KON file format
17. Section type name
Section size
File type signature
TITL section body
INFO section body
HEAD section body
datc_start_addr
Entry point addr
File header of “bootld_C_V2.kon” from firmware:
So, we know bootloader start address (0x40000) and entry point (0x40004). Trying to load in IDA
PRO and see some problem:
• Part of code has different than 0x40000 base address
• ANOTHER part of code has base address that is different from the previous two
KON file format
18. bootloader code
0x40000 0x66700
First part
0x40750
0xFFF00000
Second part
0x53078
0xFFF1292C 0xFFF3361C0xFFF20000
Third part
0xFFF02C30
0x340000 0x34FCA0
Fourth part
Hardcoded KON file0x2000
0xFFF25294
0x20948
Finally bootloader code map memory view
0x2000 - 0x20948 0x40000 - 0x66700 0x340000 – 034FCA0 0xFFF00000 – 0xFFF1292C 0xFFF20000 – 0xFFF3361C
Nested Doll - Matreshka from Germany
19. Architecture and OS of device
• pSOS+/PPC V2.0.7
• Upgrade with Ethernet module EN100
• Same CPU
• Same OS
• TCP/IP communication
• Port forwarding
29. Java applet
• CPU service
• Some proprietary 56797/udp protocol
• Some diagnostics
• Some password check on user-side
• But it’s not hardcoded
• It’s confirmation code 311299
• Some read out of bounds => crash
30. Crash >= RCE
• Defective mode
• Can be fixed only with manual reboot
• No protection
• Terminal runs in “Monitor mode”
• Tested with RETOM device
• True for core CPU bugs
Relay protection and automation testing system
32. The firmware is available as a file with the extension “.bin”. For example – “A09ma604.000.bin”.
At offset 0x100 starts loader code:
First instruction – “ba 0xFFB10104”, therefore, the base address of firmware is a 0xFFB10104.
Target device - F60
33. Loader copies 0x1D4F8 bytes from 0xFFB10270 to 0x1F80000.
F60 Firmware unpacking (1)
34. This bytes contains zlib uncompress code that unpack main firmware code from 0xFFB2D768 to 0x8000.
F60 Firmware unpacking (2)
35. If uncompressing is finished successfully, code at offset 0x1F80000 is cleared
F60 Firmware unpacking (3)
36. Finally, control is passed at offset 0x8100 in the uncompressed code.
F60 Firmware unpacking (4)
37. Global Device Objects
• Thousands of them
• Backed up by EEPROM
• Inheritance level ~ 3
• Strongly typed => Unified access
38. This values has db based view, that initialized using hardcoded value descriptions.
Example of such object
39. At start we were analyzing firmware version 6.04. So, on vendor’s website has newer one.
New in firmware v. 7.31:
• VxWorks 6.8
• And that has VxWorks symbols!
Well, knowledge of the names of functions and global variables really doing life better
Sometimes, the new version is really better
42. Modbus
• Authorization
• Different modes
• Password is a 32 bit number or username with password
• Bruteforce protection
• R/W Access control
• Old Enervista protocol
48. Comfortable terminal
VxWorks
PowerPC
FS access
VxWorks img is ELF
Symbols
Traceback with PC and LR
And something more...
49. Comfortable terminal
debugsrv
7755/tcp – stdout with additionals headers
7766/tcp – stdin
Can be switched in boot
• VxWorks console
• Internal debugger
• Arbitrary calls by name and by address
• Many more
50. Ref630 DB based
• All data in DB that is stored on file system
• Database files are divided into three types:
• Basic – plain data, no encryption and compression
• Sequential – compressed data blocks
• Secure sequential – compressed and encrypted data blocks
51. Ref630 Encrypted DB files
• Blowfish algorithm
• Encryption key depended on
interfaces IP addresses
52. Ref630 Encrypted DB files
• Two interfaces
• Loopback with IP address 127.0.0.1
• Common with external IP address
• Hardcoded string
54. IEC 61850
MMS Lite from SISCO
Cares about security
Some info about secpatches
But “SISCO does not provide detailed technical information of
any kind (security related or otherwise) on our products to
anonymous or unknown persons”
55. MMS Lite
No sources
Some dumb fuzzing => No results
Again some bug in user-hooks
56. Path traversal
u_mvl_fopen_ind
Used to read COMTRADE files
But allows to read any file on flash
Reported 26 Jul 2016
Device credentials
stored in DB file
Hashed with MD5
57. Pimp my term!
Schneider Electric
Fancy TV for your terminal
220 Service ready on KAOS system
Magelis xbtgt5330
Only one default port
58. Firmware as OS
Firmware consists of several .dlm files
.dml – is ordinary PE
x86 based
Some kind of KAOS system
But KAOS looks like Windows App
PTC Perc “Real-Time” Java machine
59. Device management
Vijeo management tool
Works through FTP
FTP has some proprietary extensions
TGID
WRDI
...
60. Smart TV
Can be integrated with bunch of terminals
Some vendors even recommend it
A lot of SW extensions
62. Third-party party
• Almost every IED (with IEC61850) uses SISCO MMSLite
• Mocana SSH
• Allegro ROM Pager
• Third-party soft is Good
• Update problems
63. Substation-ng
• Remove embedded devices
• Goodbye, VxWorks!
• Goodbye, PowerPC!
• Signal acquiring from power lines still required
• Put all protection processing in virtual machines
• Application running on Windows box
• Only HI-TECH countries
64. In the end
Still just an embedded device
Real-Time requirements
No encryption
No exploit mitigations
Updates are slow/manual/hard
A lot of people still writing their own HTTP Servers
65. @scadasl kudos
@atimorin Alexander Timorin
@_Rigmar_ Alexander Tlyapov
@arbitrarycode Alexander Zaitsev
@GiftsUngiven Alexey Osipov
Anatoly Katushin
@repdet Gleb Gritsai
Sergey Gordeychik
Sergey Sidorov
iGrids Lab
Maksim Nikandrov
Viktor Nikitin
And others
http://scadastrangelove.blogspot.com
66. iGrids Lab
• Cheboksary, home of ‘Bouquet of Chuvashia’ beer
• https://en.wikipedia.org/wiki/Chuvashia
• Substation (“releyka”) capital of RF
• Certification laboratory
• (ad) Access to numerous substation devices by subscription
• (russian) http://igrids.ru/
• Open challenges on conferences
Hello everyone, I’m Kirill Nesterov and my friend Alexander could come for some reasons. So I’ll tell you a story about our journey into world of relay protections which are the part of a digital substation. So let’s start. First of all I’ll describe what is relay protection and what role does it play in electric power transmission.
Small disclaimer. All opinions are my own and not the views of my employer. So let’s start.
Here is an awesome NIST picture with Conceptual Model shows overall lifecycle of energy management from generation sources, how the flow of electricity is controlled and how it gets to our homes, factories and other destinations. From now on we will talk only about Transmission part of this process, but interesting fact is that generation itself depends on transmission as it needs electricity to operate. For example, Nuclear Power Plant will have up to 3 substations powering it, cause in case of electricity faults reactors could be stopped to prevent overheating/melting – say hello to cascade effect.
Transmission is done by a set of substations and power lines. Basically, substations are routers for power lines coming from generation facilities to customers distribution. Those routers are controlled by various parties who manage, bill and whatsoever the economics of the energy distribution. Our point of interest is a substation and the equipment inside it. Though the same equipment for example Protection Relay Terminals, is used much wider: in distribution part, in railroad traction substations, small diesel/gas powered facilities – substations are the most important of them.
Substation differ a lot by voltage level, their role in energy network, and most interestingly by their customers. As we already mentioned Nuclear Power Plant will have 3 substations powering it, less critical destinations will not have such redundancy.
Substation can have a number of input and output lines. Electricity from them is passed through a set of primary equipment (primary system devices), for example transformer, in order to either lower voltage or make it higher, or various switchgear to route electricity in required destination.
Though, the transformer does looks like a nice spot to conduct an attack as its explosion would sweep the substation, they are actually hard to attack as most of the time only electronics they have is diagnostics, they are insulated with some dielectric or have other safety features, and in the end, many things will melt or brake before transformer, but not him. There are obvious ways to get to transformer – rtfm sabotage manuals. There are a lot of them in the internet.
Simple Sabotage Field Manual - http://svn.cacert.org/CAcert/CAcert_Inc/Board/oss/OSS_Simple_Sabotage_Manual.pdf
Here I’ll show you some demo. At first you can see two examples of triggering the circuit breaker while it’s line is powered. So It looks charming but real damage is minimal. As I have mentioned before It’ hard to get the transformer, but sometimes it’s still possible. So here you can see a fail of protection system and overheating of the transformer. But still it’s a petty damage. Because real one will look like blackout of the city or even stop of some kind of a factory. And restart of industrial objects after an accidental stop can be very expensive.
Secondary system devices are our goal as those are in charge of protection which includes overcurrent protection, differential protection, distance and other. This includes data acquisition from substation and sometimes joint substations, calculations and if required action to prevent damage due to some fault, for example trigger of circuit breakers.
IEC 61850 is how substation work. And modern substation works almost exclusively via MMS and GOOSE protocols. Which are Ethernet, but GOOSE have another Ether type. The data inside those protocols is described in substation configuration language during substation configuration. SCL is represented as XML with description of configuration of protecting relays and supporting devices, like network switches. It can hold network configuration and electrical configuration for protection processes. Of course, industrial vendors included a pack of proprietary protocols to relays which allow to reconfigure, update and monitor devices.
You can imagine that substations are part of some closed network, but this is actually the other way around. Inside substations you will find a ton of various systems from video surveillance, physical access system, various switchgear monitoring, SCADA system, network equipment, billing systems to utility communication (telephony, DECT, etc.). All these systems are not only managed by different teams by its role, but also represent up to 3 or 4 different companies.
Security of substations is a very dull thing. Current standard IEC 61850 has no security in it and this has been discussed one thousand hundred million billion times. The point is there are MMS and GOOSE protocols (and more others), they are open and non-authenticated, and provide means for reconfiguration or distribute measurements between Intelligent Electronic Devices. It’s a matter of replay or spoofing attack to start influence normal processes at substation. This could or could not cause the news headlines with words blackout, balalaika or baboon – not interesting. The point is that if your are in a subnet or have network access to protection relay on substations you’d only need ARP spoofing, packet replaying or open source MMS, GOOSE tools for attack. The fact that all protection relays are supported by a typical ICS out of this millennia software only emphasizes the legendary complexity of potential attack.
New standard IEC 62351 is meant to add security to all this madness. And while the theory is good for ideologists and marketing, the madness is not easily fought with implementation phase. But this is an answer to a very good question: What is the purpose of the talk when things are so bad? We wanted to look at substation of future that would be more or less secure at least typical office network grade and analyze the security of embedded devices in charge of safe electric power distribution.
We are kindly asking you to leave your interesting question and thoughts about real world buzz in a trash bin. We most probably made and will make mistakes related to electrical engineering terms – just skip them. The point of the talk is reverse engineering of embedded devices found on susbstations.
Элерон – это один из газов используемых в оболочках для трансформаторов. Оболочки с газами не проводящими электричество используют, чтобы когда ебнет дуги успокоить.
We have met nearly ten different relay protection terminals and here You can see the most generic things about them. So on such device you can expect to see some Real Time OS, the most popular was VxWorks. Also every device that we saw implements two protocols. It’s MMS and HTTP, which in some cases can be turned off. Another generic problem with them it’s debugging, without access to hardware it’s very hard to get even some crash info. And in the end there for some reasons vendors like to build their devices on top of PowerPC’s CPU MPC860.
Here You can see our first target which is SIPROTEC from Siemens. It’s the forth generation of such devices, we have heard that the fifth one exists but we have not met it anywhere. So it’s typical for relay protection box with small display and number of buttons. Let’s take a closer look at firmware format, because It’s complicated comparably to others.
Final firmware for the device consists of two parts – OS with bootloader and number of additional modules with different industrial protocols. But everything is packed inside of proprietary PCK format. Such PCK file is a container for different number of KON files and xml with their description.
As You can see PCK header is very simple. It have three fields: name, check sum and a size of current element.
Here is an example of enumeration of such PCK file. As you can see it contains several KON files with appropriate names. From this names you can understand that they contains firmware for different kind of devices with different CPU types.
And KON format is a set of tagged records with short name and size. This short name informs the updater about what kind of data it has. For example tag DATA means that it’s a raw data and tag DATC is a compressed record. Each type has it’s own fields. And because of code lays inside of DATA or DATC record it has appropriate field for loading address. Also you can notice that some of the records have additional check sums for some reason. Maybe because it can be a standalone file format. Also we released parser library for SIPROTEC’s firmware format.
So here is a colorful representation of a KON format based on bootloader. One of the presented sections here is DATC section which is compressed data with code. When you load it into IDA you can clearly see that whole binary can be divided into several parts depending of their offset.
If you dive deeper into the code you will see that bootloader’s startup code copy parts of itself into different memory locations. And here is some attempt to describe it’s memory map. So in the end we are able to parse and unpack firmware so let’s take a closer look into system.
So when we look inside the code of the firmare we will find that it’s based on some rare pSOS. It’s a RTOS which was born in early 80s and than after some business stuff was bought by Wind River – developer of VxWorks, they were rivals. So you can understand It’s fate. Also main CPU module can be upgraded with different kinds of communication modules. One of then is Ethernet module. Funny thing about it is that it has the same CPU, same OS and communicate with main module with some sort of TCP/IP protocol.
Here is services which we have discovered on nearly default configuration. Some of this services work on main CPU and forwarded their by EN100 module, other work on EN100 by itself. Here you can see two proprietary protocol. First one is DIGSI – it’s a protocol for the main configuration tool and on the Java’s applet protocol we will soon take a closer look. But let’s start from my favorite HTTP service. I think for some reason developers which need to implement HTTP service always have a desire to implement it from scratch as it was with SCADA system WinCC OA from same company.
And as we see developers of EN100 firmware couldn’t resist this desire. May be because HTTP always looks very simple. But one of the problems is how to hide some development or debug info inside of your new HTTP server?
The password is a good answer! So with this three ‘x’ symbols you can get inside of debug interface which allows to read whole memory and so on. So let’s take a closer look at some of “protected” options.
After the password check there are two input forms. They are rather similar. So first one tries to find a taskname substring in user request and if it finds it will try to read several integers into stack buffer. So we could get in some trouble in case of s specifier for scanf function. Because all the code relies in addresses with leading zero and because it’s a big endian system it’s impossible to overwrite return address without corrupting this zero byte. But here you just drop your payload and pivot the stack. Like in ninety's. So let’s try to write some values.
Of course the EN100 module will crash and web error log will show us crash info, which is very convenient for devices of this kind because you can see not only where it have crashed but also the state of all it’s registers. As you can see program counter is equals to ascii chars of A and this crash looks like it can lead to code execution. Moreover there are read write and execute memory rights on some of the network buffers which make the exploitation even more easier. But a little notice, the code section have only read and execute which is right. So I don’t know why execution prevention wasn’t implemented fully.
Also after we submitted bugs to Siemens and they patched everything we’ve got little stuck with CVSS scores which are 5.3 for remote code execution. It’s on some kind of a smart network card, but it’s still code execution.
So now I’ll show you some demo.
So we have found simple overflow with scanf function. But searching for vulnerabilities can be pretty simple if you taint network controlled data to the scanf calls. Because a lot of them can be vulnerable.
Another interesting service is Java applet which can be downloaded through web. But when loaded it connects to device through UDP port 56797. It allows to see device state and send some remote commands to it.
So it’s one of the services which forwarded to main CPU module. It communicates with custom protocol. Show small amount of diagnostics after you enter right password witch is checked on client side. At first we thought it’s hardcoded, but then we found some kind of document with it. This document describes actions in case of problems with the device. So it’s not really a password, but a “confirmation” code. Simple protocol fuzzing revealed some read out of bounds. In case of CPU service we can’t get any debug info through web error handler as before.
Why this read out of bound can be more serious threat than RCE in EN100 module? There is special mode inside of the device which is called Monitor mode. This mode is activated after three fails in short period of time. In this mode device turns into expensive brick and if any electrical accidents happens it will just ignore them and they will be escalated to the next level of protection dealing more damage. This case was tested with this device called RETOM, which simulates emergency situations and checks protection.
Next device is F60 from General Electrics.
This firmware format is much simpler than the previous one. It have some header and after one hundred bytes in hex it have code.
Loading process contains relocation of self part into new address.
And then it just uncompresses main part of it.
If uncompressing is finished successfully, code at this offset is cleared.
Then control is passed to uncompressed code. Nothing strange.
It’s more interesting how it works inside. There are thousands of global objects. Their initialization function even doesn’t shows in IDA graph view on default settings. They are strongly typed and every object have proper getters and setters, so you don’t have to worry about wrong usage of strcpy. Also they are build in heavy Object Oriented style with average inheritance level of 3. State of every such object is backed up by EEPROM. And when we saw this code at first there was no decompiler for PowerPC. I’m not a big fun of reversing large cplusplus code inside of just an disassembler, because of difficulties with structure offsets. Then HexRays choose PowerPC between PowerPC and MIPS and It made me on of the happiest man in the world, at least for some time.
Here is an example of initializer for such global object. I hope you can see that it has a type, a name, default value and also a Modbus number, because on this devices Modbus is one of the main protocols. And nearly every such object can be called for reading or writhing through Modbus. You can even virtually push buttons on the device through Modbus.
At first we took 6th version of firmware and wasted a lot of time on understanding it. Just because it was firmware version of lab device. But then we found that in 7th version vxworks symbols were not stripped. So sometimes it’s rather useful to analyze several firmware version at once.
Here is number of services we have found on newer firmware. As you see my favorite HTTP protocol still here.
It looks like selfmade HTTP server. But when I’ve played a little with it turned out that it’s extremely simple and have no user input at all. It’s so lazy that it ignores everything except GET and URL.
Next was Modbus. It looked promising. But it turned out to be an authorization on top of Modbus with bruteforce protection and proper access control. After some google we could found that different people had some research on security of this device and even made parts of their masters degree. Main configuration tool Enervista worked through Modbus and even update firmware through some extensions. The only bad thing about it is that it doesn’t have encryption. But even this was fixed soon.
In newer versions of firmware we have found MocanaSSH service. Mocana is an SSH implementation for embedded devices. It’s a new Enervista configuration protocol and It looks rather secure. Let’s take a glimpse at it.
I was staring on this function names for a long time. And after that I definitely see some correlation in main callback functions. Now I clearly can say that It has example substring inside of them. What can it mean?
But I won’t say what it mean. Communication is the sad part about this vendor. Before this presentation we contacted all vendors and asked about coordinated disclosure. So everyone responded with ok and provided detailed information about patches except this one. We had no response at all. I have found four potential remote code execution, some of them a really funny in my opinion. But in light of recent events I will not disclose any of found bugs because I heard that there is a Recon in Montreal and It’s hard to visit such conference when you are in jail or under sanctions.
But the main problem with multilin devices that without physical access they don’t have any any debug facilities. So after the devices crashes you will see a lonely string inside of log. Just a system exception. No debugger, no crash info, no traceback, nothing at all. And I’m pretty lazy to guess what have gone wrong in static.
And the next relay terminal with database architecture is REF630 from ABB. This DB architecture is similar to global objects in GE terminal, but it’s backed up by custom db format on an normal filesystem. Also to get a firmware you only have to get an access to an ftp service. Because this FTP gives full access to flash. As other devices it has http and iec61850 and two additional protocols SPA and ODBC. Which allows to work with different objects inside of database.
And it was the most comfortable terminal that I saw. It has Vxworks with symbols. Like the others it’s based on powerpc cpu. You can have full access to filesystem through FTP. If something gone wrong you will find a file with a traceback. And something more.
It has a debug server which can be turned on with special option in configuration file. So when it’s running it opens two ports. One for stdout with some custom headers and the second one is raw stdin. Every string received from stdin port is executed inside of vxworks shell. it’s differs a from telnet but just a bit. So if you will find some bug which can provide to write-what-where primitive you just need to turn this service on. And you don’t have to upload any code to the device at all. So this service allows to interact with native vxworks console which in most cases contains internal debugger and many other useful stuff. So we will walk through some of the services but before that I will describe format of the database files.
There are three types of database files. First one is a simple plain data format. Second one is compressed. And the last one is compressed and encrypted. So let’s take a look at that encryption.
It uses blowfish algorithm. And here is a key generation procedure. The resulting key depends on ip addresses of the device’s network interfaces.
So the ordinary story is two interfaces. First one is a loopback and the second one is Ethernet. So we concatenate VMX substring with ip addresses, then we xor the result with some hardcoded string. And this whole key generation procedure can be described with this six lines of python code.
If you need to parse this format for some reason you can get released parser and a client to communicate with devices of this kind.
Another interesting service is of course MMS. So like others it uses MMSlite from sisco. And let’s focus on this library. If you google something like sisco mmslite security you will find info about security patches. But details or any other kind of information is closed to ordinary people. Only clients have access to it.
As you can guess it’s a proprietary library. But as it often happens you can find some sources inside of Chinese internet. Of course they are not the latest one, but it still something. Also we have tried some simple fuzzing with zero results. And in the end it turned out that easy bugs can be found in user hooks code of this library.
There is an fopen implementation which allows to read comtrade files. It’s special files which contain information before and after the fault. And it doesn’t sanitize network data at all. So we can easily read any file on flash with simple path traversal. And because MMS have no authorization for such calls you can read database file with all credentials which are hashed with md5. After that you get full access to device with legitimate configuration software.
As you can notice before relay terminals have very simple display with small number of buttons. But there are a lot of different augmentation for them. One of them is HMI system from schneider electric. It have only one default port – it’s ftp with fingerprint that says that is has some kind of KAOS system on board.
But when we look through it’s firmware it seemed that KAOS is just a library for some embedded windows system. Because all imports lead to standard windows libraries. Also significant part of the system is written in java and runs under some kind of real-time java machine for internet of things. Whatever that meant.
It’s a little bit unusual for me, but device management works through ftp protocol. But with some proprietary extensions.
So this smart tv allows integration with different kind of devices and vendors. Also it has a lot of supported protocols.
Also it has some hardware extensions like this biometric switch which allows to authorize through fingerprints. I think I saw biometric switches with such design in some part of Deus Ex series. If you know what it is.
So modern relay terminal contains a lot of third party software. And it’s very good but it can have problems with one days because of update problems.
So we described pretty new technologies for substation automation build on industrial embedded devices. But do we have any real reasons to build all this equipment? It looks like ”no”. New advances bring several hardware servers where all protection relays are virtualized and run as standalone application inside Windows 7 box for example. Will it help security or not? At least, even Windows box provide more flexibility for security reconfiguration. In reality, this technology is probably only used in China nowadays and nobody had a chance to research it. Little notice here. Most of researched devices built on top of real time OS, and it’s seems that their tasks need to be real-time to get the job done. But some devices and this virtualized approach in my opinion does not provide hard real time capabilities. But still they are working somehow.
I my opinion such industrial devices still just an ordinary embedded device with some additional peripherals. So security of such device have the same weaknesses as other embedded systems. So because of real-time requirements there are no exploit mitigation techniques like canary words or address space randomization. All you can find is partial data execution prevention. But as you understand it doesn’t work when it’s partial. Most of the protocol have no encryption so they are vulnerable to man-in-the-middle attacks or even replay attacks. Updates for such devices are slow, hard or need a help of engineer. So network security of such systems relays on that fact that this network have to be isolated, but for some administrative or business reasons they still need to be accessed from the outside. And still a lot of people try to implement their own http servers AGAIN.
Here is our team, those who stay behind of this research.
And big thanks goes to our friends from certification laboratory iGrids, who build strong security community around their lab by providing access to a big number of substation equipment for security researches, create hacking challenges on various conferences, provide introduction workshops in electrical engineering for it engineers and speak on hot security topics regarding substations.
Thanks for your attention. Thank you Recon team. It’s all I’ve got. Now I can answer some questions if we’ve got some.
