FoxGuard Solutions has encountered and resolved a wide variety of problems in our monthly work of patching control systems for our OEM clients and hundreds of power utility sites. In this presentation, we will cover a list of problems you might encounter and some real-world strategies that we have helped our clients implement to deal with them.

1. FoxGuard Solutions 1
Monta Elkins
Security Architect -- FoxGuard
Solutions
www.FoxGuardSolutions.com
What to do when you don’t know what to
do:
Control system patching problems and their
solutions

2. Installed Software
FoxGuard Solutions 2
Windows Control Panel
– Programs and Features

3. Installed Software
FoxGuard Solutions 3
This powershell command shows the installed software:
Get-WmiObject win32_product | Select-Object Name,Vendor,Version

4. Finding Patches
Patch Tuesday
FoxGuard Solutions 4

5. Identifying Patches
FoxGuard Solutions 5

6. Air-gapped
FoxGuard Solutions 6
update the wsusscn2.cab manually
it usually resides in
C:UsersusernameAppDataLocalMicrosoftMBSACachewsu
sscn2.cab
download the cab file from here and “carry it”
http://download.windowsupdate.com/microsoftupdate/v6/wsus
scan/wsusscn2.cab
Now use MBSA to identify patches

7. Identifying Patches
FoxGuard Solutions 7
CLI options:
From the mbsa program folder (c:Program FilesMicrosoft Baseline Security Analyzer)
Execute Mbsacli >results.txt

8. Which are Security Patches
FoxGuard Solutions 8

9. Security Patches
FoxGuard Solutions 9

10. A Patch List
FoxGuard Solutions 10
Manually download and carry patches from the final list and install them

11. Another Approach
FoxGuard Solutions 11
Discovering Patches and Downloading them
Virtual Environment Approach:
Setup virtual machines containing all software identified
on your systems, (but not configuration information)
Connect virtual machines to the Internet
Scan to identify and download appropriate patches
Hand carry the validated patches to air gapped
machines

12. Installed Updates
FoxGuard Solutions 12

13. Another Method to Verify Patch
Installation
FoxGuard Solutions 13
Powershell:
Get-WmiObject -Class "win32_quickfixengineering"

14. Windows Update History
FoxGuard Solutions 14

15. Verifying Patch Installation
FoxGuard Solutions 15

16. Watch for Disk Space Issues
Patches will not install if there is not
enough disk space.
Recommendation:
Have at minimum 1 Gigabyte free
storage space
Troubleshooting
FoxGuard Solutions 16

17. Patch Failure
FoxGuard Solutions 17
Microsoft Patch fails to install
System Update Readiness Tool
“The System Update Readiness Tool can
help fix problems that might prevent
Windows updates and service packs from
installing
If your computer is having problems
installing an update or a service pack,
download and install the tool, which runs
automatically. Then, try installing the
update or service pack again.”

18. Missing Patches
FoxGuard Solutions 18
Detection Issue:
Update KB2645410 for Windows 7 and
Windows Server 2008 R2 Historians.
Update for Microsoft Visual Studio 2010
Service Pack 1. This update may be
required but is not detected by Shavlik
(vCenter) Protect.
Corrective Action:
FoxGuard Solutions recommends that you
manually deploy update KB2645410 on all
Windows 7 and Windows Server 2008 R2
Historians

19. FoxGuard Solutions Technical Information Notice
Notice#:20140312-01
Notice Title: AVG Virus Warning
Reason for Notice:
After applying the AVG Anti-Virus 2013 updates from
the M1 2014 release the virus “VBS/Downloader.Agent”
was found on the system.
FoxGuard Solutions has confirmed the two files
referenced are automated manufacturing process
artifacts used during the HMI manufacturing process
that were not removed prior to the system being
shipped from the factory.
AV Signature Updates Can Cause Problems
FoxGuard Solutions 19

20. The script is used to temporarily turn off User Account Control (UAC) so
that manufacturing automation tools can run successfully on the system.
FoxGuard Solutions has determined that these scripts are not infected
files, but they do contain code that triggers AVG to flag them as a virus.
Specifically, the following code is flagged by AVG:
If WScript.Arguments.length = 0 Then
Set objShell = CreateObject("Shell.Application")
objShell.ShellExecute "wscript.exe", Chr(34) & _
WScript.ScriptFullName & Chr(34) & " uac", "", "runas", 1
Else
This is effectively equivalent to right-clicking an application and choosing
“Run as administrator”. This is a common practice with scripts that
require UAC elevation to execute properly, earlier releases did not flag
these files as malware.
AV Trigger Details
FoxGuard Solutions 20

21. Validation Checklists & Signoffs
FoxGuard Solutions 21
Have a set of validation checklists to
verify operations after patching.
Include testing signoff for record
keeping

22. AV & IDS Signatures
FoxGuard Solutions 22
CIP 007-3 R4.2. The Responsible Entity shall document
and implement a process for the update of
anti-virus and malware prevention “signatures.”
The process must address installing and testing the
signatures.
Use a “virus test file”
"EICAR Standard Anti-Virus Test File“ 68 bytes
And a “malicious network traffic” file

23. Ports and Services
FoxGuard Solutions 23
Logical Network Accessible Ports
– What are they?
– Listening ports
– Document need
• What is it?
• Why is it needed?
• On this particular device
– Or Shut it off
• Host based firewall mitigation
– RPC port changes
– MS DNS 2501 (MS improper docs)
– Every 35 days (and patching / updates 010-1)
Centralized Ports and Services Auditor (CPSA)
White Paper FoxGuardSolutions.com

24. Improper Documentation for DNS
FoxGuard Solutions 24
DNS documentation from Microsoft could cause you
to fail an audit
We received this acknowledgement of our findings

25. Test Lab and Rollout
FoxGuard Solutions 25
Validation lab equipment should closely mirror production
equipment
Where direct mirroring isn’t practical, be sure to include a superset
of all installed software.
Now do it “for real”
Use phased rollout approach:
•Test lab
•Less critical machines
•More critical machines
•Patch
•Verify
•Validate
•Backup

26. FoxGuard Patching and Validation
Services
FoxGuard Solutions 26
FoxGuard Solutions' DisPatch subscriptions provide validated
patches and updates plus documentation on a monthly basis.
To learn how FoxGuard Solutions can help you with patch and update
validation, contact us at requestinfo@foxguardsolutions.com,
or by calling 877-446-4732.