15-Minute Linux Incident Response Live Analysis by Philip Polstra

1. 15-Minute Linux Live Analysis
Dr. Phil (Polstra)
Bloomsburg University of Pennsylvania

2. What is this talk about?
● Determining with some certainty if you have been hacked
● In a matter of minutes
● With minimal disturbance to the subject system
● Automating the process with shell scripting

3. Why should you care?
● Someone calls you about a suspected breach
● You need to need to figure out if they were hacked
● Quickly so as to avoid further harm to your client
● Without destroying evidence
● Without taking down a critical machine

4. Who is the handsome man before you?
● Digital forensics professor at Bloomsburg University
● Programming since age 8 (in Assembly since age 10)
● Known to hack electronics from age 12
● Linux user from its start
● Author USB
Forensics
Insert cool
Cover art

5. Roadmap
● Opening a case
● Talking to the users
● Mounting known-good binaries
● Minimizing disturbance to the system
● Collecting Data
● Automation with scripting
● Next steps if there is a breach

6. Opening a Case
● Decide on case name (date?)
● Create a case folder on your laptop
● Start making entries in your notebook
● Bound notebook with numbered pages
● Easy to carry
● Hard to insert/remove pages
● No batteries required

7. First Talk to the Users
● They know the situation better than you
● Might be able to tell a false alarm before digging in
● Why did you call me?
● What they suspect
● No internal experts or policy to use outsider?
● Why do you think there was an incident?
● Everything they know about the subject system

8. USB Response Drive
● Contains known-good binaries
● Minimum /bin, /sbin, /lib for same architecture
● Might also grab /usr/sbin, /usr/bin, /usr/lib
● Must be on an ext2, ext3, or ext4 partition
● Could contain a bootable Linux on another partition
● This partition will probably be FAT
● Should be first partition
● See http://linuxforensicsbook.com/hdvideos.html Chapter 1

9. Mounting Known-Good Binaries
● Insert response drive
● Exec your bash binary
● Set path to your binaries (and only your binaries)
● Set LD_LIBRARY_PATH
● Run all shell scripts as bash <script>
● Don't use she-bang (#!) in scripts!

10. Demo: Mounting Binaries

11. Minimize Disturbance to System
● You will always change the system a little
● Goal is to
● Minimize memory footprint
● Never write to subject media
● Two basic options
● Save everything to your USB response drive
● Send it over the network

12. Sending data over the network
● Better than USB drive due to caching
● Use netcat
● Create a listener for “log” information on forensics workstation
● Send “log” information from client
● Also create a listener for interesting files on forensics
workstation
– Spawn a new listener when files are sent

13. Setting Up Log Listener
● netcat -k -l 9999 >> case-log.txt
● (-k) keep alive (-l) listen (>>) append
● From subject
● {command} | netcat {forensic ws IP} 9999
● Let's use shell scripting to automate this
● Shell not Python because we want to minimize memory
footprint

14. Automating the Log Listener
usage () {
echo "usage: $0 <case number>"
echo "Simple script to create case folder and
start listeners"
exit 1
}
if [ $# -lt 1 ] ; then
usage
else
echo "Starting case $1"
fi
#if the directory doesn't exist create it
if [ ! -d $1 ] ; then
mkdir $1
fi
# create the log listener
`nc -k -l 4444 >> $1/log.txt` &
echo "Started log listener for case $1 on $(date)"
| nc localhost 4444
# start the file listener
`./start-file-listener.sh $1` &

15. Automating the Log Client-Part 1
usage () {
echo "usage: $0 <IP> [log port] [fn port] [ft port]"
exit 1
}
# did you specify a file?
if [ $# -lt 1 ] ; then
usage
fi
export RHOST=$1
if [ $# -gt 1 ] ; then
export RPORT=$2
else
export RPORT=4444
fi
if [ $# -gt 2 ] ; then
export RFPORT=$3
else
export RFPORT=5555
fi
if [ $# -gt 3 ] ; then
export RFTPORT=$4
else
export RFTPORT=5556
fi

16. Automating the Log Client – Part 2
# defaults primarily for testing
[ -z "$RHOST" ] && { export RHOST=localhost; }
[ -z "$RPORT" ] && { export RPORT=4444; }
usage () {
echo "usage: $0 <command or script>"
echo "Simple script to send a log entry to listener"
exit 1
}
# did you specify a command?
if [ $# -lt 1 ] ; then
usage
else
echo -e "++++Sending log for $@ at $(date) ++++n $($@) n----end----n" | nc $RHOST $RPORT
fi

17. Automating Sending Files
● Listener on forensics workstation listens for file name
● When a new file name is received
● Create a new listener to receive file
● Redirect file to one with correct name
● Also log in the main case log (optional)
● On the client side
● File name is sent
● After brief pause send file to data listener port

18. Automating the File Listener
usage () {
echo "usage: $0 <case name>"
echo "Simple script to start a file listener"
exit 1
}
# did you specify a file?
if [ $# -lt 1 ] ; then
usage
fi
while true
do
filename=$(nc -l 5555)
nc -l 5556 > $1/$(basename
$filename)
done

19. Automating the File Client
# defaults primarily for testing
[ -z "$RHOST" ] && { export RHOST=localhost; }
[ -z "$RPORT" ] && { export RPORT=4444; }
[ -z "$RFPORT" ] && { export RFPORT=5555; }
[ -z "$RFTPORT" ] && { export RFTPORT=5556; }
usage () {
echo "usage: $0 <filename>"
echo "Simple script to send a file to listener"
exit 1
}
# did you specify a file?
if [ $# -lt 1 ] ; then
usage
fi
#log it
echo "Attempting to send file $1 at $(date)" | nc
$RHOST $RPORT
#send name
echo $(basename $1) | nc $RHOST $RFPORT
#give it time
sleep 5
nc $RHOST $RFTPORT < $1

20. Cleaning Up
# close the case and clean up the listeners
echo "Shutting down listeners at $(date) at user request" | nc
localhost 4444
killall start-case.sh
killall start-file-listener.sh
killall nc

21. Collecting Data
● Date (date)
● Clock skew on subject
● Time zone on subject
● Kernel version (uname -a)
● Needed for memory analysis
● Might be useful for researching vulnerabilities

22. Collecting Data (continued)
● Network interfaces (ifconfig -a)
● Any new interfaces?
● Strange addresses assigned?
●
Network connections (netstat -anp)
● Connects to suspicious Internet addresses?
● Strange localhost connections?
● Suspicious ports?
● Programs on wrong ports (i.e malware on port 80)

23. Collecting Data (continued)
● Open files (lsof -V)
● What programs are using certain files/ports
● Might fail if malware installed
● Running processes (ps -ef and/or ps -aux)
● Things run as root that shouldn't be
● No login accounts logged in and running things
● Might fail if malware installed

24. Collecting Data (continued)
● Routing info (netstat -rn and route)
● Routed through new interface
● New gateways
● Conflicting results = malware
● Failure to run = malware

25. Collecting Data (continued)
● Mounted filesystems (mount and df)
● Things mounted that shouldn't be (especially tempfs)
● Mount options that have changed
● Filesystem filling up
● Disagreement = malware
● Kernel modules (lsmod)
● New device drivers
● Modules that have changed

26. Collecting Data (continued)
● Who is logged in now (w)
● System accounts that shouldn't be allowed to login
● Who has been logging in (last)
● System accounts that shouldn't be allowed to login
● Accounts that don't normally use this machine
● Failed logins (lastb)
● Repeated failures then success = password cracked

27. Collecting Data (continued)
● User login info (send /etc/passwd and /etc/shadow)
● Newly created login
● System accounts with shells and home directories
● Accounts with ID 0
● Accounts with passwords that shouldn't be there

28. Putting It Together with a Script
usage () {
echo "usage: $0 [listening host]"
echo "Simple script to send a log entry to listener"
exit 1
}
# did you specify a listener IP?
if [ $# -gt 1 ] || [ "$1" == "--help" ] ; then
usage
fi
# did you specify a listener IP?
if [ "$1" != "" ] ; then
source setup-client.sh $1
fi
# now collect some info!
send-log.sh date
send-log.sh uname -a
send-log.sh ifconfig -a
send-log.sh netstat -anp
send-log.sh lsof -V
send-log.sh ps -ef
send-log.sh netstat -rn
send-log.sh route
send-log.sh lsmod
send-log.sh df
send-log.sh mount
send-log.sh w
send-log.sh last
send-log.sh lastb
send-log.sh cat /etc/passwd
send-log.sh cat /etc/shadow

29. Running the Initial Scan

30. Have I been hacked?

31. Who is Johnn?
/etc/passwd

32. Why do these accounts have
passwords?
/etc/shadow

33. Who's been logging in?
Results from last

34. Who failed to login?
Results from lastb

35. Looks Like They Were Hacked
Now What?

36. Live Analysis
● Use techniques described to
● Grab file metadata for key directories (/sbin, /bin, etc.)
● Grab users' command history
● Get system log files
● Get hashes of suspicious files
● Dump RAM
● Must compile LiME (off subject machine!)
● Risky – can cause a crash

37. Dead Analysis
● Unless the machine absolutely cannot be taken offline it is
strongly recommended that you shut it down and get a
filesystem image
● If you cannot shutdown the machine
● You can still get a filesystem image with dcfldd
● You probably cannot use this evidence in court

38. More on Dead Analysis
● Filesystem analysis is much more mature and powerful
than memory analysis
● The Linux support in Volatility is somewhat lacking
● Relatively new addition to a new tool
● Seems to fall down a lot with late 3.x and 4.x kernels
● None of the investigators I've talked to could come up with
a case where evidence existed only in memory

39. Finding Out More
● Heard there was a new book out (1 kg+ of knowledge)
● Resources on http://linuxforensicsbook.com
● Feel free to talk to me later

40. Questions?