Moloch Gzip Decode

As previously mentioned, the open source packet capture tool Moloch does a number of on-the-fly decoding functions, including decompressing gzip'd data from the Web server. In the screen shot below we see a response from the server to a GET request. Because there's a large amount of code in the response it's gzipped, so we need to decompress it to analyze what's in the response.

After turning on the "Decode Gzip" option, represented by the check box in the grey options bar at top, the gzipped data is instantly decompressed for us and displayed.

SYN-FIN packet decode

Though the attack vector has shifted heavily to client-side attacks through the Web browser, there are still plenty of attacks coming against the perimeter (which supposedly doesn't exist any more, a premise I don't buy into). The result is that an intrusion analyst not only needs to learn how to accurately decipher client side attacks, he/she also STILL needs to know the basic packet/protocol decode information as well. The best course I have personal experience with to do that is the SANS SEC503 course, six days of learning starting with Day One's "Bit Boot Camp". To that end, here's a decode of a SYN-FIN packet. It's a good example for the new intrusion analyst, as it requires some bit masking to decode instead of all fields breaking nicely on 4 bit or greater boundaries. We need to check the individual bits of the TCP flags field, as shown below.

The alert:

Event : DECODE_TCP_SYN_FIN (116:420), Timestamp : 2013-12-11 07:45:20

Classification : Misc Activity

Priority : low

Detection Engine : myidsdmz2-IPS / myyidsdmz2.xxx.com

Protocol : tcp

Source IP : 10.10.10.130 (desktop1.xyz.xxx.com)

Source Port/ICMP Type : 59025/tcp

Destination IP : 69.171.245.49 (mqtt-shv-04-ash3.facebook.com)

Destination Port/ICMP Code : 443 (https)/tcp

Intrusion Policy : DMZ Passive Intrusion Policy

Rule : alert (msg:"DECODE_TCP_SYN_FIN"; sid:420; gid:116; rev:1; metadata:rule-type decode; classtype:misc-activity; )

Actions

Packet Information

FRAME 1 (Expand All)

Frame 1: 62 bytes on wire (62 bytes captured (496 bits)

Ethernet II, Src: 00:1c:f9:6b:64:00, Dst: e0:5f:b9:04:45:9b

Internet Protocol Version 4, Src: 10.10.10.130 (desktop1.xyz.com), Dst: 69.171.245.49 (mqtt-shv-04-ash3.facebook.com)

Transmission Control Protocol, Src Port: 59025 (59025), Dst Port: 443 (443), Seq: 0, Len: 0

Packet Text

._..E....kd...E..0V.@.=... P..E..1......4.....p...c...........

Packet Bytes

0000 e0 5f b9 04 45 9b 00 1c f9 6b 64 00 08 00 45 00 ._..E....kd...E. 0010 00 30 56 87 40 00 3d 06 f7 91 0a 0a 0a 82 45 ab .0V.@.=....P..E. 0020 f5 31 e6 91 01 bb 13 d5 34 b7 00 00 00 00 70 03 .1......4.....p. 0030 ff ff 63 97 00 00 02 04 05 b4 04 02 00 00 ..c...........

To verify this:

GREEN is your Ethernet header, the first 14 bytes.

BLUE is your IP header, 20 bytes. How do we know? The first byte offset from 0, (counting always starts at 0) the second nibble is a 5 (one byte equals 8 bits, half a byte is called a nibble and is 4 bits, so this is the nibble in the first byte which has a value of "5"), this is the header length field which represents the number of 32 bit words (or 4 byte words), and 4 bytes times 5 = 20 bytes)

RED is your TCP header. The TCP flags are in byte 13 offset from 0, in BOLD UNDERLINE.

The value is 0x03.

The TCP flags in the 13th byte (from 0), are in the following order:

CWR ECE URG ACK           PUSH RESET SYN FIN

We have one byte (or two nibbles or 8 bits here). 

Ignore the first two. They’re used for Explicit Congestion Notification and aren't used here. 

We have a 0 in the first (leftmost) nibble, so no flags are set.

The second nibble equals 3, so binary wise, the FIN (value of 1) flag and SYN (value of 2) flags are set, giving us a value of 3.  So when we see 03 in the TCP Flags byte, the packet is a SYN-FIN packet. 

Of course, if the ECN or CWR flags are set, that will change the value.

For example, if the CWR flag were set as well as SYN and FIN, our value would be 0x83, because in the first nibble, the CWR flag is in the 8's (2 to the power of 3) column. 

If CWR and ECE were both set,we'd have a value of 12 (8 + 4) in the first nibble, represented in hex as C, so our flags field would 0xC3. 

Moloch In-Depth

I've been using AOL Moloch (yes, that's America On-Line; the developers work for AOL) for well over a month and I have to say it's the best open source packet capture product I've ever used.

   The first system I ever worked with was the now famous SHADOW (Secondary
Heuristics Analysis for Defensive Online Warfare), written by Stephen Northcutt many years ago and then developed and maintained by Bill Ralph of the Naval Surface Warfare Center until they retired it.
After that,George Bakos extended SHADOW into IDABench when we was with the Dartmouth ISTS (Institute for Security, Technology and Society) with a framework of Perl scripts that allowed you to examine the packets and run them through ngrep, p0f, Ethereal (yes, it's been a while) and others.

   IDABench was a great tool but it wasn't scalable. The console server would pull each hours worth of packets form each capture box, which was feasible when George wrote it, but quickly scaled out as networks got faster and faster.

   After IDABench, I tried OpenFPC. The system had great promise, but it was buggy and not maintained on a consistent basis and I eventually had to abandon it. After that, our shop used daemonlogger to capture packets for quite a while. I used bash loops to extract the packets and mergecap to write them all back to one capture file and tools like ngrep, tcpdump and Wireshark to inspect them.

   Then my boss came to me one day with a system he came across, which was Moloch. He did a single install on a VM as a test platform and we really liked what we saw, so he rebuilt a rather beefy, recently retired server for the Elastic Search engine and installed the capture and viewer components on our existing packet capture boxes.

   Moloch works in this way: Each capture box writes out the packets it sniffs to it's own drive array, but the meta-data about the packets is sent off to the Elastic Search box and indexed. You can run a viewer, the web front end, off of each capture box or off of the Elastic Search box; it doesn't matter. When you do a search on any Moloch machine, you're searching the indexes on ES server. If you're searching on Box 1, you'll see the capture information from Box 1 as well as the meta-data from all the other capture boxes. When you expand a record to look at the ensuing packet data, that data is transferred to the box you're working on unless it was captured on that sensor.

   When you expand the packet you'll see a column with both sides of the conversation, color coded in red and blue, with the sources packets on the left and the destinations on the right. If you're familiar with Wireshark, it looks almost exactly like when you right click a packet and choose "Follow TCP Stream".

   If the data is gzipped, there's a check box under the header data to uncompress and redisplay it. Another check box gives you the "Show Images and Files" option. Images are rendered in the flow of the conversation, under their corresponding packet data. Files are reconsructed and a link is provided for you to download them. For example, if you're looking at an email conversation and there is a PDF file attached to the email, you're provided a hyperlink to the file. You can also download a pcap file of the session or transfer the raw data into a file from the source, destination or both.

   Moloch can save you enormous amounts of time investigating alerts from your IDS/IPS. I run my IDS in one screen and Moloch in another and as go through my alerts, I search for packets and reconstruct the session. In this way I can quickly see what triggered the alert, beyond the one or two packets my IDS captured.

   Moloch is a great product. For the functionality it provides they could have easily made this into a commercial product and charged copious amounts of money for it. If you're an intrusion analyst or work on a network team and need near-instantaneous insight into your packet captures, this might be the product for you.

Moloch Packet Capture

If you're looking for free, open source packet capture software that indexes metadata for full packet searching, you might want to look at Moloch. https://github.com/aol/moloch Moloch uses Elastic Search on the back end for indexing, allows searching on a host of fields, will reconstruct files and images on the fly and will transfer the full pcap to the viewer on the sensor you're working on from the sensor it was captured on. Designed for multi-gigabyte networks, its very fast if given beefy enough hardware and the required tuning. 

Tools on Wireshark's Wiki page

Wireshark has a great list of packet capture related tools on their Wiki page, found here. Great resource..

DerbyCon 3.0 Videos

Adrian Crenshaw, aka Iron Geek, is a co-founder and the videographer for DerbyCon. He does an incredible job of getting the videos of the DerbyCon presentations posted soon after the conference is over, and the videos for DerbyCon 3.0 are already available. You can go his site here to get them or you can go to YouTube and subscribe to his channel.

DerbyCon 3.0

I'll be traveling tomorrow to Louisville, Kentucky for the third DerbyCon security conference. If you're not familiar with it, Derbycon is security/hacker conference along the lines of DefCon (except smaller in scale) and has excellent presenters.

This years keynoters are H.D. Moore of Rapid7, author of Metasploit, and Ed Skoudis, Inguardians co-founder, SANS teacher and courseware author and speaker. There were over 1,700 attendees last year and over a 1,000 folks there for year one.

It's too late to get tickets (or a hotel room within 10 miles of the site) this year, but if you're interested in a low cost conference to attend, you might consider it for next year. Tickets are only $150.00 for all three days of the con, and there are two days of training prior to the conference for $1,000.00 per class.

This is mostly hacking/pentest/red team oriented but there are some blue team presentations, and being a good defender requires understanding the attacker and their tools. Well worth the relatively small cost. If you can't attend next year, all of the videos will be posted shortly after the conference on the Derbycon YouTube channel and on Iron Geek's web site (Adrian Crenshaw, Iron Geek, is the videographer for DerbyCon.)

Links:

DerbyCon: http://www.derbycon.com/
YouTube Channel: http://www.youtube.com/user/irongeek
Iron Geek Derbycon videos: http://www.irongeek.com/i.php?page=security/hackingillustrated

There is also a Roku channel for DerbyCon videos as well.

Packet Auditing versus Packet Capturing

If you're new to network security, you might hear the terms "packet auditing" and "packet capturing" used interchangeably. These are two related but different concepts.

The idea behind packet auditing is to have a packet log of every connection. You want to capture enough of the packets to log the IP addresses, ports and protocols. In order to successfully do this you need a snaplength (capture size) large enough to get complete headers at their maximum size. The largest an IP header can be is 60 bytes; that is the same for a TCP header. DNS can use TCP instead of UDP for large answers to queries or zone transfers and has a header size of 12 bytes. The Ethernet header is 14 bytes. So at as minimum, you probably want to set the snaplength to at least 150 bytes or so if packet auditing is all you need to accomplish. If you have the storage space, you might want to set it higher and have at least the first several bytes of data to assist you in identifying what kind of session you're looking at.

Packet capturing, usually FULL packet capture, is intended to capture the data as well as the headers and allow for session reconstruction. Having the full packet data allows you to reconstruct the entire session, including documents and binaries for examination.

Obviously if you're doing full packet captures, you'll need a lot more storage space than doing packet auditing. When implementing your first packet capture box, you should do sample captures at various times of the day to get an idea of what a full days worth of captures will take, determine how many days, weeks or months you'll want to capture and then add a good percentage for future growth.

Since you will be storing full session data, your packet captures themselves will be repositories of a lot of confidential and proprietary data. Harden the box as you would any high value asset, keep it patched, minimize the services to what you actually need and restrict access to only those who actually need it.

Moloch Packet Auditing

I recently came across another open-source packet capturing/auditing system called Moloch. It uses elasticsearch and can be run as a single node, or (by design) as a distributed network of capture boxes. It has a nice user interface that allows you to drill down into the packet data, pull a pcap of the packet you're looking at or download the entire packet capture. You can find it here. We've begun testing it where I work in a limited (single node) fashion, so I don't know how well the distributed piece works, but it looks very promising from what I've seen so far.

virustotal-submit.py

Didier Stevens has created a nice little tool to submit malware samples to VirusTotal using python. The tool can be downloaded here. This is really handy for an intrusion analyst (as well as a malware researcher).
Consider this scenario: You're investigating an alert and pull the packets for the session that generated it. You discover the user downloaded, intentionally or not, a binary from some site, purpose unknown. You wget or curl down the file and inspect it with hexedit or bless, but can't determine for certain whether it's malicious or not.
Run  virustotal-submit.py and submit the file right from your packet box. No need to transfer the file to another box to submit it (I'm assuming you're not running X on your packet audit box). And you can build this into your own scripts and tools for automated submission (the public API has a limit of four files per minute).

You'll need a couple things to run the tool. There's one dependency, the python module "poster", found here. And you'll need a VirusTotal public API key. All you need to do to get one is create an account on the site at http://www.virustotal.com. Your key will be in your Community profile after you log in.

Cisco Buys Sourcefire

Sourcefire announced today they've entered into a definitive agreement to be acquired by Cisco for $2.7 billion dollars. While it will obviously be a while before we see if this is a good thing or a bad thing for the network security industry, my experience has been that when big, bloated companies like Cisco or IBM buy out a smaller competitor and integrate them, it usually doesn't work out well. (For example, IBM buying ISS).

Some of the biggest concerns I've heard so far is that Cisco will try to integrate the Snort ruleset into their own products (ala PIX), or they will close the rule set, not allowing modifications or custom rules, or licensing will go up (bet on it) or the worse case scenario, the SF talent will be crowded out and exit, leaving us with Cisco engineers "improving" the product. If that happens, long time Sourcefire customers can meet all those folks who ran ISS at the corner pub and commiserate together over a cold one or two or ten.

Noise vs. Pattern

When does malicious traffic directed towards your network become white noise instead of part of the overall threat picture? The oft-used example is the Code Red worm. To this day there are still infected servers spewing out traffic that has faded into the white noise of the Internet. Very few have Code Red signatures enabled on their IDS and even fewer would actually look at any alerts that were generated from them.
But what about current valid exploits directed against a non-affected server? What about PHP attacks launched against an .ASP server? Should we be concerned? What about taking action?

I think there are two schools of thought on this. An instructor I had in a SANS class phrased it this way: "If you can't play nice on my network, you can't play on my network."
I'd love to be able to follow that philosophy. Unfortunately there are other factors that come into play when you're defending someone else's network, especially a large one.

Shunning malicious IPs involves one of several scenario's. If you're running IPS and are doing in line blocking, it's automatic, assuming the signature or rule is set to block. That's the good news. The bad news is: it's automatic. IT managers shudder at the thought of a customer trying to access their public web app servers and being blocked by a false positive on the IPS, as well they should. That's still the overarching issue with IPS:  How much do you trust it not to accidentally block legitimate traffic. So, many times, the IPS is set to only block really noisy, old attacks that have almost a zero percent chance of false positives. Not terribly helpful.

Another way to deal with pervasive attacks is to shun the IP at the border router. This is a process. First you must document the attacks that you are seeing, with logging and/or packets. You must then present your case to someone higher up (unless you're in management and have that authority) that the IP needs shunned and get their approval. Then you'll need to get change request approval. Even if there is a short path approval process for situations like this, this still takes time. You may find that by the time you shun the IP, the attacks have stopped and never come back again from that IP. That's a very good possibility because of the pervasiveness of botnets. An attacker never needs to use the same IP twice; he/she has many to choose from. That way his attack can't be shunned and it's very difficult for the defender to try and respond.

Another consideration is: Is there is human intelligence behind the attack? Is someone probing, looking for a way to exploit a host or is it a blind sweep of your whole address space and hundreds of other folks space as well. A real attack won't be that noisy unless the attacker is completely incompetent. Does the attack make any sense? Seeing zone transfer attempts against your web servers is a really good indication the attacker just wound up a script and is whacking against a whole range of IPs. Most of what we see today, or at least what I see, are fishing expeditions. They run for an hour, a few hours, and we never see that IP again.

So what's noise and what's of real concern? There isn't any easy answer to this. It all depends on how much analysis you can do and what can be determined, how large your public presence is and how pervasive the traffic is, how much of the traffic an organization would classify as an acceptable risk and what the return would be on your investment of time to respond and block the traffic.

Ask Yourself Questions

A lot of analysis is asking yourself questions and then using your tools and data to answer those questions. I was recently looking at some alerts that used a non-standard user agent in the client header, but the desktop team wasn't finding any indication of malware. My thought was an app was making requests and setting the user agent. All of the traffic from each machine was to one, business related site. If malware were involved, I would expect to see traffic to different sites all using that same user-agent. So my question to answer was "Does this machine use a different, standard user-agent to other sites?" To answer this I needed to go to a packet audit server that the traffic was captured at and pull all of the packets from that day in the hour they  happened and the hour before for good measure, then look at the web traffic and compare the user agents.

for i in $( ls -lah | grep 'Jul  8' | grep -w '1[7-8]' | awk '{print $9}' );do tcpdump -nn -r $i -s0 -X 'host x.x.x.x and (port 3128 or port 80)' -w work/$i.pcap;done

Each hours worth of data from that machine (and there were multiple captures per hour) on those ports is now in my work directory with a filename of the original packet capture filename and a .pcap extension. Now I need to use mergecap to concatenate all those captures into one file.

mergecap -w aggregate.pcap dmz1tap.*

Now I have all the traffic on one pcap called aggregate.pcap and can run it through  ngrep or through Wireshark and use the Find Packet function to search for the "User-Agent:" string. (That's under Edit, Find Packet). I need to change the type in the "By:" field to String and change the Search In type to "Packet bytes".

As expected, only certain packets used the user agent in question. Now the next question to ask myself is "Why do these particular connections use this non-standard user agent?" Ask, find answers, and repeat until the analysis is done.

VMX to OVF Conversion

I got back from taking the SEC542 class at SANSFIRE and needed to install the two virtual machines to my home machine to recreate the labs I did. SANS uses VMWare Player; I prefer Virtual Box. VBox will only import machines in the OVF (Open Virtualization Format) but VMWare uses VMX. I found that VMWare has a tool, called VMWare OVF Tool, that will convert one to the other, both directions. You can download the tool at:
http://communities.vmware.com/community/vmtn/automationtools/ovf (you'll need to create a My VMWare account, but it's free). Then just change to the \VMWare\VMWare OVF Tool\ directory and run:
"ovftool  (path_to_vmx_file) (path_to_save_ovf_file)". I put the .ovf file in the same directory as as the .vmx file. Then open up VirtualBox and go to File\Import Appliance and point it at your new .ovf file. Once it's done importing, you're ready to fire up your virtual machine. Nice. 

NIC Offloading and Packet Capturing

If you run a packet audit server or an IDS that does full packet captures, I recommend you read this article by Doug Burks from the Security Onion blog about disabling NIC offload features, found here.
You may find you really need the full data of packets and don't have it, if you don't do this. Doug is a GSE, the lead author of the Security Onion distro and an all-around helpful guy. Good stuff.

New NTOP

The primary author of ntop, Luca Deri has announced that a new, rewritten from the ground up version of ntop has been released. Renamed ntopng, info and download links are available here. If you're not familiar with this great open-source tool, ntop monitors traffic on an interface (or multiple interfaces) and tracks connections, bandwidth usage, protocols, top talkers and the like and supports taking netflows from infrastructure equipment (supports sFlow, NetFlow and IPFIX).  Putting an ntop instance on your packet audit box can be a great tool for tracking hosts from EOI's and can show you changes in traffic patterns that might indicate increased interest in your network from the outside. It can also quickly show you all the hosts an infected box talked to and how much traffic was passed, which might give you an indication of data leakage. It's not specifically designed as a NetSec tool, but it certainly fills the role as a good correlation tool. You can also feed it your pcaps to get a snapshot in time of network traffic you're interested in.

Verify Your Tools (Even if the tool is a web site)

I was looking at an alert with some Base64 encoded data and went to one of my bookmarked sites to decode and was told “Invalid length for a Base-64 char array”. That didn't look right to me, so I went to one of my Linux boxes and ran the text through base64 -d, which decoded it just fine. I want to another web site,

http://home.paulschou.net/tools/xlate/ , and it also decoded it. My point is beware of false negatives. Had I went with the results of the first site, I would have assumed I didn't have good data to decode and just moved on. (As it turned out the data was benign, but I had no way of knowing that.) It's a good idea to use multiple tools, and if you have a favorite you use most of the time, verify it's giving you correct results.

Using packet captures to investigate IDS/IPS alerts

Packet data is an almost indispensable resource. For example, I had a series of alerts of someone hitting a web server and trying exploit a PHP vulnerability. Without knowing whether the site uses PHP or not (and if it's a vulnerable version) and not having a definitive server response in the alert, I went to the packet caps to see what transpired.

The Snort alert.

Event : SERVER-WEBAPP PHP-CGI remote file include attempt (1:22063), Timestamp : 2013-05-13 03:50:05
Classification : Attempted Administrator Privilege Gain
Priority : high
Detection Engine : mysensor-IPS / mysensor.somedomain.com
Protocol : tcp
Source IP : 85.1.1.1 (somesite.panel.com)
Source Port/ICMP Type : 38217/tcp
Destination IP : 10.1.2.3 (Cannot Resolve)
Destination Port/ICMP Code : 80 (http)/tcp
Intrusion Policy : DMZ Passive Intrusion Policy
HTTP Hostname : somewebserver.com
HTTP URI : /?-d+allow_url_include%3d1+-d+auto_prepend_file%3dphp://input

The rule. 

Rule : alert tcp $EXTERNAL_NET any -> $HOME_NET $HTTP_PORTS (msg:"SERVER-WEBAPP PHP-CGI remote file include attempt"; flow:to_server,established; content:"auto_prepend_file"; http_uri; metadata:policy balanced-ips drop, policy security-ips drop, service http; reference:cve,2012-1823; reference:cve,2012-2311; classtype:attempted-admin; sid:22063; rev:5; )
Summary : This event is generated when an attempt is made to exploit a known vulnerability in php.

Packet Information

FRAME 1 (Expand All) Frame 1: 393 bytes on wire (393 bytes captured (3144 bits)
Ethernet II, Src: 00:22:90:9d:d6:b9, Dst: 00:0b:fc:fe:1b:03
Internet Protocol Version 4, Src: 85.1.1.1 (somesite.panel.com), Dst: 10.1.2.3 (Cannot Resolve)
Transmission Control Protocol, Src Port: 38217 (38217), Dst Port: 80 (80), Seq: 1, Ack: 1, Len: 339
Hypertext Transfer Protocol
Line-based text data: application/x-www-form-urlencoded
Packet Text
......."......E..{=.@.2...U...
S...I.P}../....P.9..Z..POST /?-d+allow_url_include%3d1+-d+auto_prepend_file%3dphp://input HTTP/1.1
User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.14.0.0 zlib/1.2.3 libidn/1.18 libssh2/1.4.2
Host: somewebserver.com
Accept: */*
Content-Length: 26
Content-Type: application/x-www-form-urlencoded



So I went to to my packet audit server and looked for the May 13 captures for 3:00 AM:

[admin@mysensor mytap1]# ls -lah | grep 'May 13' | grep '03:'
-rw-r--r--. 1 root root 1.1G May 13 03:05 mytap.pcap.1368428365
-rw-r--r--. 1 root root 1.1G May 13 03:07 mytap.pcap.1368428709
-rw-r--r--. 1 root root 1.1G May 13 03:08 mytap.pcap.1368428873
-rw-r--r--. 1 root root 1.1G May 13 03:08 mytap.pcap.1368428904
-rw-r--r--. 1 root root 1.1G May 13 03:09 mytap.pcap.1368428937
-rw-r--r--. 1 root root 1.1G May 13 03:09 mytap.pcap.1368428969
-rw-r--r--. 1 root root 1.1G May 13 03:10 mytap.pcap.1368428999
-rw-r--r--. 1 root root 1.1G May 13 03:11 mytap.pcap.1368429036
-rw-r--r--. 1 root root 1.1G May 13 03:11 mytap.pcap.1368429068
-rw-r--r--. 1 root root 1.1G May 13 03:12 mytap.pcap.1368429100
-rw-r--r--. 1 root root 1.1G May 13 03:12 mytap.pcap.1368429133
-rw-r--r--. 1 root root 1.1G May 13 03:13 mytap.pcap.1368429171
-rw-r--r--. 1 root root 1.1G May 13 03:13 mytap.pcap.1368429202
-rw-r--r--. 1 root root 1.1G May 13 03:14 mytap.pcap.1368429233
-rw-r--r--. 1 root root 1.1G May 13 03:19 mytap.pcap.1368429294
-rw-r--r--. 1 root root 1.1G May 13 03:26 mytap.pcap.1368429571
-rw-r--r--. 1 root root 1.1G May 13 03:37 mytap.pcap.1368430013
-rw-r--r--. 1 root root 1.1G May 13 03:48 mytap.pcap.1368430639

I searched those pcaps for the external IP:

[admin@mysensor mytap1]# for i in $( ls -lah | grep 'May 13' | grep '03:' | awk '{print $9}' );do tcpdump -nn -r $i 'host 85.1.1.1';done
reading from file mytap.pcap.1368428365, link-type EN10MB (Ethernet)
reading from file mytap.pcap.1368428709, link-type EN10MB (Ethernet)
reading from file mytap.pcap.1368428873, link-type EN10MB (Ethernet)
reading from file mytap.pcap.1368428904, link-type EN10MB (Ethernet)
reading from file mytap.pcap.1368428937, link-type EN10MB (Ethernet)
reading from file mytap.pcap.1368428969, link-type EN10MB (Ethernet)
reading from file mytap.pcap.1368428999, link-type EN10MB (Ethernet)
reading from file mytap.pcap.1368429036, link-type EN10MB (Ethernet)
reading from file mytap.pcap.1368429068, link-type EN10MB (Ethernet)
reading from file mytap.pcap.1368429100, link-type EN10MB (Ethernet)
reading from file mytap.pcap.1368429133, link-type EN10MB (Ethernet)
reading from file mytap.pcap.1368429171, link-type EN10MB (Ethernet)
reading from file mytap.pcap.1368429202, link-type EN10MB (Ethernet)
reading from file mytap.pcap.1368429233, link-type EN10MB (Ethernet)
reading from file mytap.pcap.1368429294, link-type EN10MB (Ethernet)
reading from file mytap.pcap.1368429571, link-type EN10MB (Ethernet)
reading from file mytap.pcap.1368430013, link-type EN10MB (Ethernet)
reading from file mytap.pcap.1368430639, link-type EN10MB (Ethernet)

Nothing there. Since the alerts were at 3:50 AM, I assumed the capture must have closed with a time stamp in the 4 AM range since there was nothing in the 3 AM range. I could verify that by looking at the packets with the stat command, but since I'm fairly certain thats the case I went ahead and checked the 4 AM packets first.

[admin@mysensor mytap1]# for i in $( ls -lah | grep 'May 13' | grep '04:' | awk '{print $9}' );do tcpdump -nn -r $i 'host 85.1.1.1';done
reading from file mytap.pcap.1368431312, link-type EN10MB (Ethernet)
03:50:05.582600 IP 85.1.1.1.38217 > 10.1.2.3.80: Flags [S], seq 2100199726, win 14600, options [mss 1380,sackOK,TS val 227240121 ecr 0,nop,wscale 7], length 0
03:50:05.582660 IP 10.1.2.3.80 > 85.1.1.1.38217: Flags [S.], seq 519966411, ack 2100199727, win 32768, options [mss 1380], length 0
03:50:05.658221 IP 85.1.1.1.38219 > 10.1.2.3.80: Flags [S], seq 9029262, win 14600, options [mss 1380,sackOK,TS val 227240197 ecr 0,nop,wscale 7], length 0
03:50:05.658286 IP 10.1.2.3.80 > 85.1.1.1.38219: Flags [S.], seq 1996989516, ack 9029263, win 32768, options [mss 1380], length 0
03:50:05.704707 IP 85.1.1.1.38217 > 10.1.2.3.80: Flags [.], ack 1, win 14600, length 0
03:50:05.704894 IP 85.1.1.1.38217 > 10.1.2.3.80: Flags [P.], seq 1:340, ack 1, win 14600, length 339
03:50:05.705068 IP 10.1.2.3.80 > 85.1.1.1.38217: Flags [.], ack 340, win 32429, length 0
03:50:05.705128 IP 10.1.2.3.80 > 85.1.1.1.38217: Flags [FP.], seq 1:103, ack 340, win 32768, length 102
03:50:05.779582 IP 85.1.1.1.38219 > 10.1.2.3.80: Flags [.], ack 1, win 14600, length 0
03:50:05.779895 IP 85.1.1.1.38219 > 10.1.2.3.80: Flags [P.], seq 1:332, ack 1, win 14600, length 331
03:50:05.780021 IP 10.1.2.3.80 > 85.1.1.1.38219: Flags [.], ack 332, win 32437, length 0
03:50:05.780065 IP 10.1.2.3.80 > 85.1.1.1.38219: Flags [FP.], seq 1:103, ack 332, win 32768, length 102
03:50:05.827525 IP 85.1.1.1.38217 > 10.1.2.3.80: Flags [F.], seq 340, ack 104, win 14600, length 0
03:50:05.827569 IP 10.1.2.3.80 > 85.1.1.1.38217: Flags [.], ack 341, win 32768, length 0
03:50:05.901888 IP 85.1.1.1.38219 > 10.1.2.3.80: Flags [F.], seq 332, ack 104, win 14600, length 0
03:50:05.901903 IP 10.1.2.3.80 > 85.1.1.1.38219: Flags [.], ack 333, win 32768, length 0
03:50:06.192994 IP 85.1.1.1.36413 > 10.2.3.4.80: Flags [S], seq 48424518, win 14600, options [mss 1380,sackOK,TS val 227240732 ecr 0,nop,wscale 7], length 0
03:50:06.194918 IP 10.2.3.4.80 > 85.1.1.1.36413: Flags [S.], seq 1830927041, ack 48424519, win 8192, options [mss 1380], length 0
03:50:06.316231 IP 85.1.1.1.36413 > 10.2.3.4.80: Flags [.], ack 1, win 14600, length 0
03:50:06.316284 IP 85.1.1.1.36413 > 10.2.3.4.80: Flags [P.], seq 1:186, ack 1, win 14600, length 185
03:50:06.340483 IP 10.2.3.4.80 > 85.1.1.1.36413: Flags [.], seq 1:1381, ack 186, win 64860, length 1380
03:50:06.340534 IP 10.2.3.4.80 > 85.1.1.1.36413: Flags [.], seq 1381:2761, ack 186, win 64860, length 1380
03:50:06.462634 IP 85.1.1.1.36413 > 10.2.3.4.80: Flags [.], ack 1381, win 16560, length 0
03:50:06.462635 IP 85.1.1.1.36413 > 10.2.3.4.80: Flags [.], ack 2761, win 19320, length 0
03:50:06.464475 IP 10.2.3.4.80 > 85.1.1.1.36413: Flags [.], seq 2761:4141, ack 186, win 64860, length 1380
03:50:06.464660 IP 10.2.3.4.80 > 85.1.1.1.36413: Flags [.], seq 4141:8281, ack 186, win 64860, length 4140
03:50:06.586261 IP 85.1.1.1.36413 > 10.2.3.4.80: Flags [.], ack 4141, win 22080, length 0
03:50:06.586475 IP 85.1.1.1.36413 > 10.2.3.4.80: Flags [.], ack 5521, win 26220, length 0
03:50:06.586485 IP 85.1.1.1.36413 > 10.2.3.4.80: Flags [.], ack 6901, win 28980, length 0
03:50:06.586542 IP 85.1.1.1.36413 > 10.2.3.4.80: Flags [.], ack 8281, win 31740, length 0
03:50:06.588096 IP 10.2.3.4.80 > 85.1.1.1.36413: Flags [.], seq 8281:11041, ack 186, win 64860, length 2760
03:50:06.588265 IP 10.2.3.4.80 > 85.1.1.1.36413: Flags [.], seq 11041:15181, ack 186, win 64860, length 4140
03:50:06.588433 IP 10.2.3.4.80 > 85.1.1.1.36413: Flags [.], seq 15181:16561, ack 186, win 64860, length 1380
03:50:06.588448 IP 10.2.3.4.80 > 85.1.1.1.36413: Flags [.], seq 16561:19321, ack 186, win 64860, length 2760
03:50:06.709723 IP 85.1.1.1.36413 > 10.2.3.4.80: Flags [.], ack 9661, win 34500, length 0
03:50:06.709961 IP 85.1.1.1.36413 > 10.2.3.4.80: Flags [.], ack 11041, win 37260, length 0
03:50:06.710085 IP 85.1.1.1.36413 > 10.2.3.4.80: Flags [.], ack 12421, win 40020, length 0
03:50:06.710211 IP 85.1.1.1.36413 > 10.2.3.4.80: Flags [.], ack 13801, win 41400, length 0
03:50:06.710333 IP 85.1.1.1.36413 > 10.2.3.4.80: Flags [.], ack 15181, win 41400, length 0
03:50:06.710462 IP 85.1.1.1.36413 > 10.2.3.4.80: Flags [.], ack 16561, win 41400, length 0
03:50:06.710510 IP 85.1.1.1.36413 > 10.2.3.4.80: Flags [.], ack 17941, win 41400, length 0
03:50:06.710709 IP 85.1.1.1.36413 > 10.2.3.4.80: Flags [.], ack 19321, win 41400, length 0
03:50:06.711660 IP 10.2.3.4.80 > 85.1.1.1.36413: Flags [.], seq 19321:20701, ack 186, win 64860, length 1380
03:50:06.711707 IP 10.2.3.4.80 > 85.1.1.1.36413: Flags [.], seq 20701:22081, ack 186, win 64860, length 1380
03:50:06.711776 IP 10.2.3.4.80 > 85.1.1.1.36413: Flags [P.], seq 22081:23950, ack 186, win 64860, length 1869
03:50:06.833710 IP 85.1.1.1.36413 > 10.2.3.4.80: Flags [.], ack 22081, win 41400, length 0
03:50:06.833891 IP 85.1.1.1.36413 > 10.2.3.4.80: Flags [.], ack 23950, win 41400, length 0
03:50:06.833895 IP 85.1.1.1.36413 > 10.2.3.4.80: Flags [F.], seq 186, ack 23950, win 41400, length 0
03:50:06.835162 IP 10.2.3.4.80 > 85.1.1.1.36413: Flags [F.], seq 23950, ack 187, win 64860, length 0
03:50:06.956383 IP 85.1.1.1.36413 > 10.2.3.4.80: Flags [.], ack 23951, win 41400, length

The very first packet inspected yielded hits. So I then used tcpdump to check just that capture and used –A just to look at packet data.

03:50:05.582600 IP 85.1.1.1.38217 > 10.1.2.3.80: Flags [S], seq 2100199726, win 14600, options [mss 1380,sackOK,TS val 227240121 ecr 0,nop,wscale 7], length 0
E..<=.@.2...U...
S...I.P}.........9........d...
^M.h.........
03:50:05.582660 IP 10.1.2.3.80 > 85.1.1.1.38217: Flags [S.], seq 519966411, ack 2100199727, win 32768, options [mss 1380], length 0
E..,........
S..U....P.I....}../`...7......d..
03:50:05.658221 IP 85.1.1.1.38219 > 10.1.2.3.80: Flags [S], seq 9029262, win 14600, options [mss 1380,sackOK,TS val 227240197 ecr 0,nop,wscale 7], length 0
E..</.@.2...U...
S...K.P..........9........d...
^M.i.........
03:50:05.658286 IP 10.1.2.3.80 > 85.1.1.1.38219: Flags [S.], seq 1996989516, ack 9029263, win 32768, options [mss 1380], length 0
E..,........
S..U....P.Kw..L....`....P.....d..
03:50:05.704707 IP 85.1.1.1.38217 > 10.1.2.3.80: Flags [.], ack 1, win 14600, length 0
E..(=.@.2..AU...
S...I.P}../....P.9...........
03:50:05.704894 IP 85.1.1.1.38217 > 10.1.2.3.80: Flags [P.], seq 1:340, ack 1, win 14600, length 339
E..{=.@.2...U...
S...I.P}../....P.9..Z..POST /?-d+allow_url_include%3d1+-d+auto_prepend_file%3dphp://input HTTP/1.1
User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.14.0.0 zlib/1.2.3 libidn/1.18 libssh2/1.4.2
Host: somewebserver.com
Accept: */*
Content-Length: 26
Content-Type: application/x-www-form-urlencoded



03:50:05.705068 IP 10.1.2.3.80 > 85.1.1.1.38217: Flags [.], ack 340, win 32429, length 0
E..(........
S..U....P.I....}...P.~.O.........
03:50:05.705128 IP 10.1.2.3.80 > 85.1.1.1.38217: Flags [FP.], seq 1:103, ack 340, win 32768, length 102
E..........]
S..U....P.I....}...P...5...HTTP/1.1 301 Moved Permanently
Connection: close
Location: http://www.somewebserver.com


03:50:05.779582 IP 85.1.1.1.38219 > 10.1.2.3.80: Flags [.], ack 1, win 14600, length 0
E..(/.@.2..+U...
S...K.P....w..MP.9...........
03:50:05.779895 IP 85.1.1.1.38219 > 10.1.2.3.80: Flags [P.], seq 1:332, ack 1, win 14600, length 331
E..s/.@.2...U...
S...K.P....w..MP.9.....POST /?-d+allow_url_include%3d1+-d+auto_prepend_file%3dphp://input HTTP/1.1
User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.14.0.0 zlib/1.2.3 libidn/1.18 libssh2/1.4.2
Host: somesite.com
Accept: */*
Content-Length: 26
Content-Type: application/x-www-form-urlencoded


03:50:05.780021 IP 10.1.2.3.80 > 85.1.1.1.38219: Flags [.], ack 332, win 32437, length 0
E..(........
S..U....P.Kw..M....P.~...........
03:50:05.780065 IP 10.1.2.3.80 > 85.1.1.1.38219: Flags [FP.], seq 1:103, ack 332, win 32768, length 102
E..........U
S..U....P.Kw..M....P.......HTTP/1.1 301 Moved Permanently
Connection: close
Location: http://www.somewebserver.com

So we saw the packets in the alert hit and the server respond and throw a 301 error and close the connection.

This is why packet data is so important. I never could have gleaned that from just the IDS alerts, and without server logs to correlate, the packet data does almost just as well.

REMnux by Lenny Zeltser

Lenny Zeltser, a SANS instructor who, among others, teaches the FOR610 Reverse-Engineering Malware: Malware Analysis Tools and Techniques track has a Linux distribution called REMnux for assisting malware analysts. The tool could be helpful for intrusion analysts as well, as it includes programs to decode JavaScript, examine executables and analyze malicious documents including the PDF tools by Didier Stevens. The distro comes in .ova format for use in either VMWare or Virtual box or as a virtual appliance for VMWare Workstation only. You can download the tool and read about it at Lenny's site here. Lenny's blog is here for even more information on malware analysis.

Self-Practice

Practicing intrusion analysis can be a worthy investment of your time when an alert you begin investigating turns out to be a critical event, such as a compromised server or data leakage of your companies information.

As you add to your arsenal, whether a new tool or a new methodology you've discovered for analyzing packets and data, it's a good thing to practice on non-critical events. Waiting until an important investigation where your superiors want information immediately or a decision on whether to shut down some piece of critical infrastructure or not isn't the time to hone your skills.

If you discover a new tool, practice with it. Read the documentation on it, know all the parameters for the command, and use it to investigate some routine alerts. Determine what information you would need and make sure you can use the tool to extract it. Make sure that tool or a combination of tools would allow you to determine what happened, what the source of the attack was and what it's target(s) were, whether or not the target was compromised, and how to mitigate the attack from reoccurring.

In most cases one tool won't accomplish everything. You may be alerted by your IDS/IPS, verify the alert using a SIEM or pulling the packets involved from your packet auditing system, and use other tools  to analyze those packets and logs or recreate the session.

The important thing to determine as you practice is that you are competent and confident in how to use each tool to accomplish the goal. You more you work with it, obviously, the better you'll become and you might discover better ways to use the tool, shortcomings in it that need a work around or even another tool for the task. An intrusion analyst's tool  kit should never be static. Knowledge is increasing and new and better tools are being released all the time and to stay on the curve as attackers adjust their methods require us doing the same.

Practice examples from web sites like the Ethical Hacker Network are great, but make sure if you use them, make sure you follow the same process and use the same tools you would if you were investigating traffic on your own network. Depending on your situation, you may not be able to dedicate time at work to hone your skills. It may require you using some of your own personal time to practice. If so, try to replicate your tool kit as closely to possible as what you have at work. If you have a spare machine at home, load your favorite distro of Linux on it and build your tool set on it. If you have a way to capture packets or are running your own Snort box or some other IDS, investigate alerts from it. You'll not only be sharpening your skills but you may found out your own network is being probed and needs locked down a little tighter, a great extra benefit to your investment of time.

Port Scan versus Port Sweep

If you're new to network security, you've no doubt heard the terms "port scan" and "port sweep" but may not be sure what the difference is.
A port scan is a scan of one host for one or more ports. For example, someone may scan a server for any of the privileged ports (server ports). Privileged ports are traditionally ports 1-1023 and are called that because under Linux you must have root privileges to bind a process to one of these ports. These days there a number of services that break this model, as the need for more ports has lead to services using high (also known as ephemeral) ports, 1024 and above. Ephemeral ports, also know as client side ports, are mostly used by the client as the source port to initiate a connection to the server on and are chosen by the operating system. The entire port range is 0-65535, though port 0 technically isn't supposed to be used.
A port sweep on the other hand is a scan of multiple hosts for one port. Scanning all the addresses in a 24 bit address space (what used to be called a class C netblock) for port 80 would be an example of a port sweep. In doing reconnaissance, an attacker (or a security admin testing his own systems) may sweep an address space for common open ports, then go back and do a port scan of each system that had an open port to see what all of the offered services are.
Nmap is the most well known of all port scanners and is under constant development and improvement by it's author, who goes by the screen name of Fyodor. His real name is Gordon Lyon (it's not a secret) and you can find nmap at his site http://insecure.org, as well as at http://nmap.org.
Nmap has a large list of parameters and does more than just basic port scanning. It can do OS fingerprinting (trying to determine the operating system of a host by the way it responds to certain network stimuli) and has hundreds of scripts that it's scripting engine uses to do all sorts of discovery.
There is a Windows version of nmap, but if you're serious about getting started in network security you should really have at least one Linux box to do your testing and learning on. Most of the good NetSec tools were written natively for Linux and many of them have no Windows counterpart. (There are a few good Windows tools that don't run on Linux, but not nearly as many as the other way).
So read up on the documents and get started. Just make sure you ONLY scan devices you either own or have permission to scan. If you're scanning at work it's a really good idea to get that permission in writing, from someone who has the authority to grant it.

Tool Kit Essentials

Every intrusion analyst has to have a toolkit. It's not just the essential Linux programs you need to install on each machine you do analysis from, but it entails all those web sites you use to check sites, to help you deobfuscate malicious code, research exploits and the like, not to mention some of the good Windows-only tools you might be using (like NetWitness Investigator or Malzilla). I used to keep a flat file of all the tools I needed, but it quickly gets out of date. How do you determine what should be on that list?

I've received a flurry of emails lately from recruiters for IDS related posts they need to fill. As I was reading one, it struck me: If I took this job and needed to get up and running doing intrusion analysis on Day One, what would I need to make that happen? I realized that scenario defines what should be in my essential toolkit. Not those rarely used apps or sites that that duplicate things I can cmd line from the packet boxes, but what I HAVE to have.

So rather than create another list, I've decided I need (yet another) flash drive to keep with me at all times (at least when I'm working) that has current copies of essential tools, Windows and Linux, exported bookmarks, copies of notes I've taken and such so that I'm fairly comfortable that I could walk in the door of a new job and within a couple of hours be ready to start looking at alerts.

If you have a better method, please share. I don't have any plans to change jobs, but keeping this info close by and updated is also a way to keep up with version checking and making sure I always have the latest improvements in my tools.
Thoughts?

Pen Test Sites

I'm a blue team guy, but occasionally I across some cool red team website or tip. Here's one, thanks to a post by Ed Skoudis. A student of his shared this excellent mind map of pen test practice sites for vulnerable apps and systems. You can find the site here.

DerbyCon 3.0

DerbyCon tickets go on sale April 1st. The con will be at the Hyatt Regency Louisville once again, and will run September 27-29th. Training will be September 25-26th and runs $1,000.00. General admission tickets to the conference only is $150.00, same as last year. Full info at the web site, http://www.derbycon.com. Hope to see you there!

JavaScript Deobfuscation

I came across this site by accident (it was mentioned in the comments of a Storm Center Diary article). It's a nice site to help deobfuscate JavaScript called, appropriately enough, deobfuscatejavascript.com. Heed the warnings they post and realize if the malware doesn't use the eval() or write() functions, which are trapped by the site, the code could execute in your browser. I run a Linux box that has VirtualBox installed, running another virtual Linux machine, with Firefox on it running no-script. Use a sandbox when checking out code or sites. Or you will probably end up getting owned at some point.

ShmooCon Live Stream

ShmooCon IX ramps up later on today and will be live streaming the conference. Information and links can be found here.

Kill Chain Analysis

Good paper by LockHeed Martin on intelligence-driven computer network defense and the analysis of intrusion kill chains.  Paper is hosted by the Virginia Information Technologies Agency here.

CloudShark

CloudShark is useful site that allows you to upload a pcap and share it with other people for collaborative efforts. Say you are troubleshooting a network issue over a conference call and need to share packet captures that were collected as the issue is happening. Or a team of network security analysts are working on a difficult analysis, and need to share the packets with someone from another location to assist them. Just drag and drop a packet capture into CloudShark and send the link to whomever needs to look at the capture. If they need to do more analysis (there are a limited number of online analysis tools built into CloudShark) they can click the download link and grab the pcap and open it up in their analysis tool of choice. There's even a plugin to allow you to upload a capture you're working on in WireShark (be aware that there's no proxy support that I've found).
The site is at http://www.cloudshark.org, and there is also a virtual image you can purchase and download to run on your own server or, order their appliance.

SANSFIRE 2013

After missing out on my yearly SANS training last year because of vendor training, I'm happy that I'm already approved for SANS this year. I'll be attending my usual conference, SANSFIRE, in July. I'll be taking a red team course this year, Web App Penetration Testing and Ethical Hacking with Seth Misenar. I've not yet had a class with Seth as the instructor. Should be doubly interesting. If you'll be at SANSFIRE this year, hope to see you there.