After turning on the "Decode Gzip" option, represented by the check box in the grey options bar at top, the gzipped data is instantly decompressed for us and displayed.
Friday, December 27, 2013
Moloch Gzip Decode
As previously mentioned, the open source packet capture tool Moloch does a number of on-the-fly decoding functions, including decompressing gzip'd data from the Web server. In the screen shot below we see a response from the server to a GET request. Because there's a large amount of code in the response it's gzipped, so we need to decompress it to analyze what's in the response.
Sunday, December 15, 2013
SYN-FIN packet decode
Though the attack vector has shifted heavily to client-side attacks through the Web browser, there are still plenty of attacks coming against the perimeter (which supposedly doesn't exist any more, a premise I don't buy into). The result is that an intrusion analyst not only needs to learn how to accurately decipher client side attacks, he/she also STILL needs to know the basic packet/protocol decode information as well. The best course I have personal experience with to do that is the SANS SEC503 course, six days of learning starting with Day One's "Bit Boot Camp". To that end, here's a decode of a SYN-FIN packet. It's a good example for the new intrusion analyst, as it requires some bit masking to decode instead of all fields breaking nicely on 4 bit or greater boundaries. We need to check the individual bits of the TCP flags field, as shown below.
The alert:
The alert:
Event : DECODE_TCP_SYN_FIN (116:420), Timestamp : 2013-12-11 07:45:20 |
Classification : Misc Activity
|
Priority : low
|
Detection Engine : myidsdmz2-IPS / myyidsdmz2.xxx.com
|
Protocol : tcp
|
Source IP : 10.10.10.130 (desktop1.xyz.xxx.
|
Source Port/ICMP Type : 59025/tcp
|
Destination IP : 69.171.245.49 (mqtt-shv-04-ash3.facebook.com
|
Destination Port/ICMP Code : 443 (https)/tcp
|
Intrusion Policy : DMZ Passive Intrusion Policy
|
Rule : alert (msg:"DECODE_TCP_SYN_FIN"; sid:420; gid:116; rev:1; metadata:rule-type
decode; classtype:misc-activity; )
|
Packet Information
FRAME 1 (Expand All)
|
P..E..1......4.....p...c......
|
0000
e0 5f b9 04 45 9b 00 1c f9 6b 64 00 08 00 45 00
._..E....kd...E.
0010
00 30 56 87 40 00 3d 06 f7 91 0a 0a 0a 82 45 ab
.0V.@.=....P..E.
0020
f5 31 e6 91 01 bb 13 d5 34 b7 00 00 00 00 70
03 .1......4.....p.
0030
ff ff 63 97 00 00 02 04 05 b4 04 02 00 00 ..c...........
|
To verify this:
GREEN is your Ethernet header, the first 14 bytes.
BLUE is your IP header, 20 bytes. How do we know? The first byte offset
from 0, (counting always starts at 0) the second nibble is a 5 (one byte equals 8 bits, half a byte is called a nibble and is 4 bits, so this is the nibble in the first byte which has a value of "5"), this is the header length field which represents the number of 32 bit words (or 4 byte words), and 4 bytes times 5 = 20 bytes)
RED is your TCP header. The TCP flags are in byte 13 offset from 0, in BOLD UNDERLINE.
The value is 0x03.
The TCP flags in the 13th byte (from 0), are in the following order:
CWR ECE URG ACK PUSH RESET SYN FIN
We
have one byte (or two nibbles or 8 bits here).
Ignore the first two. They’re used
for Explicit Congestion Notification
and aren't used here.
We have a
0 in the first (leftmost) nibble, so no flags are set.
The second nibble equals 3, so binary wise, the FIN (value of 1) flag and SYN (value of 2) flags are set, giving us a value of 3. So when we see 03 in the TCP Flags byte, the packet is a SYN-FIN packet.
Of course, if the ECN or CWR flags are set, that will change the value.
For example, if the CWR flag were set as well as SYN and FIN, our value would be 0x83, because in the first nibble, the CWR flag is in the 8's (2 to the power of 3) column.
If CWR and ECE were both set,we'd have a value of 12 (8 + 4) in the first nibble, represented in hex as C, so our flags field would 0xC3.
Thursday, December 5, 2013
Moloch In-Depth
I've been using AOL Moloch (yes, that's America On-Line; the developers work for AOL) for well over a month and I have to say it's the best open source packet capture product I've ever used.
The first system I ever worked with was the now famous SHADOW (Secondary
Heuristics Analysis for Defensive Online Warfare), written by Stephen Northcutt many years ago and then developed and maintained by Bill Ralph of the Naval Surface Warfare Center until they retired it.
After that,George Bakos extended SHADOW into IDABench when we was with the Dartmouth ISTS (Institute for Security, Technology and Society) with a framework of Perl scripts that allowed you to examine the packets and run them through ngrep, p0f, Ethereal (yes, it's been a while) and others.
IDABench was a great tool but it wasn't scalable. The console server would pull each hours worth of packets form each capture box, which was feasible when George wrote it, but quickly scaled out as networks got faster and faster.
After IDABench, I tried OpenFPC. The system had great promise, but it was buggy and not maintained on a consistent basis and I eventually had to abandon it. After that, our shop used daemonlogger to capture packets for quite a while. I used bash loops to extract the packets and mergecap to write them all back to one capture file and tools like ngrep, tcpdump and Wireshark to inspect them.
Then my boss came to me one day with a system he came across, which was Moloch. He did a single install on a VM as a test platform and we really liked what we saw, so he rebuilt a rather beefy, recently retired server for the Elastic Search engine and installed the capture and viewer components on our existing packet capture boxes.
Moloch works in this way: Each capture box writes out the packets it sniffs to it's own drive array, but the meta-data about the packets is sent off to the Elastic Search box and indexed. You can run a viewer, the web front end, off of each capture box or off of the Elastic Search box; it doesn't matter. When you do a search on any Moloch machine, you're searching the indexes on ES server. If you're searching on Box 1, you'll see the capture information from Box 1 as well as the meta-data from all the other capture boxes. When you expand a record to look at the ensuing packet data, that data is transferred to the box you're working on unless it was captured on that sensor.
When you expand the packet you'll see a column with both sides of the conversation, color coded in red and blue, with the sources packets on the left and the destinations on the right. If you're familiar with Wireshark, it looks almost exactly like when you right click a packet and choose "Follow TCP Stream".
If the data is gzipped, there's a check box under the header data to uncompress and redisplay it. Another check box gives you the "Show Images and Files" option. Images are rendered in the flow of the conversation, under their corresponding packet data. Files are reconsructed and a link is provided for you to download them. For example, if you're looking at an email conversation and there is a PDF file attached to the email, you're provided a hyperlink to the file. You can also download a pcap file of the session or transfer the raw data into a file from the source, destination or both.
Moloch can save you enormous amounts of time investigating alerts from your IDS/IPS. I run my IDS in one screen and Moloch in another and as go through my alerts, I search for packets and reconstruct the session. In this way I can quickly see what triggered the alert, beyond the one or two packets my IDS captured.
Moloch is a great product. For the functionality it provides they could have easily made this into a commercial product and charged copious amounts of money for it. If you're an intrusion analyst or work on a network team and need near-instantaneous insight into your packet captures, this might be the product for you.
The first system I ever worked with was the now famous SHADOW (Secondary
Heuristics Analysis for Defensive Online Warfare), written by Stephen Northcutt many years ago and then developed and maintained by Bill Ralph of the Naval Surface Warfare Center until they retired it.
After that,George Bakos extended SHADOW into IDABench when we was with the Dartmouth ISTS (Institute for Security, Technology and Society) with a framework of Perl scripts that allowed you to examine the packets and run them through ngrep, p0f, Ethereal (yes, it's been a while) and others.
IDABench was a great tool but it wasn't scalable. The console server would pull each hours worth of packets form each capture box, which was feasible when George wrote it, but quickly scaled out as networks got faster and faster.
After IDABench, I tried OpenFPC. The system had great promise, but it was buggy and not maintained on a consistent basis and I eventually had to abandon it. After that, our shop used daemonlogger to capture packets for quite a while. I used bash loops to extract the packets and mergecap to write them all back to one capture file and tools like ngrep, tcpdump and Wireshark to inspect them.
Then my boss came to me one day with a system he came across, which was Moloch. He did a single install on a VM as a test platform and we really liked what we saw, so he rebuilt a rather beefy, recently retired server for the Elastic Search engine and installed the capture and viewer components on our existing packet capture boxes.
Moloch works in this way: Each capture box writes out the packets it sniffs to it's own drive array, but the meta-data about the packets is sent off to the Elastic Search box and indexed. You can run a viewer, the web front end, off of each capture box or off of the Elastic Search box; it doesn't matter. When you do a search on any Moloch machine, you're searching the indexes on ES server. If you're searching on Box 1, you'll see the capture information from Box 1 as well as the meta-data from all the other capture boxes. When you expand a record to look at the ensuing packet data, that data is transferred to the box you're working on unless it was captured on that sensor.
When you expand the packet you'll see a column with both sides of the conversation, color coded in red and blue, with the sources packets on the left and the destinations on the right. If you're familiar with Wireshark, it looks almost exactly like when you right click a packet and choose "Follow TCP Stream".
If the data is gzipped, there's a check box under the header data to uncompress and redisplay it. Another check box gives you the "Show Images and Files" option. Images are rendered in the flow of the conversation, under their corresponding packet data. Files are reconsructed and a link is provided for you to download them. For example, if you're looking at an email conversation and there is a PDF file attached to the email, you're provided a hyperlink to the file. You can also download a pcap file of the session or transfer the raw data into a file from the source, destination or both.
Moloch can save you enormous amounts of time investigating alerts from your IDS/IPS. I run my IDS in one screen and Moloch in another and as go through my alerts, I search for packets and reconstruct the session. In this way I can quickly see what triggered the alert, beyond the one or two packets my IDS captured.
Moloch is a great product. For the functionality it provides they could have easily made this into a commercial product and charged copious amounts of money for it. If you're an intrusion analyst or work on a network team and need near-instantaneous insight into your packet captures, this might be the product for you.
Subscribe to:
Posts (Atom)