Wireshark As A Teaching Tool

Wireshark, the most popular (and free) protocol analyzer, can be a great tool to gain familiarity in analyzing packets. The main Wireshark window is separated into three frames. The top frame is a list of all the packets Wireshark captured, showing the timestamp, source IP, destination IP, protocol and information about the packet (such as the source and destination port, the type of packet, the sequence and acknowledgement numbers, and so forth) The second frame shows the protocol headers in order, with expandable fields and the last frame is the actual packet data. If you're not familiar with the headers of a packet and the header charts aren't making sense yet, you can click on each field of the headers (Ethernet, IP, TCP, ICMP or whatever) and Wireshark highlights the field in the packet data frame for you.
This makes it very easy to see where the Ethernet header begins and ends, which parts are the IP header, TCP header, and so forth.
As you step through each field, you'll begin to become familiar with the layout of each header. With time you'll start to easily find the IP header (usually beginning with 45 00, the embedded protocol field (06 for TCP, 01 for ICMP, 11 (hex) for UDP) and so forth. This will especially be helpful for fields that don't fit neatly on byte boundaries, such as the flags field in the IP header (reserved, don't fragment, and more fragments bits) and the TCP flags bits (SYN, FIN, ACK, RST, and so forth).
Try capturing different types of traffic using the capture filters and step through each field. Some traffic will have other headers that Wireshark will display for you, such as DNS traffic. Within the Domain Name  System header, you'll be able to clearly see fields like the type of query, the transaction ID number, and the type and number of answers received from the name server.
Wireshark can analyze hundreds of protocols, is laid out very intuitively, and is a great aid to learn the inner secrets of packets. 

Book Recommendation

I'm reading a new book on packet analysis that would be good for someone new to network security (or just networking). It's by Chris Sanders and it's called "Practical Packet Analysis: Using Wireshark To Solve Real-World Problems". The first chapter on "Packet Analysis and Network Basics" would be especially helpful to someone just starting out, as it's one the clearest, easiest to understand summaries of protocols and the TCP\IP stack I've read. It's also not a terribly expensive book. I picked up my copy for about $30. If you're looking for some informative reading on packet analysis over the holidays, I'd recommend this one.

Network Security Tool list

Fyodor, author of nmap, maintains the excellent Sectools.org site, which lists the top 125 network security tools as determined by online survey. The 2011 results are posted, and there are no surprises (to me, anyways) in the top ten. This is a great site to peruse and find a new tool to do an old job better, or address a new need.
http://www.sectools.org

IDS Evasion Part II

  Another form of IDS evasion uses fragmentation to to evade the pattern matching function of the sensor. When a packet has to traverse a network with a smaller MTU than the network it originated from, the packet must be broken down into smaller chunks. In order for the destination host to be able to reassemble those fragments, IP uses an ID field to identify all of the packets that are part of the fragmentation train. Each packet will also have a fragmentation offset, which is it's order in the original packet.
Ethernet has an MTU of 1500 bytes. If the packet traverses a network with an MTU smaller than this, say 576 bytes, the maximum amount of data that could be carried in the first packet would be 576 bytes, minus 20 bytes for the IP header (assuming no IP options) and the size of the layer 4 header. If this were TCP, that would anywhere from 20 bytes (minimum) to 60 bytes (maximum).
   Let's say the TCP header is also 20 bytes for simplicity. The data in that packet would be 536 bytes. When reassembling that stream, the data in the first packet would have an offset (order) of 0 (counting begins at 0, not 1). The next packet's data offset would start at 536 (the previous packet would be contained in the first 0-535 bytes) and so forth. But what if the second packet gave a fragmentation offset of 516, instead of 536. This would produce fragmentation overlap, that is, the second packets data overlaps the first packets last 20 bytes. What does the network stack do with this data? Does it overwrite the last 20 bytes of the first packet, or, does it discard the first 20 (0-19) bytes of the second packet and start writing byte 20 at the 536 byte offset?
   What a network stack does here is identified by one of two methods, called favor old or favor new (or first or last). Not all network implementations use the same method. For example, Windows hosts use the favor old method. Linux uses favor new. In order for your IDS to properly reassemble fragments the same way as the destination host will, it needs to know whether the hosts it's protecting use favor old or favor new. Why is this an issue? Say an attacker sends a fragmented attack to a server. Withing those frag packets, he crafts one packet with some bytes of extra data that are not part of the attack, but sends the next packet with an overlapping offset to overwrite those packets, because the server is Linux based (favor new). As long as your IDS is set to favor new, it will correctly overwrite the bogus data as well, correctly detecting the attack. But if your sensor is set to keep the original data in the first packet, it will miss the exploit because it will use the bogus data in it's reassembled stream, which will not match the signature.
If you have a targeted sensor near the assets it's protecting and they are all on the same platform, like a Web farm of servers running Apache/Linux, you can set your IDS correctly to reassemble fragments the same way they servers do. But an edge sensor may be protecting assets of differing operating systems that use different methods for dealing with overlapping fragments. That's why targeted sensors that protect a subset of assets and are physically situated close to those assets on the network are preferable.

IDS Evasion Part I

Methodologies to avoid detection by IDS involve one of three types of techniques: evasion, insertion and denial of service. These are defined and described in the landmark paper "Insertion, Evasion, and Denial of Service:Eluding Network Intrusion Detection", released back in 1998 by Thomas Ptacek and Timothy Newsham. A new intrusion analyst would do well to read this paper, mostly still relevant after 13 years, and keep a copy for reference.
An insertion attack is where an IDS accepts a packet that the destination host rejects.
An evasion attack is where an IDS rejects a packet that the destination host will accept.

An example of an insertion attack would be where an IDS does not calculate TCP checksums, or does not calculate them correctly. An attacker sends a series of packets containing an exploit to host A. Within that stream of packets is one with a bad TCP checksum. If IDS B does not detect that the checksum failed, it will use that packet when it reassembles the stream and check it against it's signatures and processors. The match will fail because of the extra payload data contained in the bad packet, and the IDS will miss the attack.

Host A, however, will drop the packet with a bad checksum, reassemble the packets, and the exploit will be launched. The IDS never saw the attack. This is an unlikely scenario in 2011, but the next one I'll discuss involves at least one modern IDS that is still vulnerable to an evasion attack outlined over a decade ago.

NIC Offloading and Packet Capturing

Doug Burk, the guy who wrote and maintains the Security Onion Live DVD, posted a nice article on NIC offloading and the issues it can cause capturing network traffic here. Rather than try to regurgitate Doug's article, I'll summarize by saying it deals with the ability of modern network cards to offload the processing of traffic from the CPU, which is a great thing on fast (Gigabit) networks, but not what you want when capturing traffic. Why not? Because the controller does it's own reassembly at sizes greater than the MTU of your link, causing the sensor or sniffing program to miss part of the packet, in some cases, substantial parts of it. Doug is looking for feedback on this, so if you test and find you're getting "super-sized" packets in your packet captures (Wireshark will flag this for you as "Packet size limited during capture", let him know if you can.

IDS\IPS 101-6

Once your sensors are filtered and tuned, what you should have left are EOI's (events of interest), events that are either obviously malicious, suspicious, or enough of an anomaly to warrant investigation. How you investigate is dependent on a couple of factors. 1, What is your role? Or you strictly an intrusion analyst or do you do double duty as an incident responder as well? If  so, you'll need to go deeper in your analysis, possibly to the point of working directly on an affected server or desktop.
 But hopefully you're specialized as an intrusion analyst, and only touch alerts and the associated packets, and another team or person does the response and recovery.

Assuming that will be your role, what do you use to dig out the info you need to determine what happened?

Some IDS have a SIEM (security information event management) built in, like Enterasys' Dragon Defense, while others offer it as a separate, bolt on product (Symantec). Others are offered by SIEM-specific vendors such as Arcsight and LogRhythm. Log analysis tools like LogLogic and Splunk will also let you look at other data associated  with your alert.
But what if you work for a smaller company, and your IT security budget was just used  up with your IDS project? Fortunately there are lots of great open source tools you can implement to help you work through your alerts and determine what occurred (or didn't, as the case may more often be).

The first category of tools are the packet grabbers, the "packet auditing" tools. Your IDS will capture the packet or packets related to the alert, but not much else unless you specify it. Your IDS may not really be the place you want to collect more packets, as it's a busy box as it is. You can use another program on a separate host to do this for you, and do the analysis on that box as well, reducing the CPU cycles needed (it takes a while to decompress a 3 GB packet capture). I've mentioned many of these on this blog before, but I'll summarize them again here.

1. Daemonlogger (http://www.snort.org/users/roesch/Site/Daemonlogger/Daemonlogger.html) Written by Marty Roesch, author of snort, Daemonlogger will capture packets and write them out to a pcap file for you, or it can also act as a software-based tap, meaning it can capture on one interface and write back out to another. 

2. tcpdump - tcpdump has the ability to rotate pcap files based on size, doing the file naming for you, for x number of captures. Once you've reached the limit you specified, it will overwrite the oldest file with the newest. 

3. IDABench - based on SHADOW, IDABench is a perl based framework that captures packets into hourly pcap files, writes a summary of them out to a web page and allows you to search on multiple hours worth of captures using tcpdump, ethereal (yes it's old and outdated) and ngrep.Still a great tool. Contact me if you'd like a copy of it; it's no longer available anywhere on the Internet as far as I know.
Analysis Tools
1. Wireshark Wireshark (http://www.wireshark.org/) is one of the the most prolific protocol analyzers available. Wireshark supports hundreds of protocols and provides an interface that easily separates out the different headers to aid analysis, as well as allowing you to carve out content data.
2. Netwitness Investigator (http://netwitness.com/products-services/investigator) is a free form investigative tool that allows you to search and drill down into packets using contextual keywords instead of by header.
3. Network Miner (http://sourceforge.net/projects/networkminer/) Network Miner parses through pcap files or traffic on a network interface and identifies host names, operating systems, and open ports, as well as pulling out any files it identifies and filing them under directories for inspection.

IDS\IPS 101-5

Once you've determined what, where and how to deploy, and stand up your sensors, the real work begins. Once your IDS/IPS is up and running, the tuning phase begins (and never ends). If you're fortunate enough to be able to run the system in baseline mode for a couple of weeks to tune down the majority of the white noise, you can accomplish quite a bit using your trending or summary tools. Find the top talkers and start filtering out the obvious false positives that constitute large volumes of alerts. Lot's of internal processes can generate alerts that are benign, but look like events to the IDS, especially Windows networking processes. The IDS alerts themselves being forwarded to a mail server can even regenerate another alert for the same event, if you're including packet data (and the signature is especially broad in nature). Realize that once you've tuned your systems down to an acceptable level of alerts and feel confident you've about completed the process, you'll be doing this on a continual basis (or else see the white noise/false positive rate increase more and more over time). New signatures, additional network segments, new types of servers/services offered will all introduce FP's (false positives) into your system and will need filtered as they are identified. Your company may separate out the sysadmin duties from the analyst duties if the staff is large enough. For medium to small companies, you may wear both hats. I've found that's both an advantage and a disadvantage. The obvious disadvantage is that the more time you spend applying new signatures, filtering and tuning, and possibly even doing maintenance and patching, the less time you have to look at and analyze alerts. The advantage is that you, as the intrusion analyst, are uniquely qualified to make decisions or recommendations as to what traffic needs filtered, and what new signatures are relevant to your companies network infrastructure.

IDS\IPS 101-4

HIDS:
A HIDS (host based intrusion detection system) is installed directly on the server, and it's responsibility is to protect the system. The HIDS does not monitor traffic on the interface(s). Instead, it monitors it's application logs (such as Apache or IIS), and system logs (like the event logs on a Windows machine or syslog on a Linux box). Another important task of a HIDS is to check for changes to critical system files and alert when they have been modified (in the same manner as Tripwire). A HIDS will have a signature set, similar to the network IDS, that looks for strings associated with exploits being levied against the server. Since the packets reaching the host must be complete and formatted correctly, many of the evasion tactics used on the wire to bypass the NIDS will not be effective against the HIDS engine (such as fragmented packets with overlapping offsets, low TTL packets added to thwart the reassembly on the NIDS but dropped before they reach the destination, malformed packets for the same reason, etc.) A balanced, defense-in-depth security posture should include a HIDS on each public facing server at a minimum. Another good place to deploy host based detection is on critical systems such as database servers, human resources/payroll, servers that store PII (personally identifiable information) of clients or employees, and domain controllers and name servers.

IDS/IPS 101-3

Deployment:
One of the most critical decisions you will make concerning a new deployment is where to situate your sensors. Correctly locating your units will allow you advantageously monitor interesting traffic with less white noise to filter out. I was taught that there are two main schools of thought on the matter (and this is a 50,000 foot view of it):
1. The "umbrella sensor", as one of my SANS instructors called it. An umbrella sensor is one that sits at the periphery of your network, somewhere near your edge routers, firewalls and switches, and monitors all of the traffic coming in and going out. This may be fine (and the only financially allowable) solution for small, simple networks, but it doesn't scale well for complex networks. Large networks have many segments that need monitored, some public and some private, requiring many sensors. Encrypted tunnels such as VPN may make monitoring traffic impossible and infrastructure equipment on private segments with partners and clients may not be under your companies control. Sensors situated outside firewalls and routers may see so much white noise traffic as to make it nearly impossible to filter. If, however, your public infrastructure is small or budgetary limitations dictate you starting out with the "umbrella" approach, I would recommend placing your sensors behind your filtering equipment such as routers with ingress ACL's and firewalls. There are several reasons. The most obvious reason is noise. Unless you have near unlimited resources to look at and analyze events, you'll probably want to see just the traffic that made it past your router ACL's and firewall rules. Seeing what is traversing your network is critical and mandatory. Seeing what bounced off your hardened exterior is optional.
Another reason is that you'll be monitoring outbound traffic as well. Unless you're inside of any devices doing NAT/PAT such as firewalls, load balancer and proxies, you'll not see the real internal IP of an outbound connection. If that box was compromised and starts connecting back out to the attacker, and especially if data leakage is taking place, you'll need to shut that box down immediately. If what you are seeing is a NAT'd address, it may take you a while to find it in the logs (assuming you're logging on that device). With the real IP, you can locate and shut the offending device down easily (or isolate the port).
2. Targeted sensors. Targeted sensors sit close to the assets they are defending (like a Web farm). Because they are only a hop or two away, they see much less traffic (theoretically only traffic destined for one of those devices), so white noise is much less. The sensor can be tuned down to a smaller signature set specific to the protected assets. If your entire Web farm is Apache web servers running on RedHat, with no other services except what's needed for remote administration, your rulebase becomes a much smaller, more manageable subset of the whole. Hardware requirements drop as you're seeing much less bandwidth passing through the sensor, or the monitoring device attached to it. You'll probably want to still have a sensor behind the edge of your main ingress points as a first point of defense and a correlating packet collector, but your internal, targeted sensors will get most of your attention. If an attack has made it past all the filtering from the edge of your network down into one of your service segments, this would be a great time to see if your infrastructure blocking needs tightened up. IDS often is a validator of policy on your other security devices.

IDS/IPS 101-2

IDS vs IPS:
The difference between IDS and IPS is both in form and function. At a high level, an intrusion detection system is a passive device that monitors packets and identifies attacks using signatures and intelligence built into the IDS engine. The usual example is the string "get /etc/passwd", which used to mean (before shadowed password files) an attempt by someone to read the password file on a Linux/UNIX system. The IDS detects the string in the packets, and depending on how it's configured, alerts the analyst by some means, such as an email or via syslog or SNMP. This is a passive monitoring of the event (though some IDS have the ability to "shoot" down sessions by spoofing RST (reset) packets to both ends of the connection.) The IDS will have a management port with an IP address on it that allows it to forward it's data on to a collection/management console and allows remote access to it for administration (this may be done via an OOB (out of band) private network to limit access to the systems. It's monitoring port or ports will connect to a tap inserted at the desired point in the network, or a SPAN port on a switch.
Intrusion prevention systems sit inline, that is, they are connected directly to the network and all packets must pass through them. This is done via two NIC's (network interface cards), each attached to one end of the connection, such as between a firewall and a switch. Since all traffic passes through the IPS, like a router, the system has the ability to shun sessions it detects as being malicious by simply dropping the packets. No need to generate additional traffic in the form of reset packets. No notification is made to the attacker that the active response is taking place. The session, as the name implies, is prevented from continuing. Of course, if the event is a false positive, the very real possibility of dropping legitimate, business related traffic exists. Careful tuning and monitoring of the traffic must take place on an on-going basis to ensure the IPS doesn't interfere with normal traffic. Since the IPS sits inline, no tap or SPAN port is needed, but redundancy must be dealt with in case the device crashes or reboots. Most IPS are configured to fail open, which means if the software fails and stops working, the device will route all packets. Even then some traffic may be prohibited in the process of failing over. Some shops use a by-pass tap that routes all packets through the IPS as long as the interfaces are detected as being connected, but bypasses the device and routes the traffic directly if the interface goes down. Other use a second IPS, set up in fail over mode, so if the first device goes down, the second IPS takes over. The advantage here is twofold: you do not lose connectivity, the primary concern, AND you also do not lose your intrusion detection and prevention coverage while the first device is offline.

IDS/IPS 101-1

IDS (intrusion detection systems) and IPS (intrusion prevention systems) are a core part of network defense in-depth yet today, despite Gartner's proclamation that "IDS is dead" in 2003. The IDS/IPS market is still kicking, and for the intrusion analyst, most of the alerting he looks at will include events from IDS. That's even if they are aggregated in a SEIM (security event information manager) with firewall, router and host logs and alerting from anti-virus, web application firewalls and proxies.
If you're just getting started or have been tasked with standing up an IDS/IPS where none was previously, there are a number of things to take into consideration, even before you make your first contact with a vendor.
Some things to start working into you project plan would include:

1. Are you going to deploy NIDS (network intrusion detection systems), HIDS (host-based intrusion detection systems) or both, or are you going to deploy NIPS/HIPS (network or host-based intrusion prevention systems)
2. Where are you going to deploy your sensors? Depending on the budget being allocated, will you settle for a few "umbrella" sensors, boxes deployed along the perimeter of your network to try and see as many packets as possible? Or will you do targeted deployments, putting sensors in front of web farms, DNS servers, core switches, and critical internal assets such as database servers, payroll and HR?
3. How will your sensors monitor packets (this will be determined a lot by whether you go IDS or IPS). Will you use network taps or span ports for IDS? If you use inline IPS, how will you address redundancy? Does fail-open circuitry on your IPS box make you feel confident your phone won't ring at 3 AM when the sensor dies or reboots?
4.   How will IDS/IPS integrated into your network teams architectural plans? If the network topology changes, will you be able to adapt to things such as a change in media or major increases in bandwidth?
5. Who will monitor alerts? What will be your coverage window? If you will be the primary analyst, how is alerting covered during off-hours? Holidays and days off? If a team will handle the duties, is there coverage  if a member is out for an extended period of time? Are you considering an MSSP (managed security services provider) to handle the level one monitoring and alerting?
6. If this is a new addition to your corporate security infrastructure, will there be training provided for the analysts? (like the SANS Sec 503 Track). What about incident response? Will the intrusion analysts also be the incident handlers? If so, will there be training for that as well? 

More Packet Sites

I'd posted a few sites where you could download packet captures to study network traffic and practice your analysis skills, especially if you're new to network security or if investigating alerts is only one of many things that you do (that's more common, especially in smaller shops, than you may realize.) One of the security listserv's I subscribe to had someone post a question on where to find packet captures and the aggregate experience on that list came up with a nice list of additional sites.
Back in March, I posted the packet repositories found on pcapr.net,  the HoneyNet Challenges and the Inter-Service Academy Cyber Warfare Competition (http://jeffsoh.blogspot.com/2011/03/practice-makes-perfect.html) . Add to that list these sites from that post..
1. https://www.openpacket.org/
2. https://www.evilfingers.com/
3.http://packetlife.net/captures/
4. http://forensicscontest.com/ (Excellent place to practice, as this is a scenario you must solve, like the Honeynet challenges..)
5. http://chrissanders.org/packet-captures/

A list of pcap sites can be found here

It can be tough after working all day to then sit at home and look at MORE packets, but the more you work with captures, you more familiar you'll become with the both the data and the tools and methods you'll learn to work through huge amounts of data to find that one packet that confirms or denies your investigations theory.
If you're a programmer as well, you'll soon write up all sort of nifty scripts to automate things. If you're not, others have written tons of them, and reinventing the wheel isn't really necessary anyways. Enjoy.

Packet Analysis Tools

I'm often asked if doing intrusion analysis requires a lot of high-cost tools
(actually I don't recall anyone ever asking that of me, but it sounded good as a
lead in to this topic). The answer, if someone SHOULD ask me that, is no, it
doesn't have to.
After identifying alerts of interest, my current tool kit for analysis includes the following:

- Snorby: Open source front end to snort databases. I just recently started using this instead of BASE
- IDABench: Packet auditing system written by George Bakos, based on Shadow.
(Hasn't been actively supported since 2003, but still works fine. Going to look at OpenFPC as a possible replacement.)
- ngrep: The ever popular packet grepper from Jordan Ritter.
- Wireshark. The best free protocol analyzer available
- xtractr - packet indexer from Mu Dynamics
- Netwitness Investigator - packet indexer/search tool from NetWitness Corporation
- p0f, tcpdump, dsniff, chaosreader, and other various cmd line tools

All of those tools are open-source and freely available. Of course, we use multiple commercial IDS systems and a SIEM and a commercial log analyzer, but all the tools I use to do actual analysis are free.
Does that answer the question you didn't ask? Good!

chaosreader

I've been tailoring most of my posts towards "if you're new to network security", and lately I've been seeing a good number of stuff on Twitter about how we need to increase mentoring in our industry and share info more effectively. That said, I'm going to use this blog primarily to try and share info with those who are new to security full time (realizing that doesn't mean new to the IT industry a lot of the time).
That said, here's another tool that might be helpful. It's old, but still very useful.

chaosreader:

What it is: chaosreader.pl is a Perl script, written by Brendan Gregg, that takes a libpcap packet capture and indexes all of the connections and extracts a fair amount of the data and organizes it all into a nice web site. You take all the output and load it up in a Web browser (you could do that locally if you're running X on the box you ran it from, or share it via a web server, or tar it up and move to a web server you own.) I personally find it easy to just copy everything to /var/www or /var/www/html, depending on what you're running, since I don't serve web pages out of the default location anyway.

You have lots of options, but the basic ones you need to know tell chaosreader what kind of files to create:


chaosreader infile      # Create application session files, indexes
       chaosreader -v infile   # Verbose - Create ALL files
       chaosreader -i infile   # Create info files
       chaosreader -r infile   # Create raw files

Verbose will create all file types chaosreader knows about, -i, just the info files and -r, the raw files. Raw files are data files from TCP or UDP transfers, and info files are descriptor files listing the source, destination and such of the connection. chaosreader also generates a Get/POST report, an image report, and an HTTP proxy report.

Rather than me screenshot chaosreader, the author has created a live sampling here  or here that you can browse and get an idea of what the tool does.

How to use it: Download the latest version and put it in a separate directory (it creates a lot of files; you REALLY don't want to run it in the root of your home dir). You can run it live on an interface, or, as I do, run it against a pcap file. Use the switches show above to determine how verbose you want it to be.
For example, if you want to see it all, and had a pcap called web.pcap, you would run:
" perl chaosreader.pl -v web.pcap". With the latest version, you'll see a list of connections scroll as chaosreader reads in the packets and processes them. After it's done, you'll have an assortment of .html, .info, .raw, raw1 and raw2 files, as well as other like JPEG's and GIF's.

It's a very useful tool, but one caution. Very large captures can take a really long time to process, and if your box is running with minimal memory, it might run out before the process finishes. A strong processor and lots of RAM make a good chaosreader box.

Foundstone Tools

I hadn't been out to the Foundstone site recently (now owned by McAfee), until this morning, and found there have been quite a few new tools added. I keep Foundstone tools on a thumb drive, along with Sysinternals and others (like IceSword and some malware scanners), so I always have a ready copy if needed. If you've never looked at the free offerings available there, you can download any or all at there site found here.
Categories of tools available here are as follows:


Assessment Utilities
Forensic Tools
Foundstone SASS Tools
Intrusion Detection Tools
Scanning Tools
Stress Testing Tools
Spam Submission
Stinger
Utilities

IDABench Lives On

I just did another install of IDABench, the intrusion analyst's toolkit based on SHADOW. This project hasn't been maintained or updated since 2003 when George Bakos left ISTS at Dartmouth, and yet it still remains a very useful tool (to me at least). I'm at a new job and the folks here never heard of IDABench, so I was happy to install it and show them it's usefulness. Still hoping some Infosec Perl jockey will come across it someday and decide it's worthy of their time to resurrect  the project and continue extending it.

Tools from SANSFIRE

I love finding new tools. Maybe the functionality they provide is simply a shortcut to something I can do with other tools, eliminating a step or two and saving time, or else they might provide a way to do something I hadn't considered before. I came across a good number of new ones at SANSFIRE. I'll post the ones that caught my interest as I review my notes and books, in case one might fill a niche for you.
The first one is pcapcat. This is a perl script which parses through a packet capture, identifies and displays all the sessions by source IP and port and destination IP and port and displays them for you. You then supply a filename and pcapcat writes out the session to a new capture file. Not only does this save some keystrokes and time over using tcpdump, but it's also a handy way to see all off the sessions quickly and determine which ones are of interest. The tool was written by Kristinn Guojonsson to help answer a challenge on forensicscontest.com, which is found here. The actual code can be found here to paste into a new script on you analysis box.
Try it out and see if it'll help save you a little time (or if you're new to network security and aren't sure about all the tcpdump syntax yet, use this until you get up to speed)

SANSFIRE Day3

It's 2:09 AM Wednesday morning, day four of SANSFIRE. Insomnia has sent in, so it's time to update my blog on the training event. Day 3 saw more excellent teaching in the FOR 558 class. This class is both a logical next step after SEC 503 (Intrusion Detection In Depth) and a different perspective from that track. Going beyond the role of intrusion analyst, it shows you how to become a forensics investigator into events from the network perspective. More than just incident response (identify, eradicate and recover), we're seeing how to take packet data found all over the network and pull the artifacts from the packets. Hex editors and protocol specific tools for SMTP, AIM, Squid and others as well as general packet parsing tools all combine to not just reconstruct the session and see what happened, but extract the data much like a traditional forensics investigator would from a hard drive. We're told day four will be the most hard core yet. From a survey of the book, looks like we're going into Netflow and wireless, as well as concepts in digital evidence.

SANSFIRE Days 1 and 2

Saturday began SANSFIRE for me with SEC 546:IPv6 Essentials. This was also Day 2 of the Security IPv6 Summit. The class was taught by Dr. Johannes Ullrich, a SANS senior instructor, Dean of Faculty and Chief Research Officer. If you're not familiar with IPv6 and the myriad of things to consider when preparing for migrating, I recommend you take this course (tell your boss you want to add it on to your five or six day course to help the security department prepare for IPv6). It was a huge amount of information packed into a one day course that will serve you well in any future discussion.


On Sunday, I began the FOR 558 course, which is network forensics. This is being taught by Jonathan Ham, an incredibly knowledgeable instructor, whose bio can be found here. This was my first class with Johnathan. The motto for FOR 558 is "No hard drive, no problem". As this suggests, the class is all about doing forensics analysis, not from the computer in question itself, but from the footprint (or fingerprint) it's network traffic has left. Day 1 took off at full bore after a quick review of networking essentials, using SNIFT, the FOR 558 equivalent of the SIFT tool used in the traditional forensics classes, to create and examine packet captures, examine them using tcpdump, Wireshark and a hex editor, and pulling data from the packets. It's not the ordinary course that has you carving graphics files with a hex editor out of bytes pulled from a pcap file. If you're an intrusion analyst, incident response team member or security investigator, you might want to look at this class. Though the class is called a forensics class, it's a perfect complement and next step to SEC 503:Intrusion In-Depth. If you have your GCIA and want to take your intrusion analysis skills a level higher, this is a great class to do so. Here's the details.

SANSFIRE Pre-Conference

I'm at the hotel that's the site of SANSFIRE 2011 (the Washington Hilton in D.C.) and ready for tomorrow. I'll start out with a one day course with Johannes Ullrich, SEC 546, which is a primer on IPv6. Then Sunday we start on Forensics 558, five days with Jonathan Ham on Network Forensics. It's going to be a good week. Saw Stephen Northcutt earlier and got to say hello to him and tell him what classes I was taking, and hear from him how well the IPv6 Summit today went. I love the atmosphere of this conference. Eager to get started tomorrow.

SANSFIRE 2011 Evenings

For SANSFIRE this year (starting July 15th) we have the following evening sessions to choose from:

Cyber Security Trends and Lessons Learned from Large Scale Incidents in Japan

- Tomohisa Ishikawa
- Sunday, July 17 * 9:15pm - 10:15pm

FACEROUTE: Mapping and Harvesting Social Media Sites

- Rob VandenBrink
- Monday, July 18 * 7:15pm - 8:15pm

Securing The Kids

- Lance Spitzner
- Monday, July 18 * 7:15pm - 8:15pm

Ninja Developers: Penetration Testing and Your SDLC

- Kevin Johnson
- Monday, July 18 * 8:15pm - 9:15pm

Securing The Human

- Lance Spitzner
- Monday, July 18 * 8:15pm - 9:15pm

Is IPv6 the Wolf in IPv4s Clothing?

- Richard Porter
- Tuesday, July 19 * 7:15pm - 8:15pm

Security Testing Automation and Reporting

- Adrien de Beaupre
- Tuesday, July 19 * 7:15pm - 8:15pm

Are your tools ready for IPv6?

- Jim Clausing
- Tuesday, July 19 * 8:15pm - 9:15pm

Nurturing the Next Gen of InfoSec Geeks: Creating Kid-Friendly Challenges for Fun and Mayhem

- Ed Skoudis, Kevin Johnson, Josh Wright
- Tuesday, July 19 * 8:15pm - 9:15pm

DNS Sinkhole Peer into your network while you sleep

- Guy Bruneau
- Wednesday, July 20 * 7:15pm - 8:15pm

Using Scapy to Craft an IDS/IPS Evasion

- Judy Novak
- Wednesday, July 20 * 7:15pm - 8:45pm

Adding Network Flow Analysis to your Security Architecture

- Sidney Faber
- Tuesday, July 20 * 8:15pm - 9:15pm

Cisco Malware; A New Risk to Consider in Perimeter Security Designs

- Manuel Humberto Santander Pelaez
- Wednesday, July 20 * 8:15pm - 9:15pm

SQL Ginsu: Better Living (and Data Reduction) through Databases

- Philip Hagen
- Thursday, July 21 * 7:15pm - 8:15pm

More Parsing for Beginners

If you're new to NetSec, you don't have to learn a ton about any one subject to get started doing your job. There are more resources out there than you could possibly ever use to quickly get enough knowledge to take care of the job at hand and then go back later and learn more. Say you have a directory of packet captures. You've been using tcpdump or Daemonlogger to get raw packets as an audit trail (or IDABench, an outdated but still useful favorite of mine). Your boss wants you to do a sanity check for some sort of PII, to validate some control. A search on doing iteration in bash will probably lead you to the for command, where you can feed ngrep with your packet captures, one by one...
Here's an example...


for i in $( ls | grep -v .gz );do ngrep -q -I $i ' [0-9]{3}-[0-9]{2}-[0-9]{4} ';done

This is simply using the for command to use ls to search for any of your captures that aren't gzipped (if you use bzip, you'll need to modify it) and populate a variable called $i, then run ngrep against the value contained in that variable (one of your packet captures) and look for numbers in the Social Security format. There are all kinds of regex sites with common searches you can use without having to become a regex guru first.

Handle the job at hand, then backtrack and learn...

Reload

I was given a really nice laptop at work as a scan/test box, so I loaded it up with Fedora 15 and Windows 7. I'll take this one to SANS with me in a couple of weeks. I realized as I started adding tools that it had been a really long time since I had updated my list that comprise my toolbox. So I just  added a few basics while I gave it some thought. I do very little vuln assessment and no pen testing, so the majority of any tools I install would be analysis related, with the exception of some to generate packets or do a quick port scan. Here's what I've put on so far..

nmap
ngrep
hping
dsniff
etherape
iptraf
scapy
ipcalc
p0f
hexedit
wireshark

And to be installed...

netdude
chaosreader.pl
xprobe
xtractr


Now to find that old list and update it with the new stuff I've found lately..

Suggestions?

SANSFIRE 2011

After a four year hiatus, I'll be returning to SANSFIRE this year, held on July 15 through the 24th. The event will be at the Hilton Washington Towers , and besides all the usual SANS @Night events will include a round of NetWars.  This will by my fourth conference in D.C. and my sixth overall.

IDS\IPS Placement

Determining where to place the sensors in an intrusion detection/prevention system is one of the first decisions that need to be made when deploying a new system (or replacing an existing system). To the extent that it has been determined what kind of sensors will be deployed (IPS or IDS, in line or promiscuous), placement can then be considered, regardless of what vendor will be used.

There are various types of placement deployments that can be used, such as using edge or “umbrella” sensors. This deployment model identifies the egress points at the edge of the network and situates a sensor there to monitor all traffic in and out. This is usually (optimally) deployed at some point behind the firewall that is performing address translation on the IP addresses. This allows traffic associated with a compromised host to be readily identified. Because of its location behind the firewall, the analyst does not need to deal with traffic that has been shunned at the edge router or dropped by the firewall.

Vint Cerf, one of the “founding fathers” of the Internet, once famously said, “The wonderful thing about the Internet is that you’re connected everyone else. The terrible thing about the Internet is that you’re connected to everyone else”.  So it is with edge sensors. The wonderful thing about them is they see all traffic entering as network at that particular point. And that is their weakness as well. Tuning and filtering is a constant challenge at that point, especially if the sensors monitoring point is place outside the firewall, and all the malicious traffic that would be rejected is added to the flow.

Tuning can be accomplished much more readily with a dedicated sensor near the assets it will be protecting. For example if you have an Apache Web farm off your DMZ, a sensor located in front of it can run a tuned, minimalistic base of signatures looking for exploits against Apache and the OS you’ve deployed the Web servers on. Umbrella sensors need to run a comparatively larger signature set to be able to detect anything in the general flow of traffic coming in.

Consider fragmentation reassembly. If your sensor will protecting hosts that reassemble with both the favor old and favor new methods for handling overlapping offsets, your IDS/IPS will need to be able to reassemble traffic this way as well. Some IDS platforms, like Enterasys Dragon, give you an either/or choice. You can only choose which method the majority of the hosts use and hope for the best with the others, if you’re employing only an edge sensor. However, if your entire Web server segment is Linux servers, and all your database servers are running MS-SQL , you can tune the focus sensor according to the methodology used.

Note: Snort’s frag4 processor allows you to specify ranges of IP’s to be processed in one of the two methods.

Another consideration concerning focused sensors is reduced bandwidth. Obviously, a sensor only monitoring traffic to a specific segment or group of servers will see considerably less traffic than the sensor at the perimeter of the network. This can translates to a lower initial cost on licensing in addition to decrease in tuning and filtering.

All said, does that mean sensors should never be deployed at the perimeter? That is a decision for the intrusion analysis team to discuss with management and depends on the breadth of the deployment. The more places a flow of traffic is seen, the more likely an alert will be triggered. This loosely fits in with the general concept of defense in-depth (though true defense in depth would deploy different technologies to mitigate attacks).  However, if the perimeter sensor is sitting physically beyond filtering devices such as firewalls, UTM, reverse proxies, application firewalls and the like, aggressive tuning to filter our noise will probably be in order.

Another consideration is whether or not to deploy the sensors exclusively from one vendors offering. Having a heterogeneous environment has the advantage of different signatures and security policies monitoring the traffic. You may have multiple monitoring points going deep into your internal network, but if an attack evades a signature, and all sensors are running the same signature, you’re blind to the traffic. Instead, by deploying another vendor’s sensor at some of the monitoring points, you now have two possible chances at detection.

In the end, where (and what) to deploy will need careful consideration and input from multiple parties. There is no one size fits all solution and this important decision should be at the front end of any deployment of intrusion detection technology.

IDS Signatures

I'll soon be moving into a new job, where I'll be supporting an IDS with a closed signature set. If you're new to NetSec and are tasked with researching out a companies first IDS/IPS, NOT buying a system with a closed signature set is a really good idea. What I mean by a closed set is that the signatures are not editable, or even viewable, by the analyst. Companies will claim "that's our proprietary information and we can't let anyone see it to prevent theft of our intellectual property".
Amazingly enough, other IDS companies have open signature sets and do not have any those predicted issues. What closing the signature set DOES do, to the analyst using the system, includes:

• Blinding the analyst to what traffic actually triggered the alert, making it difficult at best to determine if the alert was a false or true positive.
• Prevents the analyst from modifying the signature to narrow the scope to prevent too many false positives, or using it as the template for a new signature customized the company's individual needs.
• It keeps the analyst from sanity checking new signatures, to see if the vendor is putting out intelligent, well researched signatures, or just throwing out a very generic signature that will also trigger on hundreds of other packets with similar packet data. Companies will do this to allow them to say their signature base detects threat xyz, regardless of whether it's a well written signature or not.

One of the first questions you should ask any vendor (in my opinion) is whether or not they have an open signature set and can the analyst/admin modify or remove signatures (through the GUI, and not through a request process to the vendor), AND, can custom signatures be added.
Another thing to ask, if the signature set is open, is whether or not the signatures are written in a proprietary language or not. Being able to write signature by just following the prescribed format (like Snort or Dragon) is a great boon to the analyst, and cuts way down on the learning curve to get operational. Anyone can learn to write a Snort signature in a short time. However, learning to write a good Snort signature takes a good deal more investment of time..

IDS Placement

A discussion on the SANS Alumni Group on LinkedIn started up concerning IDS placement. I weighed in with this:


"I've had this discussion before when a company I worked for mandated ONE IDS at the ingress points, with additional ones optional. I've always believed behind the firewall would be where you would put the IDS if you could only have one. My reasoning is that 1) behind the firewall shows you what traffic just made it past your router ACL's, firewall rules, reverse proxying,etc. 2) behind the firewall lets you see the real IP of outbound traffic instead of the hide address, so when boxes get infected or compromised and start connecting out, you can shut down the traffic and easily trace the box to clean it. 3) Outside the firewall keeps down the white noise of junk your perimeter defenses and allows you to concentrate on alerts of interest. If you have both, you tune your external IDS for the traffic you need to see, like specific high value servers or boxes that attract a lot of attacks, to gauge what preventive shunning needs done.
Mike Poor talks about the umbrella sensor, that is the sensor right at the edge of the network that's supposed to detect all traffic to everything behind it and how it's not a good architecture. Ideally, you should have layers of sensors at strategic points, like in front of your web farm, in front of your name servers, in front of WAN links with partners and clients, etc and then you can tune each one appropriately with what signatures you run and with filtering. That's the ideal set up. For many, especially those smaller companies, that umbrella sensor is all you're going to get. In that case, behind the firewall is the only place to put it..."

What do you think (especially those new to the network security field)? Assuming you had to work with an umbrella sensor and could only have one, where would you put it?

Practice Makes Perfect

If you're just starting out in network security or working towards that end, packet practice will increase your familiarity and improve your speed in doing analysis. It's best to get better and faster now, when there's no pressure or duress, then when an incident is happening and folks are yelling for answers... or what you find determines whether a production system should be taken off-line.
By practice, I mean spending time working with packets. If you have monitoring access to your companies network, that would be the place to start. Not only will it give you access to a wider range of traffic than using your home network, but you'll start becoming familiar with what is normal. It's very difficult to find the anomalous if you don't know what's normal. If you don't have access to your companies traffic yet (and ONLY use your companies network if you have been given explicit permission, in writing, to do so), there are tons of places you can pull down interesting packet captures to play with and practice.  pcapr.net has a huge repository, the HoneyNet project has monthly challenges, the Army ITOC posts captures from the Inter-Service Academy Cyber Defense Competition (the NSA provides a Red Team against the the service academies Blue Teams), and so forth.
There all different kinds of ways to practice. You can do scenarios; choose an attribute that would indicate possibly malicious activity, then see if you can find any on your network. Do a realtime sniff for any packets whose IP header is greater than 20 bytes, indicating that IP options are in use. Look for large sized ICMP packets (greater than 100 bytes), and if you find some and they aren't malicious, determine why they're large (Microsoft is one culprit in this category).
You can redirect the captures into text, and use awk and sed and grep and other utilities to start building up a collection of scripts and filters for things you'll want to look for often. If you're a perl programmer, you can use all the nifty perl networking modules to make yourself some automated programs to break apart captures and extract data.
You can use tools like xtractr and NetWitness Investigator to see your high level categories, then drill down and look at the individual packets when you see something interesting. I've been using both to look at events from our IDS.
 You can run the packets through dnsiff and see if there are clear text authentication tokens you weren't aware of. Run them through ngrep and use keywords specific to your corporation and look for data leakage going beyond the perimeter of your network. You can load them in EtherApe and watch for the blooms of color.. readily identifying "top talkers", which could be indicative of malicious activity, a network issue or a bandwidth hog, like someone using BitTorrent.
There are dozens and dozens of tools you can run those captures through, and the more you do it, the more familiar you'll become with your network and the better at identifying those folks who refuse to play nice.
Links to packet captures:

• pcapr - http://www.pcapr.net/home
• HoneyNet - http://www.honeynet.org/challenges
• ITOC - http://www.itoc.usma.edu/research/dataset/ (bottom half of the page.. scroll... they're there)

More NetWitness Investigator

To import packets into Investigator, you need to create a collection (the default collection, as we saw, is simply called "Demo Collection". Create a new collection (Ctrl-L or use the menu), and then you can import your pcap file. The capture must be 1 GB or less, and you can have 25 simultaneous captures at any one time.
Double click your new collection, and after a second or so, you should see it's status change to ready. Now you can right-click on it and choose "Import Packets". Browse to where you've saved your pcap file (hopefully you captured it with full packet data), and you'll see the import process begin. Depending on how much of your 1 GB limit you used, this may take a little while. Once you get the "ready" status once more, double click your collection once more and open it up..

Here we see all of our source and destinations, aliases, content types, ports and services used... and in the lower part of the report, (below) we have counties and cities involved (mine always has Beijing, as does probably yours) and organizations.

Since we see some of our hourly packets from China there, we can now drill down and see that kind of traffic they entail. Clicking on the number in parentheses (the packet count) after Beijing takes us to a nice sessions screen with our summaries for each packet...

I'm redacting my IP address here, replacing it with x.x.x.x, but you get the idea.. Now I can zoom in on the packet data.. and see that as we mouse over each field in the packet header, we get a little pop up telling us the field and doing the hex conversion for us.

We have all the summary information of all packets in the capture to look at. We can, in a few mouse clicks, go from the high overhead view down to the single packet, to any field in that packet, and see it's identification and conversion, if needed. Pulling all this information out manually, then drilling down to granular of a level, even with scripting, would take far, far longer.
Next we need to explore using Investigator to do the data capture itself.

Another Free Packet Tool - NetWitness Investigator

NetWitness Investigator is a  free tool for packet analysis. This one runs on Windows (including Windows 7) only (except for the commercial version, which runs on Linux) and has functionality that includes wireless support, real time layer 7 analysis, IPv6 support, SSL decryption (with the server cert) and full context searching including regex. The freeware version license supports 25 simultaneous 1 Gb captures.

To get started, grab a copy from the download page here and install.
When opening the program for the first time, you'll be prompted to login, or create an account. Go ahead and create a community account and after activating Netwitness (from the email you'll receive), you'll be presented with the Investigator window. In the left hand pane will be a folder marked Demo Collection, a packet capture that allows you to test out the app immediately, without needing to do any capturing of your own.

Double-click the Demo collection icon, and a new tab will open with the contents of that collection, which will look like this:

We can see a whole host of attributes that can be drilled into to explore. At the very top, we see Investigator has flagged three alerts for us, non-standard http, cleartext passwords and irc file transfer. Clicking the "Alert" attribute will drill down to the next level where all the attributes are related to these three alerts. We see the services used, source and destination addresses, ports and protocols, and all our layer 7 info broken out for us, like email addresses, user accounts, and attachments.

Clicking on the "sample_vulnerability:cleartextpasswords:other (1)" link takes us down another level, showing us all the attributes of the packets that triggered this alert. We now know the IP addresses involved, the service type (POP3), the user account involved and the email address involved. When we drill down even further in our later 7 data, Investigator will do reconstructions for us. For example, the email address attribute has one session associated with it (the number in green in parentheses after the field shows the number of sessions). When we click on the number after the email address, NetWitness reconstructs the email header for us:

   
We can do the same for the other data presented, such as the Creditcards.txt attachment. Investigator will ask us how we want to open the file, allowing us to open it in a default app for the data type, choose a different app or use Investigator itself to open it. 


This is just the very basic starting point of what you can do with this tool. Like xtractr, it's a very good tool to investigate a capture and you need to drill down deep in a visual way to ascertain what action took place. 

BSides Cleveland

Went to Security BSides Cleveland yesterday (February 18) and had a great time. The conference was at the House of Blues in downtown Cleveland, and there was a great lineup of speakers. You could truly say there was something for everyone, because the topics ran the spectrum of everything from security education and GRC (high level talks) to thicknet (SQL injection), using Netflow for security and claims based identity management.
I'm definitely looking forward for the next BSides that comes within range of me attending.
The videos are up and available at Ustream (http://www.ustream.tv/channel/security-justice---test)

Learning awk with Practical Examples

Learning awk with Practical Examples Ten Days of Awk at bashshell.net. I just found this site today. It looks like it will be a good resource for beginners at Linux (including those just moving over to NetSec).

OpenFPC

OpenFPC Network traffic capture tool. Three components here. A client device used by the analyst to request a packet capture. A master device (or devices) that receive the request, look up what device would have seen the traffic and pull the capture file and display it back to the client. One or more slaves, which are the sniffers doing the packet captures. Looks very interesting; added to my growing list of tools I'd like to test..

BPF's For Beginners

If you've just started looking at packets, no doubt you've already found the need to pull out packets based on more than just the primitives built into tcpdump and other pcap aware tools (such as src, host, port, net, etc). tcpdump supports the use of the Berkeley Packet Filters expressions, allowing you to search on any field that tcpdump records. The basic syntax is field : operator : operand. For example, if you wished to see all IP packets whose length is greater than 20 (meaning IP options are in use) you would add the filter:
 'ip[0] & 0x0F > 5' This says to look at the first field in the IP header (offset 0, headers begin counting at 0), and using a bitmask, filter out the first nibble (the first four bits, 0), looking at the last four bits, (F) and show any that are greater than 5. We know the IP header length field is expressed as a 32 bit word (four bytes), so a 5 in that field (5 x 4 bytes) equals 20, the length of a normal IP header (also the minimum size). So any size greater than 5 would mean a header more than 20 bytes. This means IP options have been added to the end of the header, such as strict source routing, or loose source routing, rarely used and often a sign of a malicious attempt to route packets through a particular node for sniffing or interception.
   Enclosing your filter in quotes to avoid syntax errors is a good habit to get into. You'll eventually need them as your filters get more complex.
You can search on any field in any of the embedded protocols as well. If you needed to see SYN packets only, you could filter on the TCP header with a filter of 'tcp[13] = 0x02'. Here we would see only packets with a value of 2 in byte 13, which would have only the SYN flag set.
   If we needed to look at just a single bit or combination of bits, instead of a whole byte or a whole nibble as in the first filter, we would go back to our bitmasking. Bitmasking bits here is simply applying a bitwise AND  to filter out any bits we don't wish to look at, and preserving the bits we do. If we AND a zero to a bit, we end up with zero, if we AND a one, we end up with a 1. So if need to see all SYN-ACK packets, we would want to see all packets with byte 13 in the TCP header that have the least most significant bit set in in the first nibble and the second least significant bit set in the second nibble.

                    C E  U A   P R  S  F
                    1  1  1  1    1  1  1  1
Our mask     0  0  0  1    0  0  1  0

So our bitmask for SYN-ACK packets would be 0x12 and we would apply it as this:

'tcp[13] = 0x12' (no other flags should be set here as the SYN-ACK combination is only used in the TCP handshake.)

You can extend this to any field, down to a single bit, and chain them together to get as granular as you need. When you begin writing complex filters and using them over and over again, you can put each one in a file, and use the tcpdump -F parameter to load it, instead of retyping it each time. Very powerful stuff for dissecting packets, and very good to learn. You can't always be sure that you'll have access to Wireshark or some other protocol parser when you do to do network forensics. If you learn the command line way to do things first, you'll hopefully never get stuck.

If you want great training in bits and bytes, protocols and packets, and everything in between (especially if you want to be an intrusion analyst) I'd recommend looking at the SANS 503: Intrusion Detection In-Depth course. Taking it live, in my opinion, is always best, as you'll get the most interaction with the instructor, work with other folks interested in the same knowledge and absorb much more than just the class from the atmosphere of a SANS conference. But take it any way if you can, if you possibly can. And if you CAN go live, use the time you have there to its fullest. Sign up for the free SANS @Night classes, go to the Storm Center presentation, go listen to the keynote speakers in the evening, do the lunch and learns and sit in on a BOF (Bird of a Feather) session. That's why I'd never to go to a SANS conference at Disney World or Virginia Beach. Too many temptations to waste your time doing other things, and the opportunity to learn practically all day long is too valuable. But's that's just the way I roll. =-)

Packet Analysis With xtractr - Continued

Now that we have our packets indexed, we need to start the web service and load them. The default address to bind to is localhost, on port 8080. If you're working remotely on your xtractr box, and especially if you want others to be able to look at the packets with you, you'll want to change to bind to your networked interface (realize no one can connect to the box and look at the packets or run queries without a Mu Dynamics account).
So we now use the browse parameter, like thus: xtractr browse (index_directory) --host (network_ip_address).

So using our example above, and assuming our box has an IP address of 192.168.100.10, our browse command would be: xtractr browse index_dns --host 192.168.100.10
If we wanted to use a port other than 8080, we could add the --port parameter to bind to a port other than 8080.

Point your browser to ip_address_of_the_xtractr_box:8080 or, if you specified a different port, ip_address:(port_number). Your browser will be directed to the Mu Dynamics xtractr site, where you'll be prompted to login with that account you created at step 1. Now, as you query and drill down, your data will be streamed through the Mu site where all the processing will take place (again note that unless you choose to upload your pcaps, they will remain on your box.) 

Mu has a large repository of user supplied pcaps available to the community, to test with and experiment. They encourage you to contribute to the library, and there are some very obscure and esoteric packet captures available to you there.

You can go to the xtractr web site's live demo, at http://www.pcapr.net/xtractr/demo#/flows, and see what all this packet parsing goodness looks like, and play before you download.

(Folks at Mu.. if you come across this post and I've screwed up anything, please Twitter me @JeffSoh and I'll correct it ASAP)

Packet Analysis With xtractr

I've mentioned xtractr by Mu Dynamics before, and over the last week or so I've been using it more and more to look at EOI's. If you've not used it before, here's a little overview of this tool.

• xtractr runs on Linux (the latest version is 4.5.40426) and can be downloaded here. 
• You will need an account (free) with Mu Dynamics to use the query services.
• xtractr can be used in stand-alone mode, which means your pcaps, queries and labels never leave your machine. It can also be used Mu Studio to convert the data into a stateful test case.
• More than one person can look at the data at a time, and if you need to look at more than one capture at the same time, you can run multiple instances of xtractr.
• The free lite version can index a capture of either 10 million packets or 1 Gig of pcaps.

That said, here's what you'll need to do to use xtractr. After creating your account, and downloading the tarball and installing, you're ready to index your packets. You'll want full length packet captures here if you're going to do network forensics.

Create yourself a workspace directory of whatever name you want, and copy (or capture, if you're testing) your pcap there. I'd suggest giving it a meaningful name, so you know later what that pcap is without having to run it.

Make a sub-directory, again, whatever you wish to call it, to store your indices in.
Now you need to index the pcap. The syntax is: xtractr index (index_directory)  --mode (basic|forensics) (pcap-file).

So if we had a pcap called "dns-traffic.2.4.11.pcap" and a sub-directory called "index_dns", and we wanted full data (forensic), we would run: xtractr index index_dns --mode forensics dns-traffic.2.4.11.pcap . Depending on how big your pcap is, this might take a little while to run, xtractr will give you a progress meter while running and return to prompt when down. You can omit the mode parameter, by the way, and xtractr will default to basic.
 (Continued - Blogger is not co-operating today)

Enterasys Dragon IDS

I've managed a Dragon IDS system for ten years now. When the company I worked for tasked me with researching and evaluating intrusion detection, some of the major players today didn't even exist yet, and snort had not yet been rolled into the commercial Sourcefire product.We looked at a number of the available offerings back then and settled on Dragon.
At the time, Dragon was winning IDS "bake-offs" and garnering a number of awards, and it seemed to be a decent choice. We did all of our own testing; there were no monies for calling in a security consultant to evaluate our needs and pair us with a product.

Dragon was written by Ron Gula, who is now the CTO of Tenable Network Security. Ron had just sold Dragon to Enterasy, which made some of the early support calls interesting.
But some of Ron's staff had gone with him to Enterasys (like Rick Olesek) and were excellent folks to deal with and very knowledgeable. We started out with Dragon 5, which by today's standards, had a rather primitive management interface. Changes were made by editing the various configuration files directly, either through the browser or using vi on the server itself (I actually missed that, once the interface became sophisticated enough to not require it anymore. Gave you a degree of confidence the changes you were making were actually getting made).

Over the last ten years, I've maintained a love/hate relationship with the Dragon (the original version we used called the HIDS component a "Squire" and the tools were in the sorcery directory. The product was originally named "Dragon Fire IDS). It's fairly easy to maintain, signatures are released for new vulnerabilities in a timely manner, the reporting is easy to set up and filter, and the newest interface has a really nice looking console that is easy to drill down into.

I also remember, however, the early versions of version 7, when I spent days on end on the phone with their developers, with them remoted into my desktop as they tried to fix issue after issue. I remarked more than once that I didn't remember signing on to be a beta tester, but I evidently was one anyway. Fortunately, as of version 7.4.1, all of the bugs have seemingly scurried back into the darkness, and the systems are running without intervention again. Which allows the intrusion analyst to spend his time actually investigating alerts, as well as tuning and filtering. What a concept.

Enterasys was merged with the Siemens Group in 2008.

Practicing Packet Analysis

Here's one way to get started quickly practicing packet analysis, using nothing but open source, free tools.

You'll need a Linux box with sufficient drive space to store your capture packets and a monitoring interface patched to the network you wish to capture from (use your own network, or make sure you have permission to monitor the network you'll be using).

Packet Capturing

You can several different tools to produce packet captures.

1. First is Daemonlogger, written by Marty Roesch of Snort fame. Daemonlogger can write packets to disk (what we want here) or act as a software tap, rewriting the packets it captures to a second interface.

2. If you have Wireshark installed, you can use the command line utility tshark to sniff. tshark, like Wireshark, is a network  analyzer that can display tons of information on the packets it receives, not what you want here.  The options are many, and if you're just starting out, tshark may not be the best choice. Make sure you use quiet mode so the capture isn't displayed to the screen (you'll drop packets as that slows things down considerably) and set the snaplength (the size of the capture) to 0 or 65535. If you're going to analyze packets, you need full packet captures unless you're just doing statistical analysis/trending. -x will capture the hex and ascii data.

3. If the box already has snort installed, you can simply run snort in packet logging mode. After all, that was the whole reason Marty wrote snort in the first place. He didn't like the sniffers that were available at the time, so he wrote his own. It developed into an IDS later.

4. IDABench. IDABench is a now defunct packet auditing tool written by George Bakos. Even though development ended on it several years ago, it's still a wonderful tool that I use daily. Apart from the Perl framework that runs various tools and creates web pages of summaries, the first thing IDABench does is simply run tcpdump and create packet captures, one for each hour of the day. For what we're doing here, its overkill just to get dumps, so unless you want to use the toolbox IDABench provides you, that leads us to number 5.

5. Just use tcpdump itself. There is a parameter (-C) you can pass that tells tcpdump to roll over according to the size of the capture. Whatever you pass to -w (write to file) will be used along with a number as the filename (if you use “-w capture”, you'll end up capture1, capture2, and so forth).

Analysis Tools

Obviously, there are many, many tools available to analyze packets. I'm just going to mention a few that I use or have used in the past here to allow you to drill down into the packets and inspect either the header fields or payload, or both. Once you have a (libpcap compatible) packet capture, you can use any or all of these tools to start digging.


1. Wireshark. Wireshark is the most complete network analysis tool that I know of, at least of the free tools, and has dissectors, or decoders, for literally thousands of protocols. Open your packet capture in Wireshark and you can drill down to any level in the protocol fields, examine the payload, follow a TCP stream, save off reassembled files that are contained in it (including VOIP traffic that can be saved as a .wav file and replayed). It’s an excellent tool. It may also be yet another case of overkill, though, depending on what you're looking for. If all you want to do is extract any login authentication pairs found in the packet stream, for example, it may not be the tool to use.

2. dsniff suite. dsniff, however, IS the tool to find clear text logins. There are also tools that will capture email traffic (mailsnarf), older versions of instant messaging (msgsnarf), extract URL's (urlsnarf), files (filesnarf), and even surf along with a user in the stream using webspy, which will redirect any URL's it finds to your web browser (more fun than functional).

3. ngrep is a command line program that will search for content in a packet stream or capture, much the way that grep does text files. It supports regex and is a powerful way to quickly find strings in packet captures.

4. Netwitness Investigator. Investigator is a Windows GUI that performs "free form contextual analysis" of your packets. Install the software, create a new collection, import your pcap file into it and load it. You'll be presented with a visual screen of things like alerts, hostnames, user accounts, ports, protocols, even countries of origin, that have categorized info below them you can drill down to find the exact information you need, and finally look at the raw packet capture of. A very nice tool to experiment with.
You can capture new traffic on the box it's installed on , as long as WinPcap is installed, which is very handy to get up and practicing quickly.

5. chaosreader - chaosreader is a big perl script that takes your packet capture and parses it down to TCP/UDP sessions and application data. It will pull telnet and ftp sessions, HTTP, email. etc. and index it all in html files that you can feed to your web server, or open directly into your browser. Be warned, though, that parsing of large files will take a long time and use up all of your available resources on your machine. It's better to use smaller packet captures, or use filters to pull traffic of interest out and write it to a separate dump file and then feed that to chaosreader.

6. snort. Snort can be run in stand-alone mode, which means you can feed it a packet capture as a parameter and run the traffic through all of snorts pre-processers and signature libraries. You might want to do this to get a second opinion on events captured by another IDS or to find alerts to drill down to practice your analytical skills. Or if you're doing network forensics for someone else, you may only have access to the raw packet capture and need to regenerate the alerts for your use.

7. xtractr - xtractr is a tool provided by MuDynamics that takes a packet capture, indexes it and writes it out to an html file. The free version has a limit of 10 million packets or 1Gb of pcaps. That’s plenty enough to practice your analytical skills.

Packet Capture Tools
1. Daemonlogger
2. Wireshark
3. Snort
4. IDABench Sorry, links are no longer active. I have the install files if interested.
5. tcpdump

Analysis Tools:

1. Wireshark 
2. dsniff
3. ngrep
4. NetWitness Investigator
5. chaosreader 
6. Snort
7. xtractr