Nmap Basics Part 1

Nmap 101 Tutorial

There are two common types of scans, the syn scan and the full connect scan.

-sS - Syn scan sends a syn packet, and if it receives a syn-ack, marks the port as open and sends a reset and tears down the session.  Must be root to run a syn scan, as it manipulates the network stack to send the reset out of sequence. 

-sT - Connect scan. Completes the full TCP/IP handshake. Any user can run a connect scan as the network stack is not manipulated. 

-v verbose mode. Can be repeated for increased verbosity, will update the status more often. If -v is not used, using the space bar you can force a progress update of the scan.

--open Only report on ports that were found open at the end of scan summary.

-sV - version scan. Reports on the version of platforms of running services like http, ssh, etc. Varying degrees of effort to determine the version, use --version-all for the most accurate results at the cost of a slower scan

-O OS scan - attempts to determine the operating system of the host being scanned. Note both version and OS scans use scripts written for nmap written in NSE, Nmap Scripting Engine that extend functionality. Use --osscan-guess for the most verbose results as it will fuzz the scan, again at the cost of a slower scan.

Output can be saved in several formats. Use -oA to save in all formats in separate files, including greppable (deprecated) and XML.

-Pn - don't ping the hosts before port scanning. Useful on networks that block ICMP echo requests.

BPFs

 Introduction

What are Berkeley Packet Filters? BPF’s are a raw (protocol independent) socket interface to the data link layer that allows filtering of packets in a very granular fashion¹.

BPFs were first introduced in 1990 by Steven McCanne of Lawrence Berkeley Laboratory, according to the FreeBSD man page on bpf².

 

Working with BPF

If you use tcpdump for very long, you encounter what are called “primitives”, filter expressions to tune your results to only see certain traffic. Examples of primitives are “net”, “port” “addr” and qualifiers to those such as “src” or “dst”.

With these, we can limit our results using filters such as ‘src host  10.10.1.1’ or ‘net 10.10’. There are many of these (see the man page of tcpdump for the full list)

You can also specify protocols, such as “ip”, “tcp”, or “icmp”.  Some even make comparisons, such as “less” and “greater” for packet length.

These primitives are shortcuts for BPF’s. Each one references some field or fields in one of the network protocol headers. For example, the embedded protocol field in the IP header is the 9^th byte offset from 0. If the value contained there is a 6, the packet is TCP. So the primitive “tcp” really means show me all the packets in the IP header whose 9^th byte offset from 0 contains a 6. If we wrote this as a BPF, it would look like this: ‘ip[9] = 6’ or using hex, ‘ip[9] = 0x06’ .

BPF’s can go far beyond the built-in primitives, allowing us to get as granular as needed, down the single bit level. If a field does not span the entire byte, we’ll need to write a BPF to look at the bits in question to determine the value there.

 

 

 

Let’s look at the first line of the IP header³ to see an example.

Byte	0	Byte 1	Byte 2	Byte 3
IP Version	IP Header length	Type of Service	Total Length

 

 

We see byte 0 (we start counting from 0, which is what we mean by offset from 0) that there are two fields in the byte, the IP Version field, and the IP Header Length Field.

If we wanted to see what the IP version of the packet is, how we would do this? We only want the value in the high order nibble (high order = left most as we count bits from right to left, and a nibble is 4 bits, or half a byte). To see that value we have to extract it from the byte of data somehow and look at it singularly. To do this, we employ a method known as bitmasking. Bitmasking is simply filtering out the bits we don’t wish to look at and retaining the ones we do.

To accomplish this, we’ll perform a bitwise AND operation on all of the bits in the byte. If we AND the bits, only the ones with a value of 1 will be retained. Let’s look at this.

 

Here’s a binary representation of a typical first byte in the IP header:

0 1 0 0      0 1 0 1

We’ve separated the two nibbles here for clarity. We see the low order nibble (right-most) has 0101. This is our IP header length. We want to check the high order nibble, which has the value 0100. To do this we will add 1 to each bit. In a bitwise AND, any values except two 1’s equal 0. Two 1’s equal one.

So to manipulate the bits to see the first nibble only, we want to add 1’s to the high order nibble and 0’s to the lower order. Since all 1’s will equal F in hex, we will write an expression adding hex F to the first nibble and 0 to the second.

Here’s what the BPF will look like:

ip[0] & 0xF0 =  0x04 (our search value).

Broken down, we are telling tcpdump to look at the IP header (ip), first byte offset from 0 ( [0] ), retain all the bits in the first nibble and discard all the bits in the low order nibble ( & 0xF0 ) and show us all the packets with a value of 4 in that nibble ( = 4).

 

Here’s our bit-wise operation…

0 1 0 0     0 1 0 1

1 1 1 1    0 0 0 0

 

0 1 0 0  0 0 0 0

We now see the low order nibble has been filtered (all 0’s) and we have the high order nibble left. Binary 0100 = decimal 4, so this shows us the packet has value of 4 in the high order nibble of the first byte; the IP header is set to IPv4.

 

Sample Filters

Now that we see how BPF’s work, here are some samples of filters we can search on:

'ip[9] = 0x11'                                                                                                udp

'ip[9] = 0x01'                                                                                                icmp

'tcp[2:2]'                                                                                                       2nd byte, spanning                                                                                                                             two bytes

'icmp[0] = 0x08'                                                                                           echo request packet

'tcp[2:2] < 0x14'                                                                                          tcp dest port <  20

 

 

 

 

 

Let’s create a filter for one of the more common and more complex uses: TCP Flags

 

The flags field in TCP is found at the 13^th byte offset from 0. The flags themselves inhabit all of the lower order nibble, and the two lower-order bits of the high order nibble.

The two high-order bits of the high-order nibble are used for ECN (Explicit Congestion Notification). Here’s our layout…

TCP Byte 13                                            

 

 

Flags                               CWR    ECE      Urg       Ack       Push     Reset   Syn       Fin

 

Binary Values               128       64         32         16         8           4           2           1

 

Let’s assume we wish to see all packets with the SYN and FIN flags set. This is anomalous behavior and is usually indicative of a port scanning method.

We would need to look at the whole byte and retain all bits except the two highest orders in the high order nibble. To do this we need a mask retaining all of the lower order nibble and the lower order bits of the high order nibble. Here’s our bitwise operation mask:

 

               High order                                Low order

CWR    ECE      Urg       Ack   --   Push   Reset   Syn       Fin

   0           0           1          1      --      1         1          1           1

 

So, we would have a hex F in the low order nibble (all 1’s), and a 3 in the high order nibble (a 1 in the 0 column, which is 1, and a one in the 2’s column, which is 2, equals 3).

So our mask would be 0x3F. That would show us only the bits that contain TCP flags.

 

If we use that filter and look for a value of 3, meaning the two lowest order bits are set, the Fin and Syn bit, we would end up with this:

tcp[13] & 0x3f = 0x03’

 

This filter tells the system to filter on the 13^th byte offset from 0, discarding the two highest order bits, and showing packets the have a total value of 3 in the six remaining bits, which would mean the Fin and the Syn flags were both flipped on.

 

Now that we know how to look at only the bits we need, we can apply this to any field, in any network header. You can, of course, string multiple filters together to get as specific as needed. Here’s a tcpdump query to show us all packets with the Syn flag set, and a datagram (packet) size greater than 134 bytes (probable data on the Syn packet), and an IP version that is NOT 4:

 

tcpdump –nn –i eth0 ‘tcp[13] & 0x02 = 2 and ip[2:2] > 0x86 and ip[0] & 0xF0 != 0x40’

 

Summary

Berkeley Packet Filters are a powerful tool for intrusion detection analysis. Using them will allow the analyst to quickly drill down to the specific packets he/she needs to see and reduce large packet captures down to the essentials. Even a basic knowledge of how to use them will save hours of time during the investigation of packets, or give insight into malicious traffic that wasn’t detected using other methods.

 

References

1. http://en.wikipedia.org/wiki/Berkeley_Packet_Filter
2. http://www.gsp.com/cgi-bin/man.cgi?section=4&topic=bpf
3. http://en.wikipedia.org/wiki/IPv4

Wild West Hackin' Fest 2021

Wild West Hackin' Fest - Deadwood:  I so love this conference. It's so much like DerbyCon. There aren't any "I hacked a car/plane/drone/coffee maker" talks. There aren't 10,000 people there and it won't make the national news like BlackHat or Defcon, but what it will do is give you the information you can use in your day-to-day job. Things to actually make your network safer. Flash and glitter are great and lots of fun. Huge parties and massages courtesy of IoActive at the Oasis are cool too if that's what you like. Less interesting but much more relevant is information you can apply to your job, day one after the con, and make your network just a little more secure. 

Speed Kills

 In computer science, speed is all. Everything we do, we want to do as fast as possible. Right? No. Consider slow hashing algorithms. Some hashes are intentionally computationally expensive. Why? Because attackers have multi-GPU dedicated password cracking machines that can try millions of variations per second. A hash that requires more computation power is much slower to crack because of the time it takes to try each variation. Sure, your user may have to wait for half a second longer for the authentication form to go away and his page to load, but his/her credentials are many times more resistant to breach if an attacker obtains the hashed passwords. Consider this. On a fast, multi-GPU cracking machine, as many as 180 billion password hashes could be cracked per second. With bcrypt, that number would fall to 71 thousand hashes per second. Which one would you want YOUR password encrypted with?

Packet Captures in the Age of TLS

 Ten to fifteen years ago, a company having FPC (full packet capture) was an indicator of the seriousness of the company's information security efforts. Having trained analysts that could use those packets to analyze alerts from NSM devices was an even better indicator. 

Today, the network landscape has changed to the point of having little similarity to a decade ago. The workforce was already starting to go mobile before COVID, and the pandemic forced a large swath of workers home. Mobile devices used for work became common and the services in the cloud, prevalent. 

We started seeing diminished value in some of our traditional tools. Endpoint Detection and Response suites gave us better fidelity about what was happening on our endpoints, regardless of whether the device was sitting on the corporate network, at home, or at an airport or coffee shop. AntiVirus became less useful. Yet, it still had value as a layer of defense and was good for detecting common, prevalent threats. The fact that its effectiveness was diminished didn't mean the tool was no longer useful. 

Packet captures are similar, in that though their effectiveness has been diminished by the use of SSL encrypted traffic, especially ECDH, it still has its place in layered defense. There is still much HTTP traffic on the Internet, plain text protocols are still in use even though safer alternatives have been prevalent for years, and they're a lot of cases where an attacker will use whatever means to do call-outs back to the attacking machine that uses unencrypted traffic. But there's another reason full packet capture is useful.

Consider the scenario where a host machine just made a connection to a known malicious website. The connection was made over TLS 1.2. You have packets from the connection, but the data is encrypted. An examination of the packets shows the TCP 3 way handshake, and five additional packets. In the first packet, there is some clear text obviously part of a certificate from the destination host, followed by three very small packets between the two hosts and finally a reset packet.

We can tell from the packet capture what happened, even though the data was encrypted. The client established a session with the site, the TLS handshake took place, and the session was ended. The rest was probably spoofed by a network security monitoring device if the traffic was not simply dropped by a firewall/IPS. What we can know is that there was no data exchanged between the two hosts.

If we have a full packet capture system set up, we can validate no data was exfiltrated, and no malware was downloaded, in seconds. Still very much valuable. And if there is no FPC solution, still worth the cost and effort to implement. (See Akime, formerly Moloch). 

Pcaps and the Tools That Love Them Part 3

 

From here, with header diagram in hand, you should know be able to look at a packet dump and find the value you need, by counting from offset 0 to the correct field. 

Some of the things you'll want to look for (like IPv6 traffic or only ICMP traffic or a particular IP) can be done most efficiently with primitives. 

Others, like, options in the IP header, which are rare and always suspicious can be done by by checking the header length field, but since the first byte in the IP header is broken into two fields of a nibble each, how do see just the header length, and not the IP version?

The answer is bitmasking. Bitmasking is hiding the bits we don't want to see and leaving the ones we do, simply put. For example, let's take the 13th byte (offset from zero, of course) in the TCP header.

Flags     CWR ECE Urg Ack     Push Reset Syn Fin

Binary Values 8    4     2    1          8      4             2    1

We have one byte, eight bits that comprise the TCP flags that can be set. We know from our basic networking that when one host wants to talk to another host, it sends a SYN (synchronize) packet to the IP address of the receiving host on the port the protocol should be listening on. 

If the receiving host is listening on that port and ready to communicate, it sends a SYN-ACK packet back (synchronize acknowledgment). 

The sending host confirms it's received the reply and is ready to start communicating by sending an ACK packet back to the destination. This is commonly known as the TCP three way hand shake.

So in order to see all of the attempts to initiate a connection on the wire or in a packet capture, we would want to see all of the packets that only have the SYN flag set. Looking at our header of byte 13, we see that the SYN flag bit is in the second nibble (4 bits) and the next to last bit. 

The way we write a BPF is to 1. specify the header, 2. the byte, and 3. any bitmasking needed to filter out additional bits.  So to see only SYN packets we would write a BPF like this:

tcp[13] = 2 - Note this shows us packets that ONLY the SYN packet is set. 

We can also write a BPF to show any packets that the SYN flag is set, as well as any another flags. To do this, instead of checking the full value of byte 13, we just check to see if the SYN bit is set. 

We would do this with a BPF like this: tcp[13] & 0x02 != 0. Here we're simply checking to see if the SYN bit is set to 1, or turned on. Any other flags could be active or not, we're not checking for anything other than SYN. 

That would mean the value of the entire byte is 2, meaning no other bits are set, or have any value but zero.

In binary our byte would be 00000010 which equals two.

This short hand way of writing BPFs works, up to a point. Once we have to represent a value greater than 9, this binary shortcut breaks down.

In that case you need to write the BPF in hexadecimal, so lets rewrite our BPF in hex and get used to writing them that way.

In hex our value would be 0x02, so the BPF would be tcp[13] = 0x02. In hex, each digit represents a nibble, or 4 bits, so two digits in hexadecimal equal a byte.

So we see here none of the first four bits are set (0) and in the second nibble the SYN flag is set, which is 2 to the power of 1, or 2. What's with the 0x in front of our numbers? 

0x tells our computer this is a hexadecimal value. Octal numbers are prefaced by just a o, or more recently, 0o so 0o5 would be an octal 5 (octal is base 8 and we don't need to delve into that here).

Just as binary is base2, and decimal is base10, hexadecimal is base16. That means in hex we run out of numbers before we get to 15 and move to the next higher place. So How do we accomplish that?

Hexadecimal uses letters to represent the numbers 10 through 15. A=10, B=11, C=12, D=13, E=14 and F=15. So 0x1A is the hexadecimal number 26. 16 plus 10. 

Pcaps and the Tools That Love Them Part 2

There's more to a primitive under the surface, and once we discover what it's actually doing, it opens up a whole new way for us to inspect and filter packets. 

Lets use the TCP primitive as an example. What is it actually doing? How does it show us only TCP packets? 

To see this we need to look at the IP header..not the TCP header. Here's a really simple version of a nicely color coded diagram.

Now we can see in the IP header we have a field called Protocol, or more accurately, Embedded Protocol. This signifies the next layer protocol embedded in the packet. 

If that field is 01, we have an ICMP packet. If it's 06, it's TCP. If it's 0x11 hex, or 17 in decimal, it's UDP.

So the tcp primitive is checking the Protocol field and seeing if that field contains 6, and if so it passes the packet on to be displayed or captured. 

Now, what field IS that. When counting header fields we start counting at zero. So we can count from the beginning of the packet and see that the protocol field is field 9.

Not clear from the diagram? You're right, it's not..too simple now..here's one showing the number of bits in each field.

Now we can see each line is 4 bytes, 32 bits. And starting from zero and counting the bytes we find Protocol in the ninth field.

The first two fields are divided into half bytes, call nibbles, 4 bits each. Just count them both as one byte. 

We'll cover what they do later and why the authors who wrote the protocol divided them up.

Most fields in the protocol header break on byte, or nibble (half byte) boundaries.

There are a few notable examples, which we'll cover as we look at bitmasking. Bitmasking can be simply defined as showing only the bits you need to see and masking out the others.

The same principles apply to any protocol. If there's a header, you can parse it. 

Getting yourself a good set of diagrams, in the format and layout YOU like is really going to help you. Unless you're in the rare position of doing nothing but analyzing packets all day, you're not going to memorize the protocol headers, probably. 

But from practice, you will remember the embedded protocol is in the ninth byte offset from zero in the IP header, or the flags bits are in the 13th byte in the TCP header, or the IP version is in the most significant 4 bits of byte 0 in the IP header. 

Without bunny trailing into big endian and little endian, for the purposes of packet inspection just remember that network byte order always starts from the left (you start counting at the leftmost bit and move right).

So to make this practical (and start at the beginning), the first byte of the IP header (byte zero) is split in half. The first four bits are the IP 

The second 4 bits, or nibble, is the header length. This is a value multiplied by 4. A normal IP header, with no options, is 20 bytes, so this should normally be 5.

Now that's your first visual clue when looking at a packet capture in hex. If the capture included the Ethernet header and you're looking for the start of the IP header, it SHOULD, but isn't always, be 45.

So if you see a packet capture and don't know what it is, a quick way to start orienting yourself is to look for a 45 byte.

Pcaps and the Tools That Love Them Part 1

There are many pcap tools available and which ones you use really depends on what you're using them for. Some are very good at just giving you the raw data, others parse the data and show you certain types of packets..

But maybe we should back up one step and define what we're taking about. What is a pcap? Simply put, a pcap is a binary file that contains packets captured off of a network interface.

How much data, and which fields depends on the manner that you capture the packets. We'll look at some of those options as look at tcpdump. I won't spend a lot of time on that tool because many people are familiar with it, at least the basics. It's actually a powerful tool when combined with BPFs, but we'll cover that as well later.

BPF is short for Berkeley Packet Filters and they allow pcap tools to get granular down to the bit, the BIT, level in specifying what header fields you want to see. For example, the TCP flags are each one bit in size. With BPFs you can specify to your tool to only show you packets that the SYN flag is set. Or SYN and any other combination of flags

The simplest BPFs are built into tcpdump. "tcp" and "port" are primitives. Others include udp, src dst, icmp, host and ether and net.

Other tools built on the libpcap library can use them as well, tools like ngrep and  tshark.

An example would be tcpdump -nn -i eth0 'tcp and port 23' 

This would tell tcpdump to listen on the eth0 interface, don't resolve hostnames, don't do port resolution, and only show packets that are the TCP protocol and either the source or destination port is 23.

If we only wanted to see traffic going to port 23, we could add the primitive dst in front of port.

tcpdump -nn -i eth0 'tcp and dst port 23'

Get into the habit of putting your BPF's into single quotes.

It will make no difference using simple BPFs like "port 80 or port 443", but when you get into complex BPFs using bitmasking, it will keep you from getting syntax errors.

So why use the -nn options, that disable name and port resolution? There are several reasons. One reason is speed and efficiency. 

If your sniffing a fast, busy segment and have to do a DNS lookup on every address, it will slow tcpdump down to the point of starting to drop packets. 

Another reason is if you're monitoring malicious traffic and the attacker controls his/her nameserver, they will see the DNS lookup and know they're being monitored by you. 

Port resolution is done using Linux's services file, in /etc/. This is a mapping of ports to services, so when tcpdump sees port 80 it substitutes http for 80. 

But can other services be bound to port 80? If you're root you can bind any service you like to any port you want. That port 80 traffic could be any protocol. So we'll bypass the commonly used ports and see what the traffic is ourselves.