BPF's and Bit Masking

BPF's (Berkeley Packet Filters) allow you filter down to the bit, not just the byte, of a protocol header. There a number of fields that might require you to do this, most notably, the TCP or IP flags fields.

The TCP flags indicate what type of packet it is. They are:

SYN - synchronize or begin the connection. SYN packets are only used during the initial three way TCP handshake.
ACK - acknowledgement that a packet was received
FIN - finish the connection gracefully with both sides shutting it down in an ordered manner.
RST - reset tears down the connection with no further exchange of packets, usually used when a host denies an attempted connection or a the client is terminated abruptly, like closing a web browser.
PSH - the push flag indicates to the other connection that it should process this data as soon as possible  instead of queuing it in the TCP buffer
URG - the urget flag is used to process data that must be processed immediately (such as a character in a telnet session) and uses a pointer to indicate the last byte of urgent data.

These fields are located in the TCP header at the thirteenth byte offset from zero (protocol fields start at 0, not 1). Proceeding them in this byte (highest order bits, or left-most since networking is big endian) are two flags used for differentiated services, or what used to be known as ToS (type of service). They aren't related, just know they are there as it will matter when we construct a BPF.

So our bits are laid out like this:

C E U A      P R S F

In the first nibble (4 bits) we have the two differentiated services flags, then the urgent flag and finally the acknowledgement flag.

In the second nibble we have the push flag, the reset flag, the syn flag and end with the fin flag.

So if we need to see only one type of packets to do our analysis, we need to employ bit masking to narrow our BPF down to that bit or bits. The way we do this is to write out the entire byte and mask out the bits we don't want to see and retain the bits we do. To do this, we apply a zero to bits we want to filter and a one to bits we wish to retain. For example, to see only SYN-ACK packets:

C     E     U     A     P     R     S    F
 0     0     0      1     0      0     1    0

We have zeroed out all the bits except the ones corresponding to the ACK and SYN flag. The ACK flag is the least most significant bit in the first nibble (4 bits make a nibble and two nibbles make a byte) and the SYN flag is at the second least significant bit in the second nibble.
This would give us a binary mask of 0001 0010. Convert that to hex and we have 0x12. To apply this as a BPF filter, we do a bitwise AND  operation on the byte. Any 1 ANDed with another 1 equals one, and a 1 with a 0, OR if both are 0, equals zero.

We use the "&" operator to do the bitwise operation. In BPF filters, the byte number is appended in square brackets to the protocol header we're working with. So in this case, we're working with the TCP header and byte 13. So the first part of our filter is tcp[13].

Now we add the bit shifting. So the second part of our filter is: "& 0x12". And finally, we tell it what value to compare the result to. Since we are checking for SYN-ACK packets, we want the entire byte to equal 0x12, which would be the value if only the SYN and ACK flags were set. So our entire filter would be:

'tcp[13] & 0x12 = 18' or we could write it 'tcp[13] & 0x12 = 0x12' since 12 in hex equals 18 in decimal. Either way would work.

You should always put your BPF in single quotes (on Linux boxes) or double quotes using WinDump, as a good practice. Your BPF may not be complex enough to need it every time, but if you always do it, it won't bite you when you have a compound filter that requires it.

Using this method, we can now specify any bit or bits in any byte in any header. Very powerful tool.

If you're new to packet analysis, you may need to learn or do a refresher on hex and binary, and look at bitwise operations, but that, with a copy of your protocol header charts, is really all you need to accomplish this.

You can span multiple bytes, chain multiple filters together, mix primitives (built in filters, like tcp) and complex filters, and get as specific as you need. And you can save your very exotic filters into a file and have tcpdump read them in from the command line with the -F switch.