IDS Detection

The most common way an IDS detects malicious traffic is through the use of signatures. A signature is a string that the IDS looks for in a packet and does a comparison for. If the string matches, the IDS alerts. The string can be a simple ASCII string, such as "cat /etc/passwd", or it can be a regular expression which allows much more granular control, or a binary match.
There are many ways to evade signature-based detection, using different forms of encoding, fragmentation, and other forms. IDS must be able to combat these methods to ensure what it sees will match what the packet ultimately looks like when it reaches the destination host.
Some IDS use pre-processors, modules that massage the data before running it through the rules base for efficiency. For example, if an attacker fragments his traffic with overlapping fragments (a fragment that overwrites part of a previous fragment) an IDS has to be able to reconstruct that packet the same way the host that receives it would. Some systems favor the original fragment that would have been overwritten, and some the fragment that overwrites the previous. Knowing which way your server will reconstruct this traffic is very important. Snort, for example, reassembles the packets both ways, since you may have Windows-based, Unix-based or other operating systems on the host. Others, like Dragon, force you to choose one method or the other. As long as all the systems on a segment being monitored are all on the same platform, this isn't a problem.
Signatures also specify other attributes, like which direction the packet is coming from, ports, protocols and more. Each one tunes the signature to be more specific, eliminating possible false positives.
The IDS engine does detection with built-in parameters that doesn't require a separate signature. For example, there may be an ICMP module in your IDS that alerts on packets over a certain size, or a certain number of packets in a time frame all going to one host to detect a ping flood. It may look for scanning by keeping track of the number of different hosts an external address tries to connect to one one port, or the number of ports on one host it tries to connect to (port sweeps and port scans).
Stateful inspection is important here. An IDS needs to be able to keep track of state to determine if packets are part of an existing connection or a new connection. Shell code being bounced off a server without a connection ever being made might cause the analyst to take notice and investigate what else the attacker might be doing, whereas shell code being passed on an established TCP connection would be cause to take action immediately.
Anomaly-based detection uses profiles to try and establish a baseline of what is "normal" traffic, and alerts you when traffic not fitting that profile occurs. Let's say you have a Web server that is primarily used by a few customers in the same country as you, in the morning hours Monday through Friday. The rest of the time there is little to no traffic to the box. Then one Saturday morning at 3:00 AM, the box is inundated with traffic from another country. There may be nothing in that traffic that triggers a signature or the IDS engine, but the deviation from the norm is curious at the least and probably alarming. Anomaly detection tries to determine events of this type.
This is obviously a very simplistic overview. There is much more that could be said for each type of detection, and other types like heuristics, protocol-decoding, etc., that you can read about if you'd like to learn more.