Network scanning, OS fingerprinting part 3

Distinguishing OS detection

If you have read and understood all the lines in the first two blog posts, you can congratulate yourself. You now understand how Nmap performs OS detection, which packets are sent, what are the expected responses to those probes, and how Nmap uses the responses to calculate OS fingerprint and finds the best match in its database.

If you have previously investigated network traffic, you should be able to distinguish between standard, common packets, corrupted ones and nonstandard, unique or rare ones. In this case you could realize that some packets that Nmap sends are quite uncommon or even rare. Let's say that we have discussed 15 core packets that Nmap sends. For example, how often have you seen UPD packet with character 'C' (0x43) repeated 300 times? Or TCP packet with SYN, FIN, URG, PSH flags set? Not very often, right?

We can use our knowledge to create a NGFW, IPS rule, policy or search in network monitoring systems to warn us about Nmap OS scanning. The rule could monitor for the presence of those 15 core packets previously described in (1) that Nmap uses for OS fingerprinting. If we see them within a five-minute period from the same source IP to one destination IP, we can definitively say this destination IP was scanned, and one of the possible responses could be to block the source IP address.

But there is always a mitigating circumstance. The first argument could be that correlating 15 packets together with so many variables is very difficult and therefore, poses a high risk of error. All the mistakes should be discovered during test phase of a rule creation, but the search or test conditions may become quite long, complicated and even confusing. The second argument could be that the correlation of all those 15 packets together with exactly defined variable values can lead to a situation in which  the attacker won't be detected at all even when he performs  a slight change to one of the 15 packets.

As an example and to simplify this, we will only take into account UDP packet with 300 'C' characters described earlier. We can have QRadar or Snort rule like this one:

QRadar

Apply Nmap_OS_scan_UDP_with_C_characters on flows which are detected by the system

and when the IP protocol is one of the following UDP
and when the destination payload matches the regex CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC

Snort

alert udp $EXTERNAL_NET any -> $HOME_NET any

(

msg: "Nmap OS scan UDP with 300C detected";

content:"CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC";

sid:1000001;

)

As you can see, having this rule in QRadar is quite inefficient and will likely have considerable performance impact. Also, when using flow data, we are limited with the payload length we can see. For this reason, we have in our rule only 64 'C' characters and not 300. To see the whole payload, we can use full packet capture, which will have additional performance impact. So, it makes sense to move the detection to NGFW or IPS tools like Snort.

If  an attacker changed the number of “C” characters in the UDP packet from 300 to 200, our rule waiting for 300 wouldn't fire. The same applies for cases when you change 'C' character to 'A'. In that case not even the QRadar rule will fire.

The question should be, can I change some Nmap parameters? Yes, you can. Even though it is not seen very often, the Nmap source code is publicly available, and once you understand it, you can adapt the specific file to your needs. What’s more, this UDP packet Nmap only monitors if it is returned the same way as it was sent with the proper packet length. If there was a change in a letter, Nmap still evaluates the response in the same way without any change. If you change the number of letters, then you also need to change the value for expected packet length in response and you are still OK.

So, we can approach it a little bit differently. Monitor only for the most unique packet(s) and for those values that cannot be changed, or their change will require huge intervention in Nmap code and logic of operation. One of my favorite packets to monitor is the ECN one (reserved bit, SYN, ECN, CWR flags, urgent field value of 0xF7F5, window size field=3, TCP options: Window Scale=10; NOP; MSS=1460; SACK permitted; NOP; NOP). In the WireShark display filter example in Figure 3 at part 1, it will filter out all the non-interesting packets. As you can see, we don't need to specify every single parameter (for example, windows size field omitted) in the WireShark and we have still quite good level of certainty that other "standard" packets won't be displayed.

WireShark display filter

(tcp.flags.syn==1 && tcp.flags.cwr==1 && tcp.flags.ecn==1) && (tcp.urgent_pointer == 63477) && (tcp.options.wscale.shift == 10) && (tcp.options.mss == 02:04:05:b4) && (tcp.options.nop == 01)

Personally, I find WireShark a great network investigation tool, but in this case, we won't be using it. We would have traffic dumps (.pcap file) from our environment, and manual investigation of so many files would be very ineffective. I suggest you try it during rule development process. Perform Nmap OS scan on one of your test systems. Dump the traffic, search for the Nmap specific packets and create rules based on the systems and network devices you are using.

But let's assume we have our WireShark search and we want to make a similar rule on our Snort IPS. We will face a challenge how to write a rule that will monitor TCP options and urgent pointer because Snort does not support these options out of the box. We can still use a rule to monitor for the flags, but we can expect a match even when there is no OS scan, because the condition is too generic. You can try it and you will see how common ECN and CWR flags are in your environment. You can use this simple rule and create better and more rules to identify Nmap OS scan:

alert tcp any any -> any any (flags:SCE;)

There is another problem with monitoring scanning attempts on the perimeter or IPS only. Not all devices are protected by IPS.  If an insider is trying to scan your internal network, the communication probably won't flow through a perimeter device. You could monitor endpoints directly. Generally, this is a good idea, but it depends heavily on the endpoint detection possibilities. As an example, we can describe the nature of the problem on  the iptables example in Figure x.

iptables

iptables -I INPUT -p tcp --tcp-flags SYN,RST,ACK,FIN SYN --tcp-option 4 -A LOGGING -j DROP

Here we are monitoring for the presence of SYN flag and TCP option 4, which is Selective ACK. Unfortunately, most of the iptables implementations don’t recognize ECN and CWR flags. That is why we can only focus on SYN out of the all known flags (SYN, RST, ACK, FIN). Also, we can specify only one TCP option.

Here, we have decided to monitor only for Selective ACK, since it is  one of the least used options. Urgent pointer is not usually supported by iptables. That leaves us with a general rule on which we can’t fully rely. I am not saying we can't create useful rules for iptables. It’s just not that easy and straightforward and it requires some development and testing.

Another idea could be to use a SIEM tool to detect Nmap OS scanning and create respective rules. But SIEM tools usually don't go into such packet detail. Even if they can, performance impact will be enormous and applicability throughout whole environment is questionable. We won't be far from the truth to say that there is no effective SIEM detection. Other flow-oriented tools fail when trying to monitor for TCP options and urgent pointer because they are not part of the flow standard.

There is no need to give up yet. Look at the YARA rule in Figure x. In the flags_window parameter, we have hexadecimally encoded header length – offset with reserved bit (88) SYN, CWR, ECN flags (C2) and window size field (00 03). In the pointer_options parameter we can see urgent pointer value (F7 F5) followed by the tcp options (03 03 0A 01 02 04 05 B4 04 02 01 01). Be careful with the order of TCP options, it can differ.. If we follow the condition, we are monitoring for a packet or traffic dump where both parameters are present.

YARA

rule NmapOSdetected

{

strings:

$flags_window = { 88 C2 00 03 }

$pointer_options = { F7 F5 03 03 0A 01 02 04 05 B4 04 02 01 01 }

condition:

$flags_window and $pointer_options

}

You can apply this rule on the endpoints. If you are creating traffic dumps, you can automate the detection by periodically applying the YARA rule on the newly created traffic dumps. Many modern network devices understand and can work with the YARA rules. Be careful with the network devices since some are applying the YARA rules only on the payload and not the protocol headers. If that’s the case, you still need to come up with a solution that fits your environment and tools.

As you can see, there is no silver bullet. It always depends on your tools, their capabilities and where you are able to effectively monitor. But I hope this  series helps you to understand what OS detection is, how it is carried out in terms of Nmap, and what kind of design thinking has to be done to create detection rules.

Now it only depends on you if you choose to rely on single packet or correlate multiple packets together. The solution will differ based on the technology you have available or you are using within your environment. Feel free to come up with your own innovative solution to approach this task. As usual, especially in cyber security, there is no single right answer.