HookFinder: Identifying and Understanding Malware Hooking Behaviors

Installing various hooks into the victim system is an important attacking strategy used by malware, including spyware, rootkits, stealth backdoors, and others. In order to evade detection, malware writers are exploring new hooking mechanisms. For example, a stealth kernel backdoor, deepdoor, has been demonstrated to successfully evade all existing hook detectors. Unfortunately, the state of the art of malware analysis is painstaking, mostly manual and error-prone. In this paper, we propose the first systematic approach to automatically identifying hooks and extracting the hook implanting mechanisms. We propose fine-grained impact analysis, as a unified approach to identify hooking behaviors of malicious code. Since it does not rely on any prior knowledge of hooking mechanisms,it can identify novel hooks. Moreover, we devise a semantics-aware impact dependency analysis method to provide a succinct and intuitive graph representation to illustrate the hooking mechanisms. We have developed a prototype, HookFinder, and conducted extensive experiments using representative malware samples from various categories. The experimental results demonstrated that HookFinder correctly identified the hooking behaviors for all the samples, and provided accurate insights about their hooking mechanisms.