DOC -- Detector of Obfuscated Calls

Introduction

DOC is a static analysis suite that detects obfuscations in executable, particularly procedure call and call-return obfuscations. It uses abstract interpretation (AI) to find instances where explicit call or call-return instructions are not used. A prototype is implemented as an Eclipse plugin for browsing X86 executables.

How it Works

Embedded within every executable program are tell-tale signs of the program's intent located in a table of system calls that are needed by the program. This list of system calls can be used to determine a program's behavior prior to executing the program. Consider a program that makes use of the system calls bind, send, and closesocket. It is quite clear that data transfer over a network will take place if this program is run. Malicious code writers, being wary of such transparency in their code, defend their programs' privacy by obfuscating the table of required system calls, thus obfuscating the program's true behavior. A common approach to this form of obfuscation is known as "call obfuscation."

In call obfuscation, the detection of used system calls is made difficult by replacing the existing call and ret instructions with a different, but equivalent, set of instructions. For instance, the two sequences of code shown below are equivalent; they both call the method DeleteFileA. However, the second sequence contains no call statement, making it difficult to know a call even exists at this program point. This rather simple trick is enough to confound many analysis tools.

Our tool, DOC (Detector of Obfuscated Calls), is capable of statically detecting such code obfuscation by interpreting the executable and building an abstract representation of the registers and stack, which allows us to detect pieces of code that violate standard calling conventions. The result is a more complete listing of system calls used by the program. Using this improved list of system calls, one is in a better position to accurately gauge a program's runtime behavior, a crucial first step in detecting malicious behavior.

Eclipse interface for DOC

In the screenshot of DOC on the right, the results of interpreting an executable are displayed. The in the left column marks an instruction that is a valid call statement, meaning it is not obfuscated. The marks instructions that have been found to be obfuscated. Manually tracing the code reveals that the instructions marked with do indeed simulate a call.

The bottom pane shows a list of detected obfuscated calls. The user can also choose to view all valid call-return sites, obfuscated returns, or associated instructions. The "Associated Instructions" view can be used to answer such questions as "where does this call statement transfer control to?"

The panes on the right display the internal program values at the selected instruction, which is instruction 401014 in the screenshot. Here, we can see the values located in each register plus a representation of the runtime stack. The user can scroll through the instructions to see the internal values at other locations in the program.