Tracing for Hardware, Driver and Binary Reverse

Engineering in Linux
Mathieu Desnoyers Michel R. Dagenais
École Polytechnique de Montréal École Polytechnique de Montréal
[email protected] [email protected]

Abstract and LTTV for reverse engineering and how you

can extend it further.

This paper introduces the new Linux Trace

Toolkit Next Generation (LTTng) kernel tracer
and its analysis counterpart, Linux Trace
Toolkit Viewer (LTTV), a fully extensible text
and graphical trace viewer. It will focus on how
these tools can be used in the security field, par-
1 Introduction
ticularly for reverse engineering.

Using a tracer to reverse engineer a software

"black box" can help understanding its behav- Collecting information about the behavior of a
ior. Such a software can be a either a driver, a li- computer program greatly helps understanding
brary or a multithreaded application : the tracer its inner structure. In the exercice of reverse
can log every interaction between the operating engineering, it can save a lot of time and ef-
system and the program. It can help eluding de- fort. However, the information gathering tools
tection of sandboxes and debuggers due to its themselves have a non-zero impact on the stud-
small performance impact compared to library ied system (Heisenberg effect), which can be
wrappers and debuggers. It can collect every used by a program that wishes to elude debug-
system call made by every program which can gers and tracers.
be later used for fuzzing.

It is not, however, limited to process examina- This paper presents some widely used profiling
tion : one could use the kernel instrumentation and event-based performance monitoring tools.
to reverse engineer a driver controlling a piece It then focuses on the latter class of tools by
of hardware. This tracer should be seen as a presenting Linux Trace Toolkit Next Genera-
system wide monitor for your system : It gives tion (LTTng), a kernel tracer in the Linux en-
you the opportunity to monitor the hardware, vironment. It will present the particularities
the OS, the libraries and the programs and anal- of this tracer relevant to the computer security
yse the information with integrated plugins. field. It will present real-life debugger detectors
examples and describe how LTTng can avoid
This paper will explain how you can use LTTng being detected by such techniques.

1
2 Performance and Behavior Anal- trace (LTT, LKST, SystemTAP/LKET) or as a
ysis summary (SystemTAP).

Those tools, in addition to be useful to un-

derstand performance issues, give the ability
To gain a thorough understanding of a pro-
to track a program’s behavior through a trace
gram’s internals, many approaches are cur-
of its execution. Some programs produce sys-
rently available. They mostly vary in the level
tem wide traces (LTT, LKST, LKET) which are
of detail they give, the intrusiveness of their
useful to understand multithreaded and multi-
approach (how much impact they have on the
processes applications. Other tools, like strace
running program), whether they can directly in-
and ltrace, gives a detailed list of each inter-
strument a binary object or they require the user
action a program has done with, respectively,
to recompile an instrumented version of the ap-
the operating system and libraries. Those tools,
plication. In order to fulfill their goal, some
in order to provide a detailed list of the pro-
tools will use a partially statistical approach to
gram’s action, use various mechanisms to in-
lessen their impact while others will accept the
tercept the program’s execution, making their
overall system slowdown generated by a mas-
approach more intrusive.
sive data output.
A second, well known class of behavior anal-
The profiling tools gprof, prof and tprof are ysis tool, are the debuggers. Gdb is an exam-
based on static instrumentation of the appli- ple of user space application debugger, kgdb
cations (require recompiling). They count the is a kernel debugger. This approach gives full
number of entry and exit for each function and access to the memory of a running program
use the interrupt timer to sample the ratio of and permits hardware assisted breakpoints and
time spent in each code region. They have a watchpoints to stop execution at precise loca-
relatively small impact on the running program tions or when a particular memory access is
and are good to create a list of the functions us- made. They have, however, the highest impact
ing the most CPU time within a program. Opro- on the timings of a running programs and give
file is a system-wide statistical profiler that per- a detailed view of a particular moment in the
mits profiling the Linux kernel, libraries and program execution, which is less appropriate to
programs. follow the sequential behavior of a program.

System wide resource usage summary tools

(time, top, nmon) use data gathered by the op-
erating system each time a scheduler call oc- 3 LTTng/LTTV Linux Trace
curs. They are good to give a live overview of Toolkit
the system’s resource usage or construct graphs
with a minimal impact on the system. Event-based performance monitoring tools are
called “tracers”. The following section uses
System wide performance analysis can also be two concepts related to this category exten-
done by using event-based performance moni- sively : an “event” can be defined as an infor-
toring tools like LTT[4], SystemTAP[2], LKST mation record with a time stamp. Many events
and LKET. Those tools are based on instru- gathered together are called a “trace”.
mentation of the operating system (static or dy-
namic) which extracts useful information about Linux Trace Toolkit Next Generation (LTTng)
the kernel and user space programs either to a is a tracer able to extract information from the

2
Linux kernel, user space libraries and from pro- about the program using the kernel instrumen-
grams. It is based on instrumentation of the ex- tation and monitoring the program intercation
ecutables, which enables the user to get infor- with the operating system by recording system
mation from a recompiled Linux kernel and to calls, signals, interprocess communication and
also get detailed information about a program other key events. It will give useful informa-
by instrumenting it. tion about how the program uses the filesystem,
the network, its memory maps ; everything that
Linux Trace Toolkit Viewer (LTTV) is an in- passes through the operating system. It is also
tegrated modular analysis tool, which offers a possible to use breakpoint based approaches,
GUI framework and text output. Contributors such a kprobes and SystemTAP, to instrument
can create their own view of the information by the binary executables at key points to call a
creating plugins. The analysis tool can com- LTTng probe to write relevant information in
bine information coming from multiple infor- the trace.
mation sources, such as the kernel, user space
programs, user space libraries, multiple kernels This type of instrumentation is especially use-
running in a virtualized environment and per- ful when trying to understand multi-threaded
form efficient merging of this information for and multi-process applications. For instance,
the analysis plugins. following the interaction between the X server
and a graphical client often involves gather-
This project focuses on having a very low im-
ing information from various sources, which is
pact on the system studied, both in term of per-
made easier by a system-wide tracer.
formance and system behavior.
Because of its small impact on the system, this
kind of tracer can also be used to study appli-
4 Application in Reverse Engineer- cations which elude debuggers. This topic will
ing be treated in more details in the following sec-
tions.

The LTTng tracer and its analysis tool can be

useful in various reverse engineering applica- 4.1 User Space Programs
tions. To list a few, reverse engineering pro-
grams, libraries, drivers, network stacks or the
whole operating system can be made easier 4.1.1 Anti-debugging techniques
with this tool.

Different usage scenarios are possible, depend- In Linux, both process tracers (strace, ltrace)
ing on the type of executable that has to be anal- and debuggers (gdb) use the “ptrace” mecha-
ysed. The first one is the study of a program as nism extensively. It permits them to connect to
a white box : this is the case when the source a process to read and modify its memory, in-
code is available. A programmer can then mod- tercept its signals and system calls. Gdb typi-
ify the source code and recompile to extract in- cally uses these abilities to insert breakpoint in-
formation at key points. structions instead of the normal program code
at specific locations. It also enables the debug-
The second scenario is the study of an exe- ger to use hardware watchpoints to see when a
cutable as a black box, when only the binary ex- particular memory address is accessed or mod-
ecutable is available. It is still possible to learn ified.

3
Silvio Cesare presented a technique[1] to de- each detection technique under low and heavy
tect the use of ptrace on a process based on loads. It results that the number of cycles spent
the fact that a particular process can only be in each operation doubles. We can therefore
ptraced once. Therefore, a program that does consider a slowdown of 4 for the noise level
a ptrace system call on itself will detect if there (including a security margin of 2).
is a debugger or a process tracer attached on it.
The impact of gdb on the trap and kill depends
Going further, I decided to demonstrate another on the user input, while it does not impact the
approach to detect debuggers that doesn’t im- system call because it is not monitored. The
ply using the ptrace system call. The idea is impact of strace is a slowdown of 2.42 for the
simple : using a precise time base (the CPU cy- trap, 96 for the kill and 417 for the system call.
cle counter) to detect the duration of a particu- Due do the imprecision of the calibration and
lar code section can detect if a ptrace process is the incoming events (interrupts) which depends
collecting information about the program. This on the system load, the trap technique could not
approach is possible due to the high impact the determine the presence of strace due to its small
round-trip to the debugger process has on the impact. The ltrace slowdown is 11 for the trap,
system compared to the case when it is not 217 for kill and 554 for the system call.
being debugged. The different scenarios have
been tested on a 3GHz Pentium 4. They are The LTTng impact on the operations is a slow-
summarized in table 1. down of 1.77 for the trap, 2.40 for kill and 2.42
for the system call. Those differences do not
The duration of events such as a trap (gener- offer a sufficient margin to determine the pres-
ating a SIGTRAP signal) can detect the pres- ence of LTTng.
ence of a debugger. To detect the presence of
strace and ltrace, the duration of a signal, gen-
erated by a kill() system call on the process it- 4.1.2 Virus
self can be used to determine if it is being run
under strace which intercepts the signals. An An example executable which uses such a de-
alternative method to discover strace is to call bugging technique is the Virus Linux.RST.B.
an unexisting system call and see how much Its advisory from Kaspersky Labs[3] states
time it takes. To be stealthier, one could call a that it uses anti-debugging techniques, infects
do_gettimeofday system call, which is fast and Linux binaries found in the current directory
used everywhere. It would lead to results simi- and in /bin. It also states that it adds a back-
lar to an unexisting system call without trigger- door on the network interface.
ing alarm bells.
In order to study its behavior, I created a virtual
Clearly, these methods must be called in loops machine in Qemu and installed a Linux kernel
to make sure they are not too much affected instrumented with the LTTng tracer in it. I took
by interrupts. They are also dependant on the a trace on the machine and launched the virus,
speed of the processor and the architecture on recording all its actions on the operating sys-
which the binary is executed, but with proper tem.
calibration, they can be effective and hard to
discover. The result is that I could follow a trace of its
execution, which would have been impossible
To verify how much the system load can mod- with strace due to the debugger detecing tech-
ify the number of cycles per operation, we run niques. The execution trace, in the LTTV tool,

4
 normal (min) high load (max) gdb strace ltrace LTTng
trap 35108 63097 790211960 85148 405713 62152
kill 13042 28927 521129438 1257112 2832143 31417
system call 1890 3082 1920 789653 1048102 4582
Table 1: Number of cycles required for calls to the operating system (10 tests)

shows that it uses the “ptrace” method to detect nel privileges and interact directly with the ker-
debuggers 1. nel, wich makes them harder to trace.

Following the program execution, we can The LTTng kernel tracer can be used with
determine that it executes a temporary file kprobes to add breakpoints on each exported
".para.tmp" which creates three other pro- symbol of the binary-only driver (also static
cesses. It opens and lists the current direc- symbols taken by kallsyms if debugging infor-
tory, the /root directory, and then modifies the mation is present, which is unlikely). Break-
binary files in /bin. The advisory did not re- points can also be put at each kernel site that are
port the activity on /root, so probably a kernel called by the module : each undefined symbol
tracer would have given this extra information. referred by the kernel module is a candidate for
A secondary thread then binds a socket on the instrumentation. The kernel stack dump is an-
network interface and listens for incoming con- other part of LTTng that can help understanding
nexions. the drivers better : at specified locations in the
kernel, we can obtain the complete call stack of
the program (just like a kernel OOPS). It will
4.1.3 Password snooping
be recorded in the trace with the rest of the in-
formation.
Being able to extract any kind of information
about the system can be very powerful when LTTng can also be used for system call
used appropriately. For instance, having the fuzzing : it can be used to extract the normal pa-
ability to dump the first bytes of each read() and rameters passed to the kernel by standard pro-
write() system calls enables us to dump each grams in the first run. Then, this information
character received and sent by each program. can be feeded to a fuzzer that can test the sys-
Clearly, this includes “su” and “ssh”, which tem call error handlers by slightly modifying
makes extraction of the password typed by the these parameters.
users very easy. After a trace is taken, one has
to find the interesting process (su), select the
interesting information by using the LTTV fil- 4.3 Hardware
ter (Figure 2) and see the password typed (Fig-
ure 3).
Very precise information can be extracted from
the kernel about the hardware connected to
4.2 Kernel it. The instrumentation of interrupt handlers
makes the frequency of interrupts available. In-
Kernel reverse engineering is particularly use- formation about their maximum, minimum and
ful for binary-only drivers. They run with ker- average duration is also given.

5
 Figure 1: Virus anti-debugging

We can also think of polling the device for its the GPL license at http://ltt.polymtl.
state by doing a memory read of its registers ca.
either periodically (hooked on the timer inter-
rupt) or when an IRQ from the device is re-
ceived.
References
[1] Silvio Cesare. Fooling the debugger. In
4.4 Conclusion
VX Heavens, http://www.vx.
netlux.org/lib/vsc04.html,
Overall, in the land of performance and be- 1999.
havior analysis tools, tracers offer precious in- [2] Vara Prasad, William Cohen, Frank Ch.
formation about the execution of complex pro- Eigler, Martin Hunt, Jim Keniston, and
grams. The LTTng tracer and LTTV, the anal- Brad Chen. Locating system problems
ysis tool, provide an efficient, extensible and using dynamic instrumentation. In OLS
low impact system-wide tracing, which bene- (Ottawa Linux Symposium) 2005
fits reverse engineers for various usage scenar- http://sourceware.org/
ios, from tracing of binary-only executables to systemtap/systemtap-ols.pdf,
drivers. 2005.

You are welcome to try the tool and post your [3] Costin Raiu. Virus.linux.rst.b. In Virus
enhancements. The project is available under Encyclopedia, http://www.

6
 Figure 2: LTTV filter expression to find a password in a trace

viruslist.com/en/viruses/
encyclopedia?virusid=21734,
2002.

[4] Karim Yaghmour and Michel R. Dagenais.

The linux trace toolkit. May 2000.

7
Figure 3: Expected password in a trace coming from su