3. Getting Started

3.1. Installation
3.2. A Simple Example
3.3. Call Monitoring (Call Tracing)
3.4. Fault Simulation
3.5. Detecting Memory Leaks

This section shows how to install KEDR framework and how to use it to analyze a simple kernel module.

Warning

KEDR framework can do much harm if it gets out of control. USE IT ONLY IF YOU KNOW WHAT YOU ARE DOING. There is no warranty. If you use KEDR, you do so at your own risk.

Main components of KEDR framework operate in the kernel space. The system instruments the modules under analysis and allows custom kernel modules to alter the behaviour of these modules. This creates both a security hole and a potential for system instability, especially if the kernel modules under analysis are faulty.

It is not recommended to use KEDR on the machines holding important data or providing important services.

3.1. Installation

It is recommended to build KEDR from source. This allows to avoid a lot of issues concerning the differences between many versions and variants of the Linux kernel. For the present, there is no official binary distribution of KEDR.

The source code of the framework can be downloaded from the project site at BerliOS Developer.

To be able to build KEDR, you need the following:

  • Your Linux system should have kernel version 2.6.31 or newer. uname -r command should tell you what kernel version you are currently using. x86 and x86-64 architectures are currently supported.

  • CMake build system (http://cmake.org/) version 2.6 or newer (version 2.8 or newer is preferable)

  • GNU C and C++ compilers version 4.0 or newer

  • GNU Make

  • Other tools and packages necessary to build kernel modules: on some systems it can be kernel-*-devel, kernel-*-source, kernel-*-syms, linux-headers-* or other packages. On some systems you may also need to install DKMS package or something like that.

After all the prerequisites have been met, unpack the archive with the sources (kedr-0.2.1.tar.bz2) and create another directory, say, kedr-build where KEDR will be built.

Note

It is highly recommended to use an out-of-source build, i.e. not to build KEDR from the directory containing its sources. With an out-of-source build, you leave the source tree of KEDR unchanged, which can be convenient. The source tree can even be read-only in this case. You can also configure and build the framework from different build directories with different options if you want to. An in-source build would make this impossible.

Change current directory to kedr-build and configure the package using the following command:

cmake ../kedr-0.2.1/

During configuration phase, the information about the environment is analyzed and appropriate settings are prepared for KEDR to tune it properly to your system. This is done automatically.

By default, KEDR will be installed to /usr/local/. If you would like to install KEDR to some other location, configure the package as follows:

cmake -DCMAKE_INSTALL_PREFIX=<install_directory> <path-to-kedr-sources>

Example:

cmake -DCMAKE_INSTALL_PREFIX=/opt/kedr/ ../kedr-0.2.1/

The package will be configured to be installed to /opt/kedr/.

If the configuration stage completes successfully, you can type make to build the package and make install - to install it. You may need to execute make install as a root user.

To remove the files installed with make install, you can use make uninstall command.

Note

Currently, make uninstall does not remove directories, only files.

KEDR package also contains a set of tests for KEDR framework. You may want to run these tests after KEDR is built but before it is installed to see if the tools provided by the framework correctly operate on your system. To do so, just execute make check (as root user).

3.2. A Simple Example

Let us consider an example of how to analyze a simple kernel module with KEDR. We assume below that KEDR has been installed to /usr/local. Unless specifically stated, the control and helper tools mentioned below should be executed by a user with root privileges.

You can use sample_target module as a kernel module to be analyzed. It can be found among the examples installed with KEDR (see /usr/local/share/kedr/examples/). Copy the contents of sample_target directory to a place of your choice and run make there. You should get kedr_sample_target.ko file as a result. It is a kernel module we will use to demonstrate the abilities of KEDR.

Run the control script (as root):

/usr/local/bin/kedr start kedr_sample_target

This will start the core components of KEDR and instruct them to process the kernel module with the specified name (kedr_sample_target). Note that kedr_sample_target itself is not loading at this stage. But as soon as this module is loaded, KEDR core will detect it and will connect to it automatically.

Now load our module to be analyzed (target module). The easiest way to do this is to execute a helper script provided with that module:

./kedr_sample_target load

This should be done from the directory where the compiled kedr_sample_target module is located. The thing is that the module actually creates character devices. So, apart from loading the module itself, the helper script creates special files /dev/cfake0 and /dev/cfake1 that represent these character devices in your filesystem.

Now that KEDR core is up and running and the target module is loaded, we can perform the anaysis. What can actually be done depends on the additional steps in the process of loading KEDR and the target module. This will be described in the following sections (see Section 3.3, “Call Monitoring (Call Tracing)” and Section 3.4, “Fault Simulation”).

When you are done with KEDR, it can be stopped. Please unload the target module first. To do this, execute the helper script as follows:

./kedr_sample_target unload

Now stop KEDR (and actually unload its components):

/usr/local/bin/kedr stop

3.3. Call Monitoring (Call Tracing)

In this section, we will show how to use KEDR for call monitoring, that is, for gathering information about function calls made by the target module. This will be demonstrated on kedr_sample_target module.

Actually, the command

/usr/local/bin/kedr start kedr_sample_target

called without additional arguments also loads the necessary modules to perform call monitoring (in addition to loading the KEDR core).

Information about kernel function calls is temporarily stored in a file in debugfs filesystem. So, to see this information, you need to have this filesystem mounted. Usually, it mounted by default to /sys/kernel/debug. If it is not the case for your system, you can mount it manually by executing

mount debugfs -t debugfs /sys/kernel/debug

You can instruct KEDR to output the information about the calls to kernel functions. To do this, open another terminal and execute

/usr/local/bin/kedr_capture_trace

From this moment, all tracing information will be output into that terminal. Leave it for a time, and switch to the terminal, in which you have started KEDR.

Everything is now ready to load the target kernel module:

./kedr_sample_target load

Now you can do something with the character device created by the module. E.g., write zeroes to it:

dd if=/dev/zero of=/dev/cfake0 bs=1 count=10

Switch to the terminal, in which you have run kedr_capture_trace tool. You can find the records like the following ones there:

insmod-6416 [001] 805.997300: target_session_begins: 
    target_name: "kedr_sample_target", payload_name: "kedr_cm_cmm"
...
insmod-6416 [001] 805.997320: target_session_begins: 
    target_name: "kedr_sample_target", payload_name: "kedr_cm_vmm"
insmod-6416 [001] 805.997615: called___kmalloc: ([<ffffffffa00e70b9>] init+0xb9) 
    arguments: (320, d0), result: ffff8800165a8000
dd-6438     [000] 858.641942: called___kmalloc: ([<ffffffffa01d661e>] core+0x61e) 
    arguments: (4000, d0), result: ffff88001659e000
dd-6438     [000] 858.642074: called_copy_from_user: ([<ffffffffa01d642a>] core+0x42a) 
    arguments: (ffff88001659e000, 000000000137d000, 1), result: 0
...

The first record says that KEDR payload with name kedr_cm_cmm has detected the loading of the target module and has prepared to monitor function calls in it. The next several lines state the same but for other payloads, ending with kedr_cm_vmm.

Next (4th line in the listing above) line shows information about the first detected call to a kernel function, __kmalloc. It shows the values of the parameters passed to the function (size=320, flags=0xd0) and its return value (address 0xffff8800165a8000). The operation was performed in the context of insmod process (its PID was 6416).

Note

([<ffffffffa00e70b9>] init+0xb9) specifies the memory address of that call to __kmalloc (0xffffffffa00e70b9). To be exact, it is technically the address of the next instruction after that call. init+0xb9 indicates that the call instruction is located in init area of the module (in a function marked with __init in the source code) right before the offset 0xb9. If the target module has debug information, this allows to determine the place in the source code of the module where the call is made. This may significantly simplify the analysis of the trace. The detailed explanation of how to analyze the trace and find the fragments of the source code corresponding to the trace records is given in Section 4.7, “Analyzing the Trace”.

The fifth record shows another detected call to __kmalloc. It was made when we were writing zeroes to /dev/cfake0 (it follows from the fact that the call was made in the context of dd process that we had launched).

The sixth record shows a detected call to another kernel function, copy_from_user, which was also made when we were writing data to /dev/cfake0 with dd.

The remaining records in the trace are similar to those we have just described.

Unload the target module using the following command (from the terminal in which the target module was loaded):

./kedr_sample_target unload

After that, the lines like the following ones should appear in the trace:

rmmod-6441 [001] 869.438875: called_kfree: ([<ffffffffa01d60d8>] core+0xd8) 
    arguments: (ffff88001659e000)
rmmod-6441 [001] 869.438879: called_kfree: ([<ffffffffa01d60d8>] core+0xd8) 
    arguments: ((null))
rmmod-6441 [001] 869.438881: called_kfree: ([<ffffffffa01d6108>] core+0x108) 
    arguments: (ffff8800165a8000)
rmmod-6441 [001] 869.438885: target_session_ends: 
    target_name: "kedr_sample_target", payload_name: "kedr_cm_cmm"
...
rmmod-6441 [001] 869.438895: target_session_ends: 
    target_name: "kedr_sample_target", payload_name: "kedr_cm_vmm"

There will be no new records in the trace file until the target module is loaded again. You can stop capturing the trace by pressing Ctrl+C in the terminal where kedr_capture_trace runs.

The last step is stopping KEDR and unloading its components:

/usr/local/bin/kedr stop

3.4. Fault Simulation

This section shows how to use KEDR for fault simulation. This way, we can, for example, model a situation when the system has low resources or put the target module in some other conditions that relatively seldom take place.

In general, it can be useful to see how a kernel module behaves on the code paths it rarely executes. The errors in the corresponding parts of the target module can remain hidden for a long time. The ability of KEDR to put the target module in such rare conditions without affecting the rest of the system is demonstrated here. kedr_sample_target is used again as the target module.

KEDR provides a special configuration file for fault simulation. To use this configuration instead of the default one, pass additional option to the KEDR control script (fsim.conf is installed with KEDR, so it is not necessary to specify the full path to this file):

/usr/local/bin/kedr start kedr_sample_target -f fsim.conf

Now KEDR is loaded and prepared to simulate, e.g., the conditions when memory allocation (e.g., via __kmalloc) fails. Note that this simulation will be performed only with respect to the target module, other parts of the kernel will not be affected.

You can choose a scenario according to which KEDR will make calls to __kmalloc fail. A number of possible scenarios is already provided by KEDR and custom scenarios can also be developed. In this example, we will show how to configure and use a pre-defined scenario. To make it available, you should load the corresponding module:

insmod /usr/local/lib/modules/`uname -r`/misc/kedr_fsim_indicator_kmalloc.ko

This module implements a scenario named kmalloc. Inform KEDR core that it should use this scenario for __kmalloc function:

echo "kmalloc" > /sys/kernel/debug/kedr_fault_simulation/points/__kmalloc/current_indicator

We assume in the last listing that debugfs is mounted to /sys/kernel/debug.

kmalloc is actually a name of a whole set of scenarios, so choose one of them:

echo "1" > /sys/kernel/debug/kedr_fault_simulation/points/__kmalloc/expression

This meaning of this scenario is fail always. That is, the answer to the question whether to make a call fail or not is always 1 (yes) in this scenario.

As you have seen in the section about call monitoring, kedr_sample_target module calls __kmalloc during initialization. So we can predict that loading of the entire module should fail, because it will not be able to allocate memory for its own use. Let us check if it is the case.

./kedr_sample_target load
insmod: error inserting 'kedr_sample_target.ko': -1 Cannot allocate memory

As it was expected, the loading has failed.

Let us now consider a slightly more complex scenario.

echo "size > 2000" > /sys/kernel/debug/kedr_fault_simulation/points/__kmalloc/expression

This means that only the allocation requests for memory blocks bigger than 2000 bytes will fail.

./kedr_sample_target load

This time loading of the target module should succeed. But the attempts to write to it or read from it will fail because the target module needs to allocate a 4000-byte buffer the first time the device is opened. Try this:

dd if=/dev/zero of=/dev/cfake0 bs=1 count=10

You should see an error message similar to the following as a result:

dd: opening '/dev/cfake0': Cannot allocate memory

To turn off fault simulation, just set the scenario to 0 (that means, never make the calls fail):

echo "0" > /sys/kernel/debug/kedr_fault_simulation/points/__kmalloc/expression

When you are done with fault simulation, unload the scenario module:

rmmod kedr_fsim_indicator_kmalloc

The scenario for __kmalloc will be cleared automatically, that is no fault simulation will be performed for the function since that moment until a new scenario is loaded and configured.

Finally, you can unload the target module

./kedr_sample_target unload

and then stop KEDR

/usr/local/bin/kedr stop

3.5. Detecting Memory Leaks

To demonstrate how memory leaks can be detected with KEDR, let us intentionally create a memory leak in kedr_sample_target module. What is needed is just to comment out one or more calls to kfree(), for example, in cfake_destroy_device():

/* Destroy the device and free its buffer */
static void
cfake_destroy_device(struct cfake_dev *dev, int minor,
    struct class *class)
{
    BUG_ON(dev == NULL || class == NULL);
    device_destroy(class, MKDEV(cfake_major, minor));
    cdev_del(&dev->cdev);
    /* kfree(dev->data); */ /* Memory leak */
    return;
}

Now rebuild kedr_sample_target and we are all set.

Load KEDR with a special configuration profile (leak_check.conf):

/usr/local/bin/kedr start kedr_sample_target -f leak_check.conf

Then load the target module and work with it as usual, for example, do something with the character devices that it creates:

./kedr_sample_target load
dd if=/dev/zero of=/dev/cfake0 bs=1 count=10
echo 0123456789ABCDEF > /dev/cfake1

Unload the target module:

./kedr_sample_target unload

By this moment, KEDR should have prepared a report about memory leaks it has detected. It should be available in kedr_leak_check directory in debugfs.

The file named info presents the summary:

Target module: "kedr_sample_target", 
    init area at 0xffffffffa0730000, core area at 0xffffffffa072d000
Memory allocations: 3
Possible leaks: 2
Unallocated frees: 0

possible_leaks provides information about the detected memory leaks, namely, address and size for each memory block that was not freed and the call stack of the allocation:

Block at 0xffff880024285000, size: 4000; stack trace of the allocation:
[<ffffffffa072d35e>] cfake_open+0x6e/0xb8 [kedr_sample_target]
[<ffffffff81156e3a>] chrdev_open+0x10a/0x200
[<ffffffff81151295>] __dentry_open+0xe5/0x330
[<ffffffff811515f4>] nameidata_to_filp+0x54/0x70
[<ffffffff8115e358>] finish_open+0xe8/0x1d0
[<ffffffff8115f7b6>] do_last+0x86/0x460
[<ffffffff81161aeb>] do_filp_open+0x21b/0x660
[<ffffffff81151039>] do_sys_open+0x69/0x170
[<ffffffff81151180>] sys_open+0x20/0x30
[<ffffffff8100a0f2>] system_call_fastpath+0x16/0x1b
[<ffffffffffffffff>] 0xffffffffffffffff
+1 more allocation(s) with the same call stack.

The line we have commented out in the sources of the module was to be executed twice (once for each device), so we have got two memory leaks. The call stack shows where the memory blocks were allocated. You can use, for example, objdump tool to find out where in the source code of the module this call stack corresponds to.

According to the stack trace, those memory blocks were allocated by the calls made from cfake_open() function in the target module. The corresponding call instruction is just before the offset of 0x6e from the beginning of the function in the binary file of the module. Let us find that place with objdump (in fact, it is not really needed in this example as it is already obvious where in the source code are the necessary lines, but still). This will disassemble the target module and store the listing in module.disasm file:

objdump -dSlr kedr_sample_target.ko > module.disasm

Look for cfake_open() in the disassembled code and you will find a fragment like this:

00000000000002f0 <cfake_open>:
cfake_open():
/home/tester/temp/sample_target/cfake.c:57
static struct class *cfake_class = NULL;
/* ============== */

int 
cfake_open(struct inode *inode, struct file *filp)
{
 2f0:	55                   	push   %rbp
 2f1:	48 89 e5             	mov    %rsp,%rbp
 ...

As you can see, the offset of cfake_open() in the .text section of the binary file is 0x2f0. The call we look for should be before the offset 0x2f0+0x6e=0x35e, so scroll down to that offset:

 350:	48 8b 7b 08          	mov    0x8(%rbx),%rdi
 354:	be d0 80 00 00       	mov    $0x80d0,%esi
 359:	e8 00 00 00 00       	callq  35e <cfake_open+0x6e>
			35a: R_X86_64_PC32	__kmalloc-0x4
cfake_open():
/home/tester/temp/sample_target/cfake.c:85
	
	/* if opened the 1st time, allocate the buffer */
	if (dev->data == NULL)
	{
		dev->data = (unsigned char*)kzalloc(dev->buffer_size, GFP_KERNEL);
		if (dev->data == NULL)
 35e:	48 85 c0             	test   %rax,%rax

So the leaked memory blocks were allocated by the calls to kzalloc() at line 84 of cfake.c.

When you are done with the reports, you can stop KEDR as usual:

/usr/local/bin/kedr stop