In this post, I'll describe how I build a simple kprobe-based tracer for the 2nd assignment of the Linux Kernel Labs course.

Assignment

Link

We want to build a kernel module named tracer that'll be a kernel operations surveillant. With this surveillant, we aim to intercept calls from processes:

• kmalloc and kfree calls

• schedule calls

• up and down_interruptible calls

• mutex_lock and mutex_unlock calls

The surveillant will hold, at the process level, the number of calls for each of the above functions.

Some key points:

• for the kmalloc and kfree calls, the total quantity of allocated and deallocated memory will be shown

• we'll use __kmalloc because kmalloc is inline

• kretprobe will be used for the instrumentation

• we'll use the miscdevice interface instead of cdev to define and manage the device

• /proc/tracer will display the information (stats), and we will interact with the driver through /dev/tracer.

You can look at the link above to get more details about the assignment.

Step-by-Step

In the following section, we'll go step by step through the implementation of our instrumentation module.

NB: We'll first probe all functions except kmalloc and kfree, because those two require some additional work

Before starting, here is the content of the tracer.h file already available for the assignment

/*
 * SO2 kprobe-based tracer header file
 *
 * This is shared with user space.
 */
#ifndef TRACER_H__
#define TRACER_H__ 1
#include 
#ifndef KERNEL
#include 
#endif /* KERNEL */
#define TRACER_DEV_MINOR 42
#define TRACER_DEV_NAME "tracer"
#define TRACER_ADD_PROCESS	_IOW(_IOC_WRITE, 42, pid_t)
#define TRACER_REMOVE_PROCESS	_IOW(_IOC_WRITE, 43, pid_t)
#endif /* TRACER_H_ */

In the ioctl section, we'll explain what _IOW, TRACER_ADD_PROCESS, and TRACER_REMOVE_PROCESS are for.

Module creation

As we did in the post on kernel module, let's set up our module:

tracer.c

#include <linux/init.h>
#include <linux/kernel.h>
#include <linux/module.h>

#include "tracer.h"

MODULE_DESCRIPTION("Kernel operations surveillant: kprobe-based tracer");
MODULE_AUTHOR("Tawaliou ALAO <alaotawaliou@gmail.com>");
MODULE_LICENSE("GPL");

static int tracer_init(void)
{
	pr_info("Misc device registered! Look at /dev/%s\n",
		TRACER_DEV_NAME);
	return 0;
}

static void tracer_exit(void)
{
	pr_info("Misc device %s unregistered!\n", TRACER_DEV_NAME);
}

module_init(tracer_init);
module_exit(tracer_exit);

Driver interface definition with miscdevice and file_operations

Registration

Now, let's define our driver interface, which will allow us to register our driver and create the /dev/tracer entry. In the last post on Character device driver - Part 1, to define our driver interface, we used to:

• set the major and minor numbers,

• initialize and add an instance of struct cdev to the kernel,

• create an entry for the driver in /dev manually with mknod, or programmatically with device_create,

• then define the file operations,

And I said that it's too long, right? Yes, for this assignment, we'll use struct miscdevice instead (Miscellaneous Character Device 😉). miscdevice will definitely simplify driver creation and save us from all the steps listed above. In short, it'll do all of these steps for us in the background.

Here is the link to the definition of miscdevice

struct miscdevice {
	int minor;
	const char *name;
	const struct file_operations *fops;
	struct list_head list;
	struct device *parent;
	struct device *this_device;
	const struct attribute_group **groups;
	const char *nodename;
	umode_t mode;
};

So let's define our driver:

tracer.c

#include <linux/miscdevice.h>

static struct miscdevice tracer_miscdevice = { 
	.minor = TRACER_DEV_MINOR,
	.name = TRACER_DEV_NAME,
};

I don't think I need to provide any additional explanation 😁.

Then, we'll register it with misc_register

int misc_register(struct miscdevice *misc)

In our init function, we'll register it

static int tracer_init(void)
{
int ret;
	ret = misc_register(&tracer_miscdevice);
	if (ret < 0) {
		pr_err("cannot register miscdev on minor=%d name=%s (err=%d)\n",
		       TRACER_DEV_MINOR, tracer_miscdevice.name, ret);
		return ret;
	}

	pr_info("Misc device registered! Look at /dev/%s\n",
		tracer_miscdevice.name);

	return 0;
}

Don't forget to check the return value of misc_register because the registration can fail.

You can notice that we don't set the major number with miscdevice. We don't need to do that because miscdevice has a default major number, MISC_MAJOR, which is set to 10 and defined in major.h

#define MISC_MAJOR 10

Unregistration

We unregister the driver with:

void misc_deregister(struct miscdevice *misc);

in our exit function.

static void tracer_exit(void)
{
	misc_deregister(&tracer_miscdevice);
	pr_info("Misc device %s unregistered!\n", tracer_miscdevice.name);
}

File operations

For the file operations, we'll just implement the ioctl operations. So, let's define an instance of file_operations

static const struct file_operations tracer_fops = { 
	.owner = THIS_MODULE,
	.unlocked_ioctl = tracer_ioctl,
};

Before moving on to ioctl, let's set tracer_fops as the file-operations handler for tracer_miscdevice

static struct miscdevice tracer_miscdevice = { 
	...
	.fops = &tracer_fops,
};

Now, we'll define the tracer_ioctl function.

So, what is IOCTL?

IOCTL, short for Input/Output Control, allows us to perform custom actions with the driver. You know, with the open, read, and write handlers of file_operations, we can only perform a limited set of actions: we can either read data sent from user space to the driver and do something with it, or write data back to user space. Now, imagine we want to do something like:

• check whether some data exists

• remove an entry from the driver's list if a flag is set

So, ioctl will help our driver do more than just read or write; it will let us control what the driver can do and how it does it.

In fact, what will happen is that we'll define the ioctl commands (I'll show how to do this in the following lines), and a user-space application will send these commands to the driver with the appropriate arguments.

Our driver will handle two commands:

• TRACER_ADD_PROCESS: to add the pid of a process to trace

• TRACER_REMOVE_PROCESS: to remove a process from the tracing list

IOCTL command

Many macros exist to define an IOCTL command; you can read about them here.

To define read and write commands, we can use the following macros: _IOR and _IOW.

// For read operations
#define _IOR(type,nr,size)	_IOC(_IOC_READ,(type),(nr),(_IOC_TYPECHECK(size)))

// For write operations
#define _IOW(type,nr,size)	_IOC(_IOC_WRITE,(type),(nr),(_IOC_TYPECHECK(size)))

where:

• the first parameter (type) is an arbitrary 8-bit character unique to the driver, for example, 'd' or 't',...

• the second parameter (nr) is an incrementing number that uniquely identifies the command

• the last parameter (size) is the "type" of the argument passed to the command, such as int or struct my_data

So, in our case:

#define MAGIC_BYTE 't' 

#define TRACER_ADD_PROCESS _IOW(MAGIC_BYTE, 42, pid_t)
#define TRACER_REMOVE_PROCESS _IOW(MAGIC_BYTE, 43, pid_t)

Inside tracer_ioctl, which we set as the unlocked_ioctl handler of the file_operations instance above:

static long tracer_ioctl(struct file *file, unsigned int cmd, unsigned long arg)
{
	switch (cmd) {
	case TRACER_ADD_PROCESS:
		tracer_add_proc(arg);
		break;

	case TRACER_REMOVE_PROCESS:
		tracer_remove_proc(arg);
		break;
	default:
		return -ENOTTY;
	}

	return 0;
}

where:

• cmd: the command sent from user space (TRACER_ADD_PROCESS and TRACER_REMOVE_PROCESS)

• arg: the value passed with the command (here, it's the process's pid); it could be more complex data.

So, based on the value of cmd, we know which operations to handle:

TRACER_ADD_PROCESS		--> tracer_add_proc
TRACER_REMOVE_PROCESS	--> tracer_remove_proc

In the next section, after defining our driver data, we'll implement the two handlers for ioctl

Driver data and ioctl operations

• struct tracer_stats

• static struct list_head tracer_head;

We'll store statistics for each process in a structure, and a linked list will hold all added processes.

struct tracer_stats

struct tracer_stats {
	pid_t tr_pid;		// process pid added
	int tr_alloc;		// number of calls to kmalloc
	int tr_free;  		// number of calls to kfree
	int tr_mem;			// memory allocated with kmalloc
	int tr_mem_free;	// memory freed with kfree
	int tr_sched;		// number of calls to schedule
	int tr_up;			// number of calls to up
	int tr_down;		// number of calls to down_interruptible
	int tr_lock;		// number of calls to mutex_lock
	int tr_unlock;		// number of calls to mutex_unlock

	struct list_head tr_pid_list; // list node reference
};

Most of the struct tracer_stats fields, except tr_pid, tr_mem, and tr_mem_free, will be incremented each time the traced function is called.

Then, we define our linked-list head

static struct list_head tracer_head;

You can check my article on the kernel linked list here.

We also need to define a lock on the list to avoid race conditions

static DEFINE_SPINLOCK(tracer_list_lock);

Now, there are some helper functions we need to write to work on our list based on the ioctl commands we'll handle. In fact, based on the cmd:

• TRACER_ADD_PROCESS: we'll add a new process ID to the list

• TRACER_REMOVE_PROCESS: we'll remove a process ID from the list

Add new process

First, we need to check whether the PID exists in the list when adding or removing a process.

So, the first function to write is tracer_find_proc, which returns a pointer to the tracer_stats instance if the PID to add/remove exists, or NULL otherwise.

static struct tracer_stats *tracer_find_proc(pid_t pid)
{
	struct tracer_stats *tr_proc;
	struct list_head *pos;
	unsigned long flags;
spin_lock_irqsave(&tracer_list_lock, flags);
list_for_each (pos, &tracer_head) {
	tr_proc = list_entry(pos, struct tracer_stats, tr_pid_list);
	if (tr_proc->tr_pid == pid) {
		spin_unlock_irqrestore(&tracer_list_lock, flags);
		return tr_proc;
	}
}
spin_unlock_irqrestore(&tracer_list_lock, flags);

return NULL;

}

NB: In that code we use

spin_lock_irqsave(&tracer_list_lock, flags);
...
...
spin_unlock_irqrestore(&tracer_list_lock, flags);

spin_lock_irqsave

• disables kernel preemption

• disables local hardware interrupts (irq)

• save the CPU's current interrupt state into flags (save)

• acquires the lock

spin_lock_irqsave() saves the interrupt flags, and spin_unlock_irqrestore() restores whatever interrupt state existed before.

If we don't use an IRQ lock here, an interrupt that executes kmalloc or kfree could happen (and we need to update the malloc stat if the relevant process is registered in the list) while the CPU is executing proc_tracer_show or an ioctl command, causing a race condition.

Then, the tracer_add_proc function for adding a process is trivial

static void tracer_add_proc(pid_t pid)
{
	struct tracer_stats *tr_proc;
	unsigned long flags;
if (tracer_find_proc(pid))
	return;

tr_proc = tracer_stat_alloc(pid);
if (!tr_proc)
	return;
spin_lock_irqsave(&tracer_list_lock, flags);
list_add(&tr_proc->tr_pid_list, &tracer_head);
spin_unlock_irqrestore(&tracer_list_lock, flags);

}

And yes, there is the tracer_stat_alloc function that allocates memory for a new process's tracer_stats and initializes it to zero.

Remove an existing process

The removal implementation is also straightforward: we iterate over the list with list_for_each_safe, then use list_entry to check whether the entry's PID is the one we want to remove by comparing it with pid. We can then use list_del to delete the entry from the list and free its memory with kfree(tr_proc). Below is the implementation for the tracer_remove_proc

static void tracer_remove_proc(pid_t pid)
{
	struct list_head *pos;
	struct list_head *n;
	struct tracer_stats *tr_proc;
	unsigned long flags;
spin_lock_irqsave(&tracer_list_lock, flags);
list_for_each_safe (pos, n, &tracer_head) {
	tr_proc = list_entry(pos, struct tracer_stats, tr_pid_list);
	if (tr_proc->tr_pid == pid) {
		list_del(pos);
		spin_unlock_irqrestore(&tracer_list_lock, flags);
		kfree(tr_proc);
		return;
	}
}
spin_unlock_irqrestore(&tracer_list_lock, flags);

}

As we implement the handler for our ioctl operations, I'll quickly show how an ioctl command can be used from user space to send a command to the driver

1. First, get a file descriptor by opening the driver with int open (const char *file, int flag), like this:

int fd = open("/dev/tracer", O_RDONLY); 

The flag can be O_RDONLY, O_WRONLY, or O_RDWR

1. Second, use int ioctl (int fd, unsigned long int request) to send a command:

int rc = ioctl(fd, TRACER_ADD_PROCESS, pid);

int rc = ioctl(fd, TRACER_REMOVE_PROCESS, pid);

Easy.

/proc/tracer entry creation

OK, it's good to see that we can handle adding and removing PIDs, but to test our driver quickly and manually, we can use the old method: print. At the start of this post, we listed that one requirement of the assignment was to display the stats through cat /proc/tracer.

So, we'll see how to automatically create an entry named tracer in the /proc directory, and then we'll iterate over the list to display its content.

Create an entry in /proc

In our init function, we'll use proc_create_single(name, mode, parent, show), like this:

static int tracer_init(void)
{
	...
	proc_create_single(TRACER_DEV_NAME, 0, NULL, tracer_proc_show);
	...
}

where

• name: is the path to the file inside /proc—tracer in our case. It could be /tracer/stats/whatever_file

• mode: the permissions/rights (read-only, write, ...)

• parent: NULL if /proc

• show: a pointer to the function that will be called to render the file's content

/proc/tracer content with tracer_proc_show

tracer_proc_show below is simple: it iterates over the list with list_for_each and, for each entry (process), displays the content of its attributes.

static int tracer_proc_show(struct seq_file *m, void *v)
{
	struct list_head *pos;
	struct tracer_stats *tr_proc;
	unsigned long flags;
seq_printf(m, "%-6s %-10s %-8s %-12s %-12s %-8s %-6s %-8s %-6s %-8s\n",
	   "PID", "kmalloc", "kfree", "kmalloc_mem", "kfree_mem",
	   "sched", "up", "down", "lock", "unlock");

spin_lock_irqsave(&tracer_list_lock, flags);
list_for_each (pos, &tracer_head) {
	tr_proc = list_entry(pos, struct tracer_stats, tr_pid_list);
	seq_printf(
		m,
		"%-6d %-10d %-8d %-12d %-12d %-8d %-6d %-8d %-6d %-8d\n",
		tr_proc->tr_pid, tr_proc->tr_alloc, tr_proc->tr_free,
		tr_proc->tr_mem, tr_proc->tr_mem_free,
		tr_proc->tr_sched, tr_proc->tr_up, tr_proc->tr_down,
		tr_proc->tr_lock, tr_proc->tr_unlock);
}
spin_unlock_irqrestore(&tracer_list_lock, flags);
return 0;

}

Here are a few details

• Our tracer_proc_show takes a pointer to struct seq_file, which is a buffer for /proc

• We use seq_printf instead of pr_info to write the formatted text into the struct seq_file *m buffer.

• %-6s and %-12d are used to set a minimum width of 6 or 12 characters with left alignment (-). s (string) and d (integer) indicate the type of data. With that, we ensure that the data and column headers are well aligned when displayed.

Now, let's talk about the instrumentation itself.

Instrumentation with kretprobe 1: schedule, up, down_interruptible, mutex_lock and mutex_unlock

So, how does probing a kernel function with a kprobe work? When we register a kprobe (probe handler) on a kernel function, depending on the architecture, the kernel replaces the first byte of the target function's instructions with a breakpoint instruction. When the kernel function is called and the CPU starts executing its instructions and encounters the breakpoint, an exception is triggered. The CPU then switches to execute the probe handler and afterward switches back to the normal flow. That's how kprobe works.

In our case, we'll use kretprobe (ret -> return): it runs the handler at the function exit.

Now, let's see how to register a handler to probe a kernel function with kretprobe.

Initialize an instance of struct kretprobe

static struct kretprobe schedule_probe = {
	.entry_handler = schedule_probe_entry_handler,
	.maxactive = 32,
	.kp = { .symbol_name = "schedule" }
};

where:

• schedule_probe_entry_handler is our probe handler, which is executed when schedule returns.

• maxactive = 32: the maximum number of concurrent executions of the probed function that will be tracked

• .symbol_name = "schedule" is used to specify the function to track or probe

Register the probe

In the init function, after registering the miscdevice, we'll use register_kretprobe to register schedule_probe.

static int tracer_init(void)
{	
	...
	ret = register_kretprobe(&schedule_probe);
	...
}

Implement the handler

static int schedule_probe_entry_handler(struct kretprobe_instance *ri,
					struct pt_regs *regs)
{
	struct tracer_stats *tr_proc;
	pid_t running_pid;
running_pid = ri->task->pid;
tr_proc = tracer_find_proc(running_pid);
if (tr_proc)
	tr_proc->tr_sched += 1;

return 0;

}
NOKPROBE_SYMBOL(schedule_probe_entry_handler);

schedule_probe_entry_handler is simple; it takes:

• an instance of struct kretprobe_instance as its first argument, which contains a pointer to the process that called the probed function (for example, schedule)

• a pointer to a copy of the CPU registers saved at that moment. We'll see later how to get the values of these registers.

NB: NOKPROBE_SYMBOL prevents our probe handler from being probed by kprobe/kretprobe or traced by other techniques available in the kernel.

So, each time schedule is called by a process and our handler is fired, we go through the kretprobe_instance and check whether the process ID is already on our list, so we can increment tr_sched, the number of schedule calls for this process.

Probe up, down_interruptible, mutex_lock and mutex_unlock

Basically, we'll do the same thing to probe up, down_interruptible, mutex_lock, and mutex_unlock, except that in the case of mutex_lock, we'll rather probe mutex_lock_nested instead of mutex_lock directly because mutex_lock is a macro that calls mutex_lock_nested.

And what's the problem with the macro? Macros in C are inlined directly at each call site, which means that the assembly code associated with the macro is always pasted directly wherever the macro is called. So, there is no explicit call to mutex_lock, but rather assembly code where mutex_lock_nested is called.

You can go to the GitHub repository to see the full implementations.

Next

In this article, we learnt:

• A new way to create a character device driver with miscdevice

• How to define the file_operations used by drivers

• How to interact with the /proc file

• How to define a handler to kprobe some kernel functions

• An introduction to the famous ioctl

In the next article, we'll continue the kprobe-based tracer by handling kmalloc and kfree.