static struct miscdevice tracer_miscdevice = {
.minor = TRACER_DEV_MINOR,
.name = TRACER_DEV_NAME,
};
```
I don't think I need to provide any additional explanation 😁.
Then, we'll register it with `misc_register`
```c
int misc_register(struct miscdevice *misc)
```
In our init function, we'll register it
```c
static int tracer_init(void)
{
int ret;
ret = misc_register(&tracer_miscdevice);
if (ret < 0) {
pr_err("cannot register miscdev on minor=%d name=%s (err=%d)\n",
TRACER_DEV_MINOR, tracer_miscdevice.name, ret);
return ret;
}
pr_info("Misc device registered! Look at /dev/%s\n",
tracer_miscdevice.name);
return 0;
}
```
Don't forget to check the return value of `misc_register` because the registration can fail.
**You** can notice that we don't set the major number with `miscdevice`. We don't need to do that because `miscdevice` has a default major number, `MISC_MAJOR`, which is set to `10` and defined in [major.h](https://elixir.bootlin.com/linux/v7.1.4/source/include/uapi/linux/major.h#L26)
```c
#define MISC_MAJOR 10
```
### Unregistration
We unregister the driver with:
```c
void misc_deregister(struct miscdevice *misc);
```
in our exit function.
```c
static void tracer_exit(void)
{
misc_deregister(&tracer_miscdevice);
pr_info("Misc device %s unregistered!\n", tracer_miscdevice.name);
}
```
### File operations
For the file operations, we'll just implement the `ioctl` operations. So, let's define an instance of `file_operations`
```c
static const struct file_operations tracer_fops = {
.owner = THIS_MODULE,
.unlocked_ioctl = tracer_ioctl,
};
```
Before moving on to `ioctl`, let's set `tracer_fops` as the file-operations handler for `tracer_miscdevice`
```c
static struct miscdevice tracer_miscdevice = {
...
.fops = &tracer_fops,
};
```
Now, we'll define the `tracer_ioctl` function.
**So, what is** `IOCTL`**?**
**IOCTL**, short for **Input/Output Control**, allows us to perform custom actions with the driver. You know, with the `open`, `read`, and `write` handlers of `file_operations`, we can only perform a limited set of actions: we can either read data sent from user space to the driver and do something with it, or write data back to user space. Now, imagine we want to do something like:
* check whether some data exists
* remove an entry from the driver's list if a flag is set
So, `ioctl` will help our driver do more than just read or write; it will let us control what the driver can do and how it does it.
In fact, what will happen is that we'll define the `ioctl` commands (I'll show how to do this in the following lines), and a user-space application will send these commands to the driver with the appropriate arguments.
Our driver will handle two commands:
* `TRACER_ADD_PROCESS`: to add the `pid` of a process to trace
* `TRACER_REMOVE_PROCESS`: to remove a process from the tracing list
**IOCTL command**
Many macros exist to define an IOCTL command; you can read about them [here](https://elixir.bootlin.com/linux/v7.1.4/source/include/uapi/asm-generic/ioctl.h).
To define `read` and `write` commands, we can use the following macros: `_IOR` and `_IOW`.
```c
// For read operations
#define _IOR(type,nr,size) _IOC(_IOC_READ,(type),(nr),(_IOC_TYPECHECK(size)))
// For write operations
#define _IOW(type,nr,size) _IOC(_IOC_WRITE,(type),(nr),(_IOC_TYPECHECK(size)))
```
where:
* the first parameter (`type`) is an arbitrary 8-bit character unique to the driver, for example, `'d'` or `'t'`,...
* the second parameter (`nr`) is an incrementing number that uniquely identifies the command
* the last parameter (`size`) is the "type" of the argument passed to the command, such as `int` or `struct my_data`
So, in our case:
```c
#define MAGIC_BYTE 't'
#define TRACER_ADD_PROCESS _IOW(MAGIC_BYTE, 42, pid_t)
#define TRACER_REMOVE_PROCESS _IOW(MAGIC_BYTE, 43, pid_t)
```
Inside `tracer_ioctl`, which we set as the `unlocked_ioctl` handler of the `file_operations` instance above:
```c
static long tracer_ioctl(struct file *file, unsigned int cmd, unsigned long arg)
{
switch (cmd) {
case TRACER_ADD_PROCESS:
tracer_add_proc(arg);
break;
case TRACER_REMOVE_PROCESS:
tracer_remove_proc(arg);
break;
default:
return -ENOTTY;
}
return 0;
}
```
where:
* `cmd`: the command sent from user space (`TRACER_ADD_PROCESS` and `TRACER_REMOVE_PROCESS`)
* `arg`: the value passed with the command (here, it's the process's `pid`); it could be more complex data.
So, based on the value of `cmd`, we know which operations to handle:
```plaintext
TRACER_ADD_PROCESS --> tracer_add_proc
TRACER_REMOVE_PROCESS --> tracer_remove_proc
```
In the next section, after defining our driver data, we'll implement the two handlers for `ioctl`
## Driver data and `ioctl` operations
* `struct tracer_stats`
* `static struct list_head tracer_head`;
We'll store statistics for each process in a structure, and a linked list will hold all added processes.
### struct tracer\_stats
```c
struct tracer_stats {
pid_t tr_pid; // process pid added
int tr_alloc; // number of calls to kmalloc
int tr_free; // number of calls to kfree
int tr_mem; // memory allocated with kmalloc
int tr_mem_free; // memory freed with kfree
int tr_sched; // number of calls to schedule
int tr_up; // number of calls to up
int tr_down; // number of calls to down_interruptible
int tr_lock; // number of calls to mutex_lock
int tr_unlock; // number of calls to mutex_unlock
struct list_head tr_pid_list; // list node reference
};
```
Most of the `struct tracer_stats` fields, except `tr_pid`, `tr_mem`, and `tr_mem_free`, will be incremented each time the traced function is called.
Then, we define our linked-list head
```c
static struct list_head tracer_head;
```
You can check my article on the kernel linked list [here](https://tawaliou.com/linux-kernel-labs-kernel-module-and-kernel-api#list-api).
We also need to define a lock on the list to avoid race conditions
```c
static DEFINE_SPINLOCK(tracer_list_lock);
```
Now, there are some helper functions we need to write to work on our list based on the `ioctl` commands we'll handle. In fact, based on the `cmd`:
* `TRACER_ADD_PROCESS`: we'll **add** a new process ID to the list
* `TRACER_REMOVE_PROCESS`: we'll **remove** a process ID from the list
### Add new process
First, we need to `check` whether the PID exists in the list when adding or removing a process.
So, the first function to write is [tracer\_find\_proc](https://github.com/tawaliou/kernel-labs-OS-2/blob/58155db4d34484de260b7d0131552cf652b886b4/1-tracer/tracer.c#L106), which returns a pointer to the `tracer_stats` instance if the PID to add/remove exists, or `NULL` otherwise.
tracer_find_proc(pid_t pid)
static struct tracer_stats *tracer_find_proc(pid_t pid)
{
struct tracer_stats *tr_proc;
struct list_head *pos;
unsigned long flags;
spin_lock_irqsave(&tracer_list_lock, flags);
list_for_each (pos, &tracer_head) {
tr_proc = list_entry(pos, struct tracer_stats, tr_pid_list);
if (tr_proc->tr_pid == pid) {
spin_unlock_irqrestore(&tracer_list_lock, flags);
return tr_proc;
}
}
spin_unlock_irqrestore(&tracer_list_lock, flags);
return NULL;
}
**NB**: In that code we use
```c
spin_lock_irqsave(&tracer_list_lock, flags);
...
...
spin_unlock_irqrestore(&tracer_list_lock, flags);
```
`spin_lock_irqsave`
* disables kernel preemption
* disables local hardware interrupts (**irq**)
* save the CPU's current interrupt state into `flags` (**save**)
* acquires the lock
`spin_lock_irqsave()` saves the interrupt flags, and `spin_unlock_irqrestore()` restores whatever interrupt state existed before.
If we don't use an IRQ lock here, an interrupt that executes `kmalloc` or `kfree` could happen (and we need to update the malloc stat if the relevant process is registered in the list) while the CPU is executing `proc_tracer_show` or an **ioctl command**, causing a race condition.
Then, the [tracer\_add\_proc](https://github.com/tawaliou/kernel-labs-OS-2/blob/58155db4d34484de260b7d0131552cf652b886b4/1-tracer/tracer.c#L165) function for adding a process is trivial
tracer_add_proc(pid_t pid)
static void tracer_add_proc(pid_t pid)
{
struct tracer_stats *tr_proc;
unsigned long flags;
if (tracer_find_proc(pid))
return;
tr_proc = tracer_stat_alloc(pid);
if (!tr_proc)
return;
spin_lock_irqsave(&tracer_list_lock, flags);
list_add(&tr_proc->tr_pid_list, &tracer_head);
spin_unlock_irqrestore(&tracer_list_lock, flags);
}
And yes, there is the [tracer\_stat\_alloc](https://github.com/tawaliou/kernel-labs-OS-2/blob/58155db4d34484de260b7d0131552cf652b886b4/1-tracer/tracer.c#L151) function that allocates memory for a new process's `tracer_stats` and initializes it to zero.
### Remove an existing process
The removal implementation is also straightforward: we iterate over the list with `list_for_each_safe`, then use `list_entry` to check whether the entry's PID is the one we want to remove by comparing it with `pid`. We can then use `list_del` to delete the entry from the list and free its memory with `kfree(tr_proc)`. Below is the implementation for the [tracer\_remove\_proc](https://github.com/tawaliou/kernel-labs-OS-2/blob/58155db4d34484de260b7d0131552cf652b886b4/1-tracer/tracer.c#L182)
tracer_remove_proc(pid_t pid)
static void tracer_remove_proc(pid_t pid)
{
struct list_head *pos;
struct list_head *n;
struct tracer_stats *tr_proc;
unsigned long flags;
spin_lock_irqsave(&tracer_list_lock, flags);
list_for_each_safe (pos, n, &tracer_head) {
tr_proc = list_entry(pos, struct tracer_stats, tr_pid_list);
if (tr_proc->tr_pid == pid) {
list_del(pos);
spin_unlock_irqrestore(&tracer_list_lock, flags);
kfree(tr_proc);
return;
}
}
spin_unlock_irqrestore(&tracer_list_lock, flags);
}
As we implement the handler for our `ioctl` operations, I'll quickly show how an `ioctl` command can be used from user space to send a command to the driver
1. First, get a file descriptor by opening the driver with `int open (const char *file, int flag)`, like this:
```c
int fd = open("/dev/tracer", O_RDONLY);
```
The flag can be `O_RDONLY`, `O_WRONLY`, or `O_RDWR`
1. Second, use `int ioctl (int fd, unsigned long int request)` to send a command:
```c
int rc = ioctl(fd, TRACER_ADD_PROCESS, pid);
int rc = ioctl(fd, TRACER_REMOVE_PROCESS, pid);
```
Easy.
## /proc/tracer entry creation
OK, it's good to see that we can handle adding and removing PIDs, but to test our driver quickly and manually, we can use the old method: print. At the start of this post, we listed that one requirement of the assignment was to display the stats through `cat /proc/tracer`.
So, we'll see how to automatically create an entry named **tracer** in the `/proc` directory, and then we'll iterate over the list to display its content.
### Create an entry in /proc
In our init function, we'll use `proc_create_single(name, mode, parent, show)`, like this:
```c
static int tracer_init(void)
{
...
proc_create_single(TRACER_DEV_NAME, 0, NULL, tracer_proc_show);
...
}
```
where
* `name`: is the path to the file inside `/proc`—`tracer` in our case. It could be `/tracer/stats/whatever_file`
* `mode`: the permissions/rights (read-only, write, ...)
* `parent`: NULL if `/proc`
* `show`: a pointer to the function that will be called to render the file's content
### /proc/tracer content with tracer\_proc\_show
[tracer\_proc\_show](https://github.com/tawaliou/kernel-labs-OS-2/blob/58155db4d34484de260b7d0131552cf652b886b4/1-tracer/tracer.c#L125) below is simple: it iterates over the list with `list_for_each` and, for each entry (process), displays the content of its attributes.
tracer_proc_show(struct seq_file *m, void *v)
static int tracer_proc_show(struct seq_file *m, void *v)
{
struct list_head *pos;
struct tracer_stats *tr_proc;
unsigned long flags;
seq_printf(m, "%-6s %-10s %-8s %-12s %-12s %-8s %-6s %-8s %-6s %-8s\n",
"PID", "kmalloc", "kfree", "kmalloc_mem", "kfree_mem",
"sched", "up", "down", "lock", "unlock");
spin_lock_irqsave(&tracer_list_lock, flags);
list_for_each (pos, &tracer_head) {
tr_proc = list_entry(pos, struct tracer_stats, tr_pid_list);
seq_printf(
m,
"%-6d %-10d %-8d %-12d %-12d %-8d %-6d %-8d %-6d %-8d\n",
tr_proc->tr_pid, tr_proc->tr_alloc, tr_proc->tr_free,
tr_proc->tr_mem, tr_proc->tr_mem_free,
tr_proc->tr_sched, tr_proc->tr_up, tr_proc->tr_down,
tr_proc->tr_lock, tr_proc->tr_unlock);
}
spin_unlock_irqrestore(&tracer_list_lock, flags);
return 0;
}
Here are a few details
* Our `tracer_proc_show` takes a pointer to `struct seq_file`, which is a buffer for `/proc`
* We use `seq_printf` instead of `pr_info` to write the formatted text into the `struct seq_file *m` buffer.
* `%-6s` and `%-12d` are used to set a minimum width of **6** or **12** characters with left alignment (**\-**). **s** (string) and **d** (integer) indicate the type of data. With that, we ensure that the data and column headers are well aligned when displayed.
Now, let's talk about the instrumentation itself.
## Instrumentation with kretprobe 1: `schedule`, `up`, `down_interruptible`, `mutex_lock` and `mutex_unlock`
So, how does probing a kernel function with a kprobe work? When we register a kprobe (probe handler) on a kernel function, depending on the architecture, the kernel replaces the first byte of the target function's instructions with a breakpoint instruction. When the kernel function is called and the CPU starts executing its instructions and encounters the breakpoint, an exception is triggered. The CPU then switches to execute the probe handler and afterward switches back to the normal flow. That's how `kprobe` works.
In our case, we'll use `kretprobe` (**ret** -> **return**): it runs the handler at the function exit.
Now, let's see how to register a handler to probe a kernel function with `kretprobe`.
### Initialize an instance of `struct kretprobe`
```c
static struct kretprobe schedule_probe = {
.entry_handler = schedule_probe_entry_handler,
.maxactive = 32,
.kp = { .symbol_name = "schedule" }
};
```
where:
* `schedule_probe_entry_handler` is our probe handler, which is executed when `schedule` returns.
* `maxactive = 32`: the maximum number of concurrent executions of the probed function that will be tracked
* `.symbol_name = "schedule"` is used to specify the function to track or probe
### Register the probe
In the init function, after registering the miscdevice, we'll use `register_kretprobe` to register `schedule_probe`.
```c
static int tracer_init(void)
{
...
ret = register_kretprobe(&schedule_probe);
...
}
```
### Implement the handler
schedule_probe_entry_handler
static int schedule_probe_entry_handler(struct kretprobe_instance *ri,
struct pt_regs *regs)
{
struct tracer_stats *tr_proc;
pid_t running_pid;
running_pid = ri->task->pid;
tr_proc = tracer_find_proc(running_pid);
if (tr_proc)
tr_proc->tr_sched += 1;
return 0;
}
NOKPROBE_SYMBOL(schedule_probe_entry_handler);
[schedule\_probe\_entry\_handler](https://github.com/tawaliou/kernel-labs-OS-2/blob/58155db4d34484de260b7d0131552cf652b886b4/1-tracer/tracer.c#L329-L342) is simple; it takes:
* an instance of `struct kretprobe_instance` as its first argument, which contains a pointer to the process that called the probed function (for example, `schedule`)
* a pointer to a copy of the CPU registers saved at that moment. We'll see later how to get the values of these registers.
**NB**: `NOKPROBE_SYMBOL` prevents our probe handler from being probed by kprobe/kretprobe or traced by other techniques available in the kernel.
So, each time `schedule` is called by a process and our handler is fired, we go through the `kretprobe_instance` and check whether the process ID is already on our list, so we can increment `tr_sched`, the number of `schedule` calls for this process.
### Probe `up`, `down_interruptible`, `mutex_lock` and `mutex_unlock`
Basically, we'll do the same thing to probe `up`, `down_interruptible`, `mutex_lock`, and `mutex_unlock`, except that in the case of `mutex_lock`, we'll rather probe `mutex_lock_nested` instead of `mutex_lock` directly because `mutex_lock` is a macro that calls `mutex_lock_nested`.
And what's the problem with the macro? Macros in C are inlined directly at each call site, which means that the assembly code associated with the macro is always pasted directly wherever the macro is called. So, there is no explicit **call to** `mutex_lock`, but rather **assembly code where** `mutex_lock_nested` **is called**.
You can go to the [GitHub repository](https://github.com/tawaliou/kernel-labs-OS-2/blob/main/1-tracer/tracer.c) to see the full implementations.
# Next
In this article, we learnt:
* A new way to create a character device driver with **miscdevice**
* How to define the **file\_operations** used by drivers
* How to interact with the `/proc` file
* How to define a handler to kprobe some kernel functions
* An introduction to the famous `ioctl`
In the next article, we'll continue the kprobe-based tracer by handling `kmalloc` and `kfree`.