How to Write a Linux Kernel Module That Actually Builds

<p>A Linux <code>kernel module</code> is a small piece of code that can be loaded into the running kernel without rebuilding the entire kernel.</p> <p>That sounds simple enough, but even a minimal module produces a surprising amount of machinery around it: object files, metadata, exported and unresolved symbols, and a final <code>.ko</code> file that is quite different from an ordinary executable.</p> <p>Here's a complete, working Linux kernel module. It's just twenty-two lines, seven of which are includes and metadata:</p> <pre><code class="language-c">#include &lt;linux/init.h&gt; #include &lt;linux/module.h&gt; #include &lt;linux/kernel.h&gt; MODULE_LICENSE("GPL"); MODULE_AUTHOR("Chris Roy"); MODULE_DESCRIPTION("A minimal loadable kernel module"); MODULE_VERSION("0.1"); static int __init hello_init(void) { pr_info("hello: loaded, module at %pS\n", hello_init); return 0; } static void __exit hello_exit(void) { pr_info("hello: unloaded\n"); } module_init(hello_init); module_exit(hello_exit); </code></pre> <p>Compiled on the machine I'm writing this on, that produces a file of about 106,000 bytes. Strip the debug information out and the same module is 4,864 bytes. Ninety-five percent of what the build gave you isn't code.</p> <p>Your total will differ from mine, and not by a predictable amount. Part of it is where you built: the debug information records the directory you compiled in, so a deeply nested path costs a few hundred bytes that a short one doesn't. Your compiler version and kernel configuration move it further. The proportion is what holds. The exact byte count is only what this machine produced.</p> <p>That gap is a good place to start, because most kernel module tutorials show you the listing above, tell you to run <code>make</code>, and stop.</p> <p>This one follows what the build actually produced, what your module already depends on before you wrote anything useful, and why the tutorial you found from 2014 no longer compiles.</p> <h2 id="heading-table-of-contents">Table of Contents</h2> <ul> <li><p><a href="#heading-what-you-need">What You Need</a></p> </li> <li><p><a href="#heading-the-smallest-module-that-works">The Smallest Module That Works</a></p> </li> <li><p><a href="#heading-the-makefile-is-stranger-than-it-looks">The Makefile is Stranger Than it Looks</a></p> </li> <li><p><a href="#heading-what-the-build-actually-did">What the Build Actually Did</a></p> </li> <li><p><a href="#heading-whats-inside-a-ko-file">What's Inside a .ko File</a></p> </li> <li><p><a href="#heading-your-hello-world-already-depends-on-three-things">Your Hello World Already Depends on Three Things</a></p> </li> <li><p><a href="#heading-vermagic-and-why-your-module-refuses-to-load">vermagic, and Why Your Module Refuses to Load</a></p> </li> <li><p><a href="#heading-passing-parameters-at-load-time">Passing Parameters at Load Time</a></p> </li> <li><p><a href="#heading-loading-it-and-where-the-output-goes">Loading it, and Where the Output Goes</a></p> </li> <li><p><a href="#heading-four-build-errors-and-what-they-mean">Four Build Errors and What They Mean</a></p> </li> <li><p><a href="#heading-why-the-tutorial-you-found-doesnt-compile">Why the Tutorial You Found Doesn't Compile</a></p> </li> <li><p><a href="#heading-conclusion">Conclusion</a></p> </li> <li><p><a href="#heading-epilogue">Epilogue</a></p> </li> </ul> <h2 id="heading-what-you-need">What You Need</h2> <p>To follow along here, you'll need a Linux machine you're willing to load code into, the headers for the kernel you're running, and a compiler.</p> <p>On Debian or Ubuntu:</p> <pre><code class="language-bash">sudo apt install build-essential linux-headers-$(uname -r) </code></pre> <p>On Fedora, the equivalent is <code>kernel-devel</code> and <code>kernel-headers</code>, and on Arch it's the <code>linux-headers</code> package matching your kernel.</p> <p>Check that the headers landed where the build expects them:</p> <pre><code class="language-bash">ls -d /lib/modules/$(uname -r)/build </code></pre> <p>That path is a symlink into the headers package, and its absence is the single most common reason a module build fails with an error that mentions nothing about headers.</p> <p>Two things will stop you from loading a module even after it builds. Secure Boot rejects unsigned modules, and kernel lockdown blocks loading in confidentiality mode. Check both:</p> <pre><code class="language-bash">mokutil --sb-state cat /sys/kernel/security/lockdown </code></pre> <p>On the machine here, Secure Boot is disabled and lockdown reports <code>[none] integrity confidentiality</code>, with the brackets marking the active mode. If yours shows Secure Boot enabled, you'll need to sign the module or disable Secure Boot in firmware before it will load.</p> <p>I'm on Ubuntu 22.04 with kernel 5.15.0-190-generic and gcc 11.4. Your versions will differ, and the article says where that matters.</p> <h2 id="heading-the-smallest-module-that-works">The Smallest Module That Works</h2> <p>Save the code from the beginning of this article as <code>hello.c</code>. Here it is again for reference:</p> <pre><code class="language-c">#include &lt;linux/init.h&gt; #include &lt;linux/module.h&gt; #include &lt;linux/kernel.h&gt; MODULE_LICENSE("GPL"); MODULE_AUTHOR("Chris Roy"); MODULE_DESCRIPTION("A minimal loadable kernel module"); MODULE_VERSION("0.1"); static int __init hello_init(void) { pr_info("hello: loaded, module at %pS\n", hello_init); return 0; } static void __exit hello_exit(void) { pr_info("hello: unloaded\n"); } module_init(hello_init); module_exit(hello_exit); </code></pre> <p>Four things in it are doing real work.</p> <p><code>module_init</code> and <code>module_exit</code> register the functions the kernel calls when your module is loaded and unloaded. They aren't <code>main</code>. A module has no single entry point that runs and returns. It has hooks that fire on two specific events, and it does nothing in between unless something else calls into it.</p> <p><code>__init</code> and <code>__exit</code> are section markers. <code>__init</code> tells the kernel this code runs once and its memory can be freed afterward, which is why you'll see "Freeing unused kernel memory" in your boot log. <code>__exit</code> tells the build that this function is only needed if the module can be unloaded at all.</p> <p><code>MODULE_LICENSE("GPL")</code> isn't paperwork. The kernel checks it at load time, and a module declaring a non-GPL license is denied access to symbols marked <code>EXPORT_SYMBOL_GPL</code>, which is most of the interesting ones. Omit the macro entirely and the kernel taints itself and logs a complaint.</p> <p><code>pr_info</code> is the modern spelling of <code>printk(KERN_INFO ...)</code>. It writes to the kernel ring buffer, not to your terminal, which trips up nearly everyone the first time.</p> <p>The <code>MODULE_AUTHOR</code>, <code>MODULE_DESCRIPTION</code>, and <code>MODULE_VERSION</code> macros are metadata rather than behavior, and they end up in the file for <code>modinfo</code> to read. Leave them out and nothing breaks, but recent kernels warn at build time about a missing <code>MODULE_DESCRIPTION</code>, which is reason enough to write all three from the start.</p> <h2 id="heading-the-makefile-is-stranger-than-it-looks">The Makefile is Stranger Than it Looks</h2> <pre><code class="language-makefile">obj-m += hello.o all: make -C /lib/modules/$(shell uname -r)/build M=$(PWD) modules clean: make -C /lib/modules/$(shell uname -r)/build M=$(PWD) clean </code></pre> <p>This looks like a Makefile, and mostly isn't one. <code>obj-m += hello.o</code> isn't a Make variable you invented. It's a declaration read by kbuild, the kernel's own build system.</p> <p>The <code>make -C</code> line changes directory into the kernel headers and runs the kernel's build system there, passing <code>M=$(PWD)</code> to say "the module source is over here." Your Makefile is a thin wrapper that hands the job to a build system you didn't write and can't easily replace.</p> <p>That indirection is why module builds fail in ways that seem unrelated to your code. You're not compiling against the kernel headers the way you compile against libc headers. You're running the kernel's build, on your file, with its flags and its rules.</p> <h2 id="heading-what-the-build-actually-did">What the Build Actually Did</h2> <p>Run <code>make</code> and read the output rather than skipping it:</p> <pre><code class="language-text">make -C /lib/modules/5.15.0-190-generic/build M=/home/chris/lkm modules make[1]: Entering directory '/usr/src/linux-headers-5.15.0-190-generic' CC [M] /home/chris/lkm/hello.o MODPOST /home/chris/lkm/Module.symvers CC [M] /home/chris/lkm/hello.mod.o LD [M] /home/chris/lkm/hello.ko BTF [M] /home/chris/lkm/hello.ko Skipping BTF generation for /home/chris/lkm/hello.ko due to unavailability of vmlinux </code></pre> <img src="https://cdn.hashnode.com/uploads/covers/6a783a81a29db580b40f1bc8/9e38d600-4a16-4f2d-8f45-8f4897f5f44c.png" alt="Diagram of the kernel module build pipeline: hello.c compiles to hello.o, MODPOST checks undefined symbols against the kernel export table and generates hello.mod.c, that compiles to hello.mod.o, the linker combines both into hello.ko at roughly 106 KB of which only 4,864 bytes survive stripping, and a final BTF step is skipped because Ubuntu ships no vmlinux" style="display: block;" width="600" height="400" loading="lazy"> <p>Five steps, and only the first is the compile you expected.</p> <p><code>CC [M] hello.o</code> compiles your source. Ordinary.</p> <p><code>MODPOST</code> is the step worth knowing about. It scans your object file for symbols you referenced but didn't define, checks each one against the kernel's table of exported symbols, and fails the build if you used something the kernel doesn't offer you. It also generates <code>hello.mod.c</code>, a small file of glue containing your module's metadata.</p> <p><code>CC [M] hello.mod.o</code> compiles that generated glue, and <code>LD [M]</code> links it together with your object into the final <code>.ko</code>.</p> <p><code>BTF [M]</code> would attach type information used by tracing tools. Here it was skipped, because generating BTF needs the uncompressed <code>vmlinux</code> image and Ubuntu doesn't ship it by default. The build warns and continues, which is correct: BTF is useful, not required.</p> <h2 id="heading-whats-inside-a-ko-file">What's Inside a .ko File</h2> <p>A <code>.ko</code> is an ordinary ELF object with kernel-specific sections bolted on. Look at its metadata:</p> <pre><code class="language-bash">modinfo ./hello.ko </code></pre> <pre><code class="language-text">version: 0.1 description: A minimal loadable kernel module author: Chris Roy license: GPL srcversion: 39D86510C9FF65D797EAF90 depends: retpoline: Y name: hello vermagic: 5.15.0-190-generic SMP mod_unload modversions </code></pre> <p>All of that lives in one ELF section, stored as null-separated strings. You can read it straight out of the file:</p> <pre><code class="language-bash">objcopy -O binary --only-section=.modinfo hello.ko /dev/stdout | tr '\0' '\n' </code></pre> <p>Which brings us back to the number from the opening. The module is about 106,000 bytes on disk:</p> <pre><code class="language-bash">ls -l hello.ko cp hello.ko /tmp/ &amp;&amp; strip --strip-debug /tmp/hello.ko &amp;&amp; ls -l /tmp/hello.ko </code></pre> <pre><code class="language-text">105984 hello.ko 4864 /tmp/hello.ko </code></pre> <p>The actual module is under five kilobytes. Everything else is DWARF debug information the build keeps so that tools like <code>crash</code> and <code>gdb</code> can make sense of your code if it panics. When you load the module, the kernel doesn't load the debug sections, so the memory cost is the small number rather than the large one.</p> <h2 id="heading-your-hello-world-already-depends-on-three-things">Your Hello World Already Depends on Three Things</h2> <p>This is the part I'd have wanted someone to show me first. Ask the object what it needs from the kernel:</p> <pre><code class="language-bash">nm -u hello.ko </code></pre> <pre><code class="language-text">U __fentry__ U _printk U __x86_return_thunk </code></pre> <p><code>U</code> means undefined: symbols your module references and the kernel must supply at load time.</p> <p><code>_printk</code> you can account for, since <code>pr_info</code> expands to it.</p> <p><code>__fentry__</code> is a call the compiler placed at the top of every one of your functions, because the kernel is built with function tracing support. Every function you write in a module gets that hook whether you asked for it or not, and it's what lets <code>ftrace</code> instrument your code later without recompiling anything.</p> <p><code>__x86_return_thunk</code> is a Spectre mitigation. Your compiler replaced ordinary return instructions with a call to a thunk that avoids the speculative execution path the vulnerability relies on. It appears in a module that prints one line, because the mitigation applies to all kernel code on this machine, module or not.</p> <p>Two of the three symbols in your hello world are infrastructure the machine imposed on you. That's a fair picture of what writing kernel code is like.</p> <p>MODPOST verified all three exist before the link succeeded. Had you called a function the kernel doesn't export, the build would have failed there with an "undefined symbol" error rather than producing a module that fails at load.</p> <h2 id="heading-vermagic-and-why-your-module-refuses-to-load"><code>vermagic</code>, and Why Your Module Refuses to Load</h2> <p>Look again at that line from <code>modinfo</code>:</p> <pre><code class="language-text">vermagic: 5.15.0-190-generic SMP mod_unload modversions </code></pre> <p>The kernel compares that string against its own before loading anything, and refuses on a mismatch. It covers the release, whether the kernel is SMP, whether module unloading is compiled in, and whether symbol versioning is on.</p> <p>This is why a module built on one machine usually won't load on another, and why upgrading your kernel means rebuilding your modules.</p> <p>There's no ABI stability guarantee inside the Linux kernel. Internal structures change between releases, and a module compiled against one layout that ran against another would corrupt memory rather than fail cleanly. Refusing to load is the kernel being careful.</p> <p>It's also why DKMS exists. If you have VirtualBox, ZFS, or an NVIDIA driver installed, you already have a module being rebuilt this way. On the machine here:</p> <pre><code class="language-bash">modinfo vboxdrv | head -3 </code></pre> <pre><code class="language-text">filename: /lib/modules/5.15.0-190-generic/updates/dkms/vboxdrv.ko version: 6.1.50_Ubuntu r161033 (0x00320000) license: GPL </code></pre> <p>Note the <code>updates/dkms/</code> in that path. DKMS keeps the source and rebuilds the module each time you install a new kernel, which is the maintenance cost the vermagic check makes unavoidable.</p> <h2 id="heading-passing-parameters-at-load-time">Passing Parameters at Load Time</h2> <p>A module that always does the same thing is rarely what you want. <code>module_param</code> exposes a variable so its value can be set at load time. Save this as <code>param.c</code> alongside <code>hello.c</code>:</p> <pre><code class="language-c">#include &lt;linux/init.h&gt; #include &lt;linux/module.h&gt; MODULE_LICENSE("GPL"); MODULE_DESCRIPTION("A module that takes parameters"); static char *who = "world"; static int times = 1; module_param(who, charp, 0444); MODULE_PARM_DESC(who, "who to greet"); module_param(times, int, 0644); MODULE_PARM_DESC(times, "how many times to greet"); static int __init param_init(void) { int i; for (i = 0; i &lt; times; i++) pr_info("param: hello, %s\n", who); return 0; } static void __exit param_exit(void) { pr_info("param: unloaded\n"); } module_init(param_init); module_exit(param_exit); </code></pre> <p>Add it to the Makefile, which takes a list:</p> <pre><code class="language-makefile">obj-m += hello.o param.o </code></pre> <p>The three arguments are the variable, its type, and the permissions on the file that will represent it under <code>/sys/module/&lt;name&gt;/parameters/</code>. A mode of <code>0444</code> makes it readable and fixed once loaded. <code>0644</code> lets root write to that file and change the value while the module is running, which is useful. It's also how you introduce a race if the module reads the variable without expecting it to change.</p> <p><code>charp</code> is a char pointer, and the other common types are <code>int</code>, <code>bool</code>, <code>long</code> and <code>charp</code> arrays via <code>module_param_array</code>. Pass a type that doesn't match the variable and the build fails rather than misbehaving later.</p> <p><code>MODULE_PARM_DESC</code> puts the description into the module metadata, where <code>modinfo</code> finds it:</p> <pre><code class="language-text">name: param parm: who:who to greet (charp) parm: times:how many times to greet (int) </code></pre> <p>Set them at load time as <code>name=value</code> pairs:</p> <pre><code class="language-bash">sudo insmod ./param.ko who=kernel times=3 </code></pre> <p>Anyone can read what parameters a module accepts before loading it, which is the main reason to bother with <code>MODULE_PARM_DESC</code> at all.</p> <h2 id="heading-loading-it-and-where-the-output-goes">Loading it, and Where the Output Goes</h2> <pre><code class="language-bash">sudo insmod ./hello.ko sudo dmesg | tail -2 lsmod | grep hello sudo rmmod hello </code></pre> <p>The <code>pr_info</code> output goes to the kernel ring buffer, so it appears in <code>dmesg</code> rather than your terminal. If <code>dmesg</code> refuses without root, that's <code>kernel.dmesg_restrict</code>, and <code>sudo journalctl -k | tail</code> reads the same messages through the journal.</p> <p>The <code>%pS</code> in the format string prints a kernel pointer as a symbol name and offset instead of a raw address, which is how you get something readable out of a log line rather than a hexadecimal number.</p> <p><code>lsmod</code> reads <code>/proc/modules</code> and shows three columns: the module name, its size in memory, and a use count with the names of anything depending on it. A module with a non-zero use count can't be unloaded, which is the most common reason <code>rmmod</code> refuses.</p> <p>One caution before you load anything. A bug in userspace crashes your program. But a bug here can take the machine down or corrupt a filesystem. Do this in a virtual machine the first several times. The cost of a snapshot is far lower than the cost of a corrupted disk.</p> <h2 id="heading-four-build-errors-and-what-they-mean">Four Build Errors and What They Mean</h2> <p>These four account for most of the time people lose, and each says something specific once you know what to read.</p> <p><code>No rule to make target 'modules'. Stop.</code> The kernel headers are missing or the symlink at <code>/lib/modules/$(uname -r)/build</code> points nowhere. Install the headers package matching the exact kernel you're running, which is often not the newest one installed if you haven't rebooted since an update.</p> <p><code>ERROR: modpost: "some_function" [hello.ko] undefined!</code> You referenced a symbol the kernel doesn't export. Here's what that looks like from a real build:</p> <pre><code class="language-text">ERROR: modpost: "this_symbol_does_not_exist" [bad.ko] undefined! make[2]: *** [scripts/Makefile.modpost:133: Module.symvers] Error 1 </code></pre> <p>Retrying won't help. Either the function is internal to the kernel and never exported, or it's exported with <code>EXPORT_SYMBOL_GPL</code> and your module declares a non-GPL license. Check with <code>grep the_symbol /proc/kallsyms</code>, where a capital <code>T</code> in the second column means it's a global text symbol.</p> <p><code>insmod: ERROR: could not insert module: Invalid module format</code>: The build succeeded but vermagic doesn't match the running kernel. Compare <code>modinfo ./hello.ko | grep vermagic</code> against <code>uname -r</code>. Rebuilding against the correct headers fixes it.</p> <p><code>insmod: ERROR: could not insert module: Operation not permitted</code>: Usually Secure Boot rejecting an unsigned module, or lockdown in confidentiality mode. Check <code>mokutil --sb-state</code> and <code>cat /sys/kernel/security/lockdown</code> before assuming your code is at fault.</p> <p>One more thing, which is easier to learn now than to debug later. Loading any out-of-tree module sets a taint flag on the kernel, which is recorded and reported in any subsequent oops or panic:</p> <pre><code class="language-bash">cat /proc/sys/kernel/tainted </code></pre> <p>The value is a bitmask. It reads 4096 on the machine here, which is bit 12, <code>TAINT_OOT_MODULE</code>, meaning an out-of-tree module has been loaded at some point.</p> <p>Bit 13 is the neighboring one people confuse it with, <code>TAINT_UNSIGNED_MODULE</code>, which is what Secure Boot cares about.</p> <p>The full list is in <code>include/linux/panic.h</code> in the kernel source. Kernel developers will ask you to reproduce a bug on an untainted kernel before they look at it, and this is the file that tells them whether you did.</p> <h2 id="heading-why-the-tutorial-you-found-doesnt-compile">Why the Tutorial You Found Doesn't Compile</h2> <p>Most module tutorials on the web predate several changes, and these are the ones that bite.</p> <p><code>printk(KERN_INFO "...")</code> still works, but <code>pr_info</code> is the current spelling and carries the log level for you.</p> <p><code>init_module</code> and <code>cleanup_module</code> as bare function names were the old convention. Use <code>module_init</code> and <code>module_exit</code> with your own names instead, which lets a file define both without collisions.</p> <p><code>MODULE_LICENSE</code> was once optional in practice. It's now load-bearing, since it gates access to GPL-only exported symbols.</p> <p>The <code>M=</code> argument used to be spelled <code>SUBDIRS=</code>. That spelling was removed, and a tutorial using it fails with an error that doesn't mention <code>SUBDIRS</code> anywhere.</p> <p>Header paths moved. Anything referring to <code>/usr/src/linux</code> predates the split into per-kernel headers packages and is old enough that the rest of it needs checking too. A smaller sign: <code>&lt;linux/module.h&gt;</code> has included <code>&lt;linux/moduleparam.h&gt;</code> for years, so a tutorial that carefully includes both is copying from something old, even though including both is harmless.</p> <p>If a tutorial builds without warnings on your kernel, it's current enough. If it doesn't, the kernel version it targeted is usually printed in the first error.</p> <h2 id="heading-conclusion">Conclusion</h2> <p>You can now build a kernel module, read what the build produced, and explain every symbol it depends on.</p> <p>More usefully, you know why it fails in the specific ways it does. A missing <code>/lib/modules/$(uname -r)/build</code> is a headers problem. A vermagic mismatch is a rebuild. An undefined symbol at MODPOST means the kernel doesn't export what you asked for, and no amount of retrying will change that.</p> <p>There are a few directions to go from here. Register a <code>/proc</code> entry with <code>proc_create</code> and read from it, which is the smallest useful thing a module can do.</p> <p>Read the kernel's own <code>Module.symvers</code> under <code>/usr/src/linux-headers-$(uname -r)/</code> to see the table MODPOST checked against, which is 26,420 exported symbols on this machine and marks each one <code>EXPORT_SYMBOL</code> or <code>EXPORT_SYMBOL_GPL</code>.</p> <p>Or trace your own module's functions, which works because of the <code>__fentry__</code> hook that was there from the first build. There's no <code>ftrace</code> command to run. It's an interface under <code>/sys/kernel/tracing</code>, so you drive it by writing to files:</p> <pre><code class="language-bash">sudo sh -c 'echo hello_init &gt; /sys/kernel/tracing/set_ftrace_filter' sudo sh -c 'echo function &gt; /sys/kernel/tracing/current_tracer' sudo cat /sys/kernel/tracing/trace </code></pre> <p>On older systems, that path is <code>/sys/kernel/debug/tracing</code> instead. If you would rather not write to files by hand, <code>trace-cmd</code> wraps the same interface.</p> <h2 id="heading-epilogue">Epilogue</h2> <p>Everything above assumes you're allowed to do it, and that assumption is the part I find interesting. A loaded module runs with the same authority as the kernel itself. It can read any memory, patch any function, and ignore any policy the system thought it was enforcing, because by the time it runs there's nothing left above it to say no.</p> <p>That makes module loading the one operation a permission model can't contain, which is why the kernel guards it with signatures and lockdown rather than with permissions.</p> <p>I ran into that floor while working on a capability-backed desktop OS, one where a program's manifest is the whole of what it may do, and the Debian ecosystem still has to work underneath it. Modules are where that model stops being expressible, so working out exactly what the kernel checks before accepting one stopped being a detail and became a design constraint.</p> <p>I write about systems and their mysteries at <a href="https://thechris.in">thechris.in</a>.</p>