405.Course: Advanced Character Device Driver Development and Debugging
Module 1: Debugging Tools for Linux Character Device Drivers
Rationale: Establish troubleshooting skills first. Developers will inevitably cause kernel panics in later modules; they need the tools to fix them independently from day one.
1a: Memory and Concurrency Sanitizers
Character drivers handle user-space buffers and shared hardware states, making them prone to memory leaks and race conditions. The kernel offers built-in debug configurations for this. KASAN detects out-of-bounds memory accesses and use-after-free bugs. Kmemleak acts like a tracing garbage collector for kmalloc()/kzalloc() allocations. Lockdep detects potential deadlocks and incorrect lock usage at runtime. KCSAN dynamically detects data races, finding missing locks or improper atomic operations.
1b: Interactive Kernel Debuggers
Deep, instruction-level debugging requires specialized tools. KGDB allows debugging the Linux kernel using GDB via a Host/VM setup or serial connection. KDB is a built-in shell that allows memory and register inspection directly from the console upon a kernel oops. Hardware JTAG/SWD Debuggers (like OpenOCD) are used in embedded Linux to halt the CPU at the hardware level.
1c: Crash Dump Analysis
When a driver causes a Kernel Panic, these tools analyze the aftermath. Objdump inspects object files. Kdump / Kexec boots a secondary kernel upon a crash to capture the memory state into a core dump file (vmcore). The crash utility is then used to navigate the vmcore dump, view backtraces, and pinpoint the exact line of code that failed.
1d: User-Space Interaction Tools
Because drivers interact directly with user-space applications, debugging system calls is necessary. strace monitors these calls, making it incredibly useful for verifying if open(), read(), write(), and ioctl() successfully reach the character device and what error codes they return.
Module 2: Advanced Device Operations & POSIX Semantics
Rationale: With debugging tools in hand, trainees expand their basic SCULL driver to behave like a robust, professional, and POSIX-compliant device. This introduces blocking behavior, which is foundational for the next module.
2a: The ioctl and check_flags Methods
Theory & Concept: Executing tasks beyond standard file I/O, such as formatting devices. This covers ioctl magic numbers, direction bits, and 32-bit vs 64-bit compatibility.
Live Demo: Implementing check_flags to handle dynamic fcntl modifications.
Lab: Add an ioctl command to wipe SCULL's memory footprint and implement check_flags to reject invalid changes.
2b: Device Flushing and POSIX Locking
Theory & Concept: Managing duplicated file descriptors. This explores the difference between release (called when the last reference is gone) and flush (called on every close).
Live Demo: Implementing flush to catch process exits, and lock to support flock() and fcntl() advisory locking.
Lab: Implement advisory locking inside the driver. Write a user-space app that fails elegantly if another app already holds an exclusive lock on /dev/scull0.
2c: Blocking I/O and Wait Queues
Theory & Concept: Standardizing sleep and wake behaviors. Reading an empty device should yield the CPU, not return EOF. This covers wait_queue_head_t and the "Thundering Herd" problem.
Lab: Create /dev/scullpipe (a producer-consumer queue). Readers block until a writer provides data, and writers block if the buffer is full.
Module 3: Multiplexing, Polling, and Asynchronous I/O
Rationale: Directly depends on the Wait Queues introduced in Module 2. Students learn how to integrate their blocking drivers into modern event loops.
3a: Multiplexed I/O (poll)
Theory & Concept: Interaction between standard select(), poll(), modern O(1) epoll, and drivers via poll masks (POLLIN, POLLOUT).
Live Demo: Implementing poll using poll_wait.
Lab: Write a user-space chat application using select() with two terminal windows communicating exclusively through /dev/scullpipe.
3b: High-Performance I/O Polling (iopoll)
Theory & Concept: Bypassing hardware interrupts for microsecond latency. Introduction to how the modern io_uring subsystem interacts with kernel drivers.
Live Demo: Overview of the iopoll API (crucial knowledge for modern kernel developers, even if rare for pure character devices).
3c: Asynchronous Notification (fasync)
Theory & Concept: Reversing the polling paradigm. The kernel aggressively sends a SIGIO or SIGPOLL signal to a user-space process the exact microsecond data becomes available.
Live Demo: Setting up fasync_struct and invoking kill_fasync.
Lab: Modify the chat application to use signal-driven Asynchronous I/O instead of blocking in a select() loop.
Module 4: Interrupts, Hardware, and Time
Rationale: Up to this point, wait queues and signals were driven by software (one process waking another). Now, students learn to drive these events using real-world, asynchronous hardware and timers.
4a: Hardware Interrupts & ISRs (Top Halves)
Theory & Concept: IRQ lines, APIC, and ARM GIC. Understanding why Interrupt Service Routines (Top Halves) must be blindingly fast and never sleep.
Live Demo: Using request_irq and free_irq to write an atomic Top Half.
Lab: Write a driver that registers a shared interrupt handler for the host's keyboard/mouse, logging a precise timestamp when it fires.
4b: Deferring Work - Bottom Halves (SoftIRQs & Tasklets)
Theory & Concept: Moving heavy processing out of the Top Half. Comparing softirqs vs. tasklets, and discussing Preempt-RT patchset behaviors.
Live Demo: Scheduling a tasklet from within an ISR.
Lab: Update the Interrupt driver so the Top Half acknowledges the interrupt and schedules a Tasklet to parse the data.
4c: Deferring Work - Workqueues
Theory & Concept: Overcoming Tasklet limitations (inability to sleep or allocate large memory). Exploring Concurrency Managed Workqueues (cmwq) which run in process context.
Live Demo: Creating custom workqueues and scheduling a struct work_struct.
Lab: Modify the bottom half to use a Workqueue, safely allocating memory using GFP_KERNEL and grabbing a Mutex.
4d: Kernel Timers and Time Management
Theory & Concept: Understanding jiffies, HZ, High-Resolution Timers (HRT), the classic jiffies wrap-around bug, and sleep/delay variants.
Live Demo: Setting up timer_list and add_timer.
Lab: Build a software "blinker" module that fires a callback every 500ms to periodically inject simulated hardware events into the wait queues.
Module 5: Modern Kernel Interfaces - /proc, sysfs & debugfs
Rationale: Now that the driver features complex states (wait queues, interrupt counts, locked files, and deferred work), students have meaningful metrics and statistics to expose safely to user-space.
5a: The /proc Filesystem and seq_file
Theory & Concept: The legacy /proc interface is notorious for buffer overruns. seq_file was invented to solve this using a safe, stateful iterator pattern.
Live Demo: Creating a /proc/scull_stats entry and writing the iterator methods.
5b: File Descriptor Info (show_fdinfo)
Theory & Concept: Exposing open-file details to user-space tools like lsof via /proc//fdinfo/.
Live Demo: Implementing show_fdinfo.
Lab: Add show_fdinfo to SCULL so admins can see exactly which file offset and access mode a specific process is currently holding.
5c: The Linux Device Model & sysfs & debugfs
Theory & Concept: The Unified Device Model, exploring kobjects, ksets, and hotplug events (uevents).
Lab: Expose SCULL's tunable configuration via /sys/class/scull/scull0/ using DEVICE_ATTR. Set up a debugfs node for dumping raw memory states.
Module 6: Mapping Memory - The mmap Interface
Rationale: This is a standalone, highly advanced topic focusing on performance optimization. It requires deep architectural understanding and serves as an excellent capstone.
6a: The Virtual Memory System & VMA
Theory & Concept: A deep dive into the Linux VM manager. Covering vm_area_struct (VMA), TLB, PMD, PTE, and ASLR.
Live Demo: Modifying SCULL to allocate page-aligned, contiguous physical memory blocks using alloc_pages.
6b: Implementing mmap
Theory & Concept: Implementing mmap using remap_pfn_range to inject kernel physical page frames directly into the user-space process's page table.
Lab: Allow a program to call mmap() on /dev/scull0. Write data to the mapped memory in user space and read it from kernel space via a /proc node, proving zero-copy transfer.
6c: Advanced Mapping (get_unmapped_area)
Theory & Concept: Handling strict hardware memory alignments by helping the kernel find suitable virtual addresses before mmap is called.
Live Demo: Overriding get_unmapped_area to enforce custom address boundaries for simulated hardware.
Would you like me to suggest specific assessment criteria or project milestones to evaluate the students' progress across these redesigned modules?