Windows Kernel Debugging and Rootkit Detection

Operating systems isolate core operations in kernel space (Ring 0) from standard applications in user space (Ring 3). Rootkits operate in kernel space, giving them total control over the OS and the ability to hide from user-mode security software.

To detect and analyze rootkits, security researchers use WinDbg. A typical setup involves two machines: a host debugger and a target VM linked via a virtual COM port or network debugging (KDNET). Using KDNET is the standard method today, offering high-speed kernel inspection.

The System Service Descriptor Table (SSDT) maps user-mode APIs to kernel-mode functions. Rootkits overwrite SSDT pointers to redirect kernel operations. For example, replacing the pointer for NtQuerySystemInformation allows a rootkit to filter the process list, hiding itself from Task Manager.

Using WinDbg, you can inspect the SSDT base and trace hooks:

lkd> x nt!KeServiceDescriptorTable
81f23ca0 nt!KeServiceDescriptorTable = <no type information>
lkd> dps nt!KeServiceDescriptorTable L4

DKOM rootkits do not hook functions; instead, they modify kernel structures in memory directly. Windows processes are represented by an EPROCESS structure containing an ActiveProcessLinks doubly-linked list. A DKOM rootkit removes its process node from the linked list. The process continues executing since the scheduler uses thread lists, but user tools traversing the process list fail to detect it.