Understanding a Kernel Oops!

Understanding a kernel panic and doing the forensics to trace the bug is considered a hacker’s job. This is a complex task that requires sound knowledge of both the architecture you are working on, and the internals of the Linux kernel. Depending on type of error detected by the kernel, panics in the Linux kernel are classified as hard panics (Aiee!) and soft panics (Oops!). This article explains the workings of a Linux kernel ‘Oops’, helps to create a simple version, and then debug it. It is mainly intended for beginners getting into Linux kernel development, who need to debug the kernel. Knowledge of the Linux kernel, and C programming, is assumed.

An “Oops” is what the kernel throws at us when it finds something faulty, or an exception, in the kernel code. It’s somewhat like the segfaults of user-space. An Oops dumps its message on the console; it contains the processor status and the CPU registers of when the fault occurred. The offending process that triggered this Oops gets killed without releasing locks or cleaning up structures. The system may not even resume its normal operations sometimes; this is called an unstable state. Once an Oops has occurred, the system cannot be trusted any further.

Let’s try to generate an Oops message with sample code, and try to understand the dump.

Setting up the machine to capture an Oops

The running kernel should be compiled with CONFIG_DEBUG_INFO, and syslogd should be running. To generate and understand an Oops message, Let’s write a  sample kernel module, oops.c:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19

#include <linux/kernel.h> #include <linux/module.h> #include <linux/init.h> static void create_oops() {         *(int *)0 = 0; } static int __init my_oops_init(void) {         printk("oops from the module\n");         create_oops();        return (0); } static void __exit my_oops_exit(void) {         printk("Goodbye world\n"); } module_init(my_oops_init); module_exit(my_oops_exit);

The associated Makefile for this module is as follows:

obj-m   := oops.o KDIR    := /lib/modules/$(shell uname -r)/build PWD     := $(shell pwd) SYM=$(PWD) all:         $(MAKE) -C $(KDIR) SUBDIRS=$(PWD) modules

Once executed, the module generates the following Oops:

BUG: unable to handle kernel NULL pointer dereference at (null) IP: [<ffffffffa03e1012>] my_oops_init+0x12/0x21 [oops] PGD 7a719067 PUD 7b2b3067 PMD 0 Oops: 0002 [#1] SMP last sysfs file: /sys/devices/virtual/misc/kvm/uevent CPU 1 Pid: 2248, comm: insmod Tainted: P           2.6.33.3-85.fc13.x86_64 RIP: 0010:[<ffffffffa03e1012>]  [<ffffffffa03e1012>] my_oops_init+0x12/0x21 [oops] RSP: 0018:ffff88007ad4bf08  EFLAGS: 00010292 RAX: 0000000000000018 RBX: ffffffffa03e1000 RCX: 00000000000013b7 RDX: 0000000000000000 RSI: 0000000000000046 RDI: 0000000000000246 RBP: ffff88007ad4bf08 R08: ffff88007af1cba0 R09: 0000000000000004 R10: 0000000000000000 R11: ffff88007ad4bd68 R12: 0000000000000000 R13: 00000000016b0030 R14: 0000000000019db9 R15: 00000000016b0010 FS:  00007fb79dadf700(0000) GS:ffff880001e80000(0000) knlGS:0000000000000000 CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 0000000000000000 CR3: 000000007a0f1000 CR4: 00000000000006e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process insmod (pid: 2248, threadinfo ffff88007ad4a000, task ffff88007a222ea0) Stack: ffff88007ad4bf38 ffffffff8100205f ffffffffa03de060 ffffffffa03de060  0000000000000000 00000000016b0030 ffff88007ad4bf78 ffffffff8107aac9  ffff88007ad4bf78 00007fff69f3e814 0000000000019db9 0000000000020000 Call Trace: [<ffffffff8100205f>] do_one_initcall+0x59/0x154 [<ffffffff8107aac9>] sys_init_module+0xd1/0x230 [<ffffffff81009b02>] system_call_fastpath+0x16/0x1b Code: <c7> 04 25 00 00 00 00 00 00 00 00 31 c0 c9 c3 00 00 00 00 00 00 00 RIP  [<ffffffffa03e1012>] my_oops_init+0x12/0x21 [oops] RSP <ffff88007ad4bf08> CR2: 0000000000000000

Understanding the Oops dump

Let’s have a closer look at the above dump, to understand some of the important bits of information.

BUG: unable to handle kernel NULL pointer dereference at (null)

The first line indicates a pointer with a NULL value.

IP: [<ffffffffa03e1012>] my_oops_init+0x12/0x21 [oops]

IP is the instruction pointer.

Oops: 0002 [#1] SMP

This is the error code value in hex. Each bit has a significance of its own:

• bit 0 == 0 means no page found, 1 means a protection fault
• bit 1 == 0 means read, 1 means write
• bit 2 == 0 means kernel, 1 means user-mode
• [#1] — this value is the number of times the Oops occurred. Multiple Oops can be triggered as a cascading effect of the first one.

CPU 1

This denotes on which CPU the error occurred.

Pid: 2248, comm: insmod Tainted: P           2.6.33.3-85.fc13.x86_64

The Tainted flag points to P here. Each flag has its own meaning. A few other flags, and their meanings, picked up from kernel/panic.c:

• P — Proprietary module has been loaded.
• F — Module has been forcibly loaded.
• S — SMP with a CPU not designed for SMP.
• R — User forced a module unload.
• M — System experienced a machine check exception.
• B — System has hit bad_page.
• U — Userspace-defined naughtiness.
• A — ACPI table overridden.
• W — Taint on warning.

RIP: 0010:[<ffffffffa03e1012>]  [<ffffffffa03e1012>] my_oops_init+0x12/0x21 [oops]

RIP is the CPU register containing the address of the instruction that is getting executed. 0010 comes from the code segment register. my_oops_init+0x12/0x21 is the <symbol> + the offset/length.

RSP: 0018:ffff88007ad4bf08  EFLAGS: 00010292 RAX: 0000000000000018 RBX: ffffffffa03e1000 RCX: 00000000000013b7 RDX: 0000000000000000 RSI: 0000000000000046 RDI: 0000000000000246 RBP: ffff88007ad4bf08 R08: ffff88007af1cba0 R09: 0000000000000004 R10: 0000000000000000 R11: ffff88007ad4bd68 R12: 0000000000000000 R13: 00000000016b0030 R14: 0000000000019db9 R15: 00000000016b0010

This is a dump of the contents of some of the CPU registers.

Stack: ffff88007ad4bf38 ffffffff8100205f ffffffffa03de060 ffffffffa03de060  0000000000000000 00000000016b0030 ffff88007ad4bf78 ffffffff8107aac9  ffff88007ad4bf78 00007fff69f3e814 0000000000019db9 0000000000020000

The above is the stack trace.

Call Trace: [<ffffffff8100205f>] do_one_initcall+0x59/0x154 [<ffffffff8107aac9>] sys_init_module+0xd1/0x230 [<ffffffff81009b02>] system_call_fastpath+0x16/0x1b

The above is the call trace — the list of functions being called just before the Oops occurred.

Code: <c7> 04 25 00 00 00 00 00 00 00 00 31 c0 c9 c3 00 00 00 00 00 00 00

The Code is a hex-dump of the section of machine code that was being run at the time the Oops occurred.

Debugging an Oops dump

The first step is to load the offending module into the GDB debugger, as follows:

[root@DELL-RnD-India oops]# gdb oops.ko GNU gdb (GDB) Fedora (7.1-18.fc13) Reading symbols from /code/oops/oops.ko...done. (gdb) add-symbol-file oops.o 0xffffffffa03e1000 add symbol table from file "oops.o" at     .text_addr = 0xffffffffa03e1000

Next, add the symbol file to the debugger. The add-symbol-file command’s first argument is oops.o and the second argument is the address of the text section of the module. You can obtain this address from /sys/module/oops/sections/.init.text (where oops is the module name):

(gdb) add-symbol-file oops.o 0xffffffffa03e1000 add symbol table from file "oops.o" at     .text_addr = 0xffffffffa03e1000 (y or n) y Reading symbols from /code/oops/oops.o...done.

From the RIP instruction line, we can get the name of the offending function, and disassemble it.

(gdb) disassemble my_oops_init Dump of assembler code for function my_oops_init:    0x0000000000000038 <+0>:    push   %rbp    0x0000000000000039 <+1>:    mov    $0x0,%rdi    0x0000000000000040 <+8>:    xor    %eax,%eax    0x0000000000000042 <+10>:    mov    %rsp,%rbp    0x0000000000000045 <+13>:    callq  0x4a <my_oops_init+18>    0x000000000000004a <+18>:    movl   $0x0,0x0    0x0000000000000055 <+29>:    xor    %eax,%eax    0x0000000000000057 <+31>:    leaveq    0x0000000000000058 <+32>:    retq End of assembler dump.

Now, to pin point the actual line of offending code, we add the starting address and the offset. The offset is available in the same RIP instruction line. In our case, we are adding 0x0000000000000038 + 0x012 =  0x000000000000004a. This points to the movl instruction.

(gdb) list *0x000000000000004a 0x4a is in my_oops_init (/code/oops/oops.c:6). 1    #include <linux/kernel.h> 2    #include <linux/module.h> 3    #include <linux/init.h> 4     5    static void create_oops() { 6        *(int *)0 = 0; 7    }

This gives the code of the offending function.

References

The kerneloops.org website can be used to pick up a lot of Oops messages to debug. The Linux kernel documentation directory has information about Oops — kernel/Documentation/oops-tracing.txt. This, and numerous other online resources, were used while creating this article.

Cyber Attacks Explained: DoS and DDoS

The fundamental technique behind a DoS attack is to make the target system busy. In a computer server, when a network packet is being received, all components (right from the network interface card or NIC to the application running under the OS) are participating to ensure successful delivery of that packet. The NIC must monitor the Ethernet frames meant for it, align data and pass it to the network card driver, which then adds its own intelligence and passes it to the OS, which takes it to the application.

Each component involved can exhibit some form of vulnerability, and DoS attacks are devised to exploit one or more of these, to penetrate into the system.

Let’s now understand the basics of the TCP/IP protocol, which uses a handshake between the sender and receiver. Figure 1 shows how a healthy TCP handshake takes place, and how a SYN flood attack compares with it.

Figure 1: A healthy TCP handshake

When the sender wants to communicate, it sends a SYN packet with its own IP address as the source, and the receiver’s IP address as the destination. The receiver responds with a SYN-ACK packet. The sender confirms this by sending an ACK packet. Now, sender and receiver have a guarantee that they can communicate with each other.

The sender then starts sending the actual data in small chunks, and for each data packet received, the receiver sends an ACK back. When the sender sends the final data chunk, it sends a FIN signal, which is acknowledged by the receiver by sending a FIN-ACK back.

If a particular port is not supposed to respond to the request, the receiver responds with an RST packet, which means it is rejecting that request.

As you can see here, the TCP/IP stack software has to deal with complex communication, which does take some CPU and memory resources. To add to this, many handshakes are happening on a server for different source and destination addresses and for various TCP ports. All IP-based protocols such as ICMP ping, telnet, FTP, etc, actually piggyback on this framework to do their job, each working on a different dedicated socket or port.

At the application layers, the OS and the application receiving or transmitting data allocate internal memory buffers and a software process to keep track of what is being sent or received. The OS partially does this job itself, and leaves the rest to the network driver and protocol stack. Each process consumes some CPU time and memory resources.

A DoS attack exploits this situation, by tweaking TCP packets to make the server respond to malicious, fabricated network requests. TCP packets can be forged, or modified to disrupt the basic handshake process, in order to create unexpected network responses. This ultimately results in exhausting all the server resources, which when overwhelmed, stops responding.

There are various ways to do this, each using a different technical approach. Please refer to the following table — it shows you how various DoS techniques map to the OSI model of network layers.

Application	Web DoS, Email Spam
Presentation	Malformed SSL Requests
Session	Telnet DDoS
Transport	SYN Flood, Smurf Attack
Network	ICMP Flooding
Data	MAC Flooding
Physical	Dummy Packet Attack

We will now discuss each of these techniques in more detail.

Network layer DoS attacks

The MAC flood

This is a rare Layer 1 attack, in which the attacker sends multiple dummy Ethernet frames, each with a different MAC address. Network switches treat MAC addresses separately, and hence reserve some resources for each request. When all the memory in a switch is used up, it either shuts down or becomes unresponsive.

In a few types of routers, a MAC flood attack may cause these to drop their entire routing table, thus
disrupting the whole network under its routing domain.

The SYN flood

The attacker sends multiple SYN packets; upon receiving SYN-ACK from the target, it does not send ACK, but instead sends more SYN packets. This leaves the TCP/IP stack on the target to conclude that there is a possible network congestion or disconnect, and it waits for a specific amount of time. Thus, multiple partially-open connections are maintained by the stack for some time, in anticipation of an ACK response.

In another SYN flooding type, the SYN is sent with a spoofed source address, which becomes the destination address for the target’s SYN-ACK packet. However, since the system at the spoofed IP never sent that SYN to begin with, it will never send an ACK to this packet, and will simply drop it. The target system is not aware of this, and keeps track of it in anticipation of an ACK.

In both examples, the connection tables and memory resources on the targeted network components are filled up with bogus entries. When the entire table is filled up, the device stops responding.

The ping of death

In this case, a malformed ping packet flood is sent to the target. Since the TCP stack responds only to a certain type of ping packet, it fails to respond to this, exhausting the system resources.

TCP established connection attacks

This is an extension to the SYN flood, the difference being that it uses a complete 3-way TCP handshake. It does not spoof addresses, nor strip the ACK responses, but just establishes a TCP connection and simply does not send any data. Due to this, connections are maintained till time-out; a flood of such connections results in a DoS on the targeted device.

Since the handshake is gracefully done, this attack is difficult to trace, and needs advanced techniques to detect and stop.

Smurf attacks

This is a famous, widely used Layer-3 attack in which the attacker sends ping traffic to IP broadcast addresses. However, the source address is spoofed, and is of the victims’ machines. Thus, routers deliver replies to the victims, which send back with a ping response. On a larger and populated subnet, this can have a devastating effect, and can practically render the routing device non-responsive.

Similar to this is the Fraggle attack, wherein a UDP echo packet is sent instead of TCP

TCP RST attacks

In this new breed of DoS attack, the source IP is spoofed with the victim’s IP address, and this malformed packet is sent to a firewall. This forces the firewall to remember this connection for some time.

This attack is rarely used; its sole purpose is to fool the intrusion detection logic on cheaper firewalls. Unlike the host desktop, a firewall with UTM features works in promiscuous mode, and takes note of each and every packet. However, if the anomaly detection logic is not smart enough, it simply results in piling up connections and eventually rendering itself useless.

Application layer DoS attacks

Besides these network-layer attacks, there are a few that deal with the application layer directly. Such attacks are comparatively easy to detect and fix, but if ignored, can result in heavy downtime.

Application-layer attacks exploit vulnerabilities in the OS and the guest application. Listed below are a few such attacks.

Buffer overflows

This very well-known attack pushes the OS to consume resources to such an extent that it starts leaking memory, becomes sluggish or simply stops responding. It is a myth that a buffer overflow is observed only in Windows OS — in fact, it is very much true for Linux distros too. Applications such as a database, email and Web servers are found to have buffer overflow vulnerabilities.

Web and DNS DoS attacks

Web servers running on TCP port 80 are a common target for DoS attacks. Attackers usually send multiple HTTP requests (not malformed at all) targeted to retrieve enormous amounts of data from the backend database server.

Such request floods make the database server busy, keeping the Web server waiting for data. This creates a pile-up on both servers, which become unresponsive to further requests. This can happen unintentionally too, especially when breaking news is posted and everyone tries to access it at the same time. In case of DNS servers, the attack method is similar to Web servers, but it has serious consequences. If a DNS server becomes non-responsive, it can take down the firm’s entire network.

Distributed DoS attacks

In simple terms, a distributed DoS attack (DDoS) combines multiple attackers using the various techniques discussed above, and can result in catastrophic failure. Please refer to Figure 2 to understand how a typical DDoS attack is planned on a website.

Figure 2: A typical DDoS attack

A lethal combination of spoofed SYN flood and old-style ping-of-death attacks is typically used to disrupt an IT network that is open to the Internet.

In a modern form of DDoS, the attacker injects malicious code into a virus and spreads it to millions of computers, making them zombie machines. On a specific day and time, all infected machines start executing that code, which is usually written to access a website or plant a network-level attack onto a targeted system. Since it is difficult to find out who injected the code into a virus, it is difficult to trace the real attacker.

In another non-hazardous form, a DDoS attack is modified to access the attacker’s website and click advertisements on it, which in turn helps the attacker earn money from ad clicks.

Protecting FOSS systems

Though DoS is one of the oldest and most commonly known attacks, unfortunately, there is no fool-proof solution to stop it because practically, it is difficult to decide which network connection is legitimate, and which one is initiating an attack.

While there are specific tools for a particular type of DoS attack, it boils down to cyber-security design and monitoring to strengthen the network.

Typical symptoms of a DoS attack on a Linux server are a sluggish system or a slow website, sudden and prolonged increase in processor and memory utilisation, excessive disk thrashing without any business activity, slower file transfers, etc.

On a network monitoring system, there could be a large number of TCP packet drops, abnormal TCP resets, broken TCP SYN packets being received, or duplicate ACK packets being sent. Usually, the first-level component impacted is a router, followed by a firewall and other components like switches. Firewalls cannot protect the router, but the firmware of modern routers (e.g., Cisco 7600 or X443) contain patches to protect against DoS attacks. Though modern firewalls have many features to combat DoS attacks, they don’t always help.

Firewalls can certainly protect against network layer-based attacks, but usually fail to protect systems from application-layer attacks like on port 80 (HTTP). Here, an application-level firewall is needed to filter each request and ensure its legitimacy.

For Ubuntu and RHEL, an APF (Advanced Policy Firewall) is a great tool that can help mitigate DoS attacks. Linux FOSS systems are blessed with fantastic network drivers, as well as many built-in features such as a packet-filtering firewall, packet monitoring, network monitoring tools, kernel hardening tools, etc.

For smaller Linux networks, a nice script can be written to SYN Trap open connections and to stop bogus TCP RST connections, as a first line of defence.

For mission-critical corporate Linux networks, deploying an Intrusion Prevention System device (IPS) is the best choice. IPS devices sit on a network in promiscuous mode and use built-in anomaly detection algorithms to intercept and decode each and every packet. Since its intelligence ranges from Layer 2 to Layer 7, it can gauge which packet is legitimate, and which isn’t. It has a great alerting mechanism
to proactively inform and stop DoS and DDoS attacks.

A good combination of IPS devices, UTM firewalls and application layer security can help stop these dreaded attacks.