Linux signals — sending system-level commands to programs

A user and SysAdmin guide

Let’s say you’ve got a program running that you want to terminate. Or that you want to power off or reboot the system, and all the running programs need to be terminated before either can properly complete.

How does that work?

Then there’s that stubborn program wouldn’t terminate no matter how many times you click the “X” button. Or the command line program that got hung up and wouldn’t complete or just close out.

How can that be fixed?

The answer to both those questions is, “signals.”

What are signals?

The Linux kernel uses signals to communicate with running programs and tell them what it wants them to do. Linux has 64 signals that can be used to communicate with running programs. For the most part, that does assume those programs are listening. The full list of signals can be viewed using the kill command. That is just a bit ironic since most signals have nothing to do with killing a process.

$ kill -l
 1) SIGHUP       2) SIGINT       3) SIGQUIT      4) SIGILL       5) SIGTRAP
 6) SIGABRT      7) SIGBUS       8) SIGFPE       9) SIGKILL     10) SIGUSR1
11) SIGSEGV     12) SIGUSR2     13) SIGPIPE     14) SIGALRM     15) SIGTERM
16) SIGSTKFLT   17) SIGCHLD     18) SIGCONT     19) SIGSTOP     20) SIGTSTP
21) SIGTTIN     22) SIGTTOU     23) SIGURG      24) SIGXCPU     25) SIGXFSZ
26) SIGVTALRM   27) SIGPROF     28) SIGWINCH    29) SIGIO       30) SIGPWR
31) SIGSYS      34) SIGRTMIN    35) SIGRTMIN+1  36) SIGRTMIN+2  37) SIGRTMIN+3
38) SIGRTMIN+4  39) SIGRTMIN+5  40) SIGRTMIN+6  41) SIGRTMIN+7  42) SIGRTMIN+8
43) SIGRTMIN+9  44) SIGRTMIN+10 45) SIGRTMIN+11 46) SIGRTMIN+12 47) SIGRTMIN+13
48) SIGRTMIN+14 49) SIGRTMIN+15 50) SIGRTMAX-14 51) SIGRTMAX-13 52) SIGRTMAX-12
53) SIGRTMAX-11 54) SIGRTMAX-10 55) SIGRTMAX-9  56) SIGRTMAX-8  57) SIGRTMAX-7
58) SIGRTMAX-6  59) SIGRTMAX-5  60) SIGRTMAX-4  61) SIGRTMAX-3  62) SIGRTMAX-2
63) SIGRTMAX-1  64) SIGRTMAX
root@testvm1:~#

Developers need to be aware of these signals, especially system-level developers. Signals can be used for inter-process communication (IPC) to do such things as synchronize the work of two or more threads. Some signals are designed to be designated with specific meaning by the developer for that inter-process communication. Signals can be used by a parent process to kill a child process. Each of these signals has a specific function though some of them can be defined by the receiving program using signal handlers.

Let’s narrow this down a bit and look at the most common signals that Linux recognizes.

• SIGHUP (Hangup) (1): Typically sent to indicate the termination of the controlling process. It often prompts daemons to reinitialize or restart.
• SIGINT (Interrupt) (2): Can be sent by pressing Ctrl+C. Usually terminates a process.
• SIGQUIT (Quit) (3): Similar to SIGINT but produces a core dump upon termination.
• SIGILL (Illegal Instruction) (4): Indicates an attempt to execute an illegal or undefined instruction.
• SIGABRT (Abort) (6): Sent by the abort(3) function, indicating abnormal termination requested by the process itself.
• SIGFPE (Floating-Point Exception) (8): Indicates an arithmetic error such as dividing by zero or floating-point overflow.
• SIGKILL (Kill) (9): Unconditionally terminates a process. Cannot be caught or ignored.
• SIGSEGV (Segmentation Fault) (11): Indicates an invalid memory reference or segmentation violation.
• SIGPIPE (Broken Pipe) (13): Sent to a process attempting to write to a pipe without a process reading from it.
• SIGALRM (Alarm Clock) (14): Sent by the alarm(2) system call or set timer (2) timer expiration.
• SIGTERM (Termination) (15): A termination request sent to a process typically causing it to exit successfully.
• SIGCHLD (Child Status Changed) (17): Sent to the parent process when a child process terminates or stops.
• SIGCONT (Continue) (18): Sent to instruct a stopped process to continue executing.
• SIGSTOP (Stop) (19): Stops a process for debugging or other administrative purposes. Cannot be caught or ignored.
• SIGTSTP (Terminal Stop) (20): Sent by pressing Ctrl+Z to temporarily stop a process running in the foreground.
• SIGTTIN (Background Read) (21): Sent to a background process attempting to read from its controlling terminal.
• SIGTTOU (Background Write) (22): Sent to a background process attempting to write to its controlling terminal.
• SIGUSR1 (User Signal 1) (10): A signal that can be used for user-defined purposes.
• SIGUSR2 (User Signal 2) (12): Another signal available for user-defined purposes.

This list is a little less terse than the list in the signals(7) man page, but you should read that, too, for additional information.

Which signals do we care about

For SysAdmins — which is my personal viewpoint — we usually need to be familiar with only a few of those signals.

• SIGTERM (15) – Signal 15, SIGTERM is the default signal sent by top and the other monitor programs when the “k” key is pressed. It may also be the least effective because the program must have a signal handler built into it. The program’s signal handler must intercept incoming signals and act accordingly. So for scripts, most of which do not have signal handlers, SIGTERM is ignored. The idea behind SIGTERM is that by simply telling the program that you want it to terminate itself, it will take advantage of that and clean up things like open files and then terminate itself in a controlled manner.
• SIGKILL (9) – Signal 9, SIGKILL provides a means of killing even the most recalcitrant programs, including scripts and other programs that have no signal handlers. For scripts and other programs with no signal handler, however, it not only kills the running script but it also kills the shell session in which the script is running; this may not be the behavior that you want. If you want to kill a process and you don’t care about being nice, this is the signal you want. This signal cannot be intercepted by a signal handler in the program code.
• SIGINT (2) – Signal 2, SIGINT can be used when you want the program to die a little more nicely, for example, without killing the shell session in which it is running. SIGINT sends an interrupt to the session in which the program is running. This is equivalent to terminating a running program, particularly a script, with the Ctrl-C key combination.

Many utilities like kill, pgrep, pkill, top, atop, and htop utilities allow you to send signals to running processes.

The kill command can also be used to send signals to processes outside of the monitors. The kill -l command can be used to list all possible signals that can be sent. The use of the kill command to send signals can be confusing if you do not actually intend to kill the process. The thing to remember is that the kill command is used to send signals to processes and that at least three of those signals can be used to terminate the process with varying degrees of prejudice.

There are many other signals but these are the ones I have found that pertain to terminating a program. They are actually the only ones I use as a SysAdmin.

Signals in action

To experiment with this, open a terminal session as a non-root user, and create a file in /tmp named cpuHog and make it executable with the permissions rwxr_xr_x (755). Add the following content to the file.

#!/bin/bash 
# This little program is a cpu hog 
X=0;while [ 1 ];do echo $X;X=$((X+1));done

Open another terminal session in a different window, position them adjacent to each other so you can watch the results and run top in the new session. Run the cpuHog program with the following command.

$ /tmp/cpuHog

This program simply counts up by one and prints the current value of X to STDOUT. And it sucks up CPU cycles. The terminal session in which cpuHog is running should show a very high CPU usage in top. Observe the effect this has on system performance in top. CPU usage should immediately go way up and the load averages should also start to increase over time. If you want, you can open additional terminal sessions and start the cpuHog program in them so that you have multiple instances running but that’s not necessary for this little experiment.

Determine the PID of the cpuHog program you want to kill. The PID for my process is 265140 but the PID on your host will be different.

$ pgrep -f cpuHog
666350
$

We could kill the cpuHog with pkill, but we will use the kill command, then verify that it has been killed. If no signal is specified, the kill command sends the TERM (15) signal. Most running programs will respond to this signal.

$ kill 666350
$ pgrep -f cpuHog
$ 

Here’s what this looks like in the terminal session in which cpuHog was running.

<SNIP>
59995161
59995162
Terminated

The man page says that processes can be killed by name, but I found that to be untrue — at least for Bash scripts like cpuHog. Restart the cpuHog and then try to kill it by name.

$ kill cpuHog
bash: kill: cpuHog: arguments must be process or job IDs

Thoughts on signals

Most SysAdmins seldom need to consider signals. We use the tools that send signals to programs, but don’t think about what’s really happening underneath. And even when we do think about signals, we really only care about two or three of them.

If a process has not been terminated by signal TERM (15), it can usually be killed using signal KILL (9). However I’ve seen a couple exceptions to this over the years where a program just won’t terminate. This usually occurs when a parent process has been killed but was unable to kill all its child processes. Most times these processes show up in tools like top as “Zombies.”

Bash has provisions for creating code that can handle incoming signals. For example it can allow the Bash program to perform some cleanup before responding to a TERM signal. But that is beyond the scope of this article.