Frozen Linux System? Here are 3 Ways to Deal With It

Linux is known to be very stable, but even the most stable system can encounter situations that cause it to become unresponsive. Then comes the awful feeling as you push and hold “Power”. One can attempt a few tricks with Linux in such a case. At this point, we are not looking to save any unsaved work. We want to find a way to avoid a reboot or at least provide the cleanest reboot possible to avoid corruption.

For the purpose of this article, Linux can be seen as a set of 2 stacks: The Kernel Stack and The User Stack. Somewhere inside the User Stack, you have programs running. More than likely X server, a window manager and some programs. You can imagine that list of programs, including the X server, as another stack inside the User Stack. This concept will help newer Linux users understand some of the tasks we will perform later.

1. Kill an unresponsive program with xkill

If you are using a Linux desktop, your first step should be an attempt to use your window manager’s execute command (alt+f2 in Ubuntu) feature. If you are lucky, it will popup sooner then later.

Then you can execute xkill and point the cursor at the defunct window and click it away.

Xkill works faster than the traditional X icon or right-clicking the window on the taskbar and clicking close. Xkill kills instantly without questions asked.

2. Revive a frozen Linux system with virtual consoles (TTY)

Sometimes, you will not be so lucky, the system may seem completely dead.

This could just be an illusion. Remember the stack of programs inside the user stack? Let’s work with this image for a little bit.

While it may seem like the root of the User Stack would be X. It is not.

X Window System runs on a Virtual Console (VC), also referred to as TTY. They are represented on the filesystem as /dev/ttyX. Linux has multiple VC’s available to you. Attempting to access another one can give you back control.

You can request a different VC by pressing CTRL+ALT+F{1,6} from a GUI environment or ALT+F{1,6} from another VC. The above keystroke gives you access to VC 1 through 6. Graphical interface runs on VC 7+.

When dealing with an unstable system, even requesting a change of VC can take a little time, so be a little patient.

If your attempt to change the virtual console is successful, you will be prompted for your user and password. You will then have a text-mode console. Your system should regain some stability here; this is in parts because the virtual console where X and everything else is running is not a priority anymore.

Once you are logged in you have a few options.

• You can attempt to kill a defunct program with the kill or killall command.
• You can kill the window manager, X server or login manager. This will kill all child processes and, if all goes well, your login manager will restart presenting you with your graphical login screen.

3. When everything else fails, talk with the Kernel using SysReq

Sometimes, even switching to a TTY does not work. The whole system is dead. Don’t give up yet, once again, more than likely an illusion. Linux holds one more feature for situations like this.

If compiled in the Kernel and enabled in a file, Linux provides you with the means to send preset requests directly to the kernel. This is useful in a situation when the entire user stack is dead. It is known as the Magic SysRq key. Compiled and enabled by default in most distributions, only a Kernel Panic should be able to prevent this from working.

To confirm the option was set when the kernel was compiled, you can attempt to locate your current kernel’s config file in /boot. Look for the option CONFIG_MAGIC_SYSRQ. As previously mentioned it is compiled and enabled in most distributions. To confirm it is enabled, run the following:

sysctl kernel.sysrq

On older systems, you had 2 options: 0 & 1. It was either OFF or ON. On newer ones, there is more fine control over which commands are enabled.

Any value above 1 means some commands are enabled. More details can be found here.

To set a different value temporarily, run the following command:

sudo sysctl -w kernel.sysrq=value

To change this value permanently, edit /etc/sysctl.conf. Search for kernel.sysrq, and set the desired value.

The easiest way to see if it works is to try. I also recommend testing and learning a set of keystrokes while the system is stable. When you execute a SysRq call, the kernel will attempt to respond. This response can only be seen if you are in a virtual console 1 through 6. You will not see the responses if the focus cannot be moved away from X. This does not mean it is not working.

Numerous commands can be sent. The full list can be seen here. I recommend memorizing the following:

Keys	Description
Alt + SysRq + r	Takes the keyboard out of raw mode, taking control away from X
Alt + SysRq + e	Send SigTerm to all process, giving them a chance to quit gracefully
Alt + SysRq + i	Send SigKill to all process
Alt + SysRq + k	Kill all process in current virtual console
Alt + SysRq + s	Sync all mounted filesystem, flushing all data to disk
Alt + SysRq + u	Remount all filesystem read-only
Alt + SysRq + b	Reboot system instantly, does not sync or unmount
Alt + SysRq + o	Shutdown system

This short list of commands gives you a powerful new tool when dealing with an unstable system. When sending request, if you are unable to see the kernel’s responses, give a few seconds in between commands. Pay attention to HDD light when requesting HDD operation.

There is 2 sequence that are worth remembering.

Alt + SysRq + r e k

This sequence is good to restart an unresponsive X server. You first get the keyboard out of raw mode with Alt + SysRq + r. You then request to send a SIGTERM to all processes with Alt + SysRq + e. X server may restart with this request. If so, you have rescued your system. If not, proceed with the third request, Alt + SysRq + k. Which will send SIGKILL to all processes within the current virtual console.

What is SIGTERM? What’s the difference between SIGKILL & SIGTERM?

Both SIGTERM and SIGKILL are used for killing a process in Linux. But you should prefer using SIGTERM. Here’s why!

Linux HandbookAbhishek Prakash

If your system is still unresponsive after the above sequence, you should proceed to this next sequence.

Alt + SysRq + r e i s u b

This last command will initiate an emergency reboot.

• r : Get keyboard out of raw mode
• e : Send SIGTERM to all processes
• i : Send SIGKILL to all processes
• s : Sync all mounted filesyste
• u : Emergency remount read only
• b : Instantly reboot system

You are now equipped with the best tool against an unresponsive system that I know. You may follow this video from my YouTube channel for a demonstration.

I hope you learned something new about dealing with a frozen Linux system with this article. I would like to know your feedback on it. Please use the comment section and share your thoughts.