The Machine That Lied
A modern Linux distribution is designed to make you feel like you understand what is happening. The installer is clean. The defaults are sane. The desktop loads. The package manager works. The system updates. The machine reboots. Everything feels obvious.
It is not obvious. It is hidden.
The firmware found something bootable. The bootloader handed control to the kernel. The kernel found the root filesystem. The init system started. Services came alive. The display manager loaded. The login screen appeared. A whole chain of events had to happen in the correct order for you to see that friendly little desktop. Most people never meet the chain. They only meet the result.
Then one day they edit the wrong file. Maybe it is /etc/fstab. Maybe it is a bad kernel flag. Maybe GRUB points to the wrong place. Maybe the initramfs is missing something it needed. Maybe /boot is mounted incorrectly and the system has been pretending everything is fine for weeks. Then they reboot. And the machine stops.
No desktop. No friendly installer. No beautiful abstraction. Just a prompt. A rescue shell. A blinking cursor. A system that was working five minutes ago and now refuses to explain itself politely. That is when Linux becomes real.
Suddenly UUIDs matter. Suddenly mount points matter. Suddenly the kernel matters. Suddenly the bootloader matters. Suddenly one line in one plain text file decides whether your machine is a computer or a paperweight. This is why GRUB matters. Not as trivia. Not as a certification exam checkbox. Not because suffering makes you noble. Because when the machine stops lying, GRUB is where the conversation begins.
The Boot Chain Every Linux User Should Be Able to Recite
Before GRUB makes sense, the sequence has to make sense. When you press the power button, control passes through several distinct layers before you see a login prompt. Each layer depends on the one before it. Each failure strands you at a different depth.
The firmware runs first. On older machines this is BIOS. On anything built in the last decade it is UEFI. UEFI knows about the EFI System Partition, a small FAT32 partition holding bootloader files. BIOS knows only about the Master Boot Record, the first 512 bytes of the disk. Either way, the firmware is doing exactly one thing: finding something to hand execution to.
Then GRUB takes over.
GRUB is not one program. It is a two-stage bootloader. Stage 1 is microscopic, a few hundred bytes that fit in the MBR or the EFI partition's boot entry. Its only job is to find and load stage 2. Stage 2 is the full bootloader, the one that reads grub.cfg, displays the menu, and loads the kernel. It lives in /boot/grub/.
Then the kernel takes over.
The kernel is not a finished system when it first loads. It cannot see your filesystems, your hardware drivers, or your init system. It needs help. That help is the initramfs: a compressed archive containing a minimal root filesystem, just enough to mount the real root partition and hand off control. Once the real root is mounted, the initramfs disappears and systemd takes over.
Then the system you recognize appears.
Most people have used Linux for years without knowing this sequence. That is fine, until it is not.
What GRUB Actually Is
GRUB stands for GRand Unified Bootloader. The current version is GRUB 2, which is entirely different from the original GRUB, now called GRUB Legacy. Most modern systems use GRUB 2. The distinction matters because their configuration files, directory structures, and rescue procedures are completely different.
On a BIOS system, GRUB stage 1 is installed into the Master Boot Record. The MBR is the very first sector of the disk, 512 bytes total. The first 446 bytes hold the bootstrap code. The next 64 bytes hold the partition table. The last 2 bytes are the boot signature. GRUB stage 1 fits in those 446 bytes and does exactly one thing: load stage 2.
On a UEFI system, GRUB installs as an EFI application into the EFI System Partition. The firmware reads the UEFI boot entries and loads the appropriate .efi file. This is cleaner and more recoverable than the MBR approach, and it is why modern systems can have multiple bootloaders side by side without conflicts.
Stage 2 lives in /boot/grub/. This directory contains the GRUB modules, files with a .mod extension, along with fonts, themes, and most importantly, grub.cfg. The modules allow GRUB to read different filesystem types, support different partition schemes, and load cryptographic tools. GRUB dynamically loads modules as needed, which is why the /boot/grub directory can be surprisingly large.
grub.cfg is the file that controls what you see when GRUB starts. It specifies the menu entries, the default boot option, the timeout, and the kernel parameters. On most distributions it is auto-generated by grub-mkconfig and should not be edited directly. The actual source of truth is /etc/default/grub and the scripts in /etc/grub.d/.
The generated grub.cfg uses UUIDs to identify partitions, not device names. There is a reason for this. Device names like /dev/sda can change depending on what other disks are attached and the order the kernel discovers them. UUIDs do not change. If grub.cfg says root=UUID=abc123, GRUB will find that partition regardless of what device name it has today. If that UUID does not exist on any disk, GRUB will stop and tell you exactly that.
The Menu You See
When GRUB starts successfully, you see a menu. On most desktop distributions this menu is hidden by default with a timeout of zero, designed to be invisible unless you hold Shift during boot on BIOS systems or ESC on UEFI. This is a deliberate choice by distribution maintainers who decided that most users have no reason to see the bootloader. They are right for most users and completely wrong for the moment something breaks.
The GRUB menu is not read-only. Press E to edit any entry before it boots, which gives you direct access to the kernel command line. You can add or remove kernel parameters, change the root device, or force a different runlevel. Changes made this way are not persistent. They last only for that boot.
Press C to enter the GRUB command line directly. This is where recovery begins when grub.cfg is missing or broken.
The E key is the most important key in the GRUB menu. Every Linux user should know it exists. Pressing it on a Linux entry reveals the full boot configuration: the kernel path, the initramfs path, and every kernel parameter. Remove quiet and splash to see every kernel message during boot. Add init=/bin/bash to drop directly into a root shell without starting systemd, bypassing authentication on a physical machine.
Boot Timing: Where the Seconds Go
A modern Linux system from power button to login prompt can take anywhere from three seconds on NVMe-backed minimal systems to ninety seconds on spinning disk with a heavy service stack. Most desktop users see between eight and twenty-five seconds. The time does not vanish randomly. Each stage accounts for a measurable portion of it.
Firmware initialization is fixed hardware overhead you cannot optimize away without changing the hardware. On older BIOS systems this includes the Power-On Self-Test. On UEFI systems it includes enumeration of connected hardware. Budget one to four seconds here, sometimes more on older server hardware with many PCIe slots.
GRUB itself is fast. Reading grub.cfg, displaying the menu, and loading the kernel typically takes well under a second, plus whatever timeout you have configured. If you see a long pause at the GRUB menu, it is almost always the countdown timer waiting for user input before proceeding to the default entry. Set GRUB_TIMEOUT=0 in /etc/default/grub if you want to eliminate it.
The initramfs phase is variable. The initramfs has to run its own discovery process, find the real root partition, possibly decrypt it with LUKS, possibly assemble it with LVM or RAID, and then perform the switch_root operation. Two to eight seconds is common, sometimes more with complex storage configurations.
systemd is where most of the remaining time goes. It starts services in parallel but cannot start a service before its dependencies. The journald and udev services start first. Anything that depends on the network must wait for network-online.target, which can add several seconds on slower networks or when waiting for DHCP. Run systemd-analyze blame after boot to see which services took the longest.
When GRUB Fails
There are exactly three states GRUB can be in when something goes wrong. Normal mode, where the full bootloader loaded successfully but could not proceed. Rescue mode, where stage 1 loaded but stage 2 could not be found. And not there at all, where the firmware could not find a valid bootloader entry.
Normal mode still has the full GRUB command line, all modules, filesystem support, and the ability to manually boot a kernel.
Rescue mode has almost nothing. A bare prompt, three commands, and no filesystem support beyond what stage 1 can provide. You have to find /boot/grub/ manually, load the normal module, and then use the normal module to get to the full interface.
The most common failure is the wrong UUID in grub.cfg. This happens when you clone a disk, reinstall a system, resize a partition, or recreate a filesystem. The partition gets a new UUID. GRUB still has the old one. The machine boots into GRUB normally, displays the menu normally, tries to boot, and fails because the kernel cannot find its root partition. The kernel calls this a VFS panic.
The second most common failure is a missing /boot partition. On systems where /boot is a separate partition, if that partition is not correctly listed in fstab, grub-mkconfig will succeed because it runs on the mounted system, but the generated paths will be wrong. The system boots fine until you update the kernel, at which point the new initramfs ends up in the wrong location.
The third most common failure is a GRUB installation targeting the wrong device. grub-install takes a device argument. If you install to /dev/sda and your system boots from /dev/nvme0n1, the firmware finds nothing valid on nvme0n1 and stops. The GRUB on sda is never reached.
The Rescue Shell
When everything goes wrong, you end up here.
The grub rescue prompt accepts three commands. ls, which lists devices and partitions GRUB can see. set, which shows and sets GRUB variables. insmod, which loads a GRUB module.
The procedure for recovering from rescue mode is always the same. Run ls to see what GRUB can see. You will get entries like (hd0), (hd0,gpt1), (hd0,gpt2). Disk numbers start at zero. Partition numbers start at one.
Use ls (hd0,gpt2)/ to look at what each partition contains. When you find the partition with /boot/grub/ inside it, that is your target. If /boot is a separate partition from /, you need to know which partition each is.
Set the prefix and root variables to point at your /boot/grub directory. Load the normal module. Run normal. If that works, you are back in the full GRUB interface and you can boot as expected.
The fix is not permanent. The normal boot command only works for this session. Once the system is running, update /etc/fstab, run grub-install, and regenerate grub.cfg with update-grub or grub-mkconfig to make the repair permanent.
The Files That Matter
The bootloader configuration lives in four places and all four have to agree.
/etc/fstab declares what gets mounted and where. GRUB's grub.cfg has to agree with fstab about where the root filesystem is. If grub.cfg says root=UUID=abc and fstab says UUID=xyz, one of them is wrong, and the kernel will panic trying to mount a filesystem that does not exist at that UUID.
/etc/default/grub contains the user-editable GRUB configuration: the default menu entry, the timeout, the kernel parameters added to every boot entry. This file does not take effect until you run grub-mkconfig. Editing it and rebooting without regenerating grub.cfg does nothing.
/etc/grub.d/ contains the scripts that grub-mkconfig runs to generate grub.cfg. The 10linux script finds installed kernels and generates menu entries for them. The 40custom file is for user-defined entries. If you want a custom menu entry that persists across kernel updates, it goes in 40_custom, not in grub.cfg directly.
/boot/grub/grub.cfg is the output. It is auto-generated and editing it directly means your changes are erased the next time update-grub runs. On Debian and Ubuntu the command to regenerate it is update-grub. On Fedora and Arch it is grub-mkconfig -o /boot/grub/grub.cfg or grub2-mkconfig depending on how GRUB was packaged.
The UUIDs That Decide Everything
On any Linux system you manage, two UUIDs matter more than anything else.
The UUID of the root partition and the UUID of the boot partition if /boot is separate.
Find them with blkid. The output shows every partition and its UUID. These values have to appear consistently in grub.cfg and in /etc/fstab. When they do not match, the system cannot boot, and the error message will not be gentle about it.
When you clone a disk, the new disk gets new UUIDs by default on most tools. Update grub.cfg and fstab to match the new UUIDs. When you recreate a filesystem with mkfs, that partition gets a new UUID as well. Same result. You can assign a specific UUID when creating a filesystem with mkfs.ext4 -U your-uuid /dev/sda2, which lets you preserve existing configurations after a filesystem rebuild.
The PARTUUID is different from the UUID. UUID is assigned by the filesystem. PARTUUID is assigned by the partition table and does not change when you reformat a partition. Some GRUB configurations use PARTUUID instead of UUID. Know which one your system uses before you attempt a repair.
Recovery: The Full Procedure
A broken GRUB on a running Linux install is always recoverable without reinstalling. The procedure is the same every time.
Boot a live Linux USB. Any distribution works. Mount the root partition at /mnt. If /boot is a separate partition, mount it at /mnt/boot. Mount the virtual filesystems the chroot requires: bind /dev, /proc, /sys, and /run into the mounted tree. Chroot into /mnt. From inside the chroot, reinstall GRUB to the correct device with grub-install /dev/sda, substituting the actual disk containing your system. Regenerate grub.cfg with update-grub or grub-mkconfig. Exit the chroot and reboot.
This works every time. The only reason it might not work is if you chrooted into the wrong root partition. Verify by checking that /mnt contains your expected /etc/hostname or /etc/fstab before chrooting.
The procedure works because GRUB only needs the files in /boot/grub/ to function. Those files exist on your disk. They just need to be pointed at correctly by the MBR or EFI partition boot entry. Reinstalling GRUB writes that pointer. Regenerating grub.cfg updates the menu entries with the correct kernel paths and UUIDs. Nothing else is needed.
The Education
Once you have recovered a broken Linux system by hand, you stop looking at computers the same way. You stop treating the operating system like an appliance that works because it works. You stop assuming that a clean boot means you understand the boot. You stop believing that because grub.cfg was there yesterday, it will be there tomorrow. Something fundamental changes in how you read a running system.
Something shifts after the first recovery. You start reading /etc/fstab instead of skipping past it. You start noting partition UUIDs when you build a new system instead of letting the installer handle it silently. You start running update-grub knowing exactly what it does: reading /etc/default/grub, executing the scripts in /etc/grub.d/, writing the output to /boot/grub/grub.cfg, instead of running it and hoping the machine comes back. You stop treating bootloader operations as incantations and start treating them as comprehensible steps with predictable outcomes.
The bootloader is where the hiding ends. Firmware hands control to GRUB. GRUB reads the disk. GRUB finds grub.cfg. GRUB loads the kernel. The kernel uncompresses. The kernel loads the initramfs. The initramfs mounts root. The kernel pivots to real root. Init starts. Services start. Login appears. Eight distinct handoffs that have to happen in the correct sequence, every single time you see a login screen, on every Linux machine that has ever booted in front of you.
One UUID out of sync with the actual partition stops the chain cold with a kernel panic. One missing driver in the initramfs leaves you with a kernel that cannot mount its own root filesystem. One wrong device argument in grub-install writes the bootloader to a disk the firmware never reads, silently, successfully, and incorrectly. That is the machine you have been using. That is the machine most people treat like a black box. You know what is underneath it now. The next time it fails in front of you, you will not panic. You will know exactly where in the chain to start looking.