Workaround for GMO VPS (CentOS8.1) not being able to start after yum update

I was baptized that I couldn't restart if I ran yum update while building a server (CentOS8.1) on a GMO VPS. Changes to /boot/grub2/grub.cfg were made during the yum update, which seems to be the effect. To avoid this, save /boot/grub2/grub.cfg at the beginning of server construction and restore it after yum update.

What kind of symptom

Start the server (CentOS8.1) available for GMO VPS from the VPS portal in normal mode, and when it is in the "ON" state, connect with TeraTerm and log in as root. The same is true if you log in from the console of the VPS portal. Then run yum update and restart the server.


login: root
Password: ***
$ yum update -y
$ reboot

Then, you cannot connect from TeraTerm, and the GRUB selection screen opens from the console of the VPS portal, and you cannot proceed.

Why it happens

GRUB is a boot loader for selecting where to load and boot the OS image. There is an old GRUB (GRUB Legacy) and a new GRUB2. GRUB2 is from CentOS7 or later. Select whether to use GRUB Legacy or GRUB2, which OS image to use, load the OS image, and boot the OS. Normally, which OS image to use remembers the default one and the one selected by the user for the first time, and automatically performs a series of operations. However, it seems that the OS image cannot be selected automatically by executing yum update. Therefore, it times out when connecting from TeraTerm, and stops at the selection screen of GRUB Legacy and GRUB2 from the console of VPS portal.

On the VPS portal console, you may be able to deal with GRUB commands from the GRUB selection screen. I gave up early because the key bindings were different from the Windows PC, I couldn't copy and paste, and I didn't know the GRUB command.

In CentOS8.1, GRUB2 is used. Control at startup can be specified in the / etc / default / grub file. Specify the timeout seconds with GRUB_TIMEOUT and the OS image with GRUB_DEFAULT. The specified OS image is automatically started after the timeout seconds have passed. To be precise, execute the grub2-mkconfig command in advance to create the /boot/grub2/grub.cfg file based on the description of the grub file and the files under /etc/grub.d/. It is generated. The grub.cfg file is executed at startup, not the grub file is interpreted at startup. For more information on GRUB2, see https://documentation.suse.com/ja-jp/sles/12-SP4/html/SLES-all/cha-grub2.html.

In other words, yum update has rewritten the grub.cfg file, or you are in a situation where you cannot execute it as described in the grub.cfg file (for example, you do not have the necessary files).

It also refers to / boot / grub2 / grubenv at boot time. This saves the settings from the previous startup and is referenced at the next startup. Used when GRUB_DEFAULT = saved is specified. You can control the next boot by modifying the grubenv file after boot. For example, you can change saved_entry directly without going through / etc / default / grub by running the grub2-set-default command.

Steps to avoid

If you save the grub.cfg file after the first login and restore it after yum update, that's pretty much it. The procedure will be described below.

(1) Start the server in normal mode from the VPS portal and log in as root from TeraTerm. Then, check the version in the initial state. CentOS 8.1.


$ cat /etc/redhat-release	
CentOS Linux release 8.1.1911 (Core) 

(2) Save the following files in advance because you will not be able to boot after yum update. Actually, the grub.cfg file alone is fine, but it is good to save other files as well.


$ cp /etc/default/grub /etc/default/grub.original
$ cp /boot/grub2/grub.cfg /boot/grub2/grub.cfg.original
$ cp /boot/grub2/grubenv /boot/grub2/grubenv.original

In fact, the grub file has already been backed up as grub.rpmsave, and the grub.cfg file has also been backed up as grub.cfg.rpmsave. Those files may be enough, but in the above procedure, I made it explicit. By the way, / etc / default / grub is as follows.


$ cat /etc/default/grub
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="crashkernel=auto biosdevname=0 net.ifnames=0 rhgb quiet"
GRUB_DISABLE_RECOVERY="true"
GRUB_ENABLE_BLSCFG=true
GRUB_DISABLE_UUID=true
GRUB_DISABLE_LINUX_UUID=true

GRUB_TIMEOUT and GRUB_DEFAULT indicate that the last used OS image should be booted after waiting 5 seconds. GRUB_DEFAULT is specified by a number such as 0,1,2, ..., by the name of the OS image, or by " saved ". In the case of "saved", the saved_entry in the / boot / grub2 / grubenv file identifies the previously used OS image.


$ cat /boot/grub2/grubenv
saved_entry=20db05bac820110f5800ebe2e7262b58-4.18.0-193.28.1.el8_2.x86_64
kernelopts=root=/dev/vda1 ro crashkernel=auto biosdevname=0 net.ifnames=0 rhgb quiet
boot_success=1

(3) Execute yum update. Executing the yum update in question. About 200 modules will be updated.


$ yum update -y

(4) Check the version after yum update. It has been upgraded to CentOS 8.2.


$ cat /etc/redhat-release
CentOS Linux release 8.2.2004 (Core)

(5) Set so that the kernel will not be updated by yum update in the future.


$ vi /etc/yum.conf
exclude=kernel*

(6) Restore the saved grub.cfg. / etc / default / grub is unchanged, so no recovery is required. grubenv is updated at startup, but usually the content is the same as last time. If you use the grub2-set-default command, grubenv will be rewritten. In that case, recover.


$ cp /boot/grub2/grub.cfg.original /boot/grub2/grub.cfg
$ cp /boot/grub2/grubenv.original /boot/grub2/grubenv

$ awk -F\' '$1=="menuentry " {print $2}' /boot/grub2/grub.cfg
CentOS Linux (4.18.0-147.5.1.el8_1.x86_64) 8 (Core)

(7) At this point, restart and confirm that you can log in from TeraTerm.

(8) Edit the grub file and reflect it with the grub2-mkconfig command. In the grub file, only GRUB_CMDLINE_LINUX is changed. Then check if the one identified by saved_entry in the grubenv file is found in the " Found linux " message. You can also check the result of awk. It can be seen that the beginning (0th) is applicable.


#Edit grub file
$ vi /etc/default/grub
GRUB_TIMEOUT=3
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="rd.lvm.lv=centos/root rd.lvm.lv=centos/swap 
 vconsole.font=latarcyrheb-sun16 vconsole.keymap=jp106 
 biosdevname=0 net.ifnames=0 rhgb quiet "
GRUB_DISABLE_RECOVERY="true"
GRUB_ENABLE_BLSCFG=true
GRUB_DISABLE_UUID=true
GRUB_DISABLE_LINUX_UUID=true

# grub.Reflect in cfg file
$ grub2-mkconfig -o /boot/grub2/grub.cfg
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-4.18.0-193.28.1.el8_2.x86_64
Found initrd image: /boot/initramfs-4.18.0-193.28.1.el8_2.x86_64.img
Found linux image: /boot/vmlinuz-4.18.0-147.5.1.el8_1.x86_64
Found initrd image: /boot/initramfs-4.18.0-147.5.1.el8_1.x86_64.img
Found linux image: /boot/vmlinuz-0-rescue-20db05bac820110f5800ebe2e7262b58
Found initrd image: /boot/initramfs-0-rescue-20db05bac820110f5800ebe2e7262b58.img
done

$ awk -F\' '$1=="menuentry " {print $2}' /boot/grub2/grub.cfg
CentOS Linux (4.18.0-193.28.1.el8_2.x86_64) 8 (Core)
CentOS Linux (4.18.0-147.5.1.el8_1.x86_64) 8 (Core)
CentOS Linux (0-rescue-20db05bac820110f5800ebe2e7262b58) 8 (Core)

#Check the grubenv file
$ cat /boot/grub2/grubenv
saved_entry=20db05bac820110f5800ebe2e7262b58-4.18.0-193.28.1.el8_2.x86_64
kernelopts=root=/dev/vda1 ro crashkernel=auto biosdevname=0 net.ifnames=0 rhgb quiet
boot_success=1

(9) There is no problem if you can restart and log in from TeraTerm.

Supplement 1: grub2-mkconfig command

I tried to generate grub.cfg by executing the grub2-mkconfig command immediately after yum update in the above procedure. The result was NG. I can no longer log in from TeraTerm, and it stops at the GRUB2 screen from the console of the VPS portal. The description of menuentry in grub.cfg is missing. I don't know the cause. As in steps (8) and (9) above, running grub2-mkconfig after rebooting seems to work.

Supplement 2: grub2-set-default command

Looking at /boot/grub2/grubenv, saved_entry contains the name 20db05bac820110f5800ebe2e7262b58-4.18.0-193.28.1.el8_2.x86_64 instead of the number (0,1,2, ...). It seems to be matched in the latter part "4.18.0-193.28.1.el8_2.x86_64".


$ cat /boot/grub2/grubenv
saved_entry=20db05bac820110f5800ebe2e7262b58-4.18.0-193.28.1.el8_2.x86_64
kernelopts=root=/dev/vda1 ro crashkernel=auto biosdevname=0 net.ifnames=0 rhgb quiet
boot_success=1

By the way, by executing grub2-set-default, saved_entry can be changed directly without going through / etc / default / grub.


$ grub2-set-default 0
$ cat /boot/grub2/grubenv
saved_entry=0
kernelopts=root=/dev/vda1 ro crashkernel=auto biosdevname=0 net.ifnames=0 rhgb quiet
boot_success=1

Supplement 3: Repair from recovery mode

If you cannot start it, you can start the server in recovery mode from the VPS portal, log in as root from TeraTerm, and follow the steps below to repair it. The password is the initial password, not the " recovery ". The message on the VPS portal seems to be wrong. The procedure is to identify and mount the main disk before operating.


$ fdisk -l | more
  /dev/vda1
$ mount /dev/vda1 /mnt
$ cp /mnt/boot/grub2/grub.cfg.original /mnt/boot/grub2/grub.cfg

I haven't tried it, but since grub.cfg.rpmsave already existed, I might have been able to recover with that file. If you copy as follows and restart the server, you may have recovered by the steps (8) and (9) above.


$ cp /mnt/boot/grub2/grub.cfg.rpmsave /mnt/boot/grub2/grub.cfg

that's all

Recommended Posts

Workaround for GMO VPS (CentOS8.1) not being able to start after yum update
Solution for unable to update CentOS 8 (yum update / dnf update) due to existing centos-gpg-keys
Solution for Apache Tomcat 8 not starting after changing to OpenJDK 11
[Rough explanation] Causes and remedies for not being able to obtain the name from the ActiveHash model