Today, I replaced the motherboard of my Homelab server. To avoid reinstalling the system and transferring data, I directly transferred the original disks to the new motherboard. It went smoothly the first time, but this time I couldn’t boot into the system. Fortunately, with the help of AI, I quickly resolved the issue. The experience of using AI to solve problems was amazing. It not only helped me quickly identify and resolve the problem, but also enabled me to learn about Linux partitioning and mounting. Here, I will record my journey.

Background

  • Original motherboard model: Changwang J6413 NAS motherboard (M.2 NVME * 2 + SATA 3.0 * 6 + 2.5G * 3)
  • New motherboard: NAS B365 NAS motherboard + i3-8100T (M.2 NVME *2 + SATA 3.0 * 6 + 2.5G * 2)
  • Disks: NVME * 2 + HDD (system installed on one of the NVME disks)

The two motherboards are not very different. The B365 motherboard has better expandability with support for PCIE X16, allowing for a 10Gb network card upgrade and CPU replacement.

image-20240819230259420

Issue 1: System Not Found

After inserting all the disks into the new motherboard and connecting the memory/CPU/power, I turned on the computer.

However, the expected boot menu did not appear, and an error message was displayed: “Boot Device Not Found.”

image-20240904062427067

I had encountered this error before, and if there was no problem with the system disk, it should be a boot order issue, where the system did not boot from the correct disk.

So, I entered the BIOS and confirmed that it was not booting from the system disk. I adjusted the boot order and restarted.

Issue 2: Partition Not Found

After the restart, I finally entered the boot menu and selected the first option.

image-20240820222452234

Unexpectedly, another issue occurred. The following error appeared:

grub loading 
synoswa(5439): error during setup ioctl handler: -2 

You are in rescue mode. After logging in, type "journalctl -xb" to view system logs, "systemctl reboot" to reboot, "systemctl default" or "exit" to boot into default mode.
Press Enter for maintenance
(or press Contol-D to continue):
root@homelab:~#

This error, “synoswa(5439): error during setup ioctl handler: -2,” was beyond my knowledge, so I turned to AI for help.

Question:

What does the error “synoswa(5439): error during setup ioctl handler: -2” mean when Linux boots into emergency mode?

Answer:

The “synoswa(5439): error during setup ioctl handler: -2” error message is usually related to some issues during the system boot process. This error can be caused by various reasons, including but not limited to hardware issues, driver problems, file system errors, or configuration errors. Here are some possible causes and solutions:

  1. File system errors: If there are file system errors, the Linux system may enter emergency mode to allow system administrators to perform repairs. You can use the fsck command to check and repair file system errors.
  2. Incorrect /etc/fstab file configuration: If the configuration in the /etc/fstab file is incorrect, such as non-existent mount points or mismatched UUIDs, it may cause the system to fail to mount the file system correctly and enter emergency mode.
  3. Hardware or driver issues: If the system encounters hardware errors or driver problems during the boot process, such as unsupported ioctl commands or incorrect ioctl usage, it may also result in the system entering emergency mode.
  4. System update or configuration script errors: After a system update, if there are issues during the update process or errors in the configuration scripts, it may cause the system to fail to boot.

Resolving such issues typically involves the following steps:

  • After entering emergency mode, use journalctl -xb to view system logs, which can help diagnose the cause of the problem.
  • Check the /etc/fstab file to ensure that all mount points and UUIDs are correct.
  • If file system errors are suspected, you can use the fsck command to manually check and repair the file system.
  • If the problem is related to hardware or drivers, you may need to check or update the relevant drivers or disable problematic hardware options in the BIOS/UEFI.

If you are not familiar with these steps or unsure how to proceed, it is recommended to consult a professional or system administrator before making any changes.

Based on the answer from GPT and considering the disk swap action, it is highly likely that the issue is related to the file system. GPT also provided troubleshooting steps, so I followed them.

In the current recovery mode, I entered the password and used the journalctl -xb command to view the boot logs, where I found the following error:

error disk '/lvmpid/xxx/xxx' not found

I also checked the /etc/fstab file:

/dev/mapper/almaLinux-root  / xfs defaults 0 0 
UUID=113413-dadasd-asdadasd-dadasd /boot xfs defaults 0 0 
UUID=0088-A3B2 /boot/efi vfat umask=0077.shortname=wommt 0 2
/dev/mapper/almaLinux-home  /home xfs defaults 0 0 
/dev/mapper/almaLinux-swap  none swap defaults 0 0 
/dev/nvme1n1p1 /mnt/nvme2 ext4  defaults 0 0 
/dev/sda1 /mnt/sda ext4  defaults 0 0

I noticed that some entries used UUID, while others directly specified the volume group name. I was curious if this format is acceptable. So, I continued to ask GPT.

Question:

My /etc/fstab file contains the following entries… Some of them don’t have UUIDs. Is it okay to omit the UUID?

Answer:

In the /etc/fstab file, using UUID is considered a best practice as it provides a more stable and reliable way to identify partitions, especially in cases where the partition order may change or disks are replaced. However, technically, you can use device files (such as /dev/sda1) to specify partitions.

Here are some considerations regarding using or not using UUID:

  1. Benefits of using UUID:

    • When the system boots, UUID allows the system to accurately mount the correct partitions, even if the partition table changes.
    • In multi-disk or dynamic disk environments, using UUID can prevent mounting errors caused by changes in partition order.

  2. Not using UUID:

    • If you are certain that your system environment is static, meaning the partitions and disks will not change, you can use device files (such as /dev/sda1).
    • However, if you add or remove disks, or repartition, not using UUID may result in the system failing to correctly identify and mount the partitions.

In the example /etc/fstab entries you provided, most of the entries already use UUID, which is a safer and more reliable configuration. However, /dev/mapper/almaLinux-root, /dev/nvme1n1p1, and /dev/sda1 are using device files or volume group names instead of UUIDs.

If you decide not to use UUID, you need to ensure:

  • Your system environment will not change, and the partition order and disk device files (such as /dev/sda1) remain consistent.
  • You understand the risks of not using UUID and are prepared to manually update /etc/fstab when the system environment changes.

Recommendation:

  • For LVM volumes like /dev/mapper/almaLinux-root, UUIDs are typically not necessary as LVM volume names are unique.
  • For physical partitions (such as /dev/nvme1n1p1 and /dev/sda1), if you are certain they will not change, you can continue using device files. However, it is best to obtain their UUIDs using the blkid command and use those UUIDs in /etc/fstab.

Finally, when making changes to /etc/fstab, whether adding, removing, or modifying entries, it is important to test the changes after making them to ensure the system can boot and mount all necessary partitions.

In summary, UUID is the best practice, but if your system is static and you don’t modify or replace disks, it is acceptable to not use UUID. However, if you make changes to the partition order or disk devices, and they don’t match the device files specified in /etc/fstab, issues may arise.

With this information, I was able to identify the problem and the next step was to replace the device files in /etc/fstab with their corresponding UUIDs.

Question:

How can I add UUIDs to my configuration file?

Answer:

To add UUIDs to the entries in your /etc/fstab configuration file, you need to first determine the UUID for each partition. Here are the steps:

  1. Find the UUID of the partitions: Use the blkid command to list all partitions and their corresponding UUIDs:

    blkid

  2. Edit the /etc/fstab file: Use a text editor with root privileges to edit the /etc/fstab file. You can use nano or vi, for example:

    nano /etc/fstab

    or

    vi /etc/fstab

  3. Add or update the UUIDs:

    • For each entry that uses a device file name (such as /dev/sda1, /dev/mapper/almaLinux-home, etc.), find the corresponding UUID using the blkid command.
    • Replace the corresponding line in the /etc/fstab file with the UUID you found. For example, if you find that the UUID for /dev/mapper/almaLinux-root is 1234-ABCD, you should change the corresponding line to:
      UUID=1234-ABCD / xfs defaults 0 0

  4. Ensure the correctness of the UUIDs:

    • Make sure the UUIDs you copied and pasted are accurate, without any extra spaces or spelling errors.

  5. Save and close the editor:

    • In nano, you can press Ctrl+O to save the changes and then Ctrl+X to exit.
    • In vi, you can use :wq to save and exit, or :q! to discard changes and exit.

  6. Test the changes to /etc/fstab:

    • Before applying the changes, you can test the syntax of the fstab file using the following command:
      mount -a
    • If this command does not report any errors, you can restart the system to test if the changes are effective.

  7. Restart the system:

    • If you are confident that the changes are correct and the mount -a command did not report any errors, you can restart the system:
      reboot

Please note that when editing the /etc/fstab file, it is important to be very careful, as incorrect configurations can prevent the system from booting. Before making any changes, it is recommended to back up the original /etc/fstab file.

With this guidance, I proceeded to make the modifications. In the recovery mode, it was not convenient to manually input the UUIDs, so I used a workaround. I copied the /etc/fstab file to a temporary file, appended the output of the blkid command to the temporary file for easy copying in the vi editor.

# Create a temporary file
cp /etc/fstab /etc/fstab.tmp
# Append the UUIDs to the temporary file for easy copying in vi
blkid >> /etc/fstab.tmp
# Edit the temporary file
vi /etc/fstab.tmp

After editing the file, the content looked like this, with all device files replaced by UUIDs:

UUID=5b2a641d-5c35-4c78-a9a4-7798a7f7f039 /                       xfs     defaults        0 0
UUID=1afc7fbc-eaab-4d0a-966b-b3920c3198a5 /boot                   xfs     defaults        0 0
UUID=8A88-A3B2                            /boot/efi               vfat    umask=0077,shortname=winnt 0 2
UUID=3f6e6f82-9c18-4fba-a8cf-e22d17074e71 /home                   xfs     defaults        0 0
UUID=33043fe7-c007-4263-b917-00c76e3a7767 none                    swap    defaults        0 0
UUID=09492435-1637-466a-938b-63291449ff01 /mnt/nvme2              ext4    defaults        0  0
UUID=5913181b-e903-4067-8394-4ae5ffb832e0 /mnt/sda                ext4    defaults        0  0

Then, I saved the changes, replaced the original file, tested it with mount -a (no errors), and finally rebooted the system.

After the reboot, I successfully entered the system, completing the migration.

Conclusion

Although I have solved similar issues in the past, the process involved Googling various keywords, opening multiple tabs, and searching for answers one by one, which was not efficient. Now, with the help of AI, I can ask questions in real-time and receive instant answers, quickly resolving problems with high efficiency. Moreover, the accuracy of AI in addressing these fundamental issues is impressive, and it quickly learns related knowledge. In conclusion, AI is a valuable tool that should be utilized, and it undoubtedly represents the future of problem-solving.