apt update
apt upgrade
is what users do… having no idea of the “heart attack” momentum that is awaiting them X-D
after a reboot… and 1min 30sec timeout, the dreaded Control+D give root password prompt appears. “Houston we have a problem.” something has gone wrong.
first of: it is NOT your fault, just shows, updates without backups are dangerous on ANY OS and no matter how often it worked before.?
(but backups of the backups? SERIOUSLY X-D yes imho guess so … so next month will have to buy a external 4TB USB harddisk to rsync backup the storagepc)
let’s try to stay cool! ?
none of user’s data is lost
it is just a bug in the kernel (?) X-D
had exactly the same problem today on Debian 9.
a whole ext3 RAID1 “vanished” after kernel was updated from:
linux-image-4.9.0-11-amd64 4.9.189-3+deb9u2
to
linux-image-4.9.0-12-amd64 4.9.210-1
# list all installed kernels
dpkg --list | grep linux-image
ii linux-image-4.9.0-11-amd64 4.9.189-3+deb9u2 amd64 Linux 4.9 for 64-bit PCs
ii linux-image-4.9.0-12-amd64 4.9.210-1 amd64 Linux 4.9 for 64-bit PCs
rc linux-image-4.9.0-6-amd64 4.9.88-1+deb9u1 amd64 Linux 4.9 for 64-bit PCs
rc linux-image-4.9.0-8-amd64 4.9.144-3.1 amd64 Linux 4.9 for 64-bit PCs
ii linux-image-4.9.0-9-amd64 4.9.168-1+deb9u3 amd64 Linux 4.9 for 64-bit PCs
ii linux-image-amd64 4.9+80+deb9u10 amd64 Linux for 64-bit PCs (meta-package)
hostnamectl; # os used
Static hostname: storagepc
Icon name: computer-desktop
Chassis: desktop
Operating System: Debian GNU/Linux 9 (stretch)
Kernel: Linux 4.9.0-12-amd64
Architecture: x86-64
# “solution”: boot previous kernel ( in this case: linux-image-4.9.0-11-amd64 )
vim /etc/default/grub GRUB_TIMEOUT=3 <- make sure a timeout larger than 0 is defined (or no time to select any options during boot) # let grub2 do it's stuff update-grub # is the same as: update-grub2 # reboot the system (if USB keyboard is not reacting during grub boot screen, try PS2 keyboard) reboot # when grub boot screen appears
after booting linux-image-4.9.0-11-amd64 kernel, can access ext3 RAID1 AGAIN! ?
problem: grub won’t remember that choice.
to make this permanent:
vim /etc/default/grub # during boot: ## select in the first menu the second (0,1) entry #### then select in the second menu select the 3rd entry (0,1,2) GRUB_DEFAULT="1>2" # make grub2 realize the changes update-grub
… yes it is confusing i know X-D
# this is what it was supposed to look like
have two RAID1 defined.
# show status of raid
cat /proc/mdstat
Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10]
md126 : active raid1 sdc1[1] sdb1[0]
3906886464 blocks super 1.2 [2/2] [UU]
bitmap: 0/30 pages [0KB], 65536KB chunk
md127 : active raid1 sde1[0] sdd1[2]
1953381376 blocks super 1.2 [2/2] [UU]
bitmap: 0/15 pages [0KB], 65536KB chunk
# show where is what mounted
mount
/dev/md126 on /media/user/ext4RAID1 type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/md127 on /media/user/ext3RAID1 type ext3 (rw,relatime,data=ordered)
# show block devices
lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
fd0 2:0 1 4K 0 disk
sda 8:0 0 238.5G 0 disk
├─sda1 8:1 0 230.8G 0 part /
├─sda2 8:2 0 1K 0 part
└─sda5 8:5 0 7.7G 0 part [SWAP]
sdb 8:16 0 3.7T 0 disk
└─sdb1 8:17 0 3.7T 0 part
└─md126 9:126 0 3.7T 0 raid1 /media/user/ext4RAID1
sdc 8:32 0 3.7T 0 disk
└─sdc1 8:33 0 3.7T 0 part
└─md126 9:126 0 3.7T 0 raid1 /media/user/ext4RAID1
sdd 8:48 0 1.8T 0 disk
└─sdd1 8:49 0 1.8T 0 part
└─md127 9:127 0 1.8T 0 raid1 /media/user/ext3RAID1
sde 8:64 0 1.8T 0 disk
└─sde1 8:65 0 1.8T 0 part
└─md127 9:127 0 1.8T 0 raid1 /media/user/ext3RAID1
sr0 11:0 1 1024M 0 rom
# find defined raids
mdadm --examine --scan
ARRAY /dev/md/2 metadata=1.2 UUID=90642755:fa191325:0fe4ec59:2456c645 name=storagepc:2
ARRAY /dev/md/1 metadata=1.2 UUID=433fb7e1:9d7f3f17:bc5ee18b:0f4eeb52 name=storagepc:1
# show UUIDS
blkid /dev/sdb1
/dev/sdb1: UUID="90642755-fa19-1325-0fe4-ec592456c645" UUID_SUB="bee458e0-509a-c110-b577-8a1ddbe6bbb3" LABEL="storagepc:2" TYPE="linux_raid_member" PARTUUID="1fd02041-9dd2-4918-83a3-c8bafbab3bed"
blkid /dev/sdc1
/dev/sdc1: UUID="90642755-fa19-1325-0fe4-ec592456c645" UUID_SUB="7d5947f8-1ba0-0c7b-18a7-194ab4051a2c" LABEL="storagepc:2" TYPE="linux_raid_member" PARTUUID="5e4ea781-68e5-43f0-accf-26342aeb4daa"
userblkid /dev/sdd1
/dev/sdd1: UUID="433fb7e1-9d7f-3f17-bc5e-e18b0f4eeb52" UUID_SUB="bed17780-3817-27c9-6336-44d4aedfb857" LABEL="storagepc:1" TYPE="linux_raid_member" PARTUUID="f6aab6c2-01"
userblkid /dev/sde1
/dev/sde1: UUID="433fb7e1-9d7f-3f17-bc5e-e18b0f4eeb52" UUID_SUB="eb90b361-94d6-2f38-7727-d386097dce81" LABEL="storagepc:1" TYPE="linux_raid_member" PARTUUID="d2fd127f-01"
# regular filesystem checks
has nothing to do with the problem but defining this via tune2fs has the advantage, that it will automatically be performed during boot. (do this on all raid1 devices and partitions that use ext3 ext4 (other filesystems do not support this)
tune2fs -C 2 -c 1 /dev/sda1; # check filesystem on every boot (for ext3 takes rather long X-D) tune2fs -c 10 -i 30 /dev/sda1; # check sda1 every 10 mounts or after 30 days
liked this article?
- only together we can create a truly free world
- plz support dwaves to keep it up & running!
- (yes the info on the internet is (mostly) free but beer is still not free (still have to work on that))
- really really hate advertisement
- contribute: whenever a solution was found, blog about it for others to find!
- talk about, recommend & link to this blog and articles
- thanks to all who contribute!
