apt update
apt upgrade

is what users do… having no idea of the “heart attack” momentum that is awaiting them X-D

after a reboot… and 1min 30sec timeout, the dreaded Control+D give root password prompt appears. “Houston we have a problem.” something has gone wrong.

first of: it is NOT your fault, just shows, updates without backups are dangerous on ANY OS and no matter how often it worked before.?

(but backups of the backups? SERIOUSLY X-D yes imho guess so … so next month will have to buy a external 4TB USB harddisk to rsync backup the storagepc)

let’s try to stay cool! ?

none of user’s data is lost

it is just a bug in the kernel (?) X-D

had exactly the same problem today on Debian 9.

a whole ext3 RAID1 “vanished” after kernel was updated from:

    linux-image-4.9.0-11-amd64                        4.9.189-3+deb9u2                            


    linux-image-4.9.0-12-amd64                        4.9.210-1                                   
# list all installed kernels
    dpkg --list | grep linux-image
    ii  linux-image-4.9.0-11-amd64                        4.9.189-3+deb9u2                            amd64        Linux 4.9 for 64-bit PCs
    ii  linux-image-4.9.0-12-amd64                        4.9.210-1                                   amd64        Linux 4.9 for 64-bit PCs
    rc  linux-image-4.9.0-6-amd64                         4.9.88-1+deb9u1                             amd64        Linux 4.9 for 64-bit PCs
    rc  linux-image-4.9.0-8-amd64                         4.9.144-3.1                                 amd64        Linux 4.9 for 64-bit PCs
    ii  linux-image-4.9.0-9-amd64                         4.9.168-1+deb9u3                            amd64        Linux 4.9 for 64-bit PCs
    ii  linux-image-amd64                                 4.9+80+deb9u10                              amd64        Linux for 64-bit PCs (meta-package)

    hostnamectl; # os used
       Static hostname: storagepc
             Icon name: computer-desktop
               Chassis: desktop
      Operating System: Debian GNU/Linux 9 (stretch)
                Kernel: Linux 4.9.0-12-amd64
          Architecture: x86-64


# “solution”: boot previous kernel ( in this case: linux-image-4.9.0-11-amd64 )

    vim /etc/default/grub
    GRUB_TIMEOUT=3 <- make sure a timeout larger than 0 is defined (or no time to select any options during boot)

    # let grub2 do it's stuff
    # is the same as:

    # reboot the system (if USB keyboard is not reacting during grub boot screen, try PS2 keyboard)
    # when grub boot screen appears 

after booting linux-image-4.9.0-11-amd64 kernel, can access ext3 RAID1 AGAIN! ?

problem: grub won’t remember that choice.

to make this permanent:

    vim /etc/default/grub

    # during boot:
    ## select in the first menu the second (0,1) entry
    #### then select in the second menu select the 3rd entry (0,1,2)

    # make grub2 realize the changes

… yes it is confusing i know X-D

# this is what it was supposed to look like

have two RAID1 defined.

    # show status of raid
    cat /proc/mdstat 
    Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10] 
    md126 : active raid1 sdc1[1] sdb1[0]
          3906886464 blocks super 1.2 [2/2] [UU]
          bitmap: 0/30 pages [0KB], 65536KB chunk
    md127 : active raid1 sde1[0] sdd1[2]
          1953381376 blocks super 1.2 [2/2] [UU]
          bitmap: 0/15 pages [0KB], 65536KB chunk

    # show where is what mounted
    /dev/md126 on /media/user/ext4RAID1 type ext4 (rw,relatime,errors=remount-ro,data=ordered)
    /dev/md127 on /media/user/ext3RAID1 type ext3 (rw,relatime,data=ordered)

    # show block devices
    fd0         2:0    1     4K  0 disk  
    sda         8:0    0 238.5G  0 disk  
    ├─sda1      8:1    0 230.8G  0 part  /
    ├─sda2      8:2    0     1K  0 part  
    └─sda5      8:5    0   7.7G  0 part  [SWAP]
    sdb         8:16   0   3.7T  0 disk  
    └─sdb1      8:17   0   3.7T  0 part  
      └─md126   9:126  0   3.7T  0 raid1 /media/user/ext4RAID1
    sdc         8:32   0   3.7T  0 disk  
    └─sdc1      8:33   0   3.7T  0 part  
      └─md126   9:126  0   3.7T  0 raid1 /media/user/ext4RAID1
    sdd         8:48   0   1.8T  0 disk  
    └─sdd1      8:49   0   1.8T  0 part  
      └─md127   9:127  0   1.8T  0 raid1 /media/user/ext3RAID1
    sde         8:64   0   1.8T  0 disk  
    └─sde1      8:65   0   1.8T  0 part  
      └─md127   9:127  0   1.8T  0 raid1 /media/user/ext3RAID1
    sr0        11:0    1  1024M  0 rom 

    # find defined raids
    mdadm --examine --scan
    ARRAY /dev/md/2  metadata=1.2 UUID=90642755:fa191325:0fe4ec59:2456c645 name=storagepc:2
    ARRAY /dev/md/1  metadata=1.2 UUID=433fb7e1:9d7f3f17:bc5ee18b:0f4eeb52 name=storagepc:1

    # show UUIDS
    blkid /dev/sdb1
    /dev/sdb1: UUID="90642755-fa19-1325-0fe4-ec592456c645" UUID_SUB="bee458e0-509a-c110-b577-8a1ddbe6bbb3" LABEL="storagepc:2" TYPE="linux_raid_member" PARTUUID="1fd02041-9dd2-4918-83a3-c8bafbab3bed"
    blkid /dev/sdc1
    /dev/sdc1: UUID="90642755-fa19-1325-0fe4-ec592456c645" UUID_SUB="7d5947f8-1ba0-0c7b-18a7-194ab4051a2c" LABEL="storagepc:2" TYPE="linux_raid_member" PARTUUID="5e4ea781-68e5-43f0-accf-26342aeb4daa"
    userblkid /dev/sdd1
    /dev/sdd1: UUID="433fb7e1-9d7f-3f17-bc5e-e18b0f4eeb52" UUID_SUB="bed17780-3817-27c9-6336-44d4aedfb857" LABEL="storagepc:1" TYPE="linux_raid_member" PARTUUID="f6aab6c2-01"
    userblkid /dev/sde1
    /dev/sde1: UUID="433fb7e1-9d7f-3f17-bc5e-e18b0f4eeb52" UUID_SUB="eb90b361-94d6-2f38-7727-d386097dce81" LABEL="storagepc:1" TYPE="linux_raid_member" PARTUUID="d2fd127f-01"

# regular filesystem checks

has nothing to do with the problem but defining this via tune2fs has the advantage, that it will automatically be performed during boot. (do this on all raid1 devices and partitions that use ext3 ext4 (other filesystems do not support this)

    tune2fs -C 2 -c 1 /dev/sda1; # check filesystem on every boot (for ext3 takes rather long X-D)
    tune2fs -c 10 -i 30 /dev/sda1; # check sda1 every 10 mounts or after 30 days

liked this article?

  • only together we can create a truly free world
  • plz support dwaves to keep it up & running!
  • (yes the info on the internet is (mostly) free but beer is still not free (still have to work on that))
  • really really hate advertisement
  • contribute: whenever a solution was found, blog about it for others to find!
  • talk about, recommend & link to this blog and articles
  • thanks to all who contribute!