qnap raid rebuild slow – How to increase raid rebuild speed

14.May.2014

wanted to expand capacity of an QNAP Turbo TS-559 Pro 5-bay NAS server running a RAID5 array and marked one 2TB drive via webInterface for replacement and replaced it with a 4TB drive…

now the rebuild process went over days and did not complete… even worse: the NAS froze during the rebuild process (no hearable harddisk activity going on status lights flash green/red… that’s it… hard reset by holding power button… start up again and now watching the slow rebuild process…)

to observe your raid array and get status updates on your rebuild process, ssh into your qnap.

ssh admin@192.168.0.123

cat /proc/mdstat;

you can put this into a little scrtip that updates the status every second.

vim ./check_proc.sh

while true; do (clear;cat /proc/mdstat;) ; sleep 1 ; done

chmod u+x ./check_proc.sh

./check_proc.sh

you can speed up the build process like this: (the default limit was set to 500Bytes/Sec! that’s super slow!)

echo 50000 > /proc/sys/dev/raid/speed_limit_min

now it’s much faster!

 Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [multipath]
md0 : active raid5 sdb3[1] sda3[5] sde3[4] sdd3[3] sdc3[2]
      7807782400 blocks level 5, 64k chunk, algorithm 2 [5/4] [_UUUU]
      [=======>.............]  recovery = 39.4% (769880488/1951945600) finish=389.8min speed=50527K/sec
      
md5 : active raid1 sde2[2](S) sdd2[3](S) sdc2[4](S) sdb2[1] sda2[0]
      530048 blocks [2/2] [UU]
      
md13 : active raid1 sda4[0] sdd4[4] sdc4[3] sde4[2] sdb4[1]
      458880 blocks [5/5] [UUUUU]
      bitmap: 0/57 pages [0KB], 4KB chunk

md9 : active raid1 sda1[0] sdd1[4] sde1[3] sdc1[2] sdb1[1]
      530048 blocks [5/5] [UUUUU]
      bitmap: 3/65 pages [12KB], 4KB chunk

http://forum.qnap.com/viewtopic.php?t=10268

[HOWTO] How to increase raid rebuild speed

by Don » Fri Dec 12, 2008 6:06 am

I found this article on the web. Use it at your own risk as I have not tested it and make no guarantees. If yo do try it please post your results here so others may benefit.

Increase the speed of Linux Software RAID reconstruction

If you are in a situation where you sit in front of the console (or on a remote ssh connection) waiting for a Linux software RAID to finish rebuilding (either you added a new drive, or you replaced a failed one, etc.) then you might be frustrated by how slow this process is running. You are running cat on /proc/mdstat repeatedly (you should really use watch in this case ), and this seems to never finish… Obviously that there is a logical reason for this ‘slowness‘ and on a production system you should leave it running with the defaults. But in case you want to speed up this process here is how you can do it. This will place a much higher load on the system so you should use it with care.

To see your Linux kernel speed limits imposed on the RAID reconstruction use:

Code: Select all

cat /proc/sys/dev/raid/speed_limit_max 200000 cat /proc/sys/dev/raid/speed_limit_min 1000

In the system logs you can see something similar to:

Code: Select all

md: minimum _guaranteed_ reconstruction speed: 1000 KB/sec/disc. md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for reconstruction.

This means that the minimum guaranteed speed of the rebuild of the array is approx 1MB/s. The actual speed will be higher and will depend on the system load and what other processes are running at that time.
In case you want to increase this minimum speed you need to enter a higher value in speed_limit_min. For example to set this to approx 50 megabytes per second as minimum use:

Code: Select all

echo 50000 >/proc/sys/dev/raid/speed_limit_min

The results are instant… you can return to the watch window to see it running, and hope that this will finish a little faster (this will really depend on the system you are running, the HDDs, controllers, etc.):

Code: Select all

watch cat /proc/mdstat

liked this article?

only together we can create a truly free world
plz support dwaves to keep it up & running!
(yes the info on the internet is (mostly) free but beer is still not free (still have to work on that))
really really hate advertisement
contribute: whenever a solution was found, blog about it for others to find!
talk about, recommend & link to this blog and articles
thanks to all who contribute!

admin