HighPoint always was and still is “crap”, if the admin wants to upgrade old server with NVMe RAID(10 preferably), than wait for ADAPTEC SmartRaid 3200 family coming out soon for purchase (untested!, see specs here https://storage.microsemi.com/en-us/support/smartraid3200/)

  • 3200 is the new Trimode family supporting SAS, SATA and NMVe devices

if RAID1 is sufficient and replacing mobos is an option, then:

back to the new LowPoint of HighPoint, some cards are reported to have died the heat-death, with HighPoint giving no good RMA-support.

for those users that could not be scared away… it actually works with MDADM software RAID under GNU Linux Debian 10 but at half the heat and half the speed.

https://highpoint-tech.com/USA_new/series-ssd7101a-1-overview.htm

this card + x16 PCIe slot + NVMes promise MASSIVE harddisk speeds 🙂

  • for Mac/OSX users this card seems to be a good pick
  • for GNU Linux users at the current state (faulty drivers that do not compile) it is just an expensive NVMe riser card… it works with mdadm software raid but does not deliver the promised speeds
  • for Windows (Server 2012 R2) users, at the current state (faulty drivers) it can not be recommended (see screenshots at the very bottom)

possible alternatives?

  • not from renowed Adaptec / Microsemi: “Microsemi RAID adapters and controllers (ROCs) supporting NVMe architecture will sample soon.” (src)

maybe just get a simple NVMe PCI card and go with the in-build software-raid of GNU Linux (mdadm) or Windows.

what is really nice are the 1TB Kingston NVMe (KINGSTON SA2000M81000G) blue harddisk activityLEDs 🙂

and yes…

but it is probably better anyway to let GNU Linux MDADM handle the RAID 🙂

WARNING! for whatever reason the USB keyboard AND MOUSE stopped working, after installing this card… X-D

luckily this a bit old Asus mobo still had PS2 keyboard-mouse combi port, so it might be smart to assign a fixed ip to the TestPC and ssh into it, BEFORE installing the card

here is the complete dmesg.txt of that problem

hint!

some of those benchmarks are too CPU heavy… so with a faster CPU (it is only a i3) it might be possible to achieve even faster speeds in mdadm RAID10

tested on:

hostnamectl; # latest GNU Linux Debian 10
  Operating System: Debian GNU/Linux 10 (buster)
            Kernel: Linux 4.19.0-16-amd64
      Architecture: x86-64
lscpu
Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
Address sizes:       36 bits physical, 48 bits virtual
CPU(s):              4
On-line CPU(s) list: 0-3
Thread(s) per core:  2
Core(s) per socket:  2
Socket(s):           1
NUMA node(s):        1
Vendor ID:           GenuineIntel
CPU family:          6
Model:               42
Model name:          Intel(R) Core(TM) i3-2120 CPU @ 3.30GHz
Stepping:            7
CPU MHz:             3291.877
CPU max MHz:         3300.0000
CPU min MHz:         1600.0000
BogoMIPS:            6583.57
Virtualization:      VT-x
L1d cache:           32K
L1i cache:           32K
L2 cache:            256K
L3 cache:            3072K
NUMA node0 CPU(s):   0-3
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 popcnt tsc_deadline_timer xsave avx lahf_lm epb pti tpr_shadow vnmi flexpriority ept vpid xsaveopt dtherm arat pln pts


cat /etc/debian_version 
10.9

# even without the vendor driver GNU Linux manages to access the NVMes
lsblk -fs
NAME                  FSTYPE      LABEL UUID                                   FSAVAIL FSUSE% MOUNTPOINT
nvme0n1                                                                                      
nvme1n1                                                                                      
nvme2n1                                                                                      
nvme3n1

info about the card: HighPoint SSD7101A-1

On August 12, 2014, Broadcom Inc. acquired the company.[21] (src: Wiki)

lspci; # the PEX 8747 is from BroadCom
# product overview: PEX8747_Product_Brief_v1-0_20Oct10.pdf
01:00.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch (rev ca)
02:08.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch (rev ca)
02:09.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch (rev ca)
02:10.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch (rev ca)
02:11.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch (rev ca)
03:00.0 Non-Volatile memory controller: Kingston Technologies Device 2263 (rev 03)
04:00.0 Non-Volatile memory controller: Kingston Technologies Device 2263 (rev 03)
05:00.0 Non-Volatile memory controller: Kingston Technologies Device 2263 (rev 03)
06:00.0 Non-Volatile memory controller: Kingston Technologies Device 2263 (rev 03)

lspci -vvv
01:00.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch (rev ca) (prog-if 00 [Normal decode])
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR- TAbort- Reset- FastB2B-
                PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [48] MSI: Enable- Count=1/8 Maskable+ 64bit+
                Address: 0000000000000000  Data: 0000
                Masking: 00000000  Pending: 00000000
        Capabilities: [68] Express (v2) Upstream Port, MSI 00
                DevCap: MaxPayload 2048 bytes, PhantFunc 0
                        ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ SlotPowerLimit 75.000W
                DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
                        RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+
                        MaxPayload 128 bytes, MaxReadReq 128 bytes
                DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-
                LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L1, Exit Latency L0s <2us, L1 <4us
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM Disabled; Disabled- CommClk-
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 5GT/s, Width x16, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR+, OBFF Via message
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
                LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
                         EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
        Capabilities: [a4] Subsystem: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch
        Capabilities: [100 v1] Device Serial Number ca-87-00-10-b5-df-0e-00
        Capabilities: [fb4 v1] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
                AERCap: First Error Pointer: 1f, GenCap+ CGenEn- ChkCap+ ChkEn-
        Capabilities: [138 v1] Power Budgeting <?>
        Capabilities: [10c v1] #19
        Capabilities: [148 v1] Virtual Channel
                Caps:   LPEVC=0 RefClk=100ns PATEntryBits=8
                Arb:    Fixed- WRR32- WRR64- WRR128-
                Ctrl:   ArbSelect=Fixed
                Status: InProgress-
                VC0:    Caps:   PATOffset=03 MaxTimeSlots=1 RejSnoopTrans-
                        Arb:    Fixed- WRR32- WRR64+ WRR128- TWRR128- WRR256-
                        Ctrl:   Enable+ ID=0 ArbSelect=WRR64 TC/VC=01
                        Status: NegoPending- InProgress-
                        Port Arbitration Table <?>
        Capabilities: [e00 v1] #12
        Capabilities: [b00 v1] Latency Tolerance Reporting
                Max snoop latency: 0ns
                Max no snoop latency: 0ns
        Capabilities: [b70 v1] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?>
        Kernel driver in use: pcieport
...
# and the NVMe
03:00.0 Non-Volatile memory controller: Kingston Technologies Device 2263 (rev 03) (prog-if 02 [NVM Express])
        Subsystem: Kingston Technologies Device 2263
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR- dmesg|grep 01:00.0
[    0.296868] pci 0000:01:00.0: [10b5:8747] type 01 class 0x060400
[    0.296884] pci 0000:01:00.0: reg 0x10: [mem 0xf7c00000-0xf7c3ffff]
[    0.296942] pci 0000:01:00.0: PME# supported from D0 D3hot D3cold
[    0.296965] pci 0000:01:00.0: 64.000 Gb/s available PCIe bandwidth, limited by 5 GT/s x16 link at 0000:00:01.0 (capable of 126.016 Gb/s with 8 GT/s x16 link)
[    0.307565] pci 0000:01:00.0: PCI bridge to [bus 02-06]
[    0.307571] pci 0000:01:00.0:   bridge window [mem 0xf7800000-0xf7bfffff]
[    0.388724] pci 0000:01:00.0: bridge window [io  0x1000-0x0fff] to [bus 02-06] add_size 1000
[    0.388725] pci 0000:01:00.0: bridge window [mem 0x00100000-0x001fffff 64bit pref] to [bus 02-06] add_size 200000 add_align 100000
[    0.388759] pci 0000:01:00.0: BAR 15: assigned [mem 0xdfb00000-0xdfdfffff 64bit pref]
[    0.388762] pci 0000:01:00.0: BAR 13: assigned [io  0x2000-0x2fff]
[    0.388814] pci 0000:01:00.0: PCI bridge to [bus 02-06]
[    0.388816] pci 0000:01:00.0:   bridge window [io  0x2000-0x2fff]
[    0.388820] pci 0000:01:00.0:   bridge window [mem 0xf7800000-0xf7bfffff]
[    0.388823] pci 0000:01:00.0:   bridge window [mem 0xdfb00000-0xdfdfffff 64bit pref]

info about the 1TB Kingston NVMe (SA2000M81000G)

smartctl -H /dev/nvme0n1
smartctl 6.6 2017-11-05 r4594 [x86_64-linux-4.19.0-16-amd64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

smartctl -i /dev/nvme0n1
smartctl 6.6 2017-11-05 r4594 [x86_64-linux-4.19.0-16-amd64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Number:                       KINGSTON SA2000M81000G
Serial Number:                      X
Firmware Version:                   S5Z42105
PCI Vendor/Subsystem ID:            0x2646
IEEE OUI Identifier:                0x0026b7
Controller ID:                      1
Number of Namespaces:               1
Namespace 1 Size/Capacity:          1,000,204,886,016 [1.00 TB]
Namespace 1 Utilization:            13,348,388,864 [13.3 GB]
Namespace 1 Formatted LBA Size:     512
Namespace 1 IEEE EUI-64:            0026b7 6847185c85
Local Time is:                      Wed May 19 23:58:32 2021 CEST

benchmark 1TB Kingston NVMe (SA2000M81000G): sequential

this is: creating ext4 on one of the 1TB Kingston NVMes and benchmarking only this single device

download the script here: bench_harddisk.sh.zip

time /scripts/bench/bench_harddisk.sh 
=== harddisk sequential write and read bench v1 ===
starting test on the device that holds the current directory the user is in
no need to run it as root

========== writing 3GB of zeroes ==========
0+1 records in
0+1 records out
2147479552 bytes (2.1 GB, 2.0 GiB) copied, 2.33577 s, 919 MB/s

real	0m2.342s
user	0m0.001s
sys	0m1.876s
========== reading 6GB of zeroes ==========
0+1 records in
0+1 records out
2147479552 bytes (2.1 GB, 2.0 GiB) copied, 1.18685 s, 1.8 GB/s

real	0m1.193s
user	0m0.000s
sys	0m0.799s
========== tidy up remove testfile ==========

real	0m3.656s
user	0m0.004s
sys	0m2.792s

benchmark 1TB Kingston NVMe (SA2000M81000G): many small python generated files

this is: creating ext4 on one of the 1TB Kingston NVMes and benchmarking only this single device

for comparison, the Samsung SSD 860 EVO 500GB took: real 0m36.163s

download the script here: bench_harddisk_small_files.py.zip

time /scripts/bench/bench_harddisk_small_files.py 

create test folder:
0.00545001029968

create files:
9.2529399395

rewrite files:
1.83215284348

read linear:
1.67627692223

read random:
2.59832310677

delete all files:
4.72371888161

real	0m23.995s (-33% faster than the SSD 860 EVO 500GB)
user	0m3.380s
sys	0m14.668s

bench in MDADM (GNU Linux software) RAID10

now to the mdadm RAID10 benchmarks:

# create the raid
mdadm --create /dev/md0 --level=10 --raid-devices=4 /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1
mkfs.ext4 -L "ext4RAID10" /dev/md0

# wait till finished
cat /proc/mdstat;
Personalities : [raid10] 
md0 : active raid10 nvme3n1[3] nvme2n1[2] nvme1n1[1] nvme0n1[0]
      1953260544 blocks super 1.2 512K chunks 2 near-copies [4/4] [UUUU]
      [>....................]  resync =  1.1% (23010240/1953260544) finish=163.9min speed=196278K/sec
      bitmap: 15/15 pages [60KB], 65536KB chunk

# when finished it looks like this
nvme1n1                     259:0    0 931.5G  0 linux_raid_member                          347f7cac-b875-ca2e-489a-06ae968510c6
└─md0                         9:0    0   1.8T  0 ext4              /media/user/md0          7d2d0f03-f1f7-4852-bcef-5f0f06659ca9
nvme0n1                     259:1    0 931.5G  0 linux_raid_member                          347f7cac-b875-ca2e-489a-06ae968510c6
└─md0                         9:0    0   1.8T  0 ext4              /media/user/md0          7d2d0f03-f1f7-4852-bcef-5f0f06659ca9
nvme3n1                     259:2    0 931.5G  0 linux_raid_member                          347f7cac-b875-ca2e-489a-06ae968510c6
└─md0                         9:0    0   1.8T  0 ext4              /media/user/md0          7d2d0f03-f1f7-4852-bcef-5f0f06659ca9
nvme2n1                     259:3    0 931.5G  0 linux_raid_member                          347f7cac-b875-ca2e-489a-06ae968510c6
└─md0                         9:0    0   1.8T  0 ext4              /media/user/md0          7d2d0f03-f1f7-4852-bcef-5f0f06659ca9

=========== harddisk usage
/dev/md0                         1.8T   77M  1.7T   1% /media/user/md0

# big sequential files
time /scripts/bench/bench_harddisk.sh 
=== harddisk sequential write and read bench v1 ===
starting test on the device that holds the current directory the user is in
no need to run it as root

========== writing 3GB of zeroes ==========
0+1 records in
0+1 records out
2147479552 bytes (2.1 GB, 2.0 GiB) copied, 2.289 s, 938 MB/s

real	0m2.298s
user	0m0.000s
sys	0m1.949s
========== reading 6GB of zeroes ==========
0+1 records in
0+1 records out
2147479552 bytes (2.1 GB, 2.0 GiB) copied, 0.855744 s, 2.5 GB/s

real	0m0.862s
user	0m0.008s
sys	0m0.853s
========== tidy up remove testfile ==========

real	0m3.291s <- jep , so here it is -10% faster
user	0m0.008s
sys	0m2.925s


# a lot of small files bench
time /scripts/bench/bench_harddisk_small_files.py 

create test folder:
0.00363707542419

create files:
9.89691901207

rewrite files:
1.9919090271

read linear:
1.61445593834

read random:
2.6266078949

delete all files:
4.63190793991

1st run: real 0m24.858s <- the NVMes in mdadm-RAID10 is +3.59658% slower than tested individually (no RAID)
2nd run: real 0m24.737s
3nd run: real 0m24.624s

# thats kind of unexpected
# because RAID10 SHOULD actually double the speed
# while adding redundancy

another test that could be performed:

create a 10Gbyte test-file with random data (rsync (no matter the compression level?) does not seem to be a good benchmark because it is cpu heavy)

# here the cpu is the limiting factor
time head -c 10240m /dev/zero | openssl enc -aes-128-cbc -pass pass:"$(head -c 20 /dev/urandom | base64)" > testfile

real	0m49.385s
user	0m37.556s
sys	0m18.791s
# and simply copy it from fileA to fileB within the device
du -hs testfile 
11G	testfile
du -s testfile 
10485768	testfile

# copy big file A to big file B (on same device)
time cp -rv testfile testfile2
'testfile' -> 'testfile2'

real	0m7.801s <- so it copied that file onto itself with 1.28188 GBytes / s
user	0m0.033s
sys	0m7.664s

# lets check that the copy process was done correctly
time md5sum testfile*
9596d43e1385d895924ecd66c015b46e  testfile
9596d43e1385d895924ecd66c015b46e  testfile2

# yep same files with same content
real	0m43.033s <- cpu again the limiting factor here
user	0m37.943s
sys	0m5.088s

#sysbench
sysbench fileio prepare
sysbench 1.0.20 (using bundled LuaJIT 2.1.0-beta2)

128 files, 16384Kb each, 2048Mb total
Creating files for the test...
Extra file open flags: (none)
Creating file test_file.0
Creating file test_file.1
...6
Creating file test_file.7
Creating file test_file.8
...
Creating file test_file.19
Creating file test_file.20
Creating file test_file.21
Creating file test_file.22
Creating file test_file.23
Creating file test_file.24
...
Creating file test_file.37
Creating file test_file.38
Creating file test_file.39
Creating file test_file.40
Creating file test_file.41
...
Creating file test_file.56
Creating file test_file.57
Creating file test_file.58
Creating file test_file.59
...
Creating file test_file.123
Creating file test_file.124
Creating file test_file.125
Creating file test_file.126
Creating file test_file.127
2147483648 bytes written in 2.41 seconds (848.94 MiB/sec).

time sysbench fileio --file-test-mode=rndrw run
sysbench 1.0.20 (using bundled LuaJIT 2.1.0-beta2)

Running the test with following options:
Number of threads: 1
Initializing random number generator from current time


Extra file open flags: (none)
128 files, 16MiB each
2GiB total file size
Block size 16KiB
Number of IO requests: 0
Read/Write ratio for combined random IO test: 1.50
Periodic FSYNC enabled, calling fsync() each 100 requests.
Calling fsync() at the end of test, Enabled.
Using synchronous I/O mode
Doing random r/w test
Initializing worker threads...

Threads started!

ee
File operations:
    reads/s:                      4296.19
    writes/s:                     2864.13
    fsyncs/s:                     9169.20

Throughput:
    read, MiB/s:                  67.13 <- that would be very bad X-D
    written, MiB/s:               44.75

General statistics:
    total time:                          10.0089s
    total number of events:              163388

Latency (ms):
         min:                                    0.00
         avg:                                    0.06
         max:                                   10.93
         95th percentile:                        0.27
         sum:                                 9926.76

Threads fairness:
    events (avg/stddev):           163388.0000/0.00
    execution time (avg/stddev):   9.9268/0.00


real	0m10.024s
user	0m0.222s
sys	0m1.174s

# tidy up afterwards
rm -rf test_file*

testing another benchmark https://unix.stackexchange.com/questions/93791/benchmark-ssd-on-linux-how-to-measure-the-same-things-as-crystaldiskmark-does-i

bench_harddisk_fio.sh.zip

apt install fio

/scripts/bench/bench_harddisk_fio.sh

What drive do you want to test? (Default: /root on /dev/mapper )
Only directory paths (e.g. /home/user/) are valid targets. Use bash if you want autocomplete.
/media/user/md0/

How many times to run the test? (Default: 5)

How large should each test be in MiB? (Default: 1024)
Only multiples of 32 are permitted!

Do you want to write only zeroes to your test files to imitate dd benchmarks? (Default: 0)
Enabling this setting may drastically alter your results, not recommended unless you know what you're doing.

Would you like to include legacy tests (512kb & Q1T1 Sequential Read/Write)? [Y/N]

cat: /sys/block/md/device/model: No such file or directory
cat: /sys/block/md/size: No such file or directory
/scripts/bench/bench_harddisk_fio.sh: line 132: *512/1024/1024/1024: syntax error: operand expected (error token is "*512/1024/1024/1024")

Running Benchmark on: /dev/md, (), please wait...

Results:

Sequential Q32T1 Read: 5736MB/s [ 175 IOPS]
Sequential Q32T1 Write: 2857MB/s [ 87 IOPS]

4KB Q8T8 Read: 1475MB/s [ 368807 IOPS]
4KB Q8T8 Write: 816MB/s [ 204230 IOPS]

4KB Q32T1 Read: 615MB/s [ 153984 IOPS]
4KB Q32T1 Write: 435MB/s [ 108936 IOPS]

4KB Read: 61MB/s [ 15485 IOPS]
4KB Write: 197MB/s [ 49408 IOPS]

Would you like to save these results? [Y/N]

Saving at /root/md2021-05-20110739.txt

# it is always important to watch the cpu not being the limiting factor
# or it would make for an unfair bench
hdparm -tT /dev/md0 

/dev/md0:
 Timing cached reads:   9810 MB in  2.00 seconds = 4911.95 MB/sec
 HDIO_DRIVE_CMD(identify) failed: Inappropriate ioctl for device
 Timing buffered disk reads: 7790 MB in  3.00 seconds = 2596.34 MB/sec

hdparm -tT /dev/md0 

/dev/md0:
 Timing cached reads:   10560 MB in  2.00 seconds = 5288.32 MB/sec
 HDIO_DRIVE_CMD(identify) failed: Inappropriate ioctl for device
 Timing buffered disk reads: 7834 MB in  3.00 seconds = 2611.05 MB/sec

now: let’s do some long term stress testing

as it is said, that those cards can become very hot

and RAID cards that “burn through” and lose all data ain’t no good for any production use

after 100x iterations the card became warm, but not hot. (as seen in the pic above, an additional 12cm cooler was put right in front of it for additional cooling, so that might have helped too)

cat /scripts/bench/often.sh

#!/bin/bash
echo "=== doing long term testing of harddisk ===" > /scripts/bench/bench_harddisk_small_files.py.log
for ((n=1; n<=100; n++)) do echo "===== running /scripts/bench/bench_harddisk_small_files.py for the $n time =====" >> /scripts/bench/bench_harddisk_small_files.py.log
	time /scripts/bench/bench_harddisk_small_files.py >> /scripts/bench/bench_harddisk_small_files.py.log
done
# where to write the files
cd /media/user/md0

# start the marathon benchmark
/scripts/bench/often.sh &
[1] 13841

# watch what it is doing
tail -f /scripts/bench/bench_harddisk_small_files.py.log

the card and the NVMes handled this bench with ease… the card got a little warm, but not hot (as many users reported, this is largely because GNU Linux mdadm (software raid aka the CPU) does all the RAID-work and the PLX chip is actually not-in-used)

This explains why the speed is HALF (RAID10 should be roughly double the speed of a single harddisk or NVMe in this case) of what would have been expected.

This is because there are actually no known good-drivers for GNU Linux Debian 10 for this card, compiling drivers for this card resulted in an error.

so let’s take it to the next level.

removed the addition 12cm cooler in front of it (not the one on the board… just the white one placed on the right side of the card)

and stressed card + NVMes like this…

cat /scripts/bench/often_sequential.sh
#!/bin/bash
echo "=== doing long term testing of harddisk ===" > /scripts/bench/bench_harddisk.sh.log
for ((n=1; n<=300; n++)) do echo "===== running /scripts/bench/bench_harddisk.sh for the $n time =====" >> /scripts/bench/bench_harddisk.sh.log
	time /scripts/bench/bench_harddisk.sh >> /scripts/bench/bench_harddisk.sh.log
done

# run it like this
/scripts/bench/often_sequential.sh & tail -f /scripts/bench/bench_harddisk.sh.log

# to get output like this

===== running /scripts/bench/bench_harddisk.sh for the 17 time =====
=== harddisk sequential write and read bench v1 ===
starting test on the device that holds the current directory the user is in
no need to run it as root

========== writing 3GB of zeroes ==========
0+1 records in
0+1 records out
2147479552 bytes (2.1 GB, 2.0 GiB) copied, 2.29723 s, 935 MB/s

real	0m2.303s
user	0m0.000s
sys	0m1.930s
========== reading 6GB of zeroes ==========
0+1 records in
0+1 records out
2147479552 bytes (2.1 GB, 2.0 GiB) copied, 0.861939 s, 2.5 GB/s

real	0m0.868s
user	0m0.000s
sys	0m0.861s
========== tidy up remove testfile ==========

real	0m3.292s
user	0m0.007s
sys	0m2.906s

indeed the card now was running warmer… but nothing of concern. (concern would be beyond 70C == too hot to touch)

now to the Windows Server 2012 R2 benchmarks:

as this card is intended to upgrade such an server and speed up it’s file based database X-D

 

amazing RAID10 (as expected speeds) of 2x a Kingston NVMe (which has read speed of ~2 GByte/sec)

but when running ATTO Benchmark things went wrong… one NVMe was reported to have failed… shortly after…

a second NVMe was reported to have failed…

after a reboot… it starts rebuilding the RAID10… so it’s not the NVMes who have failed… but the RAID card who got out of sync.

argh HighPoint… this does not look like a RAID card that a admin should trust critical data to 🙁 (at least not under Windows Server 2012 R2 )

how to realize NVMe RAID10?

2017: https://www.pcworld.com/article/3229947/amds-ryzen-threadripper-cpus-get-free-nvme-raid-support.html

admin