Yeah, after [part 3] it should be “part 4”. The final stage. However, while I’d love to present my final ~55TiB RAID-6 to you, I cannot do so yet, because there were and probably are some severe issues with the setup, which I will talk about down below. So, since my level of trust for Seagate is rather low because of the failure rates reported by Backblaze and my own experiences at work as well as the experiences from some other administrators I know, their line of Enterprise disks was out of the game. Another option would’ve been Hitachis Helium-filled Ultrastar He8, but since the He6 was reportedly rather disastrous, I don’t really want to trust those drives either.
This Helium stuff is just so new and daring, that I don’t want to trust them to be the very base of a RAID array that’s supposed to last for many, many years just yet.
Ultimately, I decided to get myself 12 insanely expensive Hitachi Ultrastar 7K6000 disks, “The last in Air” as they call ’em themselves. That’s a classic 5-platter 10-head airfilled enterprise disk with 7200rpm rotational speed and 6TB of capacity. I got the SAS/12Gbps version which also boasts 128MiB of cache. Mechanically, that’s all the same old tech that I’ve already been using with my 8 × 1TB Deskstars and now 8 × 2TB Ultrastars, so it’s something I can trust. However, as I said, there were/are some very serious issues. Maybe you remember this image:
So my old RAID-6 based on a 3ware 9650SE-8LPML with 8 × 2TB Ultrastars is sitting on the table, while the new one has been plugged into the Chieftec 2131SAS bays and hooked up to the Areca ARC-1883ix-12. Both RAID systems are thus connected to the same host machine at the same time, making it a total of 20 drives. This is supposed to make data migration using rsync very convenient and easy.
The problem is that I didn’t have enough power connectors for this (12 × SATA for the old array, ODDs and SSD, 8 × 4P Molex for the SAS bays), so I settled for Y-adapters to hook up the new array. Then the trouble started. At first I thought it was the passive SAS bays to blame. But as I continued my tests, drives would behave slightly differently as I exchanged and rotated the Y cables. What I observed was some weird “jitter”, where the drive heads were audibly moving around were they shouldn’t have, and sometimes drives would stall for a moment as well.
Ultimately, the array ran into a massive failure during init at about 60%, and 4-5 drives successively failed, collecting tons of recoverable read AND write errors in their S.M.A.R.T. logs. Bleh… At least no unrecoverable ones, but still…
At this point I ripped out half of the Y cables and hooked two of the four bays up to a dedicated power supply (only two, because of a lack of plugs). It seems this greatly changed the behavior of the whole setup, stabilizing it significantly. Of course it’s too early to say anything for sure, because now I’m just at roughly 25% through the second initialization process. But if I’m right, then a few 1€ parts have successfully wrecked a ~8000€ RAID array, now that’s something, eh?
In any case, before getting my Ultrastars I also tried the system with some Seagate Cheetah 15k.6 and 15k.7 drives I managed to borrow at work, 300GB 15000rpm SAS pieces, just for some benchmarks. Since those showed more severe problems even than the Hitachis (probably because they’re more power hungy?), I went down to 11, then 8 drives. Some of the benches will also show sudden stalls. Yeah. That’s the power issue.
Well, it can still serve as a quick glance at the performance levels one can expect with the Areca ARC-1883ix-12, even in such a state. Let me just say: It is a nice feeling to see a RAID array based on mechanical drives push 1000-1200MiB/s over the bus on average, reading at 64kiB-1MiB block sizes. At least that part is undeniably awesome! Here are a few screenshots for you, RAID stripe block sizes are always 64kiB, read block sizes are 4kiB, 64kiB and 1MiB, write block sizes are 64kiB, 512kiB and 1MiB. For the RAID-6 setup there are also benches during init and in 2-disk degraded mode, software’s just a cheap HDTune 2.55 + HDTune Pro 5.00 for now.
Ah yes, you might be wondering why the CPU usage is so high. Well, these were just quick preliminary tests anyway, so some video transcoders were running in the background at the same time, that’s why. Here we go:
RAID-0, 8 × 15000rpm Cheetahs, reads:
RAID-0, 8 × 15000rpm Cheetahs, writes:
RAID-6, 11 × 15000rpm Cheetahs, reads in normal state:
RAID-6, 11 × 15000rpm Cheetahs, reads during initialization:
RAID-6, 11 × 15000rpm Cheetahs, reads in 2-disk degraded mode:
The performance degradation due to the initialization process is somewhat in line with what’s configured on the controller itself, giving the background process a low 20% priority. The degradation in 2-disk degraded mode is what’s really interesting though. Here we can see that the 1.2GHz dual core PowerPC RAID engine is seriously powerful. With double parity computation required on the fly, the array still delivers 64kiB transfer rates in excess of 800MiB/s! That’s insane! I was hoping for normal transfer rates over 600MiB/s, but this really waters ones mouth!
Of couse, all of this is still preliminary, my array still doesn’t work and these aren’t the final drives running through the tests, nor is the controller fully configured yet. Let’s just hope that I can get a grip on that situation soon… because all these problems are seriously pissing me off already, as you may be able to understand, given the price of the hardware and the pressing issue that I’m running out of space on my old array.
Well, let’s hope a real “part 4” can follow soon!
Edit: And finally, [here it is]!
Building the 54.5TiB “Taranis” RAID-6 array and the hardware around it, part 3½: Disasters and also some benchmarks by The GAT at XIN.at is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.