Quadzilla's BIG_DISK!

I've recently added some more storage to my server. Here's an example of a do-it-yourself software based RAID array built on fibre channel arbitrated loop (FC-AL) disks. Yes, contrary to popular belief, it's actually possible to wire up your own loop of disks. It's actually kind of amazing that this hand-wired stuff actually works with signals running over these little wires at 1.0625 gigabaud!

The heart of this project is the 'T-Card'. This little sucker attaches to the FCAL port on the back of the disk and gives you connections for power and loop in/out signals, as well as jumpers for setting the arbitrated loop physical address (ALPA).

Wiring up the drives is pretty straightforward. You attach the loop in to the loop out of the previous disk in the chain. You continue this process for all the disks you want to add, creating one large loop. The disadvantage of this type of installation is that if any disk is powered off, explodes, or falls off the loop somehow, all the disks vanish! So, obviously, you probably wouldn't want to do this with anything that's mission critical.

3 drives connected in a test loop on the workbench

I built a loop with 14 36Gb Seagate ST136403FC disks I had sitting in a box. (which didn't work in the Sun A5000 I bought, damn Sun and their proprietary firmware!) Anyway, I set their ALPAs from 100 to 113 so they wouldn't conflict with anything else on my SAN.


Inside the 14 bay SCSI box, showing the wiring.	The cabling shown here is just short pieces of regular solid CAT5 network cable, with the ends tinned. Yes, that's a DB9 PC card bracket I screwed into the back of the case.

Tower of disks from the front. Here's one creative way to use 300 watts of power.

Interestingly enough, this actually works really quite well. The Emulex LP8000 driver sees these disks as regular SCSI disks. My /proc/scsi/scsi looks like the following text. /dev/sda through /dev/sdf are the six internal disks. /dev/sdg and /dev/sdh are the LSI Logic RAID unit, and /dev/sdi through /dev/sdv are my new FCAL disks.

Attached devices:
Host: scsi0 Channel: 00 Id: 02 Lun: 00
Vendor: TOSHIBA Model: CD-ROM XM-6401TA Rev: 1009
Type: CD-ROM ANSI SCSI revision: 02
Host: scsi0 Channel: 00 Id: 04 Lun: 00
Vendor: SONY Model: CD-R CDU948S Rev: 1.0g
Type: CD-ROM ANSI SCSI revision: 02
Host: scsi1 Channel: 00 Id: 00 Lun: 00
Vendor: SEAGATE Model: ST318203LSUN18G Rev: 034A
Type: Direct-Access ANSI SCSI revision: 02
Host: scsi1 Channel: 00 Id: 01 Lun: 00
Vendor: SEAGATE Model: ST318203LSUN18G Rev: 034A
Type: Direct-Access ANSI SCSI revision: 02
Host: scsi1 Channel: 00 Id: 02 Lun: 00
Vendor: SEAGATE Model: ST318203LSUN18G Rev: 034A
Type: Direct-Access ANSI SCSI revision: 02
Host: scsi1 Channel: 00 Id: 03 Lun: 00
Vendor: SEAGATE Model: ST318203LSUN18G Rev: 034A
Type: Direct-Access ANSI SCSI revision: 02
Host: scsi1 Channel: 00 Id: 04 Lun: 00
Vendor: SEAGATE Model: ST318404LSUN18G Rev: 4207
Type: Direct-Access ANSI SCSI revision: 03
Host: scsi1 Channel: 00 Id: 05 Lun: 00
Vendor: FUJITSU Model: MAG3182L SUN18G Rev: 1111
Type: Direct-Access ANSI SCSI revision: 02
Host: scsi1 Channel: 00 Id: 06 Lun: 00
Vendor: ESG-SHV Model: SCA HSBP M6 Rev: 0.63
Type: Processor ANSI SCSI revision: 02
Host: scsi3 Channel: 00 Id: 05 Lun: 00
Vendor: LSI Model: INF-01-00 Rev: 0400
Type: Direct-Access ANSI SCSI revision: 03
Host: scsi3 Channel: 00 Id: 05 Lun: 01
Vendor: LSI Model: INF-01-00 Rev: 0400
Type: Direct-Access ANSI SCSI revision: 03
Host: scsi3 Channel: 00 Id: 100 Lun: 00
Vendor: SEAGATE Model: SX136403FC Rev: E00F
Type: Direct-Access ANSI SCSI revision: 02
Host: scsi3 Channel: 00 Id: 101 Lun: 00
Vendor: SEAGATE Model: SX136403FC Rev: E00F
Type: Direct-Access ANSI SCSI revision: 02
Host: scsi3 Channel: 00 Id: 102 Lun: 00
Vendor: SEAGATE Model: SX136403FC Rev: E00F
Type: Direct-Access ANSI SCSI revision: 02
Host: scsi3 Channel: 00 Id: 103 Lun: 00
Vendor: SEAGATE Model: SX136403FC Rev: E00F
Type: Direct-Access ANSI SCSI revision: 02
Host: scsi3 Channel: 00 Id: 104 Lun: 00
Vendor: SEAGATE Model: SX136403FC Rev: E003
Type: Direct-Access ANSI SCSI revision: 02
Host: scsi3 Channel: 00 Id: 105 Lun: 00
Vendor: SEAGATE Model: SX136403FC Rev: E00F
Type: Direct-Access ANSI SCSI revision: 02
Host: scsi3 Channel: 00 Id: 106 Lun: 00
Vendor: SEAGATE Model: SX136403FC Rev: E00F
Type: Direct-Access ANSI SCSI revision: 02
Host: scsi3 Channel: 00 Id: 107 Lun: 00
Vendor: SEAGATE Model: SX136403FC Rev: E00F
Type: Direct-Access ANSI SCSI revision: 02
Host: scsi3 Channel: 00 Id: 108 Lun: 00
Vendor: SEAGATE Model: SX136403FC Rev: E003
Type: Direct-Access ANSI SCSI revision: 02
Host: scsi3 Channel: 00 Id: 109 Lun: 00
Vendor: SEAGATE Model: SX136403FC Rev: E003
Type: Direct-Access ANSI SCSI revision: 02
Host: scsi3 Channel: 00 Id: 110 Lun: 00
Vendor: SEAGATE Model: SX136403FC Rev: E00F
Type: Direct-Access ANSI SCSI revision: 02
Host: scsi3 Channel: 00 Id: 111 Lun: 00
Vendor: SEAGATE Model: SX136403FC Rev: E003
Type: Direct-Access ANSI SCSI revision: 02
Host: scsi3 Channel: 00 Id: 112 Lun: 00
Vendor: SEAGATE Model: SX136403FC Rev: E003
Type: Direct-Access ANSI SCSI revision: 02
Host: scsi3 Channel: 00 Id: 113 Lun: 00
Vendor: SEAGATE Model: SX136403FC Rev: E00F
Type: Direct-Access ANSI SCSI revision: 02
Host: scsi3 Channel: 00 Id: 124 Lun: 00
Vendor: ADIC Model: Scalar 1000 Rev: 3.00
Type: Medium Changer ANSI SCSI revision: 02
Host: scsi3 Channel: 00 Id: 124 Lun: 01
Vendor: SONY Model: SDX-500C Rev: 0200
Type: Sequential-Access ANSI SCSI revision: 02
Host: scsi3 Channel: 00 Id: 124 Lun: 02
Vendor: SONY Model: SDX-500C Rev: 0200
Type: Sequential-Access ANSI SCSI revision: 02
Host: scsi4 Channel: 00 Id: 01 Lun: 00
Vendor: ADIC Model: FastStor DLT Rev: 0119
Type: Medium Changer ANSI SCSI revision: 02
Host: scsi4 Channel: 00 Id: 03 Lun: 00
Vendor: Quantum Model: DLT4000 Rev: D996
Type: Sequential-Access ANSI SCSI revision: 02

Using the normal Linux RAID-tools, I built a 14 disk RAID-5 as /dev/md0, formatted it (which took a little over two hours) and created an ext3 file system on the beast. Mounted it up, and now I have about 450 gigs more space!

Filesystem 1k-blocks Used Available Use% Mounted on
/dev/sda2 5039616 823060 3960556 18% /
/dev/sda1 495828 43041 427188 10% /boot
/dev/sdb1 16126904 1844816 13462892 13% /home
none 515516 0 515516 0% /dev/shm
/dev/sda5 9071072 1929112 6681164 23% /usr
/dev/sdg1 244483852 21459544 210605264 10% /shares/pub
/dev/sdh1 34653160 32828 32860012 1% /shares/tmp
/dev/md0 455023072 21459552 410449688 5% /shares/big_disk

So, for those counting, I have 6 disks internally, 20 disks in the LSI RAID unit, and 14 in the FCAL box. That's 40 disks!

Stuff to come:

The cooling of this setup leaves something to be desired. I've been monitoring the internal temperature of the disks, which seem to hover around 40c. Warm, but within the acceptable operating range of these disks. The box has really poor internal airflow characteristics despite the huge fan on the back of the case. What I've been doing is leaving the front door open slightly so air is sucked in the front and flows over the disks. So far this has been sufficient but if someone were to close the door completely, I think we'd have Seagate meltdown. I've also never felt so much hot air come out of an AT-type power supply. 300 watts will do that, I guess.

I've got a few copper to optical media converters that I'm trying to get operational with this array. At the moment its connected to the fibre channel switch with an active copper GBIC, which works fine. I'd prefer to use optical cabling and interfaces if I can, but for some reason the switch will not initialize the loop when I use the media interface adapters. If anyone has any idea why, I'd love to know!

I wired up some LEDs to the activity light connectors on the disks so I can watch them blink. The drives also have numerous other LED outputs that might be helpful, especially in debugging should something go wrong. It might be useful to hook those up too.

Each disk actually has two loop interfaces. It might be advantageous to wire up both loops, in case something goes wrong with one loop. Then of course, I could access the disk via the other loop. Of course, if one of the disks goes up in smoke, both loops will open and that would be of no help!

More to come, maybe- depending on if anything changes!

That is all!