One of the things I've learned, having been in more traditional "Enterprise" environments and "Internet" companies is that the latter have much larger scale issues, with respect to storage, by an order of magnitude or two, than the former.
Having purchased, assembled, configured, and turned up quite a number of storage arrays, where a major concern was total cost, I've come up with something of a checklist of best practices.
Use cheap, commodity, desktop SATA drives. They're as good, if not better than, "enterprise" models. They're certainly cheaper per performance.
Having recently read more and more discussion about so-called dark storage, I've been reminded of something I routinely try to impress upon managers, especially clients: unless your use case is archiving, total bytes is a poor metric for storage.
In fact, the term "storage" itself may be partly to blame for the continued misconception. One need only glance at the prices of commodity disks to recognize that there isn't anything near a linear relationship between cost and bytes stored.
As a UNIX veteran who has a vague recollection of /dev/drum, I keep thinking that it would be really nice to have a device to swap to that's somewhere between disk and memory in terms of speed and cost (total installed cost, not just each module).
Mostly, I feel constrained by the 32-48GB limits on moderately priced ($1-3k) servers. To go higher, for even modest processor speeds, is a $5-$10k premium. Moreover, DRAM doesn't really wear out, and it would be nice to put older, lower density modules to use.
The choice of the unit of measure of storage is interesting to me because it's otherwise tought to measure price for performance.
I remain agape at the price tag on high-end, supposedly high-performance, storage systems. Connected by FibreChannel or gigabit Ethernet, that's a limit of 400 and 110 MB/s, respectively. (Yes, I know of 8Gb/s FC and 10GE, but these are prohibitively expensive, if supported.Even link-aggregated GigE practically tops out at 880MB/s) I'm thinking that writes across 40 7200RPM disks could saturate an FC link, and it would take fewer than 20 15k disks. Neither of these strikes me as impractical or unusual sizes of storage arrays, even doubling those numbers for RAID 1. More importantly, such arrays don't strike me as high performance.
Recently, I had a discussion with a colleague about storage performance, and he kept talking about IOPS, whereas I have always measured it with the, perhaps more traditional, bytes per second. Since IOPS is effectively the reciprocal of latency, I have tended to ignore it for disk storage, as I have yet to see any use case which is synchronous, let alone sensitive to sub-centiseond latencies.
The alleged use case is a random write-heavy Oracle instance. I confirmed with a DBA I know that Oracle's block sizes will range from 4 to 32kiB. That suggests that the worst case random I/O can't occur, as the payload for each operaton will be between 8 and 64 sectors. Still, I no longer have the data for benchmarks I ran, to be able to quantify how much of a difference this might make.
