Storage Monkeys

Complete Data loss? How about snap or clone the data before upgrading? Or backup? or replication? That's 4 commonly used tactics to prevent data loss for any DAS, NAS, or SAN..

(Discloser: I am not affiliated, employed, receiving payments of any kind from Microsoft, HDS, Danger, T-Mobile (or it's affilates Sidekick), and virtualizarion that may or may not have been deployed; nor am I affilated with any Bloggers (other than myself), analysts, or any other long-winded pundits! LOL! I have a feeling I will get a 74 line dissertation on my flippant attitude towards disclosure... my pre-emptive response is PPPPBBBTTTTT!!!!)

LOL RBruklis and concur with Jpolk,

Does this mean that CDP = Cloud Data Problem, or, Cloud Data Protection Problem (CDPP)?

Oh no, Im probally in trouble again now with the cloud storage police or cloud crowd cheerleaders for saying that, oh well, some will or would have if not me! Besides, I use a cloud/msp/baas or what ever you choose to call it provider as part of my data proteciotn, that is, I dont rely solely on it, rather, use multiple tools/techniques.

Ok, how about CDP = Complete/Comprehensvie Data Proteciton or, Commonsense Data Protection?

In other words, if your data is valuable, why not have multiple copies in differnet locaitons, mediums, locally or remotely?

Why put all your eggs in one basket so to speak?

What ever happended to common sense or did it go out the door with the low cost craze?

I hope that was not too long winded

Cheers - gs

I particularly liked "Hey SNIA, want to make yourself relevent? Get out of your bubble and try addressing the black eye that cloud storage is taking today."

C'mon... it's SNW... the media is right there... SAY SOMETHING!

Tim I thought you were going to say "Hey, SNIA, get out of the clouds" (Think to a Rolling Stones melody)

Disclosure: Im not at SNW so cant say how many media people are actually there

Cheers gs

How the HECK do you even lose data during a SAN upgrade? And if you DO, how do you NOT have a backup of it?

I'm talking to a Sidekick user, and this has been going on for five days, and she said that early on they had her reset her phone by pulling the battery, ensuring that she'd lose all her contacts, etc. Basically, they wouldn't admit (or maybe even didn't know) that it was all their fault due to an outage. She could have re-uploaded her contacts if she didn't erase them, but she did what the tech rep said to do and voila! No more data.

BTW, opportunistic or not, this is my problem with cloud storage. How do you know that the company that is storing your data isn't as big of an idiot as these guys (or the other guys before them that have lost people's data)? How do you do that? MAYBE if you had a cloud storage provider being backed up by a different cloud backup provider?

My idea puts storage in the hands of people good at storage and backup in the hands of people good at backup. AND, yes, it lessens the chances that BOTH companies would have a catastrophic failure like the one that started this article.

I agree with your second post. My concern is: How does an end user of cloud storage verify that the provider has a tested and proven DR strategy? And if they verify it this month, how do they verify that they keep up with the policies/procedures?

As long as the primary failure gets glossed over, and the "but there should have been a backup" diversion is allowed, this sort of thing will continue.

Man1: I hear you're playing the lottery.
Man2: Yes.
Man1: What if your ticket isn't a winner?
Man2: I bought a second ticket just in case.


If you buy service from two different cloud providers, its not like the second one is automatically going to offer you a fail proof scenario for free when the first does not. No more so than having insurance coverage on both you and your spouse is going to provide 100% coverage. Coordination of benefits limits the insurance companies exposure.

Offer some marketing fluff, and guarantee nothing. Cloud is all about what people want, and not about what they need.

I'm the backup guy so that's what I talk about. What can I say?

But I definitely don't agree with your lottery analogy. My point was hiring specialists in each type of technology instead of going with one company to do both. I don't think that has anything to do with a lottery.

may I use it in my training classes?

As an end user I think this is really funny. I feel bad for the screwed over users of the Sidekick but otherwise... I think I actually lol'd when I read the original article. I just love it when companies (especially big ones) do this to themselves. It's like watching a train wreck, or Wall street, or a Kevin Costner movie. Yeah there's some damage to peoples lives and the world is never quite the same after but dang if the chaos isn't fun to watch... for a few minutes. Attention spans just aint what they used to be. Naturally this does nothing to to improve my confidence in cloud storage and I'm cynical enough to believe that this won't drive any major change in the industry. Basically a group of humans completely dropped the ball. Those out there in the cloud 'verse that know they have things under control are laughing and those that don't were ignorant (and relying on luck!) before this happened and likely still are. Curtis is absolutely correct. How are we supposed to know under which category they fall? Go cloud!

I only see the market from one perspective; my experience is effectively anecdotal.

Several of the large storage shops I've worked in look at backup and recovery as a buried cost they would as well avoid. Most of the products and services (including cloud) being aggressively marketed to us upsell removing most of the expense of managing and protecting storage.

Implied in the "cloud message" is "the vendor doesn't need to worry about backing anything up, because they don't have your stupid IT people in the way screwing everything up". And weirdly enough, the Cxx level folks I've talked to lately are eating this message up.

This recent incident just underlines the hubris of the storage vendors. Cloud isn't magic, its just storage vendors figuring out that if they don't charge themselves the normal exorbitant markup on tin, they can move into a new market segment. The fact that their equipment is still the same marginal shit it always has been is lost on them, as they're hiring people to staff out cloud who have no incentive to cry "bullshit" at the setups being built.

Look at the job postings for "build our cloud at vendor X". They are putting people on a very difficult mission, and tried and true fault tolerance is no where on that list. Instead the "magiclly copy stuff around so we avoid a failure" approach appears to be the norm. Sounds great until the "magic copy routine" does something stupid two times in a row, and then then some poor customer is screwed.

This all sounds painfully close to a page out of the Telco playbook: Offer something big, guarantee nothing.

I never understood why people didn't see through the premise of Enron either.

(I heard that joke about the lottery tickets somewhere, it didn't originate with me.)

The Man sent Chico to the store with some small amount of pocket change to get The Man a good cigar. (The Man knew Chico couldn't get a good cigar for that amount of money, but he wanted Chico to be out of his hair for a while.)

Chico came back later in the morning, with a good cigar, which The Man started to smoke.

While The Man smugly enjoyed the first few puffs of the fine cigar, he asked how Chico managed to get the cigar so cheaply.

Chico's reply: I knew there was no way I could buy a fine cigar at that price, so instead I bought a roll of tape and used it to connect a number of the fine cigar butts that I found outside the store.

The moral of the story? Don't think that somehow you're pulling a fast one on the cloud vendors and getting a better level of protection than you're paying for. The storage companies are still packaging the same commodity gear they always have, they've just found a different way to charge for it.

rclark: That;'s a riot... and I'm glad there's someone more cynical than me on here!

This is pretty interesting but considering the source, Apple, I am reluctant to believe even half of it.

http://www.appleinsider.com/articles/09/10/12/ microsofts_sidekick_pink_problems_blamed_on_dogfooding_
and_sabotage.html

The very long outage of core functionality, followed by an incapacity to recover any data, both point to the possibility that "someone with access to the servers at the datacenter must have inserted a time bomb to wipe out not just all of the data, but also all of the backup tapes, and finally, I suspect, reformatting the server hard drives so that the service itself could not be restarted with a simple reboot (and to erase any traces of the time bomb itself)."

Just so all my posts aren't negative, here is some vintage Chico And The Man: http://www.youtube.com/watch?v=VrbDcn0C57c

rclark said:

"Implied in the "cloud message" is "the vendor doesn't need to worry about backing anything up,"

Any vendor that even remotely suggested that to me would be immediately escorted from the premises. It's as bad as when a storage vendor tells me snapshots are so good that you don't need backup.

"its just storage vendors figuring out that if they don't charge themselves the normal exorbitant markup on tin, they can move into a new market segment."

No, that is not the majority of the cloud. It might be a majority of the vendors, but the bulk of cloud customers are not on traditional storage vendors' storage. They're on something like Amazon, Mozy (which was viable before EMC bought them) and Nirvanix.

"The storage companies are still packaging the same commodity gear they always have, they've just found a different way to charge for it."

Only a few companies are doing that (e.g. NetApp). Most are using completely different gear (e.g. Atmos) when addressing the cloud.

Clearly, Danger should have backed up user data as a matter of good business conduct. I don't think anyone would disagree with that. Danger screwed up royally.

Now for the part few people want to talk about...mostly because it's a touchy subject with consumers and many writers are too busy scolding businesses to take the time to reflect on the broader picture of responsibility and accountability that includes consumers.

Many consumers rarely thoroughly read the fine print of agreements that they sign, or they simply ignore the fine print altogether. How many people thoroughly read the terms of service of their ISP, web host, or bank before signing agreements to establish new accounts? Not many. I don't need to conduct a survey. The Internet is filled with uncountable posts written by irate consumers who learned about the details in the fine print only after an event such as the one discussed above.

This has shades of the non-compete agreement debates I participated in a few months ago. One camp believes that seemingly draconian fine print is evil and should not be permitted - period. The other camp believes that people shouldn't sign agreements if they're not willing and prepared to live with the consequences.

I belong to the second camp. We're all adults here. And ignorance is never a defense in court. "Your honor I didn't know that my service provider was not liable for my data loss. Had I known I would never have agreed to use its service." That doesn't fly in court. If we knowingly sign agreements that clearly waive liability and hold harmless if a company fails to deliver, then we have little or no defense when the supplier actually fails to deliver.

Many consumers choose, in the age of the Internet, to play a new game called 'I Can Do Whatever I Want Without Consequences'. That includes, apparently, signing agreements no matter the language. Why? Because society later allows them to create massive costly online PR disasters in a quest to get something (reimbursement, product replacements, free services, etc.) that they, from a legal standpoint, do not deserve. I rarely find a comment of the type I am writing now - one that questions the responsibility and accountability of the individuals signing the contracts. Comments such as mine are a quick way to attract one thousand consumers who will attempt to rationalize why they are right and their supplier is wrong - every one of them glossing over the fact that they signed an agreement that let the company get away with just about anything.

Below (in a separate comment) you will find relevant excerpts of T-Mobile and Danger Terms of Service. All emphasis (bold and underline) is mine. The language is pretty clear and typical. Danger caps damages at $200 per customer. Fair or not, this is the language of the agreement that Sidekick users signed. Perhaps consumers may someday band together and nationally boycott companies that insert such language into their terms of service...a grassroots effort to change the nature of the legal agreements that they sign. I doubt it. It is far more likely that they will continue to carelessly sign such agreements and continue to take their fight to the Internet when things go wrong - hoping that if they collectively scream loud enough - maybe throw a public temper tantrum or two - they'll get their way regardless what they agreed to in their contracts. Human behavior at its finest.

Danger was wrong to do what it did - there is no excuse for not having a backup. But I ask myself, are we any better if we expect not to be held accountable and responsible for our own actions (i.e. signing a contract that waives supplier liability)? What then, is the point of having contracts?

I'm going to read your responses with great interest.

Disclaimer: I'm not fond of T-Mobile, have never spoken to Danger and have done business (compensated and not) with Microsoft and HDS.

Exhibit A - T-Mobile Terms of Service:

http://www.t-mobile.com/Templates/Popup.aspx?WT.z_unav=f
tr__useterms&PAsset=Ftr_Ftr_TermsOfUse&print=true

"No Warranties by T-Mobile -- T-MOBILE MAKES NO EXPRESS REPRESENTATIONS OR WARRANTIES ABOUT ITS SERVICES AND DISCLAIMS ANY AND ALL IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, WARRANTIES OF TITLE OR IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. T-MOBILE DOES NOT AUTHORIZE ANYONE TO MAKE A WARRANTY OF ANY KIND ON ITS BEHALF AND YOU MAY NOT RELY ON ANY SUCH STATEMENT OF WARRANTY. T-MOBILE IS NOT THE MANUFACTURER OF ANY OF THE PRODUCTS, AND THE ONLY WARRANTY APPLICABLE TO THE PRODUCTS USED IN CONNECTION WITH THE SERVICES IS THAT PROVIDED BY THE PRODUCT MANUFACTURER(S). THIS SECTION SURVIVES TERMINATION OF THIS AGREEMENT."

"Limitation of Liability -- Neither T-Mobile, nor its parent or any of its affiliates, nor the directors, officers, employees, or other representatives of any of them, is liable for any type or amount of damages arising out of or in connection with the use of the Products. This is a comprehensive limitation of liability that applies to all damages of any kind, including without limitation compensatory, direct, indirect, consequential, incidental, special, exemplary, or punitive damages; loss of data, income, or profit; loss of or damage to property; and any and all claims of third parties."

Exhibit B - Danger's Terms of Service:

http://www.danger.com/legal/service_license_terms.php

IMPORTANT: PLEASE CAREFULLY READ THIS SERVICE AND END-USER LICENSE AGREEMENT (THE "AGREEMENT") BETWEEN YOU AND DANGER, INC., A SUBSIDIARY OF MICROSOFT CORPORATION, ("DANGER") BEFORE YOU USE YOUR WIRELESS DEVICE (THE "DEVICE"). YOUR USE OF THE DEVICE OR THE SERVICE FOR THE DEVICE SHALL CONSTITUTE YOUR BINDING ACCEPTANCE OF ALL OF THE TERMS AND CONDITIONS OF THIS AGREEMENT. IF YOU DO NOT AGREE TO ALL OF THE TERMS AND CONDITIONS OF THIS AGREEMENT, THEN YOU MAY NOT USE THE DEVICE OR THE SERVICE AND SHOULD PROMPTLY RETURN THE DEVICE TO THE PLACE WHERE YOU PURCHASED IT.

7. NO WARRANTIES. THE SERVICE AND SOFTWARE ARE PROVIDED BY DANGER ON AN "AS IS" BASIS. DANGER, ITS SUPPLIERS AND SERVICE PROVIDERS HEREBY DISCLAIM ALL REPRESENTATIONS OR WARRANTIES OF ANY KIND, WHETHER EXPRESS, IMPLIED, OR STATUTORY, RELATING TO THE SERVICE OR THE SOFTWARE (INCLUDING ANY INFORMATION, CONTENT, MATERIALS OR PRODUCTS THAT YOU MAY ACCESS THROUGH THE SERVICE), INCLUDING WITHOUT LIMITATION ANY IMPLIED WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, QUIET ENJOYMENT, NON-INFRINGEMENT OF THIRD PARTY RIGHTS, ACCURACY OF INFORMATIONAL CONTENT, AND ANY WARRANTIES ARISING FROM A COURSE OF DEALING OR USAGE OF TRADE. WITHOUT LIMITING THE FOREGOING, YOU ACKNOWLEDGE THAT DANGER, ITS SUPPLIERS AND SERVICE PROVIDERS DO NOT WARRANT OR REPRESENT THAT THE SERVICE OR THE SOFTWARE WILL MEET YOUR REQUIREMENTS, THAT THE SERVICE OR THE SOFTWARE WILL BE TIMELY, SECURE, UNINTERRUPTED, OR ERROR-FREE, THAT DEFECTS IN THE SERVICE OR THE SOFTWARE WILL BE CORRECTED, THAT ANY CONTENT OR INFORMATION CONTAINED IN THE SERVICE OR ACCESSED THROUGH YOUR USE OF THE DEVICE WILL BE ACCURATE, COMPLETE, RELIABLE, OR ERROR-FREE, THAT THE SOFTWARE OR ANY MATERIALS AVAILABLE FOR DOWNLOAD FROM THE SERVICE WILL BE FREE OF VIRUSES OR OTHER HARMFUL COMPONENTS, OR THAT THE SERVICE WILL BE FREE FROM UNAUTHORIZED ACCESS (INCLUDING THIRD PARTY HACKERS OR DENIAL OF SERVICE ATTACKS). FURTHER, AS THE WIRELESS CARRIER IS NOT CONTROLLED BY DANGER, NO WARRANTY IS MADE AS TO COVERAGE, AVAILABILITY OR GRADE OF SERVICE PROVIDED BY THE WIRELESS CARRIER. PLEASE NOTE THAT SOME JURISDICTIONS MAY NOT ALLOW THE EXCLUSION OF IMPLIED WARRANTIES, SO SOME OF THE ABOVE EXCLUSIONS MAY NOT APPLY TO YOU.

8. LIMITATION OF LIABILITY. IN NO EVENT SHALL DANGER, ITS SUPPLIERS OR SERVICE PROVIDERS BE LIABLE TO YOU FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, CONSEQUENTIAL, EXEMPLARY, PUNITIVE, OR OTHER DAMAGES (INCLUDING, WITHOUT LIMITATION, DAMAGES FOR LOSS OF PROFITS, BUSINESS INTERRUPTION, LOSS OF INFORMATION OR DATA, OR OTHER PECUNIARY LOSS) RESULTING FROM YOUR ACCESS TO, OR USE OR INABILITY TO USE THE SERVICE OR SOFTWARE (INCLUDING AS A RESULT OF AN OUTAGE OF THE SERVICE PROVIDED BY YOUR WIRELESS CARRIER OR ANY OTHER THIRD-PARTY SERVICE PROVIDER, OR WITH RESPECT TO THE INFORMATION, SERVICES, CONTENT OR ADVERTISEMENTS CONTAINED ON OR OTHERWISE ACCESSED THROUGH THE SERVICE. IN NO EVENT WILL DANGER, ITS SUPPLIERS OR SERVICE PROVIDERS BE LIABLE TO YOU IN CONNECTION WITH THE SERVICE OR THE SOFTWARE FOR ANY DAMAGES IN EXCESS OF TWO HUNDRED U.S. DOLLARS (U.S. $200).

9. Indemnity. You agree to defend, indemnify, and hold harmless Danger, its officers, directors, employees and agents, from and against any claims, actions or demands, including without limitation reasonable attorneys' fees, made by any third party due to or resulting from your violation of this Agreement.

btw, Tim, someone has got to do something about the carriage returns in the comments for this blog. That is supremely annoying. If I don't remember to put periods or some other character on empty lines between paragraphs then everything is bunched together as it is above.
.
Please, for readability sake, find a fix for it.

I damn near forgot my point during that long-winded comment - so what's my point?
.
My point is this - cloud or no cloud, if you're going to do business with providers that include waivers of liability and hold harmless clauses in their contracts - think long and hard about it. Once you sign on the dotted line, you've given them permission to fail - intentionally or not - and you may lose irreplaceable data. Proceed at your own risk.
.
I could have simply stated this earlier, but I prefer to drive home the depth of consumer recklessness in dealing with suppliers. I speak from experience having burned myself on more than one occasion in the past after failing to read the fine print.

You are right that the fine print is the device that allows companies to get away with being stupid/unreliable/whatever without much recourse (in the form of lawsuits). If I was a service provider I'd do the same thing. I'm sure Danger had no intention (as an entity) of allowing this event to occur but here we are. The data loss is lame and completely unforgivable in the professional world but I agree that there's not much that can (or should) litigiously be done... other than switch away from the Sidekick. That's what I'd do. Incidentally, Joseph, your comments are long enough that they really should be their own blog - especially since the focus shifts from cloud fail to customer fail. Go lawyers!

Legal disclaimers are great but... that simply means there's no redress for Sidekick owners. It does nothing for the image of Microsoft as not being a reliable service provider.