Is deduplication a strategy or a finger in the dike?

Posted by: Tim in De-Duplication on July 2, 2009

There is no doubt that the various deduplication technologies address a specific set of problems. For the purposes of this blog post, I'm going to focus on backup deduplication.

When most companies look at backup deduplication, they are typically trying to solve one or more of the follow problems:

1. Backup window - data growth is causing the backup process to bleed past the backup window and into production
2. Retention on disk - either retaining some portion of backups on disk or growing the number of days data is being retained on disk without ridiculously large amounts of disk storage
3. Replication - reduce the amount of data that needs to be replicated to a remote site thereby lowering the cost of bandwidth

IT departments looking for one of these solutions have identified a problem with their backup processes and are looking for a solution to fix the problem. Simple enough... but are these problems REALLY the issue or are they symptoms of a larger problem with the tradional backup?

Let's think about it - how much have IT departments changed in the last 20 years? Pretty dramatically when you think about it. One thing that has not changed is backup. The same backup software you are using today is not all that different from the backup software used 20 years ago. Functionally, you are taking daily, weekly, monthly and maybe yearly backups of your data. The process hasn't changed and for the most part, backup administrators haven't evolved much either. Ask yourself or your backup administrator why they are using backup software and the likely answer is "Well, that's what we've always done and it works just fine". If it's working for you, then you shouldn't need a deduplication solution, right?

The truth is that backup software is very outdated technology. We use it because we've always used it, not because it's the best or most effective way to protect our data - and I'm just as guilty of this as anyone else. It's the easy way but it's not the best way. Many of the leading storage and backup companies offer alternative strategies (ie snapshots) that are more effective and more efficient at data protection  - and do it at a lower cost. Each of the problems that backup deduplication solves are problems with the technology and less sophisticated IT departments opt to not solve the technology problem, they simply patch it.

Deduplication is a patch, not a solution.

So let me throw this question out - why are you using tradional backup software? Give me a technical reason, not a process reason.