The current state of Deduplication

I recently attended one of the many Enterprise IT related conferences that occurs every spring in the city and had the good fortune to hit a special channel session where one of the speakers, a specialist in storage technology, gave us an interesting heads-up as to the state of deduplication in the enterprise.

To make a long story short:

• Situation: In a conference call with a client to a vendor of storage equipment with dedupe built-in
• Topic:  Cost of 2 additional disk drives ($53,000, available from Newegg for $500.00)
• Reason for Discussion:  Explain the cost
• Justification:  Licensing for support, warranty, extended services (including dedupe) for the additional drives.

He went further on with this topic to a more interesting point, regarding to a judicial ruling regarding SOX compliance with dedupe. To Point: You cannot dedupe exact copies of docs that are transmitted to recipients at different times/dates.  So if you forgot to include the name of one person in a mailing list, or had to resend because the recipients e-mail got accidentally deleted. etc., that e-mail, if SOX relevant (and it can be argued by lawyers, that almost everything regarding related to money, taxes, or law is) then those exact copies must be stored independently, and completely (no dedupe).  If one were to continue this trend of thinking, you might expect the same ruling to eventually apply to any compliance related documentation trail(HIPPA, etc.).

He told us one additional anecdote:
A large communications provider (phone, etc), several years ago decided to acquire a dedupe solution from one of the major storage providers (part of their suite of applications/devices) at a significant sum (pick a suitable integer and add at least 7 zeros before the decimal point).  Their  goal was to reduce the growth rate of their total document storage via deduplication.  After several rounds with operations and legal, they discovered that, instead of being able to dedupe across their entire document base, they were limited to using dedupe for only a single digit percentage of their documents.  A video (published on one of the major video sites) was made of a presentation at one of these enterprise events by one of their IT execs where he pointed this out to his audience, which, because of contractual obligations, between the customer and their vendor, was quickly pulled first from the show's video list and eventually from the video site itself.

His recommendation to was was not to sell/recommend storage device integrated deduplication, but to us an add-on device that you could configure as required for a lower TCO.

This is all second hand information, so I can't state to it's veracity.  However, I do recommend that you do a thorough check with all departments (including legal, compliance) to verify how effectively you can make use of a dedupe solution before you spend a lot of money on something you may not be able to use efficiently.