Do you need primary storage deduplication?

Recently, there have been more publicity about some new main memory deduplication products, and I have found that the value of main memory deduplication has once again paid off. After all, if you can manage memory properly, there shouldn't be too much duplicate data in memory, especially on main memory. Of course, if the old data is not accessed for 90 consecutive days, we will archive that data to tape storage. Even in a well-managed system, there is still redundant data on the main memory, so the benefits of deduplication are great.

First, as I hinted earlier, storage capacity is growing fast, IT staff is too tired, and even then it is impossible to manage all storage devices. Redundant copies of data will increase. The database will keep multiple copies of the data, users will save multiple different versions of a file with different names, and will never check and clear relatively early copies of the data. There are some more sensible examples, such as the company's trademark will be inserted into each slide shown and the memo in the server. Main memory deduplication can help you find all of this redundant data.

The second area where primary storage deduplication can be applied is virtualized server and desktop image storage. The repeatability between these image files is very high. The primary storage deduplication technology also eliminates this type of redundancy, freeing up a large amount of storage capacity. In many cases, reading data from deduplicated data does not affect performance.

The third and perhaps the largest application area for primary storage deduplication is efficiency optimization, and primary storage deduplication technology reduces the need for capacity for copies, backups, snapshots, and replication of data. This does not eliminate the need for secondary backups, as it is generally better to have a separate copy of the data that is not associated with deduplication or snapshot metadata. If you can deduplicate data earlier in the data processing process, you can reduce the frequency of using separate devices, especially when the primary storage system replicates data to a similar system in the disaster recovery center.

This effect makes the backup just a copy of the same data. Backup applications can be backed up to the same storage system. No secondary storage system is required. The archive becomes a multiple copy of the file, except that the archived file is labeled with a read-only label, but the archive application copies that data to the same storage system.

If you want to achieve data optimization at this level, there is still a lot of work to be done. First, main memory vendors must provide a deduplication engine in their storage solutions. Second, the deduplication technology and its processing of metadata must prove its reliability. Only time can tell. By that time, I think users will do a lot of testing first, and then slowly deploy the deduplication program. I believe that the main memory deduplication solution will eventually prove to be reliable. I think the deduplication solution won't replace the other ones. While some users may fully trust the main storage deduplication solution, it is certainly better to keep a separate copy of the data on a completely different storage device, and it is obviously more reasonable.