How netapp dedupe works

1. How NetApp Dedupe works?
When you 'sis on' a volume, the behaviour of that volume changes, and the change takes place in
two phases:
TWO PHASE PROCESS:
PHASE 1 -> SIS enabled: Pre-process: Before the block is written to the array: Collecting Fingerprint
Note: This is true for new blocks, for the existing data blocks that were written before enabling SIS, you need to run the scan on the
existing data and pull those fingerprints into the catalogue.
PHASE 2 -> SIS Start : Post-process: After the block is written to the array: Sorting, Comparing and
deduping
PHASE 1:
The moment SIS is enabled:
 Every time SIS notices a block write request coming in, the sis process makes a call to Data
ONTAP to get a copy of the fingerprint for that block so that it can store this fingerprint in its
catalogue file.

2. Note: This request interrupts the write string and results in a 7% performance penalty for all writes
into any volume with sis enabled.
PHASE 2:
Now, at some point you'll want to dedupe the volume using the 'sis start' command manually/auto
or via schedule:
 SIS goes through the process of comparing fingerprints from the fingerprint database
catalogue file, validating data, and dedupe'ing blocks that pass the validation phase.
Note: In the end all we are really doing is adjusting some inode metadata to say "hey remember that
data that used to be here, well it’s over there now."
IMPORTANT: Nothing about the basic data structure of the WAFL file system has changed, except
you are traversing a different path in the file structure to get to your desired data block. That’s why
NetApp dedupe *usually* has no perceivable impact on read performance - all we've done is
redirect some block pointers. Accessing your data might go a little faster, a little slower, or more
likely not change at all - it all depends on the pattern of the file system data structure and the
pattern of requests coming from the application.

3. What is a fingerprint?
Fingerprint is a small digital representation of a larger data object. Basically, it is a checksum
character generated by WAFL for each BLOCK for the purpose of consistency checking (This generally
involves the creation of a hash).
Is fingerprint generated by SIS?
No. Each time a WAFL block is created, a 'checksum' character is generated for the purpose of
consistency checking. NetApp Deduplication (SIS) simply "borrows" a copy of this checksum and
stores it in a catalogue as fingerprint.
What happens during post-process dedupe?
A. The fingerprint catalogue is sorted and searched for identical fingerprints.
B. When a fingerprint "match" is made, the associated data blocks are retrieved and scanned byte-
for-byte.
C. Assuming successful validation, the inode pointer metadata of the duplicate block is redirected to
the original block.
D. The duplicate block is marked as "Free" and returned to the system, eligible for re-use.
When to use QSM vs. VSM on dedupe volumes?
Use QSM when you only want to dedupe the destination volume, and use VSM when you want to
dedupe both the source and destination volumes automatically, and save bandwidth during SM
transfers.
Courtesy: Dr. Dedupe, NetApp.
Prepared by:
ashwinwriter@gmail.com