Domino 10 has some amazing self-healing capabilities that are going to make every admin’s life much easier. Learn how automatic cluster database symmetry works to ensure both NSFs and DAOS content are kept in sync across cluster mates. Understand how the repair engine remedies any missing databases or NLOs, rebuilds full text indexes, and detects and repairs corrupt NSFs and NLOs. You will also learn how to use the new commands to manually invoke corruption repair and how the new replica synchronization tool and replication currency monitoring work. Get the ins and outs of the functionality of automatic hot view updating and how to setup deletion logging so you know who/what deleted a document.

1. Self Healing Capabilities of Domino 10
Kim Greene
Kim Greene Consulting, Inc.
kim@kimgreene.com
HCL Development Team
They Rock!!!

2. Introduction
• Owner of an IT consulting company
– Kim Greene Consulting, Inc.
– www.kimgreene.com
• Started my career at IBM, left and launched my own business … 18 years ago
• Focus areas:
– IBM collaboration software portfolio
– MongoDB
• Customers are worldwide and in multiple industries
• Blog: www.dominodiva.com
• Twitter: iSeriesDomino
MongoDB Advocate

3. Some Background to Get Started
• Domino servers running and stable
• No one “looks under the covers”
• Problems lurk admins are unaware of
• Problems happen admins struggle with how to
resolve them

4. Domino 10 Focus
• Domino 10 has a heavy focus on TCO
– Reduce administrator workload and simplify
– Improve Domino visibility and serviceability
– Improve Domino clusters for both administrators and
end users

5. Issue 1: Missing Replicas and NLOs
• Missing replicas
– Very common on non-mail servers
• Missing NLOs
– Only found if reported by user, comb the logs or
manually check

6. Issue 1: Missing Replicas and NLOs
• Prior to Domino 10
– Need to do your own analysis to detect
– To detect missing .NSFs
• Look at cluster directory database
• Write your own application to detect
• Ytria replicationEZ
– To detect missing .NLOs
• Analyze server log
• Use MISSING parameter on tell daosmgr listnlo command
• Investigate missing attachment report from user

7. Issue 1: Missing Replicas and NLOs
• Domino 10 solution: Automatic cluster database symmetry
– Automates keeping NSF and DAOS content synchronized in a
cluster

8. Automatic Cluster Database Symmetry
• Based on cluster configuration document,
directory folders are kept in sync via the “repair
engine”

9. Issue 1: Missing Replicas and NLOs
• Domino 10 solution: Automatic Database Repair
– High speed transfer of file(s) between servers
– Any required DAOS objects repaired automatically
– Replication history adjusted to correctly reflect replication currency
– Full Text indexes rebuilt as necessary

10. Issue 1: Missing Replicas and NLOs

11. Issue 1: Missing Replicas and NLOs
• Domino 10 solution: Automatic Database Repair
– What about encrypted .NLOs?
• Automatically unencrypted with source server’s ID
• Repaired / replicated
• Encrypted again with target server’s ID
– Same process applies to corrupted .NLOs
• Good version is found and asynchronously “repaired”

12. Issue1 : Missing Replicas and NLOs
• Domino 10 solution: Automatic Database Repair
– What about databases intentionally deleted?
• Get marked as “disabled” for auto synchronization in CLDBDIR
• Administrators can override “enable/disable” setting in CLDBDIR
– repair enable | disable mail/kgreene.nsf

13. Issue 2: Corrupt NSFs and NLOs
• Server crashes and other events can cause
corruption of NSFs and NLOs

14. Issue 2: Corrupt NSFs and NLOs
• Prior to Domino 10
– DBCapture Tool
• File renamed to .cor and moved to IBM_TECHNICAL_SUPPORT
folder
• Had to explicitly enable: DATABASE_CAPTURE_ENABLED=1
• Resets every time capture is done, requires server restart to
enable again

15. Issue 2: Corrupt NSFs and NLOs
• Domino 10 solution: Automatic database repair
– Upon database open event, if corruption is detected run
fixup as first step
– IF fixup is unsuccessful, second step taken:
• Replace with “good” replica
• Rename to dbfilename.pd_bad_YYYYMMDDHHMMSSSS
– Left on system for xx days
– # of days configurable in Cluster Configuration document

16. Issue 2: Corrupt NSFs and NLOs

17. Summary of Issues 1 & 2
• To enable automatic cluster database symmetry
and automatic database repair
– Add notes.ini
• D10_ENABLE_REPAIR=1
– Add these two tasks to SERVERTASKS= (or issue load)
• AutoRepair
• dbcleanup

18. Issue 3: Missing Documents
• Every customer environment I have worked with
has replication discrepancies they are not aware of

19. Issue 3: Missing Documents
• Prior to Domino 10
– Compare document counts across replicas:
• Manually
• Custom written application to do comparisons
• Ytria replicationEZ

20. Issue 3: Missing Documents
• Domino 10 solution:
– Replica synchronization tool
– Replication currency monitoring

21. Issue 3: Missing Documents
• Domino 10 solution: Replica synchronization tool
– Background resync of database without clearing
replication history
– Can manually invoke
• replicate <servername> <databasename> -f

22. Issue 3: Missing Documents
• Domino 10 solution: Replica synchronization tool
– Default is no logging of replica synchronization
• To enable logging, set this notes.ini, but it will generate a LOT
of output!
– REPL_SYNC_VERBOSE=1
• Example output
ReplCurrency Source="mailkgreene.nsf", Dest=CN=Ella/O=PNPHCL (Q:1)
mailkgreene.nsf, UpdateToRepl2="07/19/2018 10:20:04 PM",
LastRepl="07/19/2018 08:01:33 PM", TimeDiff= 287 min, TimeDiff2=425 min

23. Issue 3: Missing Documents
• Domino 10 solution: Replica synchronization tool
– New set of cluster currency statistics available
• show stat replica.cluster.currency.*
[00134C:000008-00000000000004AC] Replica.Cluster.Currency.Mail2/Renovations.Outside8Hours = 0
[00134C:000008-00000000000004AC] Replica.Cluster.Currency.Mail2/Renovations.Uninitialized = 0
[00134C:000008-00000000000004AC] Replica.Cluster.Currency.Mail2/Renovations.Within15Min = 0
[00134C:000008-00000000000004AC] Replica.Cluster.Currency.Mail2/Renovations.Within1Hour = 0
[00134C:000008-00000000000004AC] Replica.Cluster.Currency.Mail2/Renovations.Within1Min = 98
[00134C:000008-00000000000004AC] Replica.Cluster.Currency.Mail2/Renovations.Within4Hours = 0
[00134C:000008-00000000000004AC] Replica.Cluster.Currency.Mail2/Renovations.Within5Min = 0
[00134C:000008-00000000000004AC] Replica.Cluster.Currency.Mail2/Renovations.Within8Hours = 0

24. Issue 3: Missing Documents
• Domino 10 solution:
– Replication currency monitoring
• Compares replication across cluster servers, identifying
timeframes when replicas are out of sync

25. Issue 3: Missing Documents
• Domino 10 solution: Replication currency monitoring
• Let’s take a closer look

26. Issue 4: Critical Views Out of Date
• Views frequently updated and accessed do not
always contain most recent data
• Users experience
SSLLOOOWWWWW access
times when views are being
updated

27. Issue 4: Critical Views Out of Date
• Prior to Domino 10
– Limited control, update queue is FIFO
– Inline view indexing / dedicated view threads
• Heavy burden on administrators
• Not dynamic to use case / workload changes

28. Issue 4: Critical Views Out of Date
• Domino 10 solution: Automatic Hot View Updating
– Domino tracks view updates and reads in real time
– “Hot” views updated in real time without administrator’s
involvement
– Enable with notes.ini NIF_VIEW_USAGE_ENABLED=1
– Show tasks will provide output on “hot view”
• View Indexer appdisc9.nsf "All Documents" 10 sec. high usage read

29. Issue 4: Critical Views Out of Date
• Domino 10 solution: Automatic Hot View Updating
– New statistics available
• Database.ViewUsage.Calls
• Database.ViewUsage.HighUsageDisables
• Database.ViewUsage.HighUsageEnables
• Database.ViewUsage.HighUsageUpdateChecks
• Database.ViewUsage.HighUsageUpdates
• Database.ViewUsage.Skipped
• Update.View.<db-file-name>.<view-name>.<statistic> where
<statistic> is one of the following: TotalInserts TotalRemoves TotalNew
TotalReplace TotalNonReplace TotalNodataReplace Count

30. Issue 5: Who Deleted My Documents
• An issue often reported, user reports missing
documents and/or folders
• Administrator has to put the pieces together

31. Issue 5: Who Deleted My Documents
• Prior to Domino 10
– Lots of analysis required
• Examine deletion stubs
• Review user activity for a database
• Check for rogue agents
• Etc.

32. Issue 5: Who Deleted My Documents
• Domino 10 solution: Note Deletion Logging
– New functionality to log additional details about deleted
documents
– Deletion data stored in deletion log files under
IBM_TECHNICAL_SUPPORT folder
• delete_<servername>_yyyy_mm_dd@hh_mm_ss.log
– Example of deletion log entry:
12/27/2018 06:27:10 PM, branneymail.nsf, 052581F8:005CB5E7, server,
CN=Mail2/O=Renovations, SOFT, 0001,
B1A644C2:EAC0720905258203:0080D2BC,From:Samantha Daryn
– Enable at database level
• Use .IND file for whole directory

33. Issue 5: Who Deleted My Documents
• Domino 10 solution: Note Deletion Logging
– Requires transaction logging
• For server AND database
– Enable with new compact switch “-deletionLogging on”
• load compact mail/kgreene.nsf –deletionLogging on
“SendTo,From,Subject,$TITLE”
– New deletion log created with server restart

34. Issue 5: Who Deleted My Documents
• Domino 10 solution: Note Deletion Logging
– What gets logged?
• Date and time document was deleted
• Database document was deleted from
• Database replicaID
• Process that performed the deletion (sever, dbmt, replica, …)
• Name of person or server who did the deletion
• Type of document deletion (SOFT, HARD, RESTORE)
• Class of document deleted
• Document UNID
• Up to 4 items specified when deletion logging was enabled

35. Issue 5: Who Deleted My Documents
• Domino 10 solution: Note Deletion Logging
– Does NOT substitute need for doing backups!!!!!
– Recovering deleted documents
• Use the same “magic” you’ve used for years

36. A HUGE Thank You to Gary Rheaume for
the demos!!!

37. Questions?

38. Contact Information
@iSeriesDomino
www.linkedin.com/in/kimgreeneconsulting
kim@kimgreene.com