The next time you find yourself thinking there isn’t enough time in a week, consider what Drinker Biddle did for their client in 7 days. When a senior executive for a publicly traded company was fired for underperformance, he made a serious allegation on his way out the door. He claimed he was laid off because of his repeated attempts to inform officials that the company was falsifying quarterly financial reports to the public. Instead of waiting for the typical pace of discovery that could potentially cost their client at least a quarter of a million dollars, Drinker Biddle used powerful analytics technology to conduct an intelligent investigation, fast. In this session, you will learn about machine learning that makes digging through large multi-sources data sets possible. You will have a chance to see the backstage of how engineers empower legal teams to organize data, discover the truth and act on it.

1. © Relativity. All rights reserved.
7 Days of Playing Minesweeper, or
How to Shut Down Whistleblower Defense with Analytics
Elise Tropiano, Senior Technical Product Manager

2. © Relativity. All rights reserved.
Elise Tropiano
Senior Technical Product Manager, Analytics

3. © Relativity. All rights reserved.
Agenda
Who We Are
The
Problems
We Solve
How We
Solve These
Problems
Case Study

4. Who we are
• Fast-growing legal tech company
• Unstructured big data platform
enhanced with advanced
analytics, machine learning, and
powerful visualizations
• 800+ employees worldwide
• Headquartered in Chicago, with
offices in London, Kraków, Hong
Kong and Melbourne

5. Witamy w naszym
Krakowskim biurze
• Product Innovation Center
• Opened in September, 2015
• Focus on data transfer solutions
• Growing team up to 100 this year

6. © Relativity. All rights reserved.
Relativity helps manage and analyze data
relevant to litigation and investigations

7. © Relativity. All rights reserved.
organize data
discover the truth
act on it

8. © Relativity. All rights reserved.
data problems we solve

9. © Relativity. All rights reserved.
We live in a world where there is an electronic trail of evidence in every potential
litigation. Every sent email or document created could be relevant in a trial.

10. © Relativity. All rights reserved.
= 45,000,000 emails
Scale of a Hypothetical Large Case
500 people potentially involved
x 100 each sending emails per day
x 180 working days a year
x 5 years

11. © Relativity. All rights reserved.
It all could be potentially relevant in litigation.
And it’s not just emails that are relevant…

12. © Relativity. All rights reserved.
750M+ files
in the largest case in Relativity

13. © Relativity. All rights reserved.
our solution

14. © Relativity. All rights reserved.
SaaS platform

15. © Relativity. All rights reserved.
our platform
search & search workflow
machine learning
email analytics
workflow & applications
repository
data analytics & reporting

16. © Relativity. All rights reserved.
whistleblower or extortionist?

17. © Relativity. All rights reserved.
Drinker Biddle is a full-service law firm providing
litigation, regulatory and business solutions to public
and private corporations, multinational Fortune 100
companies and start-ups.

18. © Relativity. All rights reserved.
• When a senior executive for a publicly
traded company was fired for
underperformance, he made a serious
allegation on his way out the door.
• He claimed he was laid off because of
his repeated attempts to inform
officials that the company was
falsifying quarterly financial reports to
the public.

19. © Relativity. All rights reserved.
Kick off a long list of tasks for the
company, including waiting for a lawyer
to send them a demand letter, gearing up
for defense, coaxing out the facts as
knowledge evolves, and possibly settling
the case before even getting to the truth.
Two Options to Handle This Case
Traditional Approach
Start an internal investigation and figure
out exactly what happened and decide
how to handle it.
Analytical Approach
Cost: $1,500,000+ Cost: $80,000

20. © Relativity. All rights reserved.
Investigation Timeline
Days 1-3
• Collected multi-sourced data
– One million emails + thousands of complex financial reports
• Relativity Analytics: Email threading
– Grouped email conversations and found initial set of relevant
meeting notes and presentations
• Relativity Analytics: Keyword expansion
– Ran terms round in these meeting notes through keyword
expansion to find additional relevant terms
• Relativity Analytics: Clustering
– Groups documents into conceptually similar groups
– Prioritized the review of clusters containing relevant documents
to find additional sources of intelligence
DAY 1 DAY 2 DAY 3

21. © Relativity. All rights reserved.
What ACTUALLY Happened?
Former employee had been emailing
with his wife on his personal
account and forwarding those
emails to his work email.
The emails contained keywords and
phrases relevant to the investigation,
so Drinker Biddle were able to make a
case to collect from former employee’s
personal email, and load that into
Relativity for further review.
Drinker Biddle found evidence of the
former employee working with his wife,
an employment attorney, to develop a
case against the company, and they were
able to prove he was drafting emails
about fraudulent accounting before the
quarterly numbers were recorded.

22. © Relativity. All rights reserved.
Relativity Analytics helped shut down
a whistleblower defense in 7 days,
saving an estimated $1.5M+

23. © Relativity. All rights reserved.
How We Did It
Email Threading
DIVIDE emails into segments and understand each
segment’s metadata.
1
COMBINE emails into conversation threads.
2
IDENTIFY inclusive emails for optimal efficiency.
3

24. © Relativity. All rights reserved.
From: Brandon Gauthier
Sent: Tuesday, November 24, 2015 7:50 AM
To: Michael Di Salvo
Subject: Demo Email
I need some email data for a demo, can you reply
back to this email. Thanks!
From: Michael Di Salvo
Sent: Tuesday, November 24, 2015 9:52 AM
To: Brandon Gauthier
Subject: Re: Demo Email
This is me, replying to your e-mail.
You sir, are very welcome.
From: Brandon Gauthier
Sent: Tuesday, November 24, 2015 7:53 AM
To: Michael Di Salvo
Subject: Re: Demo Email
One more time! This way we can show email
segments better.
From: Michael Di Salvo
Sent: Tuesday, November 24, 2015 9:54 AM
To: Brandon Gauthier
Subject: Re: Demo Email
This is me, creating an additional segment.
From: Brandon Gauthier
Sent: Tuesday, November 24, 2015 7:50 AM
To: Michael Di Salvo
Subject: Demo Email
I need some email data for a demo, can you reply
back to this email. Thanks!
From: Michael Di Salvo
Sent: Tuesday, November 24, 2015 9:52 AM
To: Brandon Gauthier
Subject: Re: Demo Email
This is me, replying to your e-mail.
You sir, are very welcome.

27. © Relativity. All rights reserved.
How We Did It
Keyword
Expansion
CREATE multi-dimensional space from terms in document text.
1
IDENTIFY terms that are conceptually similar to a
user-provided query.
2
RETURN the terms for augmented searching.
3

28. © Relativity. All rights reserved.
“Personally-held”

30. © Relativity. All rights reserved.
How We Did It
Clustering
INDEX documents into multi-dimensional space.
1
HIERARCHICALLY GROUP documents into conceptually similar
groups using the document text.
2
VISUALIZE clusters for promoting corpus understanding
and searching.
3

31. © Relativity. All rights reserved.

33. © Relativity. All rights reserved.
“It’s incredibly unjust for companies to pay
a settlement if they’re unsure that a claim
has merit simply because they don’t have
the money or the resources to investigate
it properly.”
Chief Data Scientist and Partner, Drinker Biddle
Bennett Borden

34. © Relativity. All rights reserved.
any questions?

35. © Relativity. All rights reserved.
thank you