A searchable encryption plugin for encrypting search indices of enterprise search platforms, Apache Solr and Elasticsearch.

1. © Hitachi Solutions America, Ltd. 2017. All rights reserved.
Zero-Knowledge Technologies for the CloudJune 7, 2017
NYC Apache Lucene/Solr Meetup

2. © Hitachi Solutions America, Ltd. 2017. All rights reserved. 1
Security Needs for Search Engine
 Gathered in one place
 Pre-processed in useful form
 Shows relativity between data
Index Data is
Valuable

3. © Hitachi Solutions America, Ltd. 2017. All rights reserved. 2
Data Breach is Real
 $4 million cost in average
 29% increased cost since 2013
 Only 25% is human error
Security Needs for Search Engine

4. © Hitachi Solutions America, Ltd. 2017. All rights reserved. 3
Index Data Encryption
 Thicker walls are not enough to
protect credential data any more.
 Effective protection even after the
attackers break in is required.
Security Needs for Search Engine

5. © Hitachi Solutions America, Ltd. 2017. All rights reserved. 4
Security Needs for Search Engine
Use FDE
platform
Customization
by engine
integrator
Lucene
6966
Existing Encryption Solution

6. © Hitachi Solutions America, Ltd. 2017. All rights reserved. 5
Security Needs for Search Engine
Keys are
accessible
for server
Server side
encryption
Decrypt for
every search
Problem of Existing Solution

7. © Hitachi Solutions America, Ltd. 2017. All rights reserved. 6
Overview of Credeon SFS
is a product developed by Hitachi Solutions in Japan,
based on an advanced cryptographic scheme born in
the research lab of Hitachi.
Credeon Secure Full-text Search

8. © Hitachi Solutions America, Ltd. 2017. All rights reserved. 7
Overview of Credeon SFS
Keys are
inaccessible
for server
Client side
encryption
Search without
decryption
Main Goal of Credeon SFS

9. © Hitachi Solutions America, Ltd. 2017. All rights reserved. 8
Overview of Credeon SFS
How dose
Credeon SFS
Work
Credeon
Codec
Credeon
Plugin
Update
Handler
Search
Component
Request
Handler
Postings
Format
Stored Fields
Format

10. © Hitachi Solutions America, Ltd. 2017. All rights reserved.
Client Side Server Side
9
Scheme like Lucene 6966
Encryption Scheme
Upload Create Index
Decrypt
Encrypt
Return Result
QUERY ? QUERY ?Send Query Search

11. © Hitachi Solutions America, Ltd. 2017. All rights reserved. 10
Scheme of Credeon SFS
Encryption Scheme
Key
Management
Server
Client Side Server Side
QUERY ? QUERY ?
Create Index
Create Query
Decrypt Result Decrypt
Upload
Search

12. © Hitachi Solutions America, Ltd. 2017. All rights reserved. 11
A Patent Cryptographic Technology
Developed by Hitachi R&D
Searchable Encryption
Secure
Practical

13. © Hitachi Solutions America, Ltd. 2017. All rights reserved. 12
Searchable Encryption
How
Secure×Practical
 AES 256 (FIPS 140-2)
 Probabilistic Encryption
 AES Encryption Speed
 None Linear to Data Size
Probabilistic
Plain Encrypted
Apple
Banana
Banana
Banana
Coconut
$k24J$jX
K#$#J%K^
G%jQ%K9(
HFv9hbvn
!=v[h-u
Terms in Index
SearchTime
Plain Text Search
Encrypted Search

14. © Hitachi Solutions America, Ltd. 2017. All rights reserved. 13
Mechanism
Searchable Encryption

15. © Hitachi Solutions America, Ltd. 2017. All rights reserved. 14
Current Model
As a Product
text keyword result
Searchable
Storage Storage
Client
Server
App
Server
OS
No Encryption OS Encryption Simple Encryption
with ES/Solr
Client-Side Credeon
SFS with ES/Solr
Credeon SFS
with ES/Solr
Security Level
No Security Decryption at storage layer
 Difficult to separate key
Decrypt first and match
 Very slow
 plaintext in memory
Match first and decrypt
 Key separation
 High Performance
 Semantically Secure
 plaintext in memory
Client Client
Decryption at client-side
 Key separation
 High Performance
 Semantically Secure
 no plaintext on server
Storage
value
Storage
value
text keyword result
Client
ES/Solr/Lucene ES/Solr/Lucene
File System Encryption
value
text keyword result
ES/Solr/Lucene
Simple Encryption
Client
value
text keyword result
ES/Solr/Lucene
Encryption
Client
value
text keyword result
Searchable Encryption
Storage
ES/Solr/Lucene
IndexIndexIndexIndexIndex

16. © Hitachi Solutions America, Ltd. 2017. All rights reserved. 15
Performance
As a Product
Plain Credeon Credeon/Plain
Indexing Time (s) 1,643 4,835 2.9
Search Time (ms) 16 19 1.2
Index Size (MB) 2,941 5,011 1.7
CPU Core i5 2.4GHz
Mem 8GB
OS Ubuntu 14.04 LTD
Engine App Solr 5
Data Wikipedia Pages 10GB (13,800,000 terms)
TestEnvironmentResult

17. © Hitachi Solutions America, Ltd. 2017. All rights reserved. 16
Limitation
As a Product
Type Feature Encrypted Field Plain Field
Search Schema Core type, Copy field, Dynamic field Yes Yes
Index Indexing, Delete, Update Yes Yes
Search N-Gram, Lower case, Phonetic, Stop, Trim, Filter Query, Boost, Boolean, Stemmer, Phrase Yes Yes
Pattern, Wildcard, Range, Grouping, Spatial search No Yes
Result Relevance, Sort*1, Highlighting*2, Pagination, Yes Yes
Spell check, Suggester, More like this No Yes
Facet Query faceting, Field-value faceting, Range faceting No Yes
Caching Filter cache, Query result cache Yes Yes
Field cache No Yes
Analyzer, Tokenizer, Filter, Transform, Response writer Yes*3 Yes
Cloud Yes Yes
Encryption Terms (.tim/.tip/.doc/.pos/.pay) Encrypted N/A
Stored field (.fdt) Encrypted N/A
Meta data Plain N/A
Search keyword Encrypted N/A
Transaction log Plain N/A
*1: Only with Score *2: Except FastVectorHighlighter *3: Need to evaluate

18. © Hitachi Solutions America, Ltd. 2017. All rights reserved. 17
 Setup
 Upload
 Search
 Select Encryption Fields
 Use Multiple Keys
 Others
Demo

19. © Hitachi Solutions America, Ltd. 2017. All rights reserved. 18
Thanks for Listening
Tong Ye tye@hitachisolutions.com
Harry Ochiai hochiai@hitachisolutions.com
@credeon
https://psg.hitachi-solutions.com/credeon/secure-full-text-search