The document describes a prototype called Journal Selector that was built using Solr, Nutch, and Hadoop. It indexes over 21,000 journals and provides recommendations to researchers on where to publish their articles based on article topics and keywords. The prototype crawls public metadata from partners like Springer and indexes the data in Solr to enable fast search and recommendations.

1. © 2013 LucidWorks

2. Edanz Journal Selector: Case Study: a
Prototype based on Solr/Nutch/Hadoop
Liang SHEN
@shenzhuxi
European Bioinformatics Institute

3. © 2013 LucidWorks
Edanz Journal Selector
a Prototype based on Solr/Nutch/Hadoop

4. © 2013 LucidWorks
English editing for scientists

5. © 2013 LucidWorks
Help scientists publish papers

6. © 2013 LucidWorks
Target journal?

7. © 2013 LucidWorks
Journal Selector

8. © 2013 LucidWorks
Open Access
PubMed

9. © 2013 LucidWorks
Journal TOCs
created in 2009
21,498 journals from
1,677 publishers
Institute for Computer
Based Learning
Heriot-Watt University

10. © 2013 LucidWorks
Partner
• Springer Metadata API
Provides metadata for over 5 million online documents
• Springer Open Access API
Provides metadata, full-text content, and images for
over 80,000 open access articles

11. © 2013 LucidWorks
Open Source Stack
• Infrastructure: Amazon Web Service
• Data processing: Hadoop/Hive
• Index: Solr/Lucene
• Web service: Drupal
• Secret Sauce/Custom Works

12. © 2013 LucidWorks
Infrastructure: Amazon EC2

13. © 2013 LucidWorks
Data processing
HDFS
Index
AP
I
Feed
s
Web
Pages

14. © 2013 LucidWorks
<script>
http://global.js.wid
get.eja.hk/ja/edan
z_ja/w.js
</script>
Web service

15. © 2013 LucidWorks
Embeddable web widget

16. © 2013 LucidWorks
Split Index for performance
Index can be divided without losing ranking, if there is always a facet field.

17. © 2013 LucidWorks
@shenzhuxi
Thanks!
Questions?