An Overview of the area and the current potential for the open technologies to be used, and some suggestions as to why they are not as heavily used as they should be.

1. Automation and Text Mining Ben O'Steen 25 May 2010

2. Hi

3. Ben O'Steen (Software Engineer, Oxford University Libraries)

4. Ben O'Steen (Software Engineer, Oxford University Libraries) Freelance Enthusiast

5. “ Text-mining” (and related techniques): Processing natural language to gain direct and contextual information, often with a means to quantify this information's accuracy.

6. Automation (in this context): Making decisions , providing additional options and increasing the amount of information understood by a system without the need for human effort.

7. “BI Search and Text Analytics”[1] says: “structured data in first place at 47%, trailed by unstructured (31%) and semi-structured data (22%).” 53%! And this was amongst data management professionals! [1] “BI Search and Text Analytics”, 2006 http://tdwi.org/research/2007/07/tdwi-best-practices-reports.aspx Natural Language Processing

10. Developing . Every year, new and better solutions are found.

12. Developing . Every year, new and better solutions are found.

13. Multi-disciplinary . The skills required by the team are broad: Machine-learning, Linguistics, Statistics, Logic and so on.

15. +“Multidisciplinary” - 25,800

16. +“Multi-disciplinary” - 10,400

18. +“Multidisciplinary” - 25,800

20. Crude, but interesting result.

22. +“Multidisciplinary” - 25,800

25. (I'll come back to why it is important to note that NLP is heavily multidisciplinary shortly)

27. Search-engines (from Google's to Apache Solr)

28. Google's Adwords: Advertising/Marketing

29. And just about any business that trades solely over the web.

32. Customer 'churn'

33. Credit ratings

35. Predictive analytics (game theory)

37. TrueKnowledge – NLP + Semantic Knowledge base

39. Specific information added to results, formatted based on the meaning of your search query.

41. Predictive analytics (game theory)

42. Search and Indexing

45. Thomson-Reuters' OpenCalais service

50. Why are people scared of NLP?

54. Access to in-depth material is reasonably good – if your library has academic subscriptions.

55. There is little middle ground between the two.

57. No easy entry-point

62. No easy entry-point

63. Cannot be blindly applied for good results (yet)

65. Requires understanding language syntax!

66. Multidisciplinary – makes it very easy for documentation to 'Not State the Obvious'

68. No easy entry-point

69. Cannot be blindly applied for good results (yet)

70. A long learning curve

72. Few have someone they know personally who can explain NLP and text-mining analyses.

74. If the barrier for new people to join in is too high, then they give up and go elsewhere....

76. If the barrier for new people to join in is too high, then they give up and go elsewhere....

77. … which means that those writing documentation never get good feedback from those they trust or know 'socially'...

79. If the barrier for new people to join in is too high, then they give up and go elsewhere....

80. … which means that those writing documentation never get good feedback from those they trust or know 'socially'...

86. From my experience, very few developers are starting with Java web apps anymore.

87. Simple API? Can we take inspiration from Google? Is it possible to offer some purposeful tools or UIMA workflows in a way that can be accessed by any browser*? (*without a Java applet)

88. What 'I' want aka: services that I or people I know have either been asked for, or have seen the desperate need for.

90. Enhanced article: http://dx.doi.org/10.1371/journal.pntd.0 ​000228.x001

91. *Cited article: http://www.ploscompbiol.org/article/info:doi/10.1371/journal.pcbi.1000361

95. Trace back to find the 'source' of a particular idea.

97. Could be applied retroactively.

99. Subject and cross-domain classification to aid searching/browsing

100. Semantic expression of textual citations

102. We need more people to understand how to perform certain tasks.

103. We need to make systems more capable of holding heuristically obtained information.

104. NLP needs simple entrypoints – more 'Hello World' type examples.

105. NLP web apps that can be accessed from the browser.