This session will first explain the index related options that are available when developing a data model and how these choices affect indexing and searching. We will cover Alfresco FTS in detail, and compare SQL 92 with the CMIS QL. We will also consider sorting and other ways to control the results returned, and how query performance may be affected by ACL evaluation.

1. Alfresco Search Internals 0 Andy Hind Senior Developer, Alfresco twitter: @andy_hind

3. Direction

4. Challenges

5. Alfresco FTS

7. Overview 3 Data Modelling Options <property name="cmis:name"> ... <type>d:text</type> ... <index enabled="true"> <tokenised>both</tokenised> <atomic>true</atomic> <stored>false</stored> </index> .... </property> Type drives analysis

8. Overview 4 Data Modelling Options <property name="cmis:name"> ... <type>d:text</type> ... <index enabled="true"> <tokenised>both</tokenised> <atomic>true</atomic> <stored>false</stored> </index> .... </property> true false

9. Overview 5 Data Modelling Options <property name="cmis:name"> ... <type>d:text</type> ... <index enabled="true"> <tokenised>both</tokenised> <atomic>true</atomic> <stored>false</stored> </index> .... </property> true false both

10. Overview 6 Data Modelling Options <property name="cmis:name"> ... <type>d:text</type> ... <index enabled="true"> <tokenised>both</tokenised> <atomic>true</atomic> <stored>false</stored> </index> .... </property> true false (d:content)

11. Overview 7 Data Modelling Options <property name="cmis:name"> ... <type>d:text</type> ... <index enabled="true"> <tokenised>both</tokenised> <atomic>true</atomic> <stored>false</stored> </index> .... </property> true false

13. Redirection by store protocol or value

14. Factories

15. AVM

16. DM

17. Unindexed

18. All lucene based with options set via properties

19. Analysis by data type and locale

21. Overview 10 Configuration properties lucene.maxAtomicTransformationTime=20 lucene.query.maxClauses=10000 lucene.indexer.cacheEnabled=true lucene.indexer.maxDocIdCacheSize=10000 lucene.indexer.maxDocumentCacheSize=100 lucene.indexer.maxParentCacheSize=10000 lucene.indexer.maxIsCategoryCacheSize=-1 lucene.indexer.maxLinkAspectCacheSize=10000 lucene.indexer.maxPathCacheSize=10000 lucene.indexer.maxTypeCacheSize=10000

22. Overview 11 Configuration properties lucene.indexer.mergerTargetIndexCount=5 lucene.indexer.mergerTargetOverlayCount=5 lucene.indexer.mergerTargetOverlaysBlockingFactor=1 lucene.indexer.mergerMergeBlockingFactor=1 lucene.indexer.maxDocsForInMemoryMerge=10000 lucene.indexer.maxRamInMbForInMemoryMerge=16 lucene.indexer.postSortDateTime=true lucene.indexer.defaultMLIndexAnalysisMode=EXACT_LANGUAGE_AND_ALL lucene.indexer.defaultMLSearchAnalysisMode=EXACT_LANGUAGE_AND_ALL lucene.indexer.maxFieldLength=10000

24. Configuration

25. system.acl.maxPermissionCheckTimeMillis=10000

26. system.acl.maxPermissionChecks=1000

27. Also set at query time

28. Read performance

29. 1.4 old model

30. 2.2/3.0 new

31. 3.4 improved read

33. Lucene with Alfresco extensions (PATH, ...)

34. Alfresco FTS

35. CMIS QL + extensions

36. DB based

37. XPath

38. Specific APIs – (using the child association table)

39. NodeService – selecting children

41. Transaction vs Snapshot

42. Query language abstraction

43. Repo reliance on the lucene index

44. Cross locale support

45. Rebuild

46. Cluster (loss of consistency)

47. Lucene limitations

48. Delete/add and reindexing

49. DB schema for properties

50. Read permission evaluation

51. One big store

52. Analyser configuration

53. Associations

56. CMIS QLSOLR DB/SQL

58. Tracking – eventual consistency

59. Not suitable for RM

60. Query time ACL filtering

61. PATH support

62. SOLR scalability and elasticity

64. Google

65. Lucene

66. Developer/App Customisation

67. Define the default namespace (e.g. Allow the user to drop cm: )

68. Disable/enable/modify certain language features

69. Define templates

70. Define the default field, simple templates for users

71. Share defines the “keywords” template as the default field

73. Phrase

74. Conjunction/Disjunction/Negation/Boosting

75. Fields

76. Wildcards

77. Ranges

78. Fuzzy matching

79. Proximity

80. Templates

82. Alfresco FTS 21 Template example – relevance tuning "%(cm:namecm:titlecm:descriptionia:whatEventia:descriptionEventlnk:titlelnk:description TEXT^2 =keywords:woof =cm:name:woof =cm:title:woof =cm:description:woof =ia:whatEvent:woof =ia:descriptionEvent:woof =lnk:title:woof =lnk:description:woof =TEXT:woof^2

84. Read-only relational view of the repository

85. Subset of SQL-92 with extensions

86. Type inheritance

87. Multi-valued properties

88. Full text search

89. CONTAINS()

90. SCORE()

91. Location

92. IN_FOLDER()

94. SELECT D.*, O.* FROM cmis:document AS DJOIN cm:ownable AS O ON D.cmis:objectId = O.cmis:objectId

95. no JOIN between types/nodes yet

96. use Alfresco FTS instead of SQL QL FTS

97. SELECT * from cmis:documentWHERE CONTAINS('cmis:name:apos;test*apos;')

98. relax some constraints

99. SCORE() can be used on its own

100. mvps can use svp syntax for IN, LIKE and comparisons