2. 10/15/2016 Ekanite
http://127.0.0.1:3999/gosf-ekanite/ekanite.slide#1 2/24
About me
Director of Data Platform Engineering at Percolate.
Previously Director of Engineering and core developer with In uxDB.
Led the backend team that built Loggly's 2nd generation indexing and search platform.
Big fan of Go, databases, and distributed systems.
3. 10/15/2016 Ekanite
http://127.0.0.1:3999/gosf-ekanite/ekanite.slide#1 3/24
About Ekanite
The goal of Ekanite is to do a couple of things, and do them well -- accept log messages over
the network, and make it easy to search the messages. What it lacks in feature, it makes up
for in focus.
You can nd the source code at github.com/ekanite/ekanite(https://github.com/ekanite/ekanite).
The current release is v1.1.1.
5. 10/15/2016 Ekanite
http://127.0.0.1:3999/gosf-ekanite/ekanite.slide#1 5/24
Why another log search system?
Built very large scale log search systems in the past.
It can be quite involved - networking, indexing, search, retention, sharding, and
performance.
Could it all be done in a single program?
Single binary Go program would make deployment really easy.
It would act as a detailed demonstration of building such a system.
New use case for bleve.
7. 10/15/2016 Ekanite
http://127.0.0.1:3999/gosf-ekanite/ekanite.slide#1 7/24
bleve
bleve is an indexing and full-text search library for Go.
Supports text analysis and faceting.
Straightforward to use.
funcmain(){
//openanewindex...ignoringallerrors.
mapping:=bleve.NewIndexMapping()
index,err:=bleve.New("example.bleve",mapping)
//indexsomedata
err=index.Index(identifier,"hello,world!")
//searchforsometext
query:=bleve.NewMatchQuery("text")
search:=bleve.NewSearchRequest(query)
searchResults,err:=index.Search(search)
}
bleve is to Go what Lucene is to Java.
18. 10/15/2016 Ekanite
http://127.0.0.1:3999/gosf-ekanite/ekanite.slide#1 18/24
Sorting hits by time
Ekanite performs time-based sorting in the application.
Earlier versions of bleve did not support sorting on custom elds.
Newer versions now do.
Complex sort method on DocIDs.
//Searchperformsasearchoftheindexusingthegivenquery.ReturnsIDsofdocuments
//whichsatisfyallqueries.ReturnsDocIDsinsortedorder,ascending.
func(i*Index)Search(qstring)(DocIDs,error){
query:=bleve.NewQueryStringQuery(q)
searchRequest:=bleve.NewSearchRequest(query)
searchRequest.Size=maxSearchHitSize
searchResults,_:=i.Alias.Search(searchRequest)
docIDs:=make(DocIDs,0,len(searchResults.Hits))
for_,d:=rangesearchResults.Hits{
docIDs=append(docIDs,DocID(d.ID))
}
sort.Sort(docIDs)
returndocIDs,nil
}
19. 10/15/2016 Ekanite
http://127.0.0.1:3999/gosf-ekanite/ekanite.slide#1 19/24
Retention enforcement is straightforward
//engine.go
//runRetentionEnforcementperiodicallyrunsretentionenforcement.
func(e*Engine)runRetentionEnforcement(){
defere.wg.Done()
for{
select{
case<-e.done:
return
case<-time.After(RetentionCheckInterval):
stats.Add("retentionEnforcementRun",1)
e.enforceRetention()
}
}
}
Shards are deleted from disk and references removed from the engine.