What you get by replicating Lucene indexes on the Infinispan Data Grid (Berlin Buzzwords 2012)

•

2 recomendaciones•3,286 vistas

At the core Infinispan is an in-memory data grid with options for multi node replication or distribution. It is also able to index and search stored data using Apache Lucene, or inverting the roles to have Lucene store any index in the grid. This talk will illustrate how to get started and what benefits (or drawbacks!) you get by storing your Lucene indexes in Infinispan. http://berlinbuzzwords.de/sessions/what-you-get-replicating-lucene-indexes-infinispan-data-grid

Tecnología

4 June 2012
Sanne Grinovero, Red Hat

What you get by replicating
Lucene indexes on the
Infinispan Data Grid

Who is that guy?
• Sanne Grinovero

• From this planet

• Team Hibernate

• Hibernate Search

• Hibernate OGM

• Team Infinispan

• Infinispan Core

• Infinispan Query

• Apache Lucene, Netty, HotSpot, ANTLR, JGroups,
Byteman, The Jokre

What are we talking
about?

• Apache Lucene
• Infinispan
• Integrations with Lucene
●
Infinispan Lucene Directory

• An in-memory datagrid

• Memory of multiple nodes

• Cluster modes

• CacheLoaders

• Integrations with Lucene

• Lucene Directory

Infinispan API?
• Map-like key/value store
• JSR 107 javax.cache.Cache interface
• JSR 347 ??
• Asynchronous API

In practice:
cache.put( “user-34”, userInstance );

cache.get( “user-34” );

cache.remove( “user-34” );

cache.putIfAbsent( “user-38”, other );

Connected via JGroups

A Toolkit for Reliable Multicast Communication
http://jgroups.org

Or remote clients via:
• Memcached
• REST
• Hot Rod (Ruby, Python, C, C#, ...)
• Netty

JBoss AS7 core
component
• Cluster nodes autodiscovery
• Session replication / failover
• Hibernate second level cache
• mod_cluster integration

In-memory volatile?
Cache Stores: durability, warm caches,
more capacity...

• Cassandra

• HBase

• JDBC

• Clouds (S3, ...)

• Plain Old Files

• Many more + custom

Queue-based clustering
(filesystem index)

Hints
• Some tuning options might have
different effects than what you're used
• Network is orders of magnitude faster
than disk (YMMV)
• But data locality helps
• Balance resources
• Get mergers to avoid segment
chunking, or readlocks will engage

“benchmarks”, stats
and more lies
Write ops/sec Queries/sec

RAMDirectory RAMDirectory

Infinispan 0 Infinispan 0

Infinispan D4 Infinispan D4

queries per second
Infinispan D40 Infinispan D40

FSDirectory FSDirectory

Infinispan Local Infinispan Local

0 50 100 150 200 250 300 350 400 0 5000 10000 15000 20000 25000

It's not about the
figures
Write ops/sec Queries/sec

RAMDirectory RAMDirectory

Infinispan 0 Infinispan 0

Infinispan D4 Infinispan D4

queries per second
Infinispan D40 Infinispan D40

FSDirectory FSDirectory

Infinispan Local Infinispan Local

0 50 100 150 200 250 300 350 400 0 5000 10000 15000 20000 25000

What's next?
• Infinispan (core) 5.2 and 6
• Lucene 4.x
• Dynamic chunk sizes
• Ad-hoc “Lucene native” CacheStore
• NIO byte buffers?

Conclusions
• Quick index replication
• Transactions
• Not a replacements for shards
• Cloud-friendly
• Delegates to any storage

Q&A

http://infinispan.org @Infinispan
http://in.relation.to @Hibernate
http://jboss.org @SanneGrinovero

Más contenido relacionado

Similar a What you get by replicating Lucene indexes on the Infinispan Data Grid (Berlin Buzzwords 2012)

Tidal scale short_story_v2

Chuck Piercey

Apache con 2011 gd

Leif Hedstrom

Next Gen Datacenter

Rui Lopes

Plugin-able POS Solutions by Javascript @HDM9 Taiwan

Rack Lin

Sun storage tek 6140 customer presentation

xKinAnx

Enhancing Live Migration Process for CPU and/or memory intensive VMs running...

Benoit Hudzia

Stathy DevOps in MSP / MKE on IAC

Stathy Touloumis

Usenix lisa 2011

Leif Hedstrom

Password Recovery Tools

Andrey Belenko

A Behind the Scenes Look at the Force.com Platform

Salesforce Developers

Nytro™ Application Acceleration Architecture Intelligent Solid-state Storage Solutions for Acceleration Datacenter Applications With Nytro™ Products, LSI delivers smarter, faster application acceleration solutions by combining SSDs with intelligent software that analyzes data hot spots to determine which data is best processed in Flash and which is best stored in HDDs, enabling you to select the right amount of flash for the price and performance you need. It’s fast, smart, and easy. FAST. Up to 12x in database transactions improvement, Up to 10x latency reduction for SAN connected storage. Powered by the LSI SandForce flash storage processor. SMART. Right-sized solutions provide a choice rather than one-size fits all. Intelligently use flash and hard drives together for to achieve the optimal price/performance. Seamless caching and acceleration software analyzes data hot spots. EASY. LSI solid-state storage solutions just plain work. No tuning, no rewrites, no hand-placing data. Transparent and seamless application acceleration with minimal impact to existing IT infrastructure, and extensive OS support. A breadth of product portfolio so solutions can be cost-effectively tailored. Nytro™ Product Family The Nytro™ Product Family is the most comprehensive, trustworthy, and highest-performance portfolio of enterprise solid-state storage solutions for customers who seek to unlock unparalleled application acceleration and reduced TCO. Because LSI technologies are so widely deployed within the datacenter and undergo rigorous internal and external qualification testing, Nytro™ products deliver ease of implementation, seamless integration, extensive OS support, reliability and future-proof scalability. That’s the smarter way to faster.

Lsi Nytro flash ssd PCIe controllers product training value proposition and k...

Kent Abendroth LSI Nytro VAR channel manager

Yahoo Communities Architecture Unlikely Bedfellows

ConSanFrancisco123

All Aboard the Databus

Amy W. Tang

Large scale preservation workflows with Taverna – SCAPE Training event, Guima...

SCAPE Project

Dynamo Systems - QCon SF 2012 Presentation

Shanley Kane

Owned By An iPod

KarlFrank99

Sun storage tek 6140 technical presentation

xKinAnx

Cette présentation est le résumé des multiples discussions autour du stockage entre Erik et moi ! En effet, Erik considère qu'une solution matérielle est la plus adaptée alors que je prône l'utilisation de logiciels d'optimisation... Pour dire vrai, c'est plus complexe que cela mais en définitif nous sommes tous les deux d'accord... Comme le disait l'un de mes professeurs émérites à l'université, "on ne peut s'engueuler qu'avec les gens avec lesquels nous sommes d'accord..."

Erik Ableson & Vincent Branger: What's best for vdi storage optimisation hard...

ILKI - Your Cloud Designer

Linux HA anno 2014

Julien Pivotto

Super cluster oracleday cl 7

Claudio Osvaldo Vargas Farfan

Similar a What you get by replicating Lucene indexes on the Infinispan Data Grid (Berlin Buzzwords 2012) (20)

Tidal scale short_story_v2

Apache con 2011 gd

Next Gen Datacenter

Plugin-able POS Solutions by Javascript @HDM9 Taiwan

Sun storage tek 6140 customer presentation

Enhancing Live Migration Process for CPU and/or memory intensive VMs running...

Stathy DevOps in MSP / MKE on IAC

Usenix lisa 2011

Password Recovery Tools

A Behind the Scenes Look at the Force.com Platform

Lsi Nytro flash ssd PCIe controllers product training value proposition and k...

Yahoo Communities Architecture Unlikely Bedfellows

All Aboard the Databus

Large scale preservation workflows with Taverna – SCAPE Training event, Guima...

Dynamo Systems - QCon SF 2012 Presentation

Owned By An iPod

Sun storage tek 6140 technical presentation

Erik Ableson & Vincent Branger: What's best for vdi storage optimisation hard...

Linux HA anno 2014

Super cluster oracleday cl 7

Último

Corporate and higher education. Two industries that, in the past, have had a clear divide with very little crossover. The difference in goals, learning styles and objectives paved the way for differing learning technologies platforms to evolve. Now, those stark lines are blurring as both sides are discovering they have content that’s relevant to the other. Join Tammy Rutherford as she walks through the pros and cons of corporate and higher ed collaborating. And the challenges of these different technology platforms working together for a brighter future.

Corporate and higher education May webinar.pptx

Rustici Software

FWD Group - Insurer Innovation Award 2024

The Digital Insurer

Abhishek Deb(1), Mr Abdul Kalam(2) M. Des (UX) , School of Design, DIT University , Dehradun. This paper explores the future potential of AI-enabled smartphone processors, aiming to investigate the advancements, capabilities, and implications of integrating artificial intelligence (AI) into smartphone technology. The research study goals consist of evaluating the development of AI in mobile phone processors, analyzing the existing state as well as abilities of AI-enabled cpus determining future patterns as well as chances together with reviewing obstacles as well as factors to consider for more growth.

Exploring the Future Potential of AI-Enabled Smartphone Processors

debabhi2

Accelerating FinTech Innovation: Unleashing API Economy and GenAI Vasa Krishnan, Chief Technology Officer - FinResults Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...

apidays

Exploring Multimodal Embeddings with Milvus

Zilliz

The Good, the Bad and the Governed - Why is governance a dirty word? David O'Neill, Chief Operating Officer - APIContext Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...

apidays

Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood

Juan lago vázquez

Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows. We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases. This video focuses on the deployment of external web forms using Jotform for Bonterra Impact Management. This solution can be customized to your organization’s needs and deployed to support the common use cases below: - Intake and consent - Assessments - Surveys - Applications - Program registration Interested in deploying web form automations for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.

Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...

Jeffrey Haguewood

Boost Fertility New Invention Ups Success Rates.pdf

sudhanshuwaghmare1

Architecting Cloud Native Applications

WSO2

Effective data discovery is crucial for maintaining compliance and mitigating risks in today's rapidly evolving privacy landscape. However, traditional manual approaches often struggle to keep pace with the growing volume and complexity of data. Join us for an insightful webinar where industry leaders from TrustArc and Privya will share their expertise on leveraging AI-powered solutions to revolutionize data discovery. You'll learn how to: - Effortlessly maintain a comprehensive, up-to-date data inventory - Harness code scanning insights to gain complete visibility into data flows leveraging the advantages of code scanning over DB scanning - Simplify compliance by leveraging Privya's integration with TrustArc - Implement proven strategies to mitigate third-party risks Our panel of experts will discuss real-world case studies and share practical strategies for overcoming common data discovery challenges. They'll also explore the latest trends and innovations in AI-driven data management, and how these technologies can help organizations stay ahead of the curve in an ever-changing privacy landscape.

TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery

TrustArc

In the thrilling conclusion to 2023, ransomware groups had a banner year, really outdoing themselves in the "make everyone's life miserable" department. LockBit 3.0 took gold in the hacking olympics, followed by the plucky upstarts Clop and ALPHV/BlackCat. Apparently, 48% of organizations were feeling left out and decided to get in on the cyber attack action. Business services won the "most likely to get digitally mugged" award, with education and retail nipping at their heels. Hackers expanded their repertoire beyond boring old encryption to the much more exciting world of extortion. The US, UK and Canada took top honors in the "countries most likely to pay up" category. Bitcoins were the currency of choice for discerning hackers, because who doesn't love untraceable money?

Ransomware_Q4_2023. The report. [EN].pdf

Overkill Security

EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER

MadyBayot

2024: Domino Containers - The Next Step. News from the Domino Container commu...

Martijn de Jong

Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving

Edi Saputra

Three things you will take away from the session: • How to run an effective tenant-to-tenant migration • Best practices for before, during, and after migration • Tips for using migration as a springboard to prepare for Copilot in Microsoft 365 Main ideas: Migration Overview: The presentation covers the current reality of cross-tenant migrations, the triggers, phases, best practices, and benefits of a successful tenant migration Considerations: When considering a migration, it is important to consider the migration scope, performance, customization, flexibility, user-friendly interface, automation, monitoring, support, training, scalability, data integrity, data security, cost, and licensing structure Next Wave: The next wave of change includes the launch of Copilot, which requires businesses to be prepared for upcoming changes related to Copilot and the cloud, and to consolidate data and tighten governance ShareGate: ShareGate can help with pre-migration analysis, configurable migration tool, and automated, end-user driven collaborative governance

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff

sammart93

ICT role in 21st century education and its challenges

rafiqahmad00786416

Webinar Recording: https://www.panagenda.com/webinars/why-teams-call-analytics-is-critical-to-your-entire-business Nothing is as frustrating and noticeable as being in an important call and being unable to see or hear the other person. Not surprising then, that issues with Teams calls are among the most common problems users call their helpdesk for. Having in depth insight into everything relevant going on at the user’s device, local network, ISP and Microsoft itself during the call is crucial for good Microsoft Teams Call quality support. To ensure a quick and adequate solution and to ensure your users get the most out of their Microsoft 365. But did you know that ‘bad calls’ are also an excellent indicator of other problems arising? Precisely because it is so noticeable!? Like the canary in the mine, bad calls can be early indicators of problems. Problems that might otherwise not have been noticed for a while but can have a big impact on productivity and satisfaction. Join this session by Christoph Adler to learn how true Microsoft Teams call quality analytics helped other organizations troubleshoot bad calls and identify and fix problems that impacted Teams calls or the use of Microsoft365 in general. See what it can do to keep your users happy and productive! In this session we will cover - Why CQD data alone is not enough to troubleshoot call problems - The importance of attributing call problems to the right call participant - What call quality analytics can do to help you quickly find, fix-, and prevent problems - Why having retrospective detailed insights matters - Real life examples of how others have used Microsoft Teams call quality monitoring to problem shoot problems with their ISP, network, device health and more.

Why Teams call analytics are critical to your entire business

panagenda

Keynote 2: APIs in 2030: The Risk of Technological Sleepwalk Paolo Malinverno, Growth Advisor - The Business of Technology Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...

apidays

"I see eyes in my soup": How Delivery Hero implemented the safety system for ...

Zilliz

What you get by replicating Lucene indexes on the Infinispan Data Grid (Berlin Buzzwords 2012)

1. 4 June 2012 Sanne Grinovero, Red Hat What you get by replicating Lucene indexes on the Infinispan Data Grid

2. Who is that guy? • Sanne Grinovero • From this planet • Team Hibernate • Hibernate Search • Hibernate OGM • Team Infinispan • Infinispan Core • Infinispan Query • Apache Lucene, Netty, HotSpot, ANTLR, JGroups, Byteman, The Jokre

3. What are we talking about? • Apache Lucene • Infinispan • Integrations with Lucene ● Infinispan Lucene Directory

4. Apache Lucene ?

5. • An in-memory datagrid • Memory of multiple nodes • Cluster modes • CacheLoaders • Integrations with Lucene • Lucene Directory

6. Infinispan API? • Map-like key/value store • JSR 107 javax.cache.Cache interface • JSR 347 ?? • Asynchronous API

7. In practice: cache.put( “user-34”, userInstance ); cache.get( “user-34” ); cache.remove( “user-34” ); cache.putIfAbsent( “user-38”, other );

8. Distributed Data

9. Connected via JGroups A Toolkit for Reliable Multicast Communication http://jgroups.org

10. Or remote clients via: • Memcached • REST • Hot Rod (Ruby, Python, C, C#, ...) • Netty

11. Consistent Hashing: DIST

12. Transactions!

13. JBoss AS7 core component • Cluster nodes autodiscovery • Session replication / failover • Hibernate second level cache • mod_cluster integration

14. In-memory volatile? Cache Stores: durability, warm caches, more capacity... • Cassandra • HBase • JDBC • Clouds (S3, ...) • Plain Old Files • Many more + custom

15. Back on Lucene: Single Writer lock

16. Queue-based clustering (filesystem index)

17. Lucene index storage

18.

19. Index stored in Infinispan

20. Example architecture : JIRA / Scarlet

21. Hints • Some tuning options might have different effects than what you're used • Network is orders of magnitude faster than disk (YMMV) • But data locality helps • Balance resources • Get mergers to avoid segment chunking, or readlocks will engage

22. “benchmarks”, stats and more lies Write ops/sec Queries/sec RAMDirectory RAMDirectory Infinispan 0 Infinispan 0 Infinispan D4 Infinispan D4 queries per second Infinispan D40 Infinispan D40 FSDirectory FSDirectory Infinispan Local Infinispan Local 0 50 100 150 200 250 300 350 400 0 5000 10000 15000 20000 25000

23. It's not about the figures Write ops/sec Queries/sec RAMDirectory RAMDirectory Infinispan 0 Infinispan 0 Infinispan D4 Infinispan D4 queries per second Infinispan D40 Infinispan D40 FSDirectory FSDirectory Infinispan Local Infinispan Local 0 50 100 150 200 250 300 350 400 0 5000 10000 15000 20000 25000

24. What's next? • Infinispan (core) 5.2 and 6 • Lucene 4.x • Dynamic chunk sizes • Ad-hoc “Lucene native” CacheStore • NIO byte buffers?

25. Conclusions • Quick index replication • Transactions • Not a replacements for shards • Cloud-friendly • Delegates to any storage

26. Q&A http://infinispan.org @Infinispan http://in.relation.to @Hibernate http://jboss.org @SanneGrinovero

What you get by replicating Lucene indexes on the Infinispan Data Grid (Berlin Buzzwords 2012)

Recomendados

Recomendados

Más contenido relacionado

Similar a What you get by replicating Lucene indexes on the Infinispan Data Grid (Berlin Buzzwords 2012)

Similar a What you get by replicating Lucene indexes on the Infinispan Data Grid (Berlin Buzzwords 2012) (20)

Último

Último (20)

What you get by replicating Lucene indexes on the Infinispan Data Grid (Berlin Buzzwords 2012)