Cassandra Lunch #23: Lucene Based Indexes on Cassandra

•

0 recomendaciones•378 vistas

During this week's Cassandra Lunch we covered packaged and DIY methods for Lucene-based indexes on Cassandra; as well as some pros and cons for using Lucene Based Indexes on Cassandra. Accompanying Blog: https://blog.anant.us/apache-cassandra-lunch-23-lucene-based-indexes-on-cassandra/ Cassandra Lunch Recording: https://youtu.be/Z0NXWmZAB8s Additional Resources: https://docs.datastax.com/en/dse/6.0/dse-dev/datastax_enterprise/search/searchTOC.html https://github.com/instaclustr/cassandra-lucene-index https://github.com/strapdata/elassandra https://github.com/tjake/Solandra Join Cassandra Lunch Weekly at 12 PM EST Every Wednesday: https://www.meetup.com/Cassandra-DataStax-DC/events/ Awesome Cassandra: https://github.com/Anant/awesome-cassandra Follow Us and Reach Us At: Anant: https://www.anant.us/ Cassandra.Link: https://cassandra.link/ Email: solutions@anant.us LinkedIn: https://www.linkedin.com/company/anant/ Twitter: https://twitter.com/anantcorp Eventbrite: https://www.eventbrite.com/o/anant-1072927283 Facebook: https://www.facebook.com/AnantCorp/

Tecnología

Version 1.0
Lucene Based Indexes on Cassandra
An Anant Corporation Story.

Types Covered + Pros and Cons
● Packaged
● DIY
● Pros
● Cons

DSE Search / Solr
● https://docs.datastax.com/en/dse/6.0/dse-dev/datastax_enterprise/search
/searchTOC.html

Cassandra Lucene Index
● instaclustr/cassandra-lucene-index: Lucene based secondary indexes for
Cassandra
a. https://github.com/instaclustr/cassandra-lucene-index
● Mutations to Cassandra -> Mutations on Disk (for that node)

Elessandra / ElasticSearch
● strapdata/elassandra: Elassandra = Elasticsearch + Apache Cassandra
a. https://github.com/strapdata/elassandra
● Cassandra + Elasticsearch
a. Mutations to Cassandra -> Mutations to Elasticsearch

Event -> CQRS -> Cassandra + Index
● Write
○ Event / Command goes into an Event Source repository (Kafka, SQL
Table, etc. )
○ Command Processor processes it into CQL / Elasticsearch or SOLR or
Amazon ... Algolia
● Request
○ Event / Command goes into an Event Source repository (Kafka, SQL
Table, etc. )
○ Command Processor goes to index / ﬁnds the data, goes to Cassandra ,
gets the data, returns.
○ Query the index --

Cassandra -> Batch -> Index
● Writes to Cassandra
● Every now and then - Index to ???
● Cassandra + Spark

Serverless Function -> Cassandra + Index

Apache Niﬁ (Lucene) -> Niﬁ Processor -> Cassandra + Index

Pros
● extremely rich search capabilities
● Geospatial
● synonym
● fuzzy logic search
● Typos
● Stemming
● packaged elastic/solr/lucene -> shorter latency
● separate index -> better separation of concerns and speed

Cons
● pure lucene -> reinventing the wheel of what Solr/ElasticSearch
● external elastic/solr/ ?? -> longer latency between ﬁnding the data / getting
the data
● packaged elastic/solr/lucene -> don't expect it to solve all your problems
● consistency issues (if DIY)
● lucene is memory heavy
● lucene is disk heavy

Strategy: Scalable Fast Data
Architecture: Cassandra, Spark, Kafka
Engineering: Node, Python, JVM,CLR
Operations: Cloud, Container
Rescue: Downtime!! I need help.
www.anant.us | solutions@anant.us | (855) 262-6826
3 Washington Circle, NW | Suite 301 | Washington, DC 20037

Resources
● tjake/Solandra: Solandra = Solr + Cassandra

Más contenido relacionado

La actualidad más candente

In this talk we’ll present how at GetYourGuide we’ve built from scratch a completely new ETL pipeline using Debezium, Kafka, Spark and Airflow, which can automatically handle schema changes. Our starting point was an error prone legacy system that ran daily, and was vulnerable to breaking schema changes, which caused many sleepless on-call nights. As most companies, we also have traditional SQL databases that we need to connect to in order to extract relevant data. This is done usually through either full or partial copies of the data with tools such as sqoop. However another approach that has become quite popular lately is to use Debezium as the Change Data Capture layer which reads databases binlogs, and stream these changes directly to Kafka. As having data once a day is not enough anymore for our bussiness, and we wanted our pipelines to be resilent to upstream schema changes, we’ve decided to rebuild our ETL using Debezium. We’ll walk the audience through the steps we followed to architect and develop such solution using Databricks to reduce operation time. By building this new pipeline we are now able to refresh our data lake multiple times a day, giving our users fresh data, and protecting our nights of sleep.

Modern ETL Pipelines with Change Data Capture

Databricks

Using Hazelcast in the Kappa architecture

Oliver Buckley-Salmon

Introduction to Data Engineer and Data Pipeline at Credit OK

Kriangkrai Chaonithi

Lambda Architecture has been a common way to build data pipelines for a long time, despite difficulties in maintaining two complex systems. An alternative, Kappa Architecture, was proposed in 2014, but many companies are still reluctant to switch to Kappa. And there is a reason for that: even though Kappa generally provides a simpler design and similar or lower latency, there are a lot of practical challenges in areas like exactly-once delivery, late-arriving data, historical backfill and reprocessing. In this talk, I want to show how you can solve those challenges by embracing Apache Kafka as a foundation of your data pipeline and leveraging modern stream-processing frameworks like Apache Kafka Streams and Apache Flink.

It's Time To Stop Using Lambda Architecture | Yaroslav Tkachenko, Shopify

HostedbyConfluent

Designing and Implementing a Real-time Data Lake with Dynamically Changing Sc...

Databricks

ASPgems - kappa architecture

Juantomás García Molina

Lambda architecture

Mario Alexandro Santini

Lambda architecture: from zero to One

Serg Masyutin

We often need to build applications that analyze Kafka data to unlock the most value from event streams, so how can organizations build these real-time analytics applications? In this talk, we examine an indexing approach that enables fast SQL analytics on data from Kafka, without data flattening or denormalization. Rockset is the real-time indexing database that builds an inverted index, a columnar index and a row index on all fields of your Kafka messages, including nested fields and arrays. This Converged Index accelerates various types of analytic queries–search, aggregations and joins–without the need to denormalize or transform data for performance reasons. With indexing delivering significant gains in query performance, we also need to index new data in a timely manner. We discuss several strategies used for efficient ingestion and indexing from Kafka, including rollups, write optimizations on the underlying RocksDB storage engine, and the disaggregation of ingest and query compute.

Sub-Second SQL Search, Aggregations and Joins with Kafka and Rockset | Dhruba...

HostedbyConfluent

Lambda architecture @ Indix

Rajesh Muppalla

Apache Spark Streaming

Bartosz Jankiewicz

Efficiently Building Machine Learning Models for Predictive Maintenance in th...

Databricks

Enforcing format, changing schema, introducing privacy filters have always been a challenge with the classical Kafka-API. In this talk we'll cover how to extend existing applications with webassembly, allowing developers to change the shape of data at runtime, per application without creating additional topics. By leveraging WebAssembly, we can extend the capabilities of the Kafka-API beyond what it was initially imagined. Come and learn about the future of the Kafka-API

Data Policies for the Kafka-API with WebAssembly | Alexander Gallego, Vectorized

HostedbyConfluent

Business leads, executives, analysts, and data scientists rely on up-to-date information to make business decision, adjust to the market, meet needs of their customers or run effective supply chain operations. Come hear how Asurion used Delta, Structured Streaming, AutoLoader and SQL Analytics to improve production data latency from day-minus-one to near real time Asurion’s technical team will share battle tested tips and tricks you only get with certain scale. Asurion data lake executes 4000+ streaming jobs and hosts over 4000 tables in production Data Lake on AWS.

Large Scale Lakehouse Implementation Using Structured Streaming

Databricks

Are you looking for a cloud-based architecture that includes the best of breed streaming and database technologies? In this session you will learn how to setup and configure the Confluent Cloud with MongoDB Atlas. We'll start the journey learning about the basic connectivity between the two cloud services and end with a brief discovery of what you can do with data once it is in MongoDB Atlas. By the end of this session you will know how to securely setup and configure the MongoDB Atlas connectors in the Confluent Cloud in both a source and sink configuration.

Streaming data in the cloud with Confluent and MongoDB Atlas | Robert Waters,...

HostedbyConfluent

Superset druid realtime

arupmalakar

AI is fundamentally transforming how we live and work. Zalando is a data driven company. We deliver an optimal customer experience that drives engagement. We continue to improve this experience by leveraging the latest technologies and machine learning techniques — such as building a cutting edge cloud based infrastructure to support our operations at scale. We provide our data scientists across Zalando with the means to implement artificial intelligence use cases, leveraging data from all parts of our company and the best machine learning techniques from across the industry. Apache Spark delivered through Databricks is at the core of this strategy. In this keynote, I’ll share our AI journey thus far, and share how we are exploring ways to unify data through A.I. with Spark and Databricks.

The Evolution of the Fashion Retail Industry in the Age of AI with Kshitij Ku...

Databricks

Javantura v3 - Real-time BigData ingestion and querying of aggregated data – ...

HUJAK - Hrvatska udruga Java korisnika / Croatian Java User Association

Uber Geo spatial data platform at DataWorks Summit

Zhenxiao Luo

Kafka Summit SF 2017 - Keynote - Managing Data at Scale: The Unreasonable Eff...

confluent

La actualidad más candente (20)

Modern ETL Pipelines with Change Data Capture

Using Hazelcast in the Kappa architecture

Introduction to Data Engineer and Data Pipeline at Credit OK

It's Time To Stop Using Lambda Architecture | Yaroslav Tkachenko, Shopify

Designing and Implementing a Real-time Data Lake with Dynamically Changing Sc...

ASPgems - kappa architecture

Lambda architecture

Lambda architecture: from zero to One

Sub-Second SQL Search, Aggregations and Joins with Kafka and Rockset | Dhruba...

Lambda architecture @ Indix

Apache Spark Streaming

Efficiently Building Machine Learning Models for Predictive Maintenance in th...

Data Policies for the Kafka-API with WebAssembly | Alexander Gallego, Vectorized

Large Scale Lakehouse Implementation Using Structured Streaming

Streaming data in the cloud with Confluent and MongoDB Atlas | Robert Waters,...

Superset druid realtime

The Evolution of the Fashion Retail Industry in the Age of AI with Kshitij Ku...

Javantura v3 - Real-time BigData ingestion and querying of aggregated data – ...

Uber Geo spatial data platform at DataWorks Summit

Kafka Summit SF 2017 - Keynote - Managing Data at Scale: The Unreasonable Eff...

Similar a Cassandra Lunch #23: Lucene Based Indexes on Cassandra

In Cassandra Lunch #58, Rahul Singh will be leading a presentation covering a Cassandra topic we're sure you won't want to miss. Accompanying Blog: Coming Soon! Accompanying YouTube: https://youtu.be/l4mAt3MDVZA Sign Up For Our Newsletter: http://eepurl.com/grdMkn Join Cassandra Lunch Weekly at 12 PM EST Every Wednesday: https://www.meetup.com/Cassandra-DataStax-DC/events/ Cassandra.Link: https://cassandra.link/ Follow Us and Reach Us At: Anant: https://www.anant.us/ Awesome Cassandra: https://github.com/Anant/awesome-cassandra Cassandra.Lunch: https://github.com/Anant/Cassandra.Lunch Email: solutions@anant.us LinkedIn: https://www.linkedin.com/company/anant/ Twitter: https://twitter.com/anantcorp Eventbrite: https://www.eventbrite.com/o/anant-1072927283 Facebook: https://www.facebook.com/AnantCorp/ Join The Anant Team: https://www.careers.anant.us

Apache Cassandra Lunch #58: Tools for Cassandra Titans

Anant Corporation

Indexing 3-dimensional trajectories: Apache Spark and Cassandra integration

Cesare Cugnasco

Pollfish is a survey platform which provides access to millions of targeted users. Pollfish allows easy distribution and targeting of surveys through existing mobile apps. (https://www.pollfish.com/). At pollfish we use Cassandra for difference use cases, eg. for application data store to maximize write throughput when appropriate and for our analytics project to find insights in application generated data. As a medium to accomplish our success so far, we use the Datastax's DSE 4.6 environment which integrates Appache Cassadra, Spark and a hadoop compatible file system (CFS). We will discuss how we started, how the journey was and the impressions gained so far along with some tips learned the hard way. This is a result of joint work of an excellent team here at Pollfish.

Cassandra at Pollfish

Pollfish

Cassandra at Pollfish

Stavros Kontopoulos

In Cassandra Lunch #96, Rahul Singh, CEO of Anant, will discuss different ways to get change data into and out of Cassandra using a few different strategies which could work out for your platform. Accompanying Blog: Coming Soon! Accompanying YouTube: https://youtu.be/HjgqAhsCQ64 Sign Up For Our Newsletter: http://eepurl.com/grdMkn Join Cassandra Lunch Weekly at 12 PM EST Every Wednesday: https://www.meetup.com/Cassandra-DataStax-DC/events/ Cassandra.Link: https://cassandra.link/ Follow Us and Reach Us At: Anant: https://www.anant.us/ Awesome Cassandra: https://github.com/Anant/awesome-cassandra Cassandra.Lunch: https://github.com/Anant/Cassandra.Lunch Email: solutions@anant.us LinkedIn: https://www.linkedin.com/company/anant/ Twitter: https://twitter.com/anantcorp Eventbrite: https://www.eventbrite.com/o/anant-1072927283 Facebook: https://www.facebook.com/AnantCorp/ Join The Anant Team: https://www.careers.anant.us

Apache Cassandra Lunch #96: Apache Cassandra Change Data Capture (CDC) Strate...

Anant Corporation

Enterprise Search is a challenging problem for most organizations. Public search technologies such as Google can index content and use link popularity to rank content in addition to the basic keyword matches. Enterprise Search is different. Sometimes it requires specially designed indexes as well as several processing steps. At the U.S. Patent & Trademark Office, part of the Department of Commerce, a team of professionals is building the next generation of search tools using open source technologies. Like any large undertaking, it’s not a simple plug and play project. Main topics to be covered in this talk: + Architectures for Large Scale Enterprise Search + Leveraging Apache Cassandra & Spark + Customizing / Configuring Apache SolR and Indexing + Writing a custom Parser for SolR in Scala

Building Enterprise Search Engines using Open Source Technologies

Rahul Singh

Building Enterprise Search Engines using Open Source Technologies

Anant Corporation

We will dissect the world famous todo app that provides a REST API (which is the foundation of microservices) with data backed by Apache Cassandra. We will leverage the TODO MVC and the TODO backend projects with the back end that we will build with Quarkus and Cassandra. Attendees will get an overview of Cassandra, including the driver for Quarkus. Through live coding (that attendees can try out later) in a cloud-based environment, primarily in Quarkus and Cassandra, attendees will understand how to implement and connect the APIs to the backend and leverage the generic client(s)provided. After attending this session attendees will walk away with a good understanding of implementing microservices using Cassandra and Quarkus. They will also get a working knowledge of how Astra (Cassandra as a service) can be leveraged in other solutions.

A Microservices approach with Cassandra and Quarkus | DevNation Tech Talk

Red Hat Developers

Big Data Technologies

Anant Corporation

Avoiding Pitfalls for Cassandra.pdf

Cédrick Lunven

Presenter: Evan Chan, Principal Software Engineer at Socrata Inc. How do you rapidly derive complex insights on top of really big data sets in Cassandra? This session draws upon Evan's experience building a distributed, interactive, columnar query engine on top of Cassandra and Spark. We will start by surveying the existing query landscape of Cassandra and discuss ways to integrate Cassandra and Spark. We will dive into the design and architecture of a fast, column-oriented query architecture for Spark, and why columnar stores are so advantageous for OLAP workloads. I will present a schema for Parquet-like storage of analytical datasets on Cassandra. Find out why Cassandra and Spark are the perfect match for enabling fast, scalable, complex querying and storage of big analytical data.

Cassandra Summit 2014: Interactive OLAP Queries using Apache Cassandra and Spark

DataStax Academy

Speaker(s): Erich Ess, CTO at SimpleRelevance An introduction to using Spark and integrating Spark with Cassandra to perform analytics and data processing. If you are using Cassandra, then you almost certainly have a large amount of data which you want upon which some form of processing needs to be done. Spark is a new distributed computing platform which makes writing big data analytics incredibly easy and it integrates with Cassandra with minimal effort. In this presentation, I will show how to get Spark setup with Cassandra and how to interact with Cassandra through Spark. Also covered will be architectural details of Spark such as how it handles failure recovery along with the main programming concepts needed to implement simple through complex processes on Spark. Then I will walk through a demonstration of using Spark to perform ETL on a dataset and save the data into Cassandra and then use Spark to do analytics on the data in Cassandra. Time permitting I will explain some things we are using Spark for at SimpleRelevance and cover Spark Streaming.

Kindling: Getting Started with Spark and Cassandra

DataStax Academy

Kafka spark cassandra webinar feb 16 2016

Hiromitsu Komatsu

Kafka spark cassandra webinar feb 16 2016

Hiromitsu Komatsu

Apache Solr! Enterprise Search Solutions at your Fingertips!

Murshed Ahmmad Khan

Spark Introduction

DataStax Academy

Apache Cassandra is rock-solid and widely deployed for OLTP and real-time applications, but it is typically not thought of as an OLAP database for analytical queries. This talk will show architectures and techniques for combining Apache Cassandra and Spark to yield a 10-1000x improvement in OLAP analytical performance. We will then introduce a new open-source project that combines the above performance improvements with the ease of use of Apache Cassandra, and compare it to implementations based on Hadoop and Parquet. First, the existing Cassandra Spark connector allows one to easily load data from Cassandra to Spark. We'll cover how to accelerate queries through different caching options in Spark, and the tradeoffs and limitations around performance, memory, and updating data in real time. We then dive into the use of columnar storage layout and efficient coding techniques that dramatically speed up I/O for OLAP use cases. Cassandra features like triggers and custom secondary indexes allow for easy data ingestion into columnar format. Next, we explore how to integrate this new storage with Spark SQL and its pluggable data storage API. Future developments will enable extreme analytical database performance, including smart caching of column projections, a columnar version of Spark's Catalyst execution planner, and how vectorization makes for fast cache- and GPU-friendly calculations - see Spark's Project Tungsten. FiloDB is a new open-source database using the above techniques to combine very fast Spark SQL analytical queries with the ease of use of Cassandra. We will briefly cover interesting use cases, such as: * Easy exactly-once ingestion from Kafka for streaming and IoT applications * Incremental computed columns and geospatial annotations. We'll discuss how FiloDB improves aggregations needed for choropleth maps over standard PostGIS solutions.

TupleJump: Breakthrough OLAP performance on Cassandra and Spark

DataStax Academy

FiloDB - Breakthrough OLAP Performance with Cassandra and Spark

Evan Chan

In these slides is given an overview of the different parts of Apache Spark. We analyze spark shell both in scala and python. Then we consider Spark SQL with an introduction to Data Frame API. Finally we describe Spark Streaming and we make some code examples. Topics:spark-shell, pyspark, HDFS, how to copy file to HDFS, spark transformations, spark actions, Spark SQL (Shark), spark streaming, streaming transformation stateless vs stateful, sliding windows, examples

11. From Hadoop to Spark 2/2

Fabio Fumarola

Breakthrough OLAP performance with Cassandra and Spark

Evan Chan

Similar a Cassandra Lunch #23: Lucene Based Indexes on Cassandra (20)

Apache Cassandra Lunch #58: Tools for Cassandra Titans

Indexing 3-dimensional trajectories: Apache Spark and Cassandra integration

Cassandra at Pollfish

Apache Cassandra Lunch #96: Apache Cassandra Change Data Capture (CDC) Strate...

Building Enterprise Search Engines using Open Source Technologies

A Microservices approach with Cassandra and Quarkus | DevNation Tech Talk

Big Data Technologies

Avoiding Pitfalls for Cassandra.pdf

Cassandra Summit 2014: Interactive OLAP Queries using Apache Cassandra and Spark

Kindling: Getting Started with Spark and Cassandra

Kafka spark cassandra webinar feb 16 2016

Apache Solr! Enterprise Search Solutions at your Fingertips!

Spark Introduction

TupleJump: Breakthrough OLAP performance on Cassandra and Spark

FiloDB - Breakthrough OLAP Performance with Cassandra and Spark

11. From Hadoop to Spark 2/2

Breakthrough OLAP performance with Cassandra and Spark

Más de Anant Corporation

QLoRA Fine-Tuning on Cassandra Link Data Set (1/2) Cassandra Lunch 137

Anant Corporation

Kono.IntelCraft.Weekly.AI.LLM.Landscape.2024.02.28.pdf

Anant Corporation

Data Engineer's Lunch 96: Intro to Real Time Analytics Using Apache Pinot

Anant Corporation

Series: Using AI / ChatGPT at Work - GPT Automation Are you a small business owner or web developer interested in leveraging the power of GPT (Generative Pretrained Transformer) technology to enhance your business processes? If so, Join us for a series of events focused on using GPT in business. Whether you're a small business owner or a web developer, you'll learn how to leverage GPT to improve your workflow and provide better services to your customers. GPT Automation: What it is and How it Works How Time-Saving GPT Automation Can Improve Your Business Cost-Effective GPT Automation: How it Can Save Your Business Money Using GPT Automation for Customer Service: Benefits and Best Practices The Power of GPT Automation for Content Creation Data Analysis Made Easy with GPT Automation Top GPT-3 Automation Tools for Businesses The Ethical Considerations of GPT Automation Overcoming Bias in GPT Automation: Best Practices The Future of GPT Automation: Trends and Predictions Since we focus on "no code" here, we'll explore the tools that are already out there such as ChatGPT plugins for Chrome, OpenAI GPT API, low-code/no-code platforms like Make/Integromat and Zapier, existing apps like Jasper/Rytr, and ecosystem tools like Everyprompt. We'll also discuss the resources available for those interested in learning more about GPT, including other people’s prompts.

NoCode, Data & AI LLM Inside Bootcamp: Episode 6 - Design Patterns: Retrieval...

Anant Corporation

Episode 3: The LLM / GPT / AI Prompt / Data Engineer Roadmap In this episode, we'll discuss the history, fundamentals, and the different flavors of LLMs available, beyond GPT/ChatGPT. This is a dry run of a session that will be on a LLM Bootcamp ( Fill out the survey on the link if you are interested in an in-person vs. virtual session) Intro / Fundamentals of LLM LLM Foundations History of LLMs Tuning, Training, or "In Context Learning" with LLMs What is "Prompt Engineering" Case for Augmenting LLMs Series: Using AI / ChatGPT at Work - GPT Automation Are you a small business owner or web developer interested in leveraging the power of GPT (Generative Pretrained Transformer) technology to enhance your business processes? If so, Join us for a series of events focused on using GPT in business. Whether you're a small business owner or a web developer, you'll learn how to leverage GPT to improve your workflow and provide better services to your customers. GPT Automation: What it is and How it Works How Time-Saving GPT Automation Can Improve Your Business Cost-Effective GPT Automation: How it Can Save Your Business Money Using GPT Automation for Customer Service: Benefits and Best Practices The Power of GPT Automation for Content Creation Data Analysis Made Easy with GPT Automation Top GPT-3 Automation Tools for Businesses The Ethical Considerations of GPT Automation Overcoming Bias in GPT Automation: Best Practices The Future of GPT Automation: Trends and Predictions Since we focus on "no code" here, we'll explore the tools that are already out there such as ChatGPT plugins for Chrome, OpenAI GPT API, low-code/no-code platforms like Make/Integromat and Zapier, existing apps like Jasper/Rytr, and ecosystem tools like Everyprompt. We'll also discuss the resources available for those interested in learning more about GPT, including other people’s prompts.

Automate your Job and Business with ChatGPT #3 - Fundamentals of LLM/GPT

Anant Corporation

In Apache Cassandra Lunch #131: YugabyteDB Developer Tools, we discussed third party developer tools that are compatible with YugabyteDB. We talked about using Yugabyte Developer Tools for data visualization and schema management. The live recording of Cassandra Lunch, which includes a more in-depth discussion and a demo, is embedded below in case you were not able to attend live. If you would like to attend Apache Cassandra Lunch live, it is hosted every Wednesday at 12 PM EST. Developer tools play a critical role in simplifying and streamlining database development and management. They allow developers and administrators to be more productive, reducing the time and effort required to create and maintain database schemas, write SQL queries, test database performance, and enable collaboration. Developer tools also make it possible to track changes over time, improving the ability to manage the entire development lifecycle.

YugabyteDB Developer Tools

Anant Corporation

In this episode we'll discuss the different flavors of prompt engineering in the LLM/GPT space. According to your skill level you should be able to pick up at any of the following: Leveling up with GPT 1: Use ChatGPT / GPT Powered Apps 2: Become a Prompt Engineer on ChatGPT/GPT 3: Use GPT API with NoCode Automation, App Builders 4: Create Workflows to Automate Tasks with NoCode 5: Use GPT API with Code, make your own APIs 6: Create Workflows to Automate Tasks with Code 7: Use GPT API with your Data / a Framework 8: Use GPT API with your Data / a Framework to Make your own APIs 9: Create Workflows to Automate Tasks with your Data /a Framework 10: Use Another LLM API other than GPT (Cohere, HuggingFace) 11: Use open source LLM models on your computer 12: Finetune / Build your own models Series: Using AI / ChatGPT at Work - GPT Automation Are you a small business owner or web developer interested in leveraging the power of GPT (Generative Pretrained Transformer) technology to enhance your business processes? If so, Join us for a series of events focused on using GPT in business. Whether you're a small business owner or a web developer, you'll learn how to leverage GPT to improve your workflow and provide better services to your customers.

Episode 2: The LLM / GPT / AI Prompt / Data Engineer Roadmap

Anant Corporation

In Data Engineer’s Lunch #89: Machine Learning Orchestration with Airflow, we discussed using Apache Airflow to manage and schedule machine learning tasks. By following the best practices of ML Ops, teams can streamline their ML workflows and build scalable, efficient, and accurate models that deliver real-world business value. Properly implemented ML Ops can help organizations stay ahead of the curve and achieve their goals in the fast-paced world of machine learning. Apache Airflow is an open-source tool for scheduling and automating workflows. Airflow allows you to define workflows in Python, with tasks defined as Python functions that can include Operators for all sorts of external tools. This makes it easy to automate repeated processes and define dependencies between tasks, creating directed-acyclic-graphs of tasks that can be scheduled using cron syntax or frequency tasks. Airflow also features a user-friendly UI for monitoring task progress and viewing logs, giving you greater control over your data pipeline.

Machine Learning Orchestration with Airflow

Anant Corporation

If you didn't attend, you don't want to miss a much shorter synopsis of what was covered and get some thoughts from us as to why they are important. We'll talk about the main topics of the event. 1. ACID transactions on Cassandra by Aaron Ploetz, Datastax 2. Apache Flink with Apache Cassandra at Satyajit Thadeswar, Netflix 3. Durable Execution built on Apache Cassandra by Loren Sands-Ramshaw, Temporal 4. Switching from Mongo to Cassandra with Mongoose & new Stargate JSON API, Valeri Karpov 5. Cloud Native and Realtime AI/ML with Patrick Mcfadin and Davor Boncaci, Datastax

Cassandra Lunch 130: Recap of Cassandra Forward Talks

Anant Corporation

Data Engineer's Lunch 90: Migrating SQL Data with Arcion

Anant Corporation

Data Engineer's Lunch 89: Machine Learning Orchestration with AirflowMachine ...

Anant Corporation

Cassandra Lunch 129: What’s New: Apache Cassandra 4.1+ Features & Future

Anant Corporation

As the demand for real-time data processing continues to grow, so too do the challenges associated with building production-ready applications that can handle large volumes of data and handle it quickly. In this talk, we will explore common problems faced when building real-time applications at scale, with a focus on a specific use case: detecting and responding to cyclist crashes. Using telemetry data collected from a fitness app, we’ll demonstrate how we used a combination of Apache Kafka and Python-based microservices running on Kubernetes to build a pipeline for processing and analyzing this data in real-time. We'll also discuss how we used machine learning techniques to build a model for detecting collisions and how we implemented notifications to alert family members of a crash. Our ultimate goal is to help you navigate the challenges that come with building data-intensive, real-time applications that use ML models. By showcasing a real-world example, we aim to provide practical solutions and insights that you can apply to your own projects. Key takeaways: An understanding of the common challenges faced when building real-time applications at scale Strategies for using Apache Kafka and Python-based microservices to process and analyze data in real-time Tips for implementing machine learning models in a real-time application Best practices for responding to and handling critical events in a real-time application

Data Engineer's Lunch #86: Building Real-Time Applications at Scale: A Case S...

Anant Corporation

Data Engineer's Lunch #85: Designing a Modern Data Stack

Anant Corporation

CL 121

Anant Corporation

Data Engineer's Lunch #83: Strategies for Migration to Apache Iceberg

Anant Corporation

In this lunch, Johnny will show us how easy it is to start monitoring your Cassandra cluster in minutes. He will explain the various aspects and features of Cassandra that need to be monitored, how to do it, and most importantly why! Approaches for backups and Cassandra repairs will be discussed and explored in detail. Learn how AxonOps significantly reduces the complexity and overhead when looking after Cassandra and ensures your Cassandra cluster is reliable and resilient. Experienced developer, DevOps, architect, and AxonOps co-founder, Johnny Miller, has worked with a wide variety of companies – from small start-ups to large enterprises. He has been working with Cassandra for many years and has a deep understanding of the challenges facing modern companies looking to adopt Apache Cassandra.

Apache Cassandra Lunch 120: Apache Cassandra Monitoring Made Easy with AxonOps

Anant Corporation

In Apache Cassandra Lunch #119, Rahul Singh will cover a refresher on GUI desktop/web tools for users that want to get their hands dirty with Cassandra but don't want to deal with CQLSH to do simple queries. Some of the tools are web-based and others are installed on your desktop. Since the beginning days of Cassandra, a lot has changed and there are many options for command-line-haters to use Cassandra.

Apache Cassandra Lunch 119: Desktop GUI Tools for Apache Cassandra

Anant Corporation

Data Engineer's Lunch #82: Automating Apache Cassandra Operations with Apache...

Anant Corporation

Data Engineer's Lunch #60: Series - Developing Enterprise Consciousness

Anant Corporation

Más de Anant Corporation (20)

QLoRA Fine-Tuning on Cassandra Link Data Set (1/2) Cassandra Lunch 137

Kono.IntelCraft.Weekly.AI.LLM.Landscape.2024.02.28.pdf

Data Engineer's Lunch 96: Intro to Real Time Analytics Using Apache Pinot

NoCode, Data & AI LLM Inside Bootcamp: Episode 6 - Design Patterns: Retrieval...

Automate your Job and Business with ChatGPT #3 - Fundamentals of LLM/GPT

YugabyteDB Developer Tools

Episode 2: The LLM / GPT / AI Prompt / Data Engineer Roadmap

Machine Learning Orchestration with Airflow

Cassandra Lunch 130: Recap of Cassandra Forward Talks

Data Engineer's Lunch 90: Migrating SQL Data with Arcion

Data Engineer's Lunch 89: Machine Learning Orchestration with AirflowMachine ...

Cassandra Lunch 129: What’s New: Apache Cassandra 4.1+ Features & Future

Data Engineer's Lunch #86: Building Real-Time Applications at Scale: A Case S...

Data Engineer's Lunch #85: Designing a Modern Data Stack

CL 121

Data Engineer's Lunch #83: Strategies for Migration to Apache Iceberg

Apache Cassandra Lunch 120: Apache Cassandra Monitoring Made Easy with AxonOps

Apache Cassandra Lunch 119: Desktop GUI Tools for Apache Cassandra

Data Engineer's Lunch #82: Automating Apache Cassandra Operations with Apache...

Data Engineer's Lunch #60: Series - Developing Enterprise Consciousness

Último

Where to Learn More About FDO _ Richard at FIDO Alliance.pdf

FIDO Alliance

Simplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdf

FIDO Alliance

The standard Salesforce Approval process can be limiting in many ways, especially in complex scenarios. What if there was a way to implement very flexible approvals where one can use Apex code to make data updates in unrelated records, dynamically generate next steps details, and compute assignees on the fly? And still use UI-based configurations to implement concrete approval processes. In this session, we will share ideas behind such a solution and show a few lines of code to get you started.

Custom Approval Process: A New Perspective, Pavel Hrbacek & Anindya Halder

CzechDreamin

Google I/O Extended 2024 Warsaw

GDSC PJATK

ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...

FIDO Alliance

Designing inclusive products is not only a social responsibility but also a business imperative. This talk delves into the journey of creating accessible hardware products that cater to diverse user needs. Key Topics Covered: 1. Introduction to Inclusive Design - Importance of accessibility in product design - Overview of Comcast's commitment to making products accessible to a wide audience 2. Case Study: Xfinity Large Button Voice Remote - Initial challenges and the evolution of the product - User research and feedback that shaped the design - Key features of the final product and their benefits 3. Designing for Diverse Needs - Understanding human-centered design and its historical context - The impact of designing for people with disabilities on overall product quality - Examples from other industries, such as architecture and industrial design 4. Integrating Accessibility from the Beginning - The cost and efficiency benefits of designing for accessibility from the start - The process of embedding accessibility as a core trait rather than an optional feature 5. Real-World Impact and Continuous Improvement - Insights from in-home studies with users having assistive needs - How continuous feedback and iterative design lead to better products - The role of inclusive research and development practices

Designing for Hardware Accessibility at Comcast

UXDXConf

Already know how to write a basic SOQL query? Great! But what about an *aggregate* SOQL query? You know, the kind that uses aggregate functions like COUNT & MAX along with GROUP BY and HAVING clauses? No? Well, get ready to learn how to slice & dice your org’s data right inside your own dev console. From finding duplicate records to prototyping summary & matrix reports, learn the ins and outs of aggregate queries during this fast-paced but admin-friendly session on advanced SOQL concepts.

SOQL 201 for Admins & Developers: Slice & Dice Your Org’s Data With Aggregate...

CzechDreamin

How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdf

FIDO Alliance

Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf

FIDO Alliance

The Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdf

FIDO Alliance

Discover the top Symfony development companies that excel in creating robust and scalable web applications. Our latest blog highlights the best firms specializing in Symfony, known for their expertise in delivering high-performance solutions. Whether you’re looking to start a new project or enhance an existing one, these companies offer the skills and experience needed to bring your vision to life. Read more to find your perfect Symfony development partner.

Top 10 Symfony Development Companies 2024

TopCSSGallery

The presentation underscores the strategic advantage of treating design systems not just as technical assets but as vital business components that require thoughtful management, robust planning, and strategic alignment with organizational goals. Key Points Covered: - Understanding Design Systems as Business Entities: Conceptualizing design systems as internal business entities can streamline their integration and evolution within a company. - Adoption and Expansion: Elaborating on the importance of tactical adoption across organizational structures, enhancing product suites to cater to user needs and broadening scope to mobile and content authoring solutions. - Data-Driven Development: Utilizing data insights for component development ensures that resources are allocated to create valuable, widely used features. - Financial Modeling for Design Systems: Developing sustainable funding models is crucial for long-term support and success of design systems. - Promoting Internal Buy-In: Stressing on strategies for promoting design systems within the organization to increase engagement and investment from internal stakeholders.

A Business-Centric Approach to Design System Strategy

UXDXConf

Ever caught yourself nodding along when someone mentions "delivering value" in Agile, but secretly wondering what the heck they actually mean? You're not alone! Join us for an eye-opening session where we'll strip away the buzzwords and dive into the heart of Agile—value delivery. But what is "value"? Is it a mythical unicorn in the world of software development, or is there more to this overused term? This isn't going to be a sit-and-get lecture. We're talking about a face-to-face, interactive meetup where YOU play a crucial role. Come along to: Define It: What does "value" really mean? We’ll build a definition that’s not just words, but a compass for your Agile journey. Contextualise It: Discover what value means specifically to you, your team, your company, and your industry. Because one size does not fit all. Deliver It: Share strategies and gather new ones for uncovering and delivering true value—no more shooting in the dark!

Unpacking Value Delivery - Agile Oxford Meetup - May 2024.pptx

David Michel

As an SEO expert specializing in the IPTV and VPN niches with over five years of experience, I navigate the unique challenges of these industries adeptly. My strategic approach encompasses competitive analysis, targeted keyword research, content optimization, and high-quality backlink creation. My goal is to optimize my clients' online visibility, generating targeted organic traffic and maximizing their return on investment. With a results-driven approach and a passion for innovation, I'm poised to assist my clients in thriving in an ever-evolving digital landscape. <a href="https://iptvreel.com">

THE BEST IPTV in GERMANY for 2024: IPTVreel

reely ones

The Metaverse: Are We There Yet?

Mark Billinghurst

ScyllaDB has the potential to deliver impressive performance and scalability. The better you understand how it works, the more you can squeeze out of it. But before you squeeze, make sure you know what to monitor! Watch our experienced Postgres developer work through monitoring and performance strategies that help him understand what mistakes he’s made moving to NoSQL. And learn with him as our database performance expert offers friendly guidance on how to use monitoring and performance tuning to get his sample Rust application on the right track. This webinar focuses on using monitoring and performance tuning to discover and correct mistakes that commonly occur when developers move from SQL to NoSQL. For example: - Common issues getting up and running with the monitoring stack - Using the CQL optimizations dashboard - Common issues causing high latency in a node - Common issues causing replica imbalance - What a healthy system looks like in terms of memory - Key metrics to keep an eye on This isn’t “Death-by-Powerpoint.” We’ll walk through problems encountered while migrating a real application from Postgres to ScyllaDB – and try to fix them live as well.

Optimizing NoSQL Performance Through Observability

ScyllaDB

Explore the core of Salesforce success in 'Salesforce Adoption – Metrics, Methods, and Motivation.' We will discuss essential metrics, effective methods to drive adoption, and the driving force behind user engagement and explore strategies for onboarding, training, and continuous support that empower users to navigate the platform seamlessly. By leveraging these tools, you can effectively measure adoption against your company’s goals and create an environment where users not only adopt Salesforce but actively contribute to its ongoing success.

Salesforce Adoption – Metrics, Methods, and Motivation, Antone Kom

CzechDreamin

Syngulon’s technology expands the capacity for selection of microorganisms. The ability to select individual microbes with a behavior of interest is essential, whether for simple cloning at the bench, or for industry-scale production. Synthetic biology uses the concept of “bioengineering” to improve or modify existing genetic systems to create microbes with desired behaviors, and Syngulon uses this approach to develop its selection technologies. This selection technology is based on bacteriocins, ribosomally-produced peptides naturally made by most bacteria to kill competitive microbial species. These bacteriocins can have a limited or wide target range against other microbial species. This technology offers advantageous over antibiotic selection for several reasons: it avoids the use of antibiotics in the first place, helping to reduce the spread of antibiotic resistant microbes. The technology also increases product yield; as bacteriocins are generally smaller peptides, they do not impose a heavy metabolic burden on the producing cell. They can have a wide target specificity, helping to avoid genetic drift. Finally, our system is 100% plasmid-based (e.g. without chromosomal mutations), making it applicable for use in any E. coli strains.

Syngulon - Selection technology May 2024.pdf

Syngulon

What's New in Teams Calling, Meetings and Devices April 2024

Stephanie Beckett

Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...

FIDO Alliance

Cassandra Lunch #23: Lucene Based Indexes on Cassandra

1. Version 1.0 Lucene Based Indexes on Cassandra An Anant Corporation Story.

2. Types Covered + Pros and Cons ● Packaged ● DIY ● Pros ● Cons

3. Packaged

4. DSE Search / Solr ● https://docs.datastax.com/en/dse/6.0/dse-dev/datastax_enterprise/search /searchTOC.html

5. Cassandra Lucene Index ● instaclustr/cassandra-lucene-index: Lucene based secondary indexes for Cassandra a. https://github.com/instaclustr/cassandra-lucene-index ● Mutations to Cassandra -> Mutations on Disk (for that node)

6. Elessandra / ElasticSearch ● strapdata/elassandra: Elassandra = Elasticsearch + Apache Cassandra a. https://github.com/strapdata/elassandra ● Cassandra + Elasticsearch a. Mutations to Cassandra -> Mutations to Elasticsearch

7. DIY

8. Event -> CQRS -> Cassandra + Index ● Write ○ Event / Command goes into an Event Source repository (Kafka, SQL Table, etc. ) ○ Command Processor processes it into CQL / Elasticsearch or SOLR or Amazon ... Algolia ● Request ○ Event / Command goes into an Event Source repository (Kafka, SQL Table, etc. ) ○ Command Processor goes to index / ﬁnds the data, goes to Cassandra , gets the data, returns. ○ Query the index --

9. Cassandra -> Batch -> Index ● Writes to Cassandra ● Every now and then - Index to ??? ● Cassandra + Spark

10. Cassandra Triggers -> Index

11. Serverless Function -> Cassandra + Index

12. Apache Niﬁ (Lucene) -> Niﬁ Processor -> Cassandra + Index

13. Pros ● extremely rich search capabilities ● Geospatial ● synonym ● fuzzy logic search ● Typos ● Stemming ● packaged elastic/solr/lucene -> shorter latency ● separate index -> better separation of concerns and speed

14. Cons ● pure lucene -> reinventing the wheel of what Solr/ElasticSearch ● external elastic/solr/ ?? -> longer latency between ﬁnding the data / getting the data ● packaged elastic/solr/lucene -> don't expect it to solve all your problems ● consistency issues (if DIY) ● lucene is memory heavy ● lucene is disk heavy

15. Strategy: Scalable Fast Data Architecture: Cassandra, Spark, Kafka Engineering: Node, Python, JVM,CLR Operations: Cloud, Container Rescue: Downtime!! I need help. www.anant.us | solutions@anant.us | (855) 262-6826 3 Washington Circle, NW | Suite 301 | Washington, DC 20037

16. Resources ● tjake/Solandra: Solandra = Solr + Cassandra

Cassandra Lunch #23: Lucene Based Indexes on Cassandra

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Similar a Cassandra Lunch #23: Lucene Based Indexes on Cassandra

Similar a Cassandra Lunch #23: Lucene Based Indexes on Cassandra (20)

Más de Anant Corporation

Más de Anant Corporation (20)

Último

Último (20)

Cassandra Lunch #23: Lucene Based Indexes on Cassandra