The presentation on "Retrieval Augmented Generation for Interactive Podcasts" outlines a method to transform podcast audio into interactive chat interfaces. It covers the design, coding, and practical aspects of using Retrieval Augmented Generation (RAG) to emulate podcast guest responses. The project involves processing a significant volume of podcast data, including episodes, transcripts, and metadata. The technical discussion focuses on ingesting audio and metadata into an AI system and querying it for conversational responses. Key concepts introduced include vector embedding, which converts text to conceptual vectors using models, and the application of Large Language Models (LLMs) with Transformer architecture for context understanding. The coding segment details microservice messages for transcript ingestion and chat functionalities, employing transcription services, embedding techniques, and vector storage solutions. Challenges in RAG project deployment are also discussed, highlighting performance, quality, regressions, and managing expectations. The presentation concludes by contrasting technical complexities with a philosophical vision of AI's potential, inspired by speculative fiction, suggesting a future where AI capabilities vastly exceed human cognitive functions. Further resources and open-source implementations are provided for those interested in the technical development of interactive podcast systems

1. Retrieval
Augmented
Generation
Making a Podcast
“interactive”
Richard Rodger, Voxgig

3. The Culture’s inhabitants “could
record their mind-states,
effectively taking a reading of the
person’s personality which could be
stored, duplicated, read, transmitted
and installed …”
Iain M Banks, Look to Windward

9. How do I build this?

10. 1. Design
2. Code
3. Practicalities

11. “Using the podcast audio
recordings, build a chat
interface that responds like a
podcast guest”

12. - ~130 episodes
- ~30 minutes per episode
- ~7000 words per transcript
- 2 new episodes per week
- Metadata:
- guest details
- show notes

13. 1. Ingestion
Get the audio and metadata into
the “AI”
2. Querying
Get the conversational responses
out of the “AI”

14. Ingestion

15. Querying

16. Seems like hard work…?
Why not just concatenate all
transcripts, and use that as
the context prompt?

17. “Vector Embedding”
Text -> Concepts
(using a “Model”)

18. But why vectors?

19. The vector database does the
embedding for you

20. Large Language Models
(LLMs)

21. Large Language Models
(LLMs)
- Very large neural networks
- Use the Transformer* architecture
- A vector embedding
- Cares about word order
- Cares about word context
* Attention is All You Need, Vaswani et al. 2017

22. Retrieval Augmented Generation (RAG)

23. Code

24. // Microservice messages
aim: ingest: {
// Get transcription
transcribe: episode: {}
// Do “embedding”
ingest: transcript: {}
}
aim: chat: {
// Use prompt context to reply
chat: query: {}
}

25. // aim:ingest, transcribe:episode
// lambda called when there’s
// a new audio file in s3
const audio = loadAudio(event)
// Use deepgram.com to get the conversation text
const result = await deepgram
.listen
.prerecorded
.transcribeFile(
audio.content,
options
)
// Save transcript to s3
saveTranscript(result)

26. // aim:ingest, ingest:transcript
// lambda called when there’s
// a new transcript file in s3
const transcript = loadTranscript(event)
// Split into “chunks”, each to be added to
// an OpenSearch vector collection
const chunks = chunkify(transcript)
for(let chunk in chunks) {
// Call AWS Bedrock, specify model
const embedding =
bedrockClient.embed(chunk, model)
// Store embedding vectors in OpenSearch
openSearchClient.store(embedding)
}

27. // aim:chat, chat:query
const query = event.body.query // HTTP POST
// Call AWS Bedrock, specify model
const embedding =
bedrockClient.embed(query, model)
// Use embedding to get context chunk text
const context = openSearchClient
.search(embedding)
// Do prompt engineering here!
const prompt = "Answer question with Context: "
+ context + "nQuestion: " + query
// Get answer!
const answer = bedrockClient.invoke(prompt)

28. // Pro Tip: use a REPL!
wovs/pdm-local> aim:chat,chat:query,query:"what
is developer relations?"
{
ok: true,
why: '',
answer: "Developer Relations is the practice of
building and maintaining relationships between
companies and developers...The goal of Developer
Relations is to make the company's products as
easy to use, understand, and integrate into a
developer's workflow as possible."
}

29. // Open Source
// Reference implementation
github.com/mikaelvesavuori/bedrock-rag-demo
// Voxgig microservice implementation
github.com/voxgig/podmind
// Blog post (next week)
richardrodger.com

30. Practicalities
…, you are face to face with the
champion privy builder of
Sangamon County

31. Paul Savage
brightbeam.com

32. It’s the Wild West
“I’ve got my stuff rigged to hit mixtral-8x7, and dolphin locally, and
3.5-turbo, and the 4-series preview all with easy comparison in
emacs and stuff, and in fairness the 4.5-preview is starting to show
some edge on 8x7 …
Until I realized Perplexity will give you a decent amount of Mistral
Medium for free ….
Who is sama kidding they’re still leading here? Mistral Medium
destroys the 4.5 preview. And Perplexity wouldn’t be giving it away
in any quantity if it had a cost structure like 4.5 …
Mistral is the new “RenTech of AI”, DPO and Alibi and sliding window
and modern mixtures are well-understood so the money is in the lag
between some new edge and TheBloke having it quantized for a Mac
Mini or 4070 Super …
https://news.ycombinator.com/item?id=38948291

33. It’s the Wild West
Here's a glossary to understand this post:
- mixtral-8x7 or 8x7: Open source model by Mistral AI.
- Dolphin: An uncensored version of the mistral model
- 3.5-turbo: GPT-3.5 Turbo, the cheapest API from OpenAI
- 4-series preview OR "4.5 preview": GPT-4 Turbo, the most capable API from OpenAI
- mistral-medium: A new model by Mistral AI that they are only serving through AI. It's in
private beta and there's a waiting list to access it.
- Perplexity: A new search engine that is challenging Google by applying LLM to search
- Sama: Sam Altman, CEO of OpenAI
- RenTech: Renaissance Technologies, a secretive hedge fund known for delivering
impressive returns improving on the work of others
- DPO: Direct Preference Optimization. It is a technique that leverages AI feedback to
optimize the performance of smaller, open-source models like Zephyr-7B1.
- Alibi: a Python library that provides tools for machine learning model inspection and
interpretation2. It can be used to explain the predictions of any black-box model,
including LLMs.
- Sliding window: a type of attention mechanism introduced by Mistral-7B3. It is used to
support longer sequences in LLMs.
- Modern mixtures: The process of using multiple models together, like "mixtral" is a
mixture of several mistral models.
- TheBloke: Open source developer that is very quick at quantizing all new models that
come out
- Quantize: Decreasing memory requirements of a new model by decreasing the precision
of weights, typically with just minor performance degradation.
- 4070 Super: NVIDIA 4070 Super, new graphics card announced just a week ago

34. https://github.com/mikaelves
avuori/bedrock-rag-demo

35. So you want to deliver a RAG
project…

36. Do you want good
performance?

37. Do you want high quality
answers?
ann-benchmarks.com

38. Do you like regressions?

39. Do you like unrealistic
expectations?

40. Do you like being unable to solve
fundamental limitations? (maybe)
…my experience over the past few
months suggests that for system
programming, LLMs almost never
provide acceptable solutions…
antirez.com/news/140
(Salvatore Sanfilippo - wrote Redis)

41. But…

42. I am not an animal brain, I am
not even some attempt to
produce an AI through
software running on a
computer. I am a Culture Mind.
We are close to gods, … we
are quicker; we live faster and
more completely than you do,
with so many more senses,
such a greater store of
memories and at such a fine
level of detail.
- Look to Windward,
Iain M Banks

43. Thanks!
richardrodger.com