案例目标本案例旨在构建一个多模态视频RAG检索增强生成系统能够处理视频内容并基于视频内容回答用户问题。系统使用OpenAI GPT4V多模态大语言模型结合CLIP模型生成多模态嵌入并使用LanceDB向量数据库进行高效的向量存储和检索。通过将视频转换为图像序列和文本转录系统可以基于视频内容进行智能问答。技术栈与核心依赖LlamaIndexLanceDBOpenAI GPT4VCLIPWhisperMoviePyPyTubeSpeechRecognitionpip install llama-index-vector-stores-lancedb pip install llama-index-multi-modal-llms-openai pip install llama-index-embeddings-clip pip install llama_index ftfy regex tqdm pip install -U openai-whisper pip install githttps://github.com/openai/CLIP.git pip install torch torchvision pip install matplotlib scikit-image pip install lancedb pip install moviepy pip install pytube pip install pydub pip install SpeechRecognition pip install ffmpeg-python pip install soundfile环境配置环境要求Python 3.8PyTorch 1.12OpenAI API密钥用于GPT4VLanceDB向量数据库import os OPENAI_API_KEY YOUR_OPENAI_API_KEY os.environ[OPENAI_API_KEY] OPENAI_API_KEY案例实现步骤1: 导入必要的库from moviepy.editor import VideoFileClip from pathlib import Path import speech_recognition as sr from pytube import YouTube from pprint import pprint from llama_index.core.indices import MultiModalVectorStoreIndex from llama_index.core import SimpleDirectoryReader, StorageContext from llama_index.vector_stores.lancedb import LanceDBVectorStore from llama_index.multi_modal_llms.openai import OpenAIMultiModal步骤2: 设置视频处理参数# 视频URL和输出路径配置 video_url https://www.youtube.com/watch?vd_qvLDhkg00 output_video_path ./video_data/ output_folder ./mixed_data/ output_audio_path ./mixed_data/output_audio.wav filepath output_video_path input_vid.mp4 Path(output_folder).mkdir(parentsTrue, exist_okTrue)步骤3: 定义视频处理函数def download_video(url, output_path): 从YouTube下载视频 yt YouTube(url) metadata {Author: yt.author, Title: yt.title, Views: yt.views} yt.streams.get_highest_resolution().download( output_pathoutput_path, filenameinput_vid.mp4 ) return metadata def video_to_images(video_path, output_folder): 将视频转换为图像序列 clip VideoFileClip(video_path) clip.write_images_sequence( os.path.join(output_folder, frame%04d.png), fps0.2 ) def video_to_audio(video_path, output_audio_path): 从视频中提取音频 clip VideoFileClip(video_path) audio clip.audio audio.write_audiofile(output_audio_path) def audio_to_text(audio_path): 使用Whisper将音频转换为文本 recognizer sr.Recognizer() audio sr.AudioFile(audio_path) with audio as source: audio_data recognizer.record(source) try: text recognizer.recognize_whisper(audio_data) except sr.UnknownValueError: print(Speech recognition could not understand the audio.) except sr.RequestError as e: print(fCould not request results from service; {e}) return text步骤4: 处理视频数据try: metadata_vid download_video(video_url, output_video_path) video_to_images(filepath, output_folder) video_to_audio(filepath, output_audio_path) text_data audio_to_text(output_audio_path) with open(output_folder output_text.txt, w) as file: file.write(text_data) print(Text data saved to file) file.close() os.remove(output_audio_path) print(Audio file removed) except Exception as e: raise e步骤5: 创建多模态向量存储# 创建文本和图像向量存储 text_store LanceDBVectorStore(urilancedb, table_nametext_collection) image_store LanceDBVectorStore(urilancedb, table_nameimage_collection) storage_context StorageContext.from_defaults( vector_storetext_store, image_storeimage_store ) # 创建多模态索引 documents SimpleDirectoryReader(output_folder).load_data() index MultiModalVectorStoreIndex.from_documents( documents, storage_contextstorage_context, )步骤6: 创建检索器retriever_engine index.as_retriever( similarity_top_k5, image_similarity_top_k5 )步骤7: 定义RAG提示模板import json metadata_str json.dumps(metadata_vid) qa_tmpl_str ( Given the provided information, including relevant images and retrieved context from the video, accurately and precisely answer the query without any additional prior knowledge.\n Please ensure honesty and responsibility, refraining from any racist or sexist remarks.\n ---------------------\n Context: {context_str}\n Metadata for video: {metadata_str} \n ---------------------\n Query: {query_str}\n Answer: )步骤8: 定义检索函数def retrieve(retriever_engine, query_str): retrieval_results retriever_engine.retrieve(query_str) retrieved_image [] retrieved_text [] for res_node in retrieval_results: if isinstance(res_node.node, ImageNode): retrieved_image.append(res_node.node.metadata[file_path]) else: display_source_node(res_node, source_length200) retrieved_text.append(res_node.text) return retrieved_image, retrieved_text步骤9: 执行查询并生成回答# 定义查询 query_str Using examples from video, explain all things covered in video regarding to gaussian function # 检索相关图像和文本 img, txt retrieve(retriever_engineretriever_engine, query_strquery_str) image_documents SimpleDirectoryReader( input_diroutput_folder, input_filesimg ).load_data() context_str .join(txt) # 显示检索到的图像 plot_images(img) # 使用GPT4V生成最终回答 openai_mm_llm OpenAIMultiModal( modelgpt-4o, api_keyOPENAI_API_KEY, max_new_tokens1500 ) response openai_mm_llm.complete( promptqa_tmpl_str.format( context_strcontext_str, query_strquery_str, metadata_strmetadata_str ), image_documentsimage_documents, ) pprint(response.text)案例效果查询结果示例当查询Using examples from video, explain all things covered in video regarding to gaussian function时系统返回了以下回答The video by 3Blue1Brown, titled A pretty reason why Gaussian Gaussian Gaussian, covers several aspects of Gaussian function, also known as the normal distribution. Heres a summary of key points discussed in the video: 1. **Central Limit Theorem**: The video begins by discussing central limit theorem, which states that the sum of multiple copies of a random variable tends to look like a normal distribution. As the number of variables increases, the approximation to a normal distribution becomes better. 2. **Convolution of Random Variables**: The process of adding two random variables is mathematically represented by a convolution of their respective distributions. The video explains concept of convolution and how it is used to find the distribution of sum of two random variables. 3. **Gaussian Function**: The Gaussian function is more complex than just \( e^{-x^2} \). The full formula includes a scaling factor to ensure the area under the curve is 1 (making it a valid probability distribution), a standard deviation parameter \( \sigma \) to describe the spread, and a mean parameter \( \mu \) to shift the center. However, the video focuses on centered distributions with \( \mu 0 \). 4. **Visualizing Convolution**: The video presents a visual method to understand the convolution of two Gaussian functions using diagonal slices on the xy-plane. This method involves looking at the probability density of landing on a point (x, y) as \( f(x) \times g(y) \), where f and g are the two distributions being convolved. 5. **Rotational Symmetry**: A key property of Gaussian function is its rotational symmetry, which is unique to bell curves. This symmetry is exploited in the video to simplify the calculation of convolution. By rotating the graph 45 degrees, the computation becomes easier because the integral only involves one variable. 6. **Result of Convolution**: The video demonstrates that the convolution of two Gaussian functions is another Gaussian function. This is a special property because convolutions typically result in a different kind of function. The standard deviation of the resulting Gaussian is \( \sqrt{2} \times \sigma \) if the original Gaussians had the same standard deviation. 7. **Proof of Central Limit Theorem**: The video explains that the convolution of two Gaussians being another Gaussian is a crucial step in proving the central limit theorem. It shows that the Gaussian function is a fixed point in the space of distributions, and since all distributions with finite variance tend towards a single universal shape, that shape must be the Gaussian. 8. **Connection to Pi**: The video also touches on the connection between the Gaussian function and the number Pi, which appears in the formula for the normal distribution. The video aims to provide an intuitive geometric argument for why the sum of two normally distributed random variables is also normally distributed, and how this relates to the central limit theorem and the special properties of the Gaussian function.系统成功检索到了与高斯函数相关的视频帧和文本内容并基于这些内容生成了详细的回答涵盖了视频中关于高斯函数的所有要点。案例实现思路本案例的实现思路基于以下核心概念视频多模态处理将视频转换为图像序列和文本转录实现视频内容的多模态表示。多模态嵌入使用CLIP模型生成图像嵌入使用文本嵌入模型生成文本嵌入将不同模态的数据映射到同一向量空间。向量存储与检索使用LanceDB向量数据库存储文本和图像的嵌入向量支持高效的相似性搜索。多模态RAG结合检索到的文本和图像内容使用GPT4V多模态大语言模型生成回答。提示工程设计专门的提示模板引导模型基于检索到的内容回答问题避免使用先验知识。系统架构如下图所示视频图像序列嵌入音频文本转录文本嵌入向量存储检索回答扩展建议功能扩展视频摘要生成自动生成视频内容的摘要便于用户快速了解视频主题时间戳定位基于查询内容定位视频中相关的时间戳实现精确的内容定位多视频检索支持同时检索多个视频内容实现跨视频的知识整合交互式查询开发交互式界面支持用户与视频内容进行更自然的交互多语言支持支持多语言视频内容的处理和查询性能优化增量索引实现视频内容的增量索引支持动态添加新视频分布式处理将视频处理和索引过程分布到多台机器提高处理效率缓存机制实现查询结果缓存减少重复查询的计算开销向量压缩使用产品量化等技术压缩向量减少存储空间总结本案例成功实现了一个基于OpenAI GPT4V和LanceDB的多模态视频RAG系统。该系统能够处理视频内容将其转换为图像序列和文本转录并基于这些内容回答用户问题。通过使用LlamaIndex框架和LanceDB向量数据库我们构建了一个高效、可扩展的多模态视频检索解决方案。该系统的核心优势包括支持视频内容的多模态处理和理解基于向量嵌入的高效相似性搜索结合文本和图像内容生成准确回答可扩展的架构易于添加新的视频内容未来该系统可以进一步扩展以支持更复杂的视频处理任务实现更精确的内容定位和更丰富的交互方式为用户提供更好的视频内容理解和检索体验。