2 minutes

27. Integrate RAG to LLM Request

Date: 2025-03-22

Status

Accepted

Context

We want our application relying on a Large Language Model (LLM) to answer user queries based on information stored in C4 diagrams within our CosmosDB vector database. Currently, the LLM’s responses are limited to its pre-trained knowledge, which may not be up-to-date or specific to our C4 diagrams. To enhance the accuracy and relevance of the LLM’s responses, we need to integrate a Retrieval Augmented Generation (RAG) system. This system will retrieve relevant information from the CosmosDB Vector Database based on the user’s query and provide it as context to the LLM.

Decision

We will implement a RAG pipeline using LangChain to retrieve relevant C4 diagram information from our CosmosDB Vector Database and augment the LLM’s input prompt. This will involve:

  • Query Embedding Generation: Generating embeddings for user queries using a compatible embedding model (e.g., Gemini Embeddings).
  • Vector Similarity Search: Performing vector similarity searches in CosmosDB to retrieve the most relevant C4 diagram chunks based on the query embeddings.
  • Context Augmentation: Constructing a prompt that includes the retrieved C4 diagram information as context for the LLM. LLM Invocation: Invoking the LLM (e.g., Gemini Flash) with the context-augmented prompt to generate a response.

Consequences

  • Improved Accuracy and Relevance: The LLM’s responses will be more accurate and relevant to the user’s queries, as they will be grounded in the specific information stored in our CosmosDB Vector Database.
  • Reduced Hallucinations: RAG will reduce the likelihood of the LLM generating inaccurate or fabricated information.
  • Increased Complexity: The introduction of RAG adds complexity to our application, requiring the management of embedding models, vector similarity searches, and prompt engineering.
  • Increased Latency: The RAG pipeline may introduce latency due to the embedding generation and vector similarity search steps. We need to optimize these steps to minimize latency.
  • Cost Implications: Vector similarity searches in CosmosDB consume Request Units (RUs), which can increase costs. We need to implement efficient query strategies and limit the number of retrieved documents to minimize costs.
  • Dependency on LangChain: We will introduce a dependency on the LangChain library, which necessitates careful version management and potential updates.
  • Data Consistency: We must ensure consistency between the data stored in CosmosDB and the embeddings generated for it. Changes to the C4 diagrams require corresponding updates to the embeddings.
  • Maintainability: The RAG pipeline requires careful documentation and maintenance to ensure its long-term stability and reliability.
  • Error Handling: Robust error handling must be implemented to manage potential failures during embedding generation, vector search, or LLM invocation.