RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMs

 


Background

Large language models (LLMs) typically utilize the top-k contexts from a retriever in retrieval-augmented generation (RAG). In this work, the authors propose a novel instruction fine-tuning framework RankRAG, which instruction-tunes a single LLM for the dual purpose of context ranking and answer generation in RAG.

What is RAG?

  • RAG is an AI framework for retrieving facts from an external knowledge base to ground large language models (LLMs) on the most accurate, up-to-date information and to give users insight into LLMs' generative process.
Generally, a dense embedding-based retriever first retrieves top-k-chunked contexts from a collection of documents or external databases for a given question. Then, LLM reads the top-k contexts to generate the answer.

Current Limitations of the RAG Pipeline
  1. Limited Capacity of Retriever - Current RAG systems usually employ sparse retrieval or moderate-size (e.g. BERT-based) embedding model as the retriever R, mainly due to efficiency consideration. These models encode questions and documents independently and calculate the similarity between questions and documents using vector similarity metrics. However, the limited capacity of embedding model and independent processing of query and documents constrain the ability to estimate textual relevance between question q and documents d, reducing their effectiveness in new tasks or domains

  1. Trade-off of Picking Top-k Contexts- Although the state-of-the-art long-context LLM can take many retrieved contexts as input for answer generation, the performance quickly saturates with increased k in practice. In general, a smaller k often fails to capture all relevant information, compromising the recall, given the limited expressibility of retriver. In contrast, a larger k improves recall but at the cost of introducing irrelevant content that hampers the LLM’s ability to generate accurate answers


OBJECTIVE (The goal of this research study): 

  • To design a RAG instruction tuning pipeline using a single language model to achieve both high-recall context extraction and high-quality content generation.


RankRAG ( Proposed solution)

  • They proposed RankRAG, which instruction-tunes a single LLM for both context ranking and answer generation in the RAG framework. Furthermore, RankRAG expands upon existing instruction-tuning data by incorporating context-rich QA, retrieval-augmented and ranking datasets, enhancing the LLM’s ability to filter out irrelevant contexts during both the retrieval and generation phases of RAG.


Approach/Method

This process involves 3 main steps:

  1. Supervised Fine-Tuning (SFT)

  2. Unified Instruction-Tuning for Ranking and Generation

  3. Inference


Supervised Fine-Tuning (SFT)- updating a pre-trained LM using labeled data to do a specific task.

SFT significantly improves the ability of LLMs to follow instructions, thus improving zero-shot results on various downstream tasks. RankRAG leverages SFT on a blend of high-quality instruction-following datasets to establish basic instruction-following capabilities. For each sample in the instruction-following dataset, we take the multi-turn conversational format, use the previous turns of conversation between the user and the assistant as the context, and compute the loss only at the last response from the assistant.


Unified Instruction-Tuning for Ranking and Generation

Stage-I’s SFT provides basic instruction-following capabilities, but LLMs still perform suboptimally on RAG tasks. RankRAG addresses this by instruction-tuning the LLM for both context ranking and answer generation. This involves:


  • SFT data from Stage I to maintain instruction-following capabilities.

  • Context-rich QA data to improve the LLM’s use of context for generation.

  • Retrieval-augmented QA data to enhance robustness against irrelevant contexts.

  • Context ranking data to empower LLMs with ranking capabilities.

  • Retrieval-augmented ranking data to train the LLM to determine the relevance of multiple contexts simultaneously, closer to real-world RAG scenarios.


Despite the variety of datasets and tasks described, they can all be cast into a standardized QA format (x, c, y), where x is the question, c is the corresponding context, and y is the target output answer. 


RankRAG Inference

As RankRAG incorporates an additional reranking step, the inference pipeline for each question is modified as a retrieve-rerank-generate pipeline, described as follows:

  1. The retriever R first retrieves top-N contexts from the corpus.

  2. The RankRAG model calculates the relevance scores between the question and the N contexts, then reranks contexts retaining only the top-k contexts.

  3. The top-k contexts, along with the question, are concatenated and fed back into the RankRAG model to generate the final answer.


Achievements

RankRAG outperforms existing RAG methods. With 8B scale, RankRAG consistently outperforms ChatQA-1.5 8B, one of the most recent open-sourced model with state-of-the-art performance. It remains competitive against models with significantly more parameters, such as InstructRetro, RA-DIT 65B, and Llama3-instruct 70B. RankRAG 70B further outperforms ChatQA-1.5 70B and other RAG baselines using InstructGPT. 


RankRAG demonstrates larger improvement on more challenging datasets. Performance gains of RankRAG over baselines are more pronounced for more challenging QA datasets.


Advantages of this Approach:

  1. RankRAG is Data-efficient.

  2. Performance v.s. Time-efficiency for RankRAG. While increasing the number of contexts for reranking from 20 to 100 improves accuracy by 5.9% to 9.1% across three tasks, it only incurs a 0.9× to 6.0× increase in time, which is significantly less than the expected 20× to 100× increase.


Link to the Research Paper: RankRAG


Comments

Popular Posts