Google's New AI Research Paves the Way for Lightening the Burden of Large Language Models

2023-07-11

"Google Research Introduces PRP Paradigm for Better Text Ranking" Large language models (LLMs) such as GPT-3 and PaLM have gained widespread attention due to their impressive performance on various natural language tasks. However, they face challenges when it comes to solving text ranking problems. Existing methods often underperform compared to trained baseline rankers unless a new strategy relying on the GPT-4 system is adopted. In this article, we explore the groundbreaking research conducted by Google Research, which introduces a paradigm called "Pairwise Ranking Prompts" (PRP) to address these limitations and demonstrate the excellent performance of medium-sized open-source LLMs in ranking. "Understanding the Challenges of LLM in Ranking Tasks:" Despite their impressive language generation capabilities, large language models perform poorly on ranking tasks due to a lack of ranking awareness in their pre-training and fine-tuning techniques. While pairwise and listwise formulations have been employed, they require LLMs to generate calibrated prediction probabilities, which is a significant challenge. Even with listwise techniques, inconsistent and meaningless outputs are still observed. Additionally, changing the order of input documents can significantly decrease ranking metrics. "Introducing the Pairwise Ranking Prompts (PRP) Paradigm:" Google Research proposes the PRP paradigm to address the complexity and calibration issues associated with LLM ranking. PRP presents queries and a pair of documents as prompts in ranking tasks and significantly reduces the task complexity of LLMs by providing a default API for generation and scoring. The simple prompt architecture of PRP enables large language models to effectively understand ranking tasks. "Achieving State-of-the-Art Ranking Performance:" The Google Research team utilizes a medium-sized open-source LLM and evaluates PRP on traditional benchmark datasets. The results are groundbreaking, surpassing previous methods in the literature and even outperforming the black-box commercial GPT-4 with smaller model sizes. On TREC-DL2020, PRP based on the FLAN-UL2 model with 20 billion parameters outperforms the previous best method by over 5% in NDCG@1 metric. On TREC-DL2019, PRP performs better than solutions like InstructGPT with 175 billion parameters across various ranking metrics. Only in NDCG@5 and NDCG@10 metrics does PRP lag behind the GPT-4 solution. "Additional Advantages of PRP:" In addition to outstanding ranking performance, PRP offers several additional advantages. It supports LLM API for scoring and generation, allowing for flexible usage. PRP is insensitive to input order, addressing the issue of changing document order affecting ranking metrics. Furthermore, the research team demonstrates the efficiency of PRP by examining various efficiency enhancement methods while maintaining good empirical performance. "Summary:" Google Research's groundbreaking work on the Pairwise Ranking Prompts (PRP) paradigm for large language models has revolutionized the field of ranking tasks. By leveraging medium-sized open-source LLMs, PRP achieves state-of-the-art ranking performance, surpassing previous methods relying on black-box commercial and larger models. The simple and effective prompt architecture of PRP enables LLMs to understand and excel in ranking tasks. Additionally, PRP provides an LLM API for scoring and generation, making it a versatile solution. By reducing the burden on LLMs in ranking tasks, Google Research paves the way for the future development of natural language processing and ranking technologies. (Source: https://www.analyticsinsight.net/new-google-ai-research-paves-the-way-to-slash-llm-burden/)

RECENT AI NEWS

RECENT AI TOOLS