"RAG is just a more advanced prompt engineering"

In the field of generative artificial intelligence, a new acronym appears every day. The latest buzzword is RAG, which stands for retrieval augmented generation. It is not just an acronym, but also represents a major breakthrough in the LLM field. But what exactly is it?

RAG has gained popularity because it combines the advantages of retrieval models and generation models. Essentially, RAG is the addition of a new database to the base model, allowing the model to retrieve new information from it and generate information.

This helps reduce the hallucination in the model. In most cases, this is a vector database, or in some cases, like GPT-4, it is the internet.

Dhruv Motwani, the founder and CEO of SpringtownAI, also discussed and demonstrated the use of RAG, exploring its architecture and functionality. Participants were able to deploy applications on their AWS accounts and test the model's hallucination, which was reduced when using RAG.

Is it really good?

"The most common standard RAG process we have currently completely lacks context about your data. Without consulting the data, it will send the query plus the original query to GPT," said Mark McQuade, co-founder of Arcee.ai.

McQuade and his team built an End2End RAG system called DALM on top of the main LLM. He said the best way to use it is to pair it with domain-specific models rather than larger models with unnecessary data.

Before delving into RAG, it is important to understand the choices AI developers make when using AI models. They can build models from scratch, fine-tune existing models, or adopt the retrieval augmented generation approach. Each method has its pros and cons, and the larger the model, the higher the likelihood of hallucination.

Building from scratch can be expensive and time-consuming. For example, OpenAI invested over $100 million to train its GPT-4 model. On the other hand, fine-tuning existing models with additional data is a viable option, but it carries the risk of "forgetting" parts of the original training data.

RAG combines retrieval models and generation models to provide context-aware answers. The retrieval model is used to access information from existing knowledge sources such as databases or online articles. Then, the generation model synthesizes this retrieved information into coherent, contextually relevant answers.

The key advantage of RAG is its ability to provide accurate and unique answers that resemble human language, rather than just summarizing the retrieved data.

Essentially, RAG is an advanced prompt engineering. Its focus is on keeping the model stable and optimizing its ability to answer specific questions by "filling" the text with contextual windows. This approach is particularly beneficial for prompt engineers who need to learn AI engineering skills and master a set of basic prompts to build RAG/Agent systems.

In more complex RAG/Agent systems, it is not just a single prompt; it involves a series of collaborative prompts to provide accurate and context-aware answers.

Extended RAG

Some researchers argue that RAG may not have as much advantage compared to longer contextual windows, as both provide similar results. A recent study titled "Retrieval meets Long Context Large Language Models" compared RAG with longer contextual windows LLM.

This paper found that open-source embedding models/retrievers outperformed the OpenAI model. Combining a simple RAG with a 4k LLM achieved the performance of a long-context LLM.

While RAG brings significant benefits, the possibility of adverse responses when retrieving information needs to be considered. Philipp Schmid, the Head of Technology at Hugging Face, questioned whether LLM can be taught to be more accurate and reliable and introduced Self-RAG. This is a novel approach for teaching the model when to retrieve information and how to use it effectively.

Self-RAG involves creating a "comment" dataset to determine when it is appropriate to retrieve information and which information is relevant. By using retrieval guidelines to create the "comment" dataset, developers can train a commenting model on synthetic datasets. Using prompts, the commenting model, and the retriever, developers can generate a RAG dataset.

After training LLM with the RAG dataset, which includes special markers to indicate when the model should retrieve or generate answers, the model adaptively generates special markers during inference to determine if retrieval is needed.

It seems that hallucination will continue for some time. But just like prompt engineering, now is the time to learn the next AI engineering skill and catch up with the buzz, this time for building a set of basic prompts for RAG systems.