Performance of Large Language Models in Long-form Question Answering

Although large language models (LLMs) like ChatGPT and GPT-4 have shown better performance in multiple benchmark tests, open-source projects such as MMLU and OpenLLMBoard have also made rapid progress in multiple applications and benchmark tests. With the rapid development of new models and methods, it becomes more important to understand their capabilities, limitations, and differences as we enter a new era of LLMs. While LLMs have demonstrated their ability to generate coherent text in tasks such as summarization, it is still necessary to understand how they perform in long-form question answering (LFQA).

One major challenge that still needs to be addressed is LFQA, which has many important real-world applications such as supporting forums, troubleshooting, and customer service. Answering such questions often requires complex thinking skills to understand the question and comprehend the material scattered throughout the source documents. The key points of the articles are compressed into summaries. They assume that a better understanding of the topics scattered across the source material will be needed to generate follow-up questions from these summaries. Additionally, other researchers have shown that answering more than one-third of long-form material is often rated as "difficult" by people.

Salesforce researchers propose a scalable evaluation approach to compare and contrast giant LLMs with smaller but successful base LLMs (such as Llama-7B, 13B) and their distilled versions (such as Alpaca-7B, 13B). To do this, they suggest having ChatGPT explicitly guide the construction of complex questions from document summaries. Their empirical study shows that creating follow-up questions from summaries provides a challenging but more realistic setting for evaluating LLMs' reasoning abilities in two aspects: the complexity of generating questions and the quality of answers from open-source LLMs. They use GPT-4 to determine the performance of answer quality in terms of coherence, relevance, factuality, and correctness, as relying solely on human evaluation for long-form question answering is expensive and difficult to scale. They also conducted a small-scale human evaluation, which showed a strong correlation between GPT-4 and human evaluation, giving their evaluation credibility.

The main conclusions drawn from this study are as follows:

- They suggest inferring longer contexts from summary summaries by running the context multiple times, which takes more than 20% of the time.

- Distilled LLMs (Alpaca-7B, 13B) generally rely less on context when generating questions from the source material, but their ability to create questions from document summaries is significantly reduced.

- For questions generated from summaries (> 16.8%), the answers produced by distilled LLMs can remain consistent across different contexts, but they often deviate from the topic, produce redundant replies, and are only partially accurate.

- Alpaca-7B and 13B are more sensitive to longer contexts (> 1024 tokens) than basic LLMs (Llama), although they typically produce reasonable responses.