Microsoft InsightPilot: Empowering Efficient Data Exploration with LLM

Data exploration is an important step in data analysis, extracting key insights through filtering, sorting, grouping, and other steps. It helps reveal patterns in the dataset and uncover potential relationships between variables. However, this process is often interactive, requiring manual exploration of the data, making it time-consuming and requiring domain expertise.

Despite the existence of various tools for general data exploration, they often fail to consider user intent and dataset characteristics, resulting in low-relevance insights. Additionally, language models generating erroneous information is a common problem, leading to unreliable generated content. To address the limitations of existing models, researchers at Microsoft have introduced InsightPilot, a system that automates the data exploration process using a large language model (LLM). The system provides accurate insights to the LLM to avoid erroneous information and presents a compact abstraction of the dataset to reduce computational costs, allowing the LLM to better answer user queries.

InsightPilot consists of the following three components:

• A UI interface that allows users to ask questions in natural language and displays the analysis results.
• An LLM that facilitates data exploration by selecting appropriate analyses based on context.
• An insight engine that performs analysis and presents results in natural language.

Initially, users pose queries in the interface, and the insight engine generates preliminary insights. Based on the context, the LLM determines the most relevant insights and continues querying the engine for more details. For example, a user may inquire about the trend in students' science scores, and based on the preliminary insights, the LLM may prompt the engine for further analysis, such as comparing scores or identifying any anomalies. The interaction between the LLM and the engine continues as long as the exploration is not complete. At the end of the data exploration process, the engine presents a coherent report consisting of the top K insights, which is then displayed to the user through the interface.

To evaluate its performance, researchers conducted user studies to simulate real-world use cases of InsightPilot. Four data science participants were asked to pose three questions and evaluate the system based on metrics such as relevance, completeness, and comprehensibility. The results showed that InsightPilot consistently outperformed the OpenAI code interpreter and Langchain Pandas agent.

A performance evaluation was also conducted through a case study based on a car sales dataset. When inquiring about the overall trend in Toyota car sales, the system not only identified "Camry" as the main driver of Toyota sales but also compared the sales situation between Toyota and Honda, providing other interesting insights.

Despite the superior performance of InsightPilot compared to other state-of-the-art systems, it often produces ambiguous answers that require manual evaluation. Therefore, testing its effectiveness on different real-life datasets is crucial. Nonetheless, it is an effective approach to extract insights from datasets using natural language queries and is expected to simplify the exploratory data analysis process, saving time and effort. Further research is needed to ensure the deployment and efficiency improvement of this approach in real-world scenarios.