The chart title, which explains complex trends and patterns, is crucial for improving readers' understanding and memory of the presented data. For individuals with visual impairments, the information in the title is often their only way to comprehend the chart.
However, writing effective and detailed titles is a time-consuming and laborious process. While automated title techniques can alleviate this burden, they often fail to describe the cognitive features that provide additional context.
To assist people in writing high-quality chart titles, researchers at MIT have developed a dataset called VisText to enhance automated title systems. With this tool, researchers can teach machine learning models to adjust the complexity and content type of chart titles based on user needs.
The researchers found that machine learning models trained on their dataset consistently generated accurate, semantically rich titles that describe data trends and complex patterns. Quantitative and qualitative analyses showed that their model outperformed other automated title systems in adding titles to charts.
The team's goal is to provide this dataset, called VisText, to other researchers as a tool for addressing the problem of automated chart titling. These automated systems can help provide titles for online charts that do not have them and improve accessibility for individuals with visual impairments, said Angie Bogust, a graduate student in electrical engineering and computer science at MIT and a member of the CSAIL Visualization Group.
"We injected many human values into the dataset so that we and other researchers do not end up with models that do not meet people's expectations and needs when building automated chart title generation systems," she said.
The co-first authors of this article are Benny J. Tang and Arvind Satyanarayan, an associate professor of computer science at MIT and the head of the CSAIL Visualization Group. The research will be presented at the annual meeting of the Association for Computational Linguistics.
Human-centered analysis
The research team drew inspiration from the Visualization Group's previous work, which explored what makes a good chart title. In that study, researchers found that users with vision and those who are blind or have low vision have different preferences for the semantic complexity of titles.
The team aimed to incorporate this human-centered analysis into automated title research. To do so, they developed VisText, a dataset that includes charts and their corresponding titles, which can be used to train machine learning models to generate accurate, semantically rich, and customizable titles.
Developing an effective automated title system is not easy. Existing machine learning methods often attempt to add titles to charts in a similar way to processing images, but humans and models interpret natural images and charts differently. Other techniques completely bypass visual content and only use the basic data table of the chart for titling. However, this data table is often unavailable after the chart is published.
Considering the limitations of using images and data tables, VisText represents charts as scene graphs. Scene graphs can be extracted from chart images and contain all the chart data, along with additional visual context.
Tang explained, "Scene graphs are like the best of both worlds. They capture almost all the information in the image and are easier to extract from the image than from the data table. Additionally, as text, we can leverage advances in modern large-scale language models for generating titles automatically."
The researchers compiled a dataset with over 12,000 charts, each represented as a data table, image, and scene graph, accompanied by titles. Each chart had two independent titles: a low-level title describing the chart's construction, such as its axis range, and a high-level title describing statistical data, data relationships, and complex trends.
The researchers used an automated system to generate low-level titles and obtained high-level titles from AI workers.
"Our titles were inspired by two key prior studies: existing guidelines for describing visual media accessibility and a conceptual model we proposed for classifying semantic content. This ensured that our titles included low-level chart elements important to visually impaired readers, such as axes, scales, and units, while still allowing for variability that humans have when writing titles," Tang said.
Translating charts
The researchers trained five machine learning models for automated chart titling using VisText and collected chart images and titles. They wanted to understand how each representation (image, data table, and scene graph) and combinations of these representations affected title quality.
"You can think of the chart automated titling models as language translation models. But instead of translating German text into English, we're translating 'chart language' into English," Bogust said.
The results showed that models trained on scene graphs performed as well as or better than models trained on data tables. Since extracting scene graphs from existing charts is easier, the researchers believe they may be a more useful representation.
Additionally, they used low-level and high-level titles separately to train the models. Through a technique called semantic prefix conditioning, they taught the models to adjust the complexity of title content.
Furthermore, they conducted a qualitative evaluation of titles generated by the best-performing method and categorized six common errors. For example, if the model states that a trend is decreasing when it is actually increasing, it results in a direction error.
This fine-grained and robust qualitative evaluation is crucial for understanding the reasons behind model errors. For instance, using quantitative methods, direction errors may receive the same penalty as repetitive errors, where the model repeats the same words or phrases. However, direction errors may be more misleading than repetitive errors. Bogust said that qualitative analysis helped them understand these nuances.
These errors also revealed the limitations of the current models and raised ethical concerns that researchers must consider when developing automated title systems.
Generative machine learning models, such as ChatGPT, have been shown to produce illusions or provide incorrect information, which can mislead users. While using these models to add titles to existing charts has clear benefits, if the chart titles are incorrect, they may propagate misinformation.
"Perhaps this means that we shouldn't just use AI to add titles to all charts. Instead, maybe we offer these automated title systems to people as editing tools. Considering these ethical concerns throughout the research process, rather than just at the final deployment of the model, is important," Bogust said.
Bogust, Tang, and their colleagues hope to continue refining the models and reducing common errors. They also aim to expand the VisText dataset to include more complex charts, such as stacked bar charts or charts with multiple lines, and gain insights into how these automated title models actually learn from chart data.
Source: https://techxplore.com/news/2023-06-ai-captions.html
Alphabet Q1 paid subscribers reached 350M, fueled by YouTube & Google One. YouTube ad revenue slightly missed at $9.88B amid Premium growth. Total Q1 revenue hit $109.9B, beating forecasts.
Anthropic releases Claude connectors for Adobe, Blender, and Ableton, enabling AI workflow automation and app control. It also sponsors the Blender Development Foundation.
Apple introduces an App Store model splitting 12-month plans into monthly installments. On iOS 26.4+, it lowers upfront costs without reducing total price, adding billing tracking.
Google launches Aadhaar digital IDs in Wallet for India, enabling secure QR/NFC verification. A phased partner rollout targets key sectors, while passport-based credentials expand globally.
YouTube tests "Ask YouTube," an AI tool blending text and video for complex queries. Initially for US Premium users, it supports interactive follow-ups, expanding Google's AI search.
Google grants US DoD classified AI access, joining OpenAI and xAI in defense deals amid Anthropic's dispute. Contractual safeguards against surveillance and lethal weapons remain unenforceable.
Mistral AI launches Workflows, a Temporal-powered orchestration platform for secure, Python-driven enterprise AI automation featuring human-in-the-loop approvals.