MIT and IBM developed an AI based on topic recommendation for documents.

2019-12-23

Even the best text parsing recommendation algorithms can be hindered by a certain size of dataset. In order to provide faster and better classification performance than existing methods, a team from MIT IBM Lab and MIT Geometric Data Processing Group has designed a technique that combines popular artificial intelligence tools such as embedding and optimal transport. They claim that this method can scan millions of possibilities and can consider either an individual's historical preferences or a group's preferences. "There is a wealth of text on the internet that can help us with our research," said the lead author of the study from MIT in a statement. To do this, the researchers categorized the algorithmic text collection into topics based on commonly used words in the set. Then, each text was divided into 5 to 15 most important topics and ranked the importance of each topic in the text. Embedding, which is a numerical representation of the data, in this case, words, helps to clarify the similarity between words, while optimal transport helps to calculate the most efficient way to move objects (or data points) between multiple destinations. Embedding enables the utilization of optimal transport twice: first by comparing topics in the set, and then by measuring the degree of overlap in common topics. The researchers claim that this method is particularly effective when scanning large volumes of books and documents. In an evaluation of 1720 pairs of books from the Gutenberg project dataset, the algorithm compared all the books in one second, which is more than 800 times faster than other methods. Furthermore, the algorithm performs better than competitors in document sorting, such as grouping books in the Gutenberg dataset by author or grouping product reviews on Amazon by department. It also provides a list of topics, allowing users to better understand why a given document is recommended, which is also more interpretable. The researchers will continue to develop an end-to-end training technique that jointly optimizes embedding, topic modeling, and optimal transport, instead of optimizing them separately as currently done. They also hope to apply their method to larger datasets and explore applications in image or 3D data modeling, which will be the main focus in the foreseeable future.

RECENT AI NEWS

RECENT AI TOOLS