Researchers report breakthrough in "Distributed Deep Learning"

2019-12-13

When shopping online, we usually use several keywords to search for the products we want. However, in a world with millions of products and shoppers, matching these ambiguous terms with the right products is one of the biggest challenges in information retrieval.


Researchers from Rice University and Amazon have employed a method that utilizes the functionality of compressive sensing, demonstrating that they can save time and computational resources required for training computers for product search and similar "extreme classification problems."Tharun Medini, the author of the paper, concluded in tests using an Amazon search dataset containing around 70 million queries and over 49 million products: "Our training time is about 7-10 times faster, and our memory footprint is about half to a quarter of the best reported performance of large-scale distributed deep learning systems."


As users perform millions of online searches every day, tech companies like Amazon, Google, and Microsoft have a wealth of data on successful and unsuccessful searches. Using this data for a machine learning technique called deep learning is one of the most effective ways to provide better results to users.


Deep learning systems, or neural network models, are collections of numerous mathematical equations that take in a set of numbers called input vectors and transform them into another set of numbers called output vectors. The network consists of matrices with multiple parameters, and the latest distributed deep learning systems contain billions of parameters, which are divided into multiple layers. During training, data is fed into the first layer, vectors are transformed, and the output is fed into the next layer, and so on."Extreme classification problems" are parameters that can produce many results, hence there are many parameters. The deep learning models used for extreme classification are so large that they usually need to be trained on powerful supercomputers, which are sets of linked graphics processing units (GPUs) where the parameters are distributed and run in parallel, often taking several days.


Researchers described it through a thought experiment, dividing one billion products randomly into three storage buckets in two different worlds, meaning products can be placed in different buckets in each world. A trained classifier can assign searches to the buckets instead of assigning the products within them, meaning the classifier only needs to map the search to one of the three product categories.


He said, "Now, I give the search to the classifier in the first world, and it shows up as the third class, and then I give the search to the classifier in the second world, and it shows up as the first class. The commonality between the two buckets is the category the searcher is most likely interested in. If you look at the buckets, there are three worlds, one world has three buckets, so there are nine possibilities in total. So, I reduced the search space to one-ninth, and I only paid the cost of creating six buckets. By doing this, I paid the cost linearly and got an exponential improvement."In experiments on training an Amazon database, Medini and his colleagues randomly divided 49 million products into 10,000 classes or buckets and repeated it 32 times. This reduced the number of parameters in the model from approximately 100 billion to 6.4 billion. The distributed approach required less time and memory for training the model compared to models with similar parameters.


Medini said, "The most important feature of distributed deep learning is that it does not require communication between parallel processors. In the thought experiment, this represents the separation, the representation of independent worlds. They don't even have to talk to each other. In principle, you can train each of the 32 on one GPU, which is something that cannot be done with non-distributed methods."

RECENT AI NEWS

RECENT AI TOOLS