Meta AI and Samsung introduce two new artificial intelligence approaches: Prodigy and Resetting

2023-06-26

Meta AI and Samsung researchers have introduced two new artificial intelligence methods, Prodigy and Resetting, for learning rate adaptation, improving the adaptation rate of the state-of-the-art D-Adaptation method. Modern machine learning heavily relies on optimization to provide effective solutions in computer vision, natural language processing, and reinforcement learning. The difficulty of achieving fast convergence and high-quality solutions largely depends on the choice of learning rate. Learning rate adjustment becomes more challenging in applications with multiple agents and their respective optimizers. While some manually tuned optimizers perform well, these methods often require specialized skills and are time-consuming. Therefore, "parameter-free" adaptive learning rate methods, such as D-Adaptation, have become increasingly popular in recent years. The research teams from Samsung AI Center and Meta AI have made two unique improvements to the D-Adaptation method, called Prodigy and Resetting, to enhance the non-asymptotic convergence rate of the D-Adaptation method, resulting in faster convergence speed and better optimization output. The authors have made two novel improvements to the original method to improve the non-asymptotic convergence rate of the D-Adaptation method. By adjusting the adaptive learning rate method, they have improved the algorithm's convergence speed and solution quality performance. To validate the proposed improvements, they established a lower bound on any method adjusting to a constant D distance from the solution. They further demonstrated that the enhanced method is the worst-case optimal constant factor compared to other methods with exponentially bounded iterative growth. Subsequently, extensive experiments were conducted to demonstrate that the enhanced D-Adaptation method can quickly adjust the learning rate, resulting in better convergence speed and optimization results. The innovative strategy of this team lies in adjusting the error term of the D-Adaptation method using a step similar to Adagrad. The researchers can now confidently take larger steps while keeping the main error term unchanged, thereby accelerating the convergence speed of the improved method. When the denominator in the step grows too large, the algorithm slows down. Therefore, they have added weights next to the gradient as a precaution. The researchers used the proposed techniques in empirical investigations to address convex logistic regression and other challenging learning tasks. In multiple studies, Prodigy demonstrated faster adaptability than any known method; D-Adaptation with Resetting achieved the same theoretical speed as Prodigy while using simpler theory than Prodigy or D-Adaptation. Additionally, the proposed method generally outperforms the D-Adaptation algorithm and can achieve test accuracy comparable to manually tuned Adam. The recently proposed two methods surpass the state-of-the-art D-Adaptation method in learning rate adaptation. Extensive empirical evidence shows that the weighted D-Adaptation variant, Prodigy, is more adaptive than existing methods. The second method, D-Adaptation with Resetting, achieves the same speed as Prodigy with simpler theory. Source: https://www.marktechpost.com/2023/06/25/researchers-from-meta-ai-and-samsung-introduce-two-new-ai-methods-prodigy-and-resetting-for-learning-rate-adaptation-that-improve-upon-the-adaptation-rate-of-the-state-of-the-art-d-adaptation-method/

RECENT AI NEWS

RECENT AI TOOLS