Breakthrough Achievement: Meta AI Researchers Implement Large Language Model (LLM) for Code Optimization

Meta AI researchers are trying to apply large language models (LLMs) to code optimization tasks in order to make them perform code optimization like conventional compilers such as LLVM. The LLVM optimizer is very complex, written in the C++ programming language with over a million lines of code, including thousands of rules and algorithms.

They believe that LLMs cannot handle this complexity as they are typically used for tasks like language translation and code generation. Compiler optimization involves various types of thinking, mathematics, and the use of complex techniques, which they believe LLMs perform poorly in. However, the results are surprising according to their methodology.

The figure above shows an overview of the approach, displaying the model inputs (prompts) and outputs (answers) during training and inference. Prompts contain unoptimized code, while answers contain optimized pass lists, instruction counts, and optimized code. During inference, only the optimized pass list is generated and then inputted into the compiler to ensure the optimized code is correct.

Their approach is simple. They first selected a large language model (LLM) architecture with 7 billion parameters from LLaMa 2 and initialized it from scratch. Then, they trained the model using a massive dataset consisting of millions of LLVM assembly examples, each paired with the best compiler options determined through a search process and the corresponding optimized assembly code. Just with these examples, the model was able to optimize code with remarkable accuracy.

Their work's most significant contribution is the first application of LLMs to code optimization tasks. They created an LLM specifically tailored for compiler optimization, demonstrating that these models achieve a 3.0% reduction in code size in a single compilation compared to search-based methods. In contrast, the current state-of-the-art machine learning methods result in regressions and require thousands of compilations. The researchers also provide additional experiments and code examples to gain a more comprehensive understanding of the potential and limitations of LLMs in code inference. Overall, they found LLMs to be highly effective in this field and believe their research will generate widespread interest.