Creating Custom Programming Languages for Efficient Visual Artificial Intelligence Systems

A photo can reveal the creator's worldview - their interest and emotion towards a certain subject or scene. But what about the creators behind the technologies that make these images possible?

Jonathan Ragan-Kelley, an associate professor in the Department of Electrical Engineering and Computer Science at MIT, is one of them. He has designed various tools, from movie visual effects to the widely used Halide programming language for photo editing and processing. As a researcher at the MIT-IBM Watson AI Lab and the Computer Science and Artificial Intelligence Laboratory, Ragan-Kelley focuses on high-performance, domain-specific programming languages and machine learning, enabling 2D and 3D graphics, visual effects, and computational photography.

Ragan-Kelley said, "Our main motivation is to develop new programming languages that make programming easier while ensuring efficient execution on increasingly complex hardware. If we want to enhance computational capabilities in practical applications - whether it's graphics, visual computing, or AI - we need to change the way we program."

Finding the Balance

In the past two decades, chip designers and programming engineers have witnessed the slowdown of Moore's Law and the shift from general-purpose computing on CPUs to more diverse and specialized computing units such as GPUs and accelerators. This shift brings trade-offs: the ability to run general code on CPUs is relatively slow, while faster and more efficient hardware requires extensive adaptation and mapping of code, along with custom programs and compilers. Improved programming with new hardware better supports various applications, such as high-bandwidth cellular radio interfaces, decoding highly compressed videos for streaming, and graphics and video processing on power-constrained smartphone cameras.

Ragan-Kelley said, "Our work is primarily aimed at unlocking the potential of the best hardware we can build, providing as much computational performance and efficiency as possible for these types of applications, which traditional programming languages cannot achieve."

To achieve this goal, Ragan-Kelley's work is divided into two directions. First, he abandons generality and focuses on capturing the structure of specific and important computational problems, using this to improve computational efficiency. This can be seen in Halide, an image processing language he co-developed, which has revolutionized the image editing industry in programs like Photoshop. Additionally, due to its specialization in fast processing of dense, regular numeric arrays (tensors), it is also suitable for neural network computations. The second focus is automation, particularly how compilers map programs to hardware. He utilizes the Exo language developed by his team in a project at the MIT-IBM Watson AI Lab.

Researchers have been striving for compiler-automated coding for years, but compilers can be a black box, still requiring performance engineers for explicit control and tuning. Ragan-Kelley and his team are developing an approach to balance the trade-offs between various techniques to achieve efficient and resource-efficient programming. The core of many high-performance programs, such as video game engines or smartphone camera processing, is optimized manually by human experts using low-level, detailed languages like C, C++, and assembly. Here, engineers make specific choices on how the program runs on the hardware.

Ragan-Kelley pointed out that programmers can choose between "very time-consuming, inefficient, and unsafe low-level code," which may introduce errors, or "safer, more efficient, higher-level programming interfaces," but these interfaces lack the ability to fine-tune in the compiler and often provide lower performance. Therefore, his team is trying to find a balance. Ragan-Kelley said, "We are trying to figure out how to provide control for key issues that human performance engineers want to control, so we are trying to build a new class of languages called user-schedulable languages, which provide safer, higher-level control knobs to control compiler operations or control program optimization."

Unlocking Hardware: Advanced and Underserved Approaches

Ragan-Kelley and his research group address this problem through two lines of work: applying machine learning and modern AI techniques to automatically generate optimized scheduling, which is an interface to the compiler for better compiler performance, and using an "external compilation" approach he is working on in the lab. He describes this approach as a way to "flip the inside of the compiler," having a compiler framework for human guidance and customization. Additionally, his team can add their custom schedulers on top, which can help with specialized hardware like IBM Research's machine learning accelerator. The scope of this work is broad: computer vision, object recognition, speech synthesis, image synthesis, speech recognition, text generation (large language models), and more.

He further advances this work through a large project in the lab. Under his guidance, lab intern William Brandon and lab research scientist Rameswar Panda are rethinking large language models (LLMs), seeking ways to slightly change the programming architecture of computation and models to make transformer-based models run more efficiently on AI hardware without sacrificing accuracy. He said their work deviates from standard thinking in important aspects and could bring significant returns in terms of cost reduction, improved capabilities, and/or downsizing LLM to reduce memory requirements and run on smaller computers.

It is this more forward-thinking approach to computational efficiency and hardware that Ragan-Kelley excels at and sees value in, especially in the long run. "I think there are some research areas that need to be pursued, but they are already mature or obvious or sufficiently aligned with common sense that many people have pursued or will pursue them," he said. "We are trying to find things that have both a large lever to actually impact the world and are underserved by other communities relative to their potential."

His course, 6.106 (Software Performance Engineering), currently being taught, embodies this idea. About 15 years ago, as devices transitioned from single processors to multi-processors, many academic projects began focusing on teaching the concept of parallelism. However, as Ragan-Kelley explains, MIT realized that students not only needed to grasp the knowledge of parallelism but also needed a deep understanding of memory optimization and the importance of leveraging dedicated hardware for optimal performance.

"By changing our programming approach, we can unlock the computational potential of new machines, allowing developers to continue rapidly creating new applications and ideas, fully utilizing increasingly complex and challenging hardware."