MIT researchers have developed a new benchmarking tool that accurately predicts the time it takes for a given code to execute on a computer chip, helping programmers optimize their code for better performance.
To make code run as fast as possible, developers and compilers (programs that convert programming languages into machine-readable code) typically use performance models that simulate the given chip architecture to run the code. Compilers use this information to automatically optimize the code, while developers use it to address performance bottlenecks in the microprocessor running the code. However, performance models for machine code are hand-written by relatively small expert teams and are not properly validated, resulting in simulated performance measurements that often deviate from actual results.
In a series of conference papers last year, researchers described a novel machine learning pipeline that automates this process, making it easier, faster, and more accurate. In a paper presented at the International Conference on Machine Learning in June, the researchers introduced Ithemal, a neural network model trained on labeled data in the form of "basic blocks" (abstracts of computational instructions) to automatically predict how long unseen basic blocks will take to execute on a given chip. The results showed that Ithemal outperformed traditional manually tuned models in terms of accuracy.
Then, at the IEEE International Symposium on Workload Characterization in November, the researchers showcased a benchmark suite of basic blocks from various domains, including machine learning, compilers, cryptography, and graphics, that can be used to validate performance models. They consolidated over 300,000 analyzed blocks into an open-source dataset called BHive. During evaluations, Ithemal predicted the speed of Intel chips running code better than Intel's own built-in performance models.
Ultimately, developers and compilers can use this tool to generate code that runs faster and more efficiently on increasingly diverse "black box" chip designs. "Modern computer processors are opaque, extremely complex, and difficult to understand. Writing computer code that performs as fast as possible on these processors is a huge challenge. This tool is an important step towards fully modeling the performance of these chips to improve efficiency," said Michael Carbin, the author of the paper.
Recently, the team proposed a new technique for automatically generating compiler optimizations in a paper submitted to the NeurIPS conference in December. Specifically, they generated an algorithm called Vemal that transforms certain code into vectors for parallel computation. Vemal outperformed manually vectorizing algorithms used in the widely-used LLVM compiler.
Learning from data
Carbin said that manually designing performance models can be "a kind of black magic." Intel provides over 3,000 pages of documentation describing the architecture of its chips. However, only a small fraction of experts currently build performance models to simulate the execution of code on these architectures. Intel's documentation is not incorrect but incomplete, and it omits certain details: it is proprietary. However, when you use data, you don't need to understand the documentation. If there's something hidden, you can learn directly from the data.
To do this, the researchers recorded the average number of cycles a given microprocessor takes to compute basic block instructions, essentially the order of startup, execution, and shutdown without any manual intervention. Automating the process allows for quick analysis of thousands of blocks.
Architecture-specific
During training, the Ithemal model analyzed millions of automatically profiled basic blocks to accurately understand how different chip architectures would execute computations. Importantly, Ithemal takes raw text as input and does not require manual feature engineering. In testing, previously unseen basic blocks and a given chip were provided to Ithemal, which generated a number indicating how fast the chip would execute the code.
The researchers found that Ithemal reduced the accuracy gap (the difference between predicted and actual speed) by 50% compared to traditional manual models. Furthermore, in their next paper, they demonstrated that Ithemal had an error rate of 10%, while Intel's performance prediction models had an error rate of 20% on various basic blocks from different domains.
The researchers said that the tool now makes it easier for people to quickly understand the performance speed of any new chip architecture. In specific domains of architecture, such as Google's new Tensor processing unit designed specifically for neural networks, which is currently under construction but not widely understood, if you want to train models on certain new architectures, you just need to collect more data from that architecture, run it through the profiler, use that information to train Ithemal, and you have a performance prediction model.
Next, the researchers are investigating methods to make the model interpretable. Machine learning is largely a black box, so it's not clear why specific models make predictions. Carbin said, "Our model says a processor needs 10 cycles to execute a basic block. Now, we're trying to figure out why. That's a good level of granularity for these types of tools."
References:
- "Ithemal: Accurate, Portable and Fast Basic Block Throughput Estimation" by Charith Mendis, Alex Renda, Saman Amarasinghe, and Michael Carbin.
- "BHive: A Benchmark Suite and Measurement Framework for Validating x86-64 Basic Block Performance Models" by Yishen Chen, Ajay Brahmakshatriya, Charith Mendis, Alex Renda, Eric Atkinson, Ondřej Sýkora, Saman Amarasinghe, and Michael Carbin.
- "Compiler Auto-Vectorization with Imitation Learning" by Charith Mendis, Cambridge Yang, Yewen Pu, Saman Amarasinghe, and Michael Carbin.
Alphabet Q1 paid subscribers reached 350M, fueled by YouTube & Google One. YouTube ad revenue slightly missed at $9.88B amid Premium growth. Total Q1 revenue hit $109.9B, beating forecasts.
Anthropic releases Claude connectors for Adobe, Blender, and Ableton, enabling AI workflow automation and app control. It also sponsors the Blender Development Foundation.
Apple introduces an App Store model splitting 12-month plans into monthly installments. On iOS 26.4+, it lowers upfront costs without reducing total price, adding billing tracking.
Google launches Aadhaar digital IDs in Wallet for India, enabling secure QR/NFC verification. A phased partner rollout targets key sectors, while passport-based credentials expand globally.
YouTube tests "Ask YouTube," an AI tool blending text and video for complex queries. Initially for US Premium users, it supports interactive follow-ups, expanding Google's AI search.
Google grants US DoD classified AI access, joining OpenAI and xAI in defense deals amid Anthropic's dispute. Contractual safeguards against surveillance and lethal weapons remain unenforceable.
Mistral AI launches Workflows, a Temporal-powered orchestration platform for secure, Python-driven enterprise AI automation featuring human-in-the-loop approvals.