Large language models (LLMs) like GPT-4 have achieved tremendous success worldwide with their remarkable natural language understanding capabilities. However, the most significant long-term opportunity for LLMs will involve a completely different type of language: the language of biology.
Through advancements in fields such as biochemistry, molecular biology, and genetics over the past century, a notable theme has emerged: biology is an interpretable, programmable, and even digital system in some respects.
DNA encodes the complete genetic instructions for every organism on Earth using just four variables (adenine, cytosine, guanine, and thymine). In comparison, modern computing systems use two variables (0 and 1) to encode global digital electronic information. One system is binary, and the other is quaternary, but these two systems have surprisingly overlapping concepts; both can be correctly viewed as digital systems.
Another example is that each protein in every organism is composed of a one-dimensional chain of amino acids connected in a specific order. Proteins can range in length from tens to thousands of amino acids, and there are 20 different amino acids to choose from.
This represents an extremely computable system, which is also a system suitable for language models to learn.
As Demis Hassabis, CEO and co-founder of DeepMind, said, "At its most fundamental level, I think biology can be seen as an information processing system, albeit a very complex and dynamic one. Just as mathematics has been shown to be the correct descriptive language of physics, biology may prove to be the perfect mechanism for applying artificial intelligence."
When large language models can fully leverage vast amounts of data filled with signals, they unleash their most powerful capabilities, inferring potential patterns and deep structures beyond the reach of human abilities. They can then generate novel and astonishingly complex outputs based on a fine understanding of the subject.
For example, tools like ChatGPT have learned to engage in thoughtful and nuanced conversations on any imaginable topic by ingesting all the text available on the internet. Text-to-image models like Midjourney have learned to generate creative original images by ingesting billions of images.
Applying large language models to biological data, enabling them to learn the language of life, will make natural language and images seem trivial.
Specifically, what will this look like?
In the near term, the most attractive opportunity for applying large language models in the life sciences field is designing novel proteins.
Proteins are the core of life itself. As renowned biologist Arthur Lesk said, "In the molecular drama of life on the scale of molecules, proteins are where the action is."
Proteins are involved in almost every important activity that occurs within every organism: digesting food, muscle contraction, oxygen transport throughout the body, and attacking foreign viruses. Your hormones are made up of proteins, as is your hair.
Proteins are so important because they are incredibly versatile. They can assume a wide range of structures and functions, far surpassing any other type of biological molecule. This incredible versatility is a direct result of how proteins are built.
As mentioned earlier, each protein is composed of building blocks called amino acids, which are strung together in a specific order. Based on this one-dimensional amino acid sequence, proteins fold into complex three-dimensional structures, enabling them to perform their biological functions.
The shape of a protein is closely related to its function. For example, antibody proteins fold into specific shapes that allow them to precisely recognize and attack foreign bodies, much like a key fitting into a lock. Enzymes, on the other hand, are proteins that can accelerate biochemical reactions. Their shapes are specifically designed to bind with particular molecules, catalyzing specific reactions. Therefore, understanding the shape that proteins fold into is crucial for understanding the function of organisms and ultimately understanding the workings of life itself.
Determining the three-dimensional structure of a protein based solely on its one-dimensional amino acid sequence has been a challenge in the field of biology for over half a century. Known as the "protein folding problem," it has plagued generations of scientists. In 2007, a commentator described the protein folding problem as "one of the most important unsolved problems in modern science."
In a groundbreaking moment in late 2020, an artificial intelligence system called AlphaFold, developed by DeepMind, solved the protein folding problem. AlphaFold accurately predicted the three-dimensional structure of proteins with an accuracy within one atomic width, surpassing any other method developed by humans before.
The significance of AlphaFold is indescribable. John Moult, an expert in protein folding who has been involved in the field for a long time, summed it up well: "This is the first time a serious scientific problem has been solved by artificial intelligence."
However, when it comes to artificial intelligence and proteins, AlphaFold is just the beginning.
AlphaFold does not rely on large language models. It relies on an earlier bioinformatics approach called multiple sequence alignment (MSA), which infers protein structure by comparing a protein sequence to evolutionarily similar proteins.
As AlphaFold clearly demonstrated, MSA is powerful but also has its limitations.
Firstly, it is slow and computationally intensive because it requires referencing many different protein sequences to determine the structure of any given protein. More importantly, MSA has limited effectiveness for so-called "orphan proteins" – proteins that lack close relatives. These orphan proteins make up approximately 20% of known protein sequences.
Recently, researchers have started exploring an intriguing alternative approach: using large language models to predict protein structure instead of relying on multiple sequence alignment.
Protein language models – LLMs trained on protein sequences rather than English words – have shown remarkable abilities to discern complex patterns and relationships between protein sequences, structures, and functions. For example, they can learn how changing specific amino acids at specific positions in a protein sequence will affect the shape it folds into. Protein language models can learn the grammar or linguistics of proteins.
The concept of protein language models can be traced back to the UniRep work by the Church Lab at Harvard University in 2019 (although UniRep used LSTM instead of the state-of-the-art Transformer models used today).
At the end of 2022, Meta released ESM-2 and ESMFold, which are among the largest and most complex protein language models to date, with 15 billion parameters (ESM-2 is the LLM itself, and ESMFold is the associated structure prediction tool).
ESM-2/ESMFold achieves nearly the same accuracy as AlphaFold in predicting protein structures. However, unlike AlphaFold, it can generate structures based solely on a single protein sequence without requiring any structural information. As a result, it is 60 times faster than AlphaFold. This speed advantage has a significant impact when researchers need to screen millions of protein sequences in protein engineering workflows. ESMFold also provides more accurate structure predictions for orphan proteins lacking evolutionary relatives compared to AlphaFold.
The ability of language models to extrapolate understanding of the "latent space" of proteins brings exciting possibilities to protein science.
However, since AlphaFold, this field has seen even more powerful conceptual breakthroughs.
In essence, these protein models can be disrupted: instead of predicting protein structure based on a sequence, models like ESM-2 can generate entirely novel protein sequences that do not exist in nature based on desired properties.
Today, all the proteins that exist in the world represent only a tiny fraction of the theoretically possible proteins. This is where the opportunity lies.
Rough data estimates suggest that the total collection of proteins in the human body – the so-called "human proteome" – is estimated to be between 80,000 and 400,000. In comparison, the number of theoretically possible proteins is on the order of 10^1300, an unimaginably large number that far exceeds the number of atoms in the universe. (It should be noted that not all of these 10^1300 possible combinations of amino acids will generate biologically viable proteins, but some subsets may.)
Over millions of years, the long process of evolution has stumbled upon tens of thousands or hundreds of thousands of viable combinations by chance. But this is just the tip of the iceberg.
As Molly Gibson, co-founder of Generate Biomedicines, a leading AI-driven protein startup, said, "The amount of sequence space sampled by nature in the history of life is roughly equivalent to a drop of water in all the Earth's oceans."
We have the opportunity to improve upon nature. After all, while evolution is a powerful force, it is not omniscient; it does not plan ahead; it does not reason or optimize from the top down. It is random and serendipitous, propagating those combinations that happen to be effective.
With artificial intelligence, we now have the ability to systematically and comprehensively explore the vast unknown territory of protein space to design completely novel proteins that have never existed in nature, tailored to our medical and commercial needs.
We will be able to design novel protein therapeutics for treating a wide range of human diseases, from infectious diseases to cancer. They will help enable gene editing. They will revolutionize materials science. They will enhance agricultural yields. They will purify pollutants in the environment. And many other applications that we cannot even imagine now.
Some early attempts at using deep learning for de novo protein design did not utilize large language models. A notable example is ProteinMPNN, which comes from the renowned lab of David Baker at the University of Washington. The ProteinMPNN architecture primarily relies on protein structure data to generate novel proteins, without using large language models.
Baker's lab recently published a more advanced and general protein design model called RFdiffusion. As the name suggests, RFdiffusion is built using a diffusion model, similar to the AI techniques driving text-to-image models like Midjourney and Stable Diffusion. RFdiffusion can generate novel and customizable protein "scaffolds" – the overall structural framework of proteins – and then stack sequences on top of them.
Structure-centric models like ProteinMPNN and RFdiffusion have achieved impressive accomplishments in the field of AI-driven protein design, driving the latest advancements in the field. However, due to the transformative capabilities of large language models, we may be on the verge of a new major revolution in this field.
Why are language models so promising compared to other computational methods for protein design? One key reason: scalability.
One of the key driving forces behind the recent tremendous progress in artificial intelligence is the so-called "scaling laws": continuously increasing the number of parameters, training data, and computational power of LLMs leads to incredible performance improvements.
With each order of magnitude increase in scale, language models have demonstrated extraordinary, unexpected, and emerging capabilities that go beyond what is achievable at smaller scales.
OpenAI's commitment to the scaling principle is more important than anything else, making the organization at the forefront of the field of artificial intelligence in recent years. With their progression from GPT-2 to GPT-3 and now GPT-4 and beyond, OpenAI has built larger models, deployed more compute, and trained on larger datasets than any other organization in the world, unleashing astonishing and unprecedented AI capabilities.
What does the scaling law have to do with proteins?
As scientific breakthroughs have made DNA sequencing cheaper and more accessible over the past two decades, the amount of DNA and protein sequence data available for training AI models has grown exponentially, far surpassing protein structure data.
Protein sequence data can be tokenized and treated in every sense as text data; after all, they are linear sequences of amino acids in a specific order, much like words in a sentence. Large language models can be trained solely on protein sequences, developing a fine understanding of protein structure and biology.
Therefore, efforts to scale up using LLMs hold great potential and may yield surprising new insights and capabilities in the field of protein science.
The first work that used Transformer-based LLMs to design entirely new proteins was ProGen, released by Salesforce Research in 2020. The initial ProGen model had 1.2 billion parameters.
Ali Madani, the lead researcher of ProGen, later founded a startup called Profluent Bio, aiming to advance and commercialize the latest technologies in LLM-based protein design.
While Madani pioneered the application of LLMs in protein design, he also recognized that using off-the-shelf language models trained solely on raw protein sequences is not the most powerful way to tackle this challenge. Integration of structure and function data is essential.
"The most significant breakthroughs in protein design will come at the intersection of carefully curated and flexibly learned multimodal modeling with diverse data," said Madani. He added, "This means leveraging all the high-signal data we have at our disposal, including protein structure and functional information derived from the laboratory."
Another interesting startup, Nabla Bio, applies LLMs to design novel protein therapeutics. Born out of George Church's lab at Harvard University and led by the UniRep team, Nabla Bio focuses on antibodies. Considering that 60% of protein therapeutics are antibodies, and the two best-selling drugs in the world are also antibody therapeutics, this comes as no surprise.
Nabla has decided not to develop its own therapeutics but rather provide its advanced technology to biopharmaceutical partners as a tool to aid them in developing their own drugs.
As the world gradually realizes that protein design represents a vast and still largely unexplored field, the seemingly magical capabilities of large language models are expected to drive more startup activity in this field in the coming months and years.
The road ahead may not be easy in the AI-driven, particularly LLM-driven, protein design field. There are still significant scientific, engineering, clinical, and commercial hurdles. Bringing these novel therapeutics and products to market will take years.
However, in the long run, few applications of AI have more promise in the market than this.
Let us look to the future. Designing entirely new proteins is not the only exciting opportunity for large language models in the life sciences.
Language models can be used to generate other categories of biomolecules, particularly nucleic acids. For example, a hot startup called Inceptive is applying LLMs to generate novel RNA therapeutics.
Other teams have broader pursuits, aiming to build generalized "biological foundation models" that can integrate diverse data types, including genomics, protein sequences, cellular structures, epigenetic states, cell imaging, mass spectrometry, spatial transcriptomics, and more.
The ultimate goal is to move from modeling individual molecules like proteins to modeling the interactions between proteins and other molecules, and then to modeling entire cells, tissues, organs, and even whole organisms.
Building an artificial intelligence system capable of understanding and designing every intricate detail of complex biological systems is awe-inspiring. But in the future, it will be within our grasp.
The 20th century was defined by fundamental advancements in physics: from Albert Einstein's theory of relativity to the discovery of quantum mechanics, from nuclear bombs to transistors. As many modern observers have pointed out, the 21st century is becoming the century of biology. Artificial intelligence and large language models will play a central role in uncovering the secrets of biology and unleashing its potential in the coming decades.
Source: https://www.forbes.com/sites/robtoews/2023/07/16/the-next-frontier-for-large-language-models-is-biology/?sh=7646f1686f05
Alphabet Q1 paid subscribers reached 350M, fueled by YouTube & Google One. YouTube ad revenue slightly missed at $9.88B amid Premium growth. Total Q1 revenue hit $109.9B, beating forecasts.
Anthropic releases Claude connectors for Adobe, Blender, and Ableton, enabling AI workflow automation and app control. It also sponsors the Blender Development Foundation.
Apple introduces an App Store model splitting 12-month plans into monthly installments. On iOS 26.4+, it lowers upfront costs without reducing total price, adding billing tracking.
Google launches Aadhaar digital IDs in Wallet for India, enabling secure QR/NFC verification. A phased partner rollout targets key sectors, while passport-based credentials expand globally.
YouTube tests "Ask YouTube," an AI tool blending text and video for complex queries. Initially for US Premium users, it supports interactive follow-ups, expanding Google's AI search.
Google grants US DoD classified AI access, joining OpenAI and xAI in defense deals amid Anthropic's dispute. Contractual safeguards against surveillance and lethal weapons remain unenforceable.
Mistral AI launches Workflows, a Temporal-powered orchestration platform for secure, Python-driven enterprise AI automation featuring human-in-the-loop approvals.