The Golden Age of Natural Language Processing

2019-12-21

Natural Language Processing (NLP) has a history of several decades, but it lacks general real-world intelligence. Machines may excel in mathematics, but it is difficult for them to convert language problems involving speech and text into their preferred mathematical formulas. However, with the latest breakthroughs in NLP technology, NLP is entering a new "golden age," with projected total revenue expected to exceed $22 billion by 2025.


The first breakthrough in NLP technology in recent times came with the introduction of the Word2Vec algorithm in 2013. This algorithm automatically reads massive text datasets and understands the associations and relationship patterns between all the words in the dataset. Word2Vec does not attempt to "understand" the text, but rather seeks associations between word embeddings. It then compresses this information into smaller dimensions to create compact encodings of the vocabulary. This compression forces the formation of logical patterns expressing word relationships. The result is a mathematical vector for each word, expressing its relationship with the compression pattern. These vectors can be used to provide highly accurate results for various NLP tasks such as machine translation, question answering, and sentiment analysis. While Word2Vec has its weaknesses, such as representing words with multiple meanings like "pit" or "bat" with a single representation, it has made a significant leap in NLP accuracy.


The next advancement in NLP came in early 2018 with ELMo (Embeddings from Language Models). Unlike Word2Vec, which uses fixed vectors for each word, ELMo considers the surrounding sentences or paragraphs to calculate the vector for an individual word. ELMo then uses deep learning and multiple hidden layers to help capture word relationships. This allows ELMo to learn different levels of language representation. For example, an earlier layer may focus on basic information like distinguishing nouns from verbs, while a lower layer may focus on higher-level information like distinguishing "Janine" from "Jamie." As a result, ELMo outperforms Word2Vec in various NLP tasks.


In the latter half of 2018, NLP made even greater strides with the release of BERT (Bidirectional Encoder Representations from Transformers). BERT is based on the attention mechanism, which first observed how humans process information by focusing on important details and eliminating noise. Intuitively, in NLP problems like machine translation, given a training corpus of relevant sentences (e.g., translating source text into another language), the attention mechanism can learn to focus on the most relevant set of information in the source sentence.


NLP researchers were able to greatly enhance the performance of the attention mechanism by using multiple attention distributions, leading to the creation of an algorithm called Transformer. BERT took a crucial step forward for Transformers by introducing "masked language modeling," where 15% of the words in a sentence are randomly masked. Then, using a Transformer, predictions are generated for each masked word based on the surrounding unmasked words (both left and right). BERT appears to be just a minor adjustment to the Transformer, which itself is a minor adjustment to the attention mechanism. These small adjustments can greatly improve the quality of NLP outputs.


Currently, BERT is the darling of NLP research, providing results surpassing human-level accuracy for certain NLP tasks like question answering. However, NLP may not maintain this state for long. Here comes a fearless prediction: within a year, we will forget about BERT and a better algorithm called ERNIE will take its place. While the name may eventually prove to be incorrect, the principle remains: we are at the dawn of the NLP era, where better algorithms are being discovered seemingly on a monthly basis, with no end in sight. Many of these algorithms are not only published as research papers but also released to the public as open-source code that can be immediately used for commercial NLP products.


As consumers, you may have already noticed the tremendous leap in computers' ability to understand human speech. From being awakened by Alexa or Siri with the day's to-do list to chatbots answering our customer service calls, we are increasingly surrounded by NLP machines. Organizations that do not embrace NLP and leverage the technology to efficiently aggregate and gain actionable insights from data will find themselves at a disadvantage and left behind.

RECENT AI NEWS

RECENT AI TOOLS