Google's AI Language Model Can Process Entire Novels

2020-01-17

Sequence data, whether it is language, music, speech, or video, is not easy for artificial intelligence and machine learning models to understand - especially when it relies on a wide surrounding context. For example, if a person or object disappears from a video and reappears after a long period of time, many algorithms will forget its appearance. Google researchers have started using Transformers to address this issue. Transformers are architectures that can scale to thousands of words and significantly improve performance in tasks such as song synthesis, image synthesis, sentence-by-sentence translation, and document summarization. However, the current Transformer is not perfect and has limitations when scaled to larger contexts. Applications that use large windows require memory ranging from gigabytes to terabytes, which means the model can only read a few segments of text or generate a small piece of music. Therefore, Google has introduced Reformer as an evolution of the Transformer, aiming to handle context windows of up to one million words. By efficiently utilizing memory and reducing complexity on long sequences through techniques like locality-sensitive hashing and reversible residual layers, Reformer can run on a single AI accelerator chip with only 16GB of memory. Like all deep neural networks, Transformers consist of neurons arranged in interconnected layers that transmit signals from input data and slowly adjust the synaptic strength (weights) of each connection. All AI models extract features and learn to make predictions in this way. However, what sets Transformers apart is that each output element is connected to every input element. In fact, the weights between them are dynamically computed. Reformer then calculates hash functions to match similar vectors instead of searching through all possible vector pairs. When assigning hash values, the sequence is rearranged, grouping elements with the same hash value together and dividing them into segments to support parallel processing. Attention is focused on shorter segments and their adjacent segments, greatly reducing the computational load. Reformer also recomputes the input of each layer as needed instead of storing it in memory, thanks to reversible memory. The research team conducted experiments using these models to generate missing details in images and process the novel "Crime and Punishment". The results showed that this method can generate full-frame images pixel by pixel and obtain text of novel length in a single training round.

RECENT AI NEWS

RECENT AI TOOLS