Google proposes a new metric for evaluating the quality of AI-generated audio and video

2019-10-23

With the development of artificial intelligence, there are numerous applications in various fields. Generating audio or video using artificial intelligence is not something new. There are many metrics for measuring the quality of these "AI masterpieces," but currently, there is no widely adopted one. For example, the Frechet Inception Distance (FID) can extract photos from the target distribution and the evaluated model, and use an artificial intelligence object recognition system to capture important features and identify similarities. Although it is one of the most popular image metrics, it is still not fully accepted by the public. To address this issue, Google developers have started their research. Currently, they have achieved two results - the Frechet Audio Distance (FAD) and the Frechet Video Distance (FVD), which can measure the overall quality of synthesized audio and video, respectively. The researchers claim that their achievements are different from peak signal-to-noise ratio, structural similarity index, or any other proposed metrics. FVD focuses mainly on the overall video quality, while FAD does not require any reference and can be used for all types of audio. Software engineers Kevin Kilgour and Thomas Unterthiner wrote in a blog post, "Reliable metrics for evaluating generative models are crucial for measuring progress in the audio and video understanding field, but currently, there is no such metric. Do the generated videos shown below look more realistic than other generated videos?" The answer is yes. In the FAD evaluation, the overlap between the distributions of two sets of audio samples (generated and real) decreases as the distortion increases, indicating lower quality of the synthesized samples. To evaluate whether the FAD and FVD metrics align with human aesthetics, the researchers conducted a large-scale study involving human evaluators. In the experiment, evaluators were tasked with examining 10,000 pairs of videos and 69,000 5-second audio clips. Specifically, they compared different video distortions and the impact on the same audio clip, and then used a model to rank the collected paired evaluation set. The final research conclusion is that the comparative results indicate that the new metrics align well with human aesthetics. Finally, Kilgour and Unterthiner stated, "We have made significant progress in generative AI models. FAD and FVD will help us maintain the measurability of this progress and are expected to guide us in improving the models for audio and video generation."

RECENT AI NEWS

RECENT AI TOOLS