Recently, an innovative visual language training framework named CLIPS, developed collaboratively by researchers from UC Santa Cruz and the University of Edinburgh, has attracted significant attention in the artificial intelligence field. By leveraging synthetic captions and novel learning methodologies, this framework successfully addresses multiple challenges in visual language training, establishing new benchmarks for cross-modal retrieval tasks.
Web-scraped image-text datasets have long been essential resources for training visual language models. However, issues such as noise, low quality, and uncertain consistency between images and text within these datasets have limited model performance. To overcome these obstacles, researchers explored using synthetic captions generated by multimodal large language models (MLLMs) as replacements for original web-scraped captions. Nevertheless, current methods face high computational costs when processing complete captions and fail to fully utilize the information contained in synthetic captions.
The CLIPS framework was developed to tackle these challenges. It employs a partial synthetic caption contrastive learning strategy, sampling segments of synthetic captions to reduce input token length while enhancing performance. This approach not only improves retrieval accuracy but also significantly lowers computational costs. Additionally, CLIPS integrates an autoregressive caption generator that produces complete synthetic captions based on web-scraped captions and their corresponding images, ensuring comprehensive utilization of synthetic caption content and enriching the semantic alignment between images and text.
Technically, the CLIPS framework preprocesses synthetic captions using a sub-caption masking strategy and combines multiple positive contrastive losses to effectively align original and shortened captions. The generation framework utilizes an autoregressive decoder guided by specially designed combined masks to achieve optimal token interactions. The decoder's output aligns with the full synthetic captions, maintaining consistency with the generation loss function during training.
After training on large-scale datasets such as DataComp-1B and evaluating on benchmarks like MSCOCO and Flickr30K, the CLIPS framework demonstrated exceptional performance. On MSCOCO, CLIPS improved text-to-image retrieval accuracy by over 5% and image-to-text retrieval by more than 3%. Similarly, on Flickr30K, the model outperformed competing frameworks in both retrieval directions. Moreover, the scalability of CLIPS was highlighted, with smaller models trained using CLIPS surpassing larger models from competing methods.
Beyond retrieval tasks, integrating the CLIPS visual encoder significantly enhanced the effectiveness of multimodal large language models across various benchmarks. This achievement not only showcases the flexibility and adaptability of the CLIPS framework but also opens up new possibilities for multimodal applications in the AI field. Ablation studies further confirmed that the generative modeling approach ensures computational efficiency while delivering substantial improvements in alignment and retrieval metrics.
The successful introduction of the CLIPS framework marks a major breakthrough in the realm of visual language training. By overcoming the challenges faced by previous attempts and utilizing synthetic captions alongside innovative learning methods, CLIPS sets new standards for cross-modal retrieval tasks. This framework represents a pivotal step toward advancing multimodal applications in artificial intelligence, providing fresh insights and directions for future research and implementations.
