{"id":"arxiv:2405.03190","kind":"preprint","title":"Adapting Dual-encoder Vision-language Models for Paraphrased Retrieval","authors":["Jiacheng Cheng","Hijung Valentina Shin","Nuno Vasconcelos","Bryan Russell","Fabian Caba Heilbron"],"author_count":5,"year":2024,"published":"2024-05-06","venue":null,"cited_by":null,"doi":null,"arxiv_id":"2405.03190","nct_id":null,"source_url":"https://arxiv.org/abs/2405.03190","archive_url":"https://space-kitty.com/papers/arxiv:2405.03190","updated":"2024-05-06","pmid":null,"pmcid":null,"openalex_id":null,"pdf_url":"https://arxiv.org/pdf/2405.03190","abstract":"In the recent years, the dual-encoder vision-language models (\\eg CLIP) have achieved remarkable text-to-image retrieval performance. However, we discover that these models usually results in very different retrievals for a pair of paraphrased queries. Such behavior might render the retrieval system less predictable and lead to user frustration. In this work, we consider the task of paraphrased text-to-image retrieval where a model aims to return similar results given a pair of paraphrased queries. To start with, we collect a dataset of paraphrased image descriptions to facilitate quantitative evaluation for this task. We then hypothesize that the undesired behavior of existing dual-encoder model is due to their text towers which are trained on image-sentence pairs and lack the ability to capture the semantic similarity between paraphrased queries. To improve on this, we investigate multiple strategies for training a dual-encoder model starting from a language model pretrained on a large text corpus. Compared to public dual-encoder models such as CLIP and OpenCLIP, the model trained with our best adaptation strategy achieves a significantly higher ranking similarity for paraphrased queries while maintaining similar zero-shot classification and retrieval accuracy.","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":null,"is_open_access":false,"retracted":false,"peer_reviewed":false,"categories":["cs.CV"],"details":null,"provenance":"arxiv","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv.","topics":[],"cited_in_dossiers":[]}