{"id":"arxiv:2011.01403","kind":"preprint","title":"Supervised Contrastive Learning for Pre-trained Language Model Fine-tuning","authors":["Beliz Gunel","Jingfei Du","Alexis Conneau","Veselin Stoyanov"],"author_count":4,"year":2020,"published":"2020-11-03","venue":null,"cited_by":296,"doi":null,"arxiv_id":"2011.01403","nct_id":null,"source_url":"http://arxiv.org/abs/2011.01403","archive_url":"https://space-kitty.com/papers/arxiv:2011.01403","updated":"2021-04-02","pmid":null,"pmcid":null,"openalex_id":"W3096565276","pdf_url":"https://arxiv.org/pdf/2011.01403","abstract":"State-of-the-art natural language understanding classification models follow two-stages: pre-training a large language model on an auxiliary task, and then fine-tuning the model on a task-specific labeled dataset using cross-entropy loss. However, the cross-entropy loss has several shortcomings that can lead to sub-optimal generalization and instability. Driven by the intuition that good generalization requires capturing the similarity between examples in one class and contrasting them with examples in other classes, we propose a supervised contrastive learning (SCL) objective for the fine-tuning stage. Combined with cross-entropy, our proposed SCL loss obtains significant improvements over a strong RoBERTa-Large baseline on multiple datasets of the GLUE benchmark in few-shot learning settings, without requiring specialized architecture, data augmentations, memory banks, or additional unsupervised data. Our proposed fine-tuning objective leads to models that are more robust to different levels of noise in the fine-tuning training data, and can generalize better to related tasks with limited labeled data.","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":null,"is_open_access":true,"peer_reviewed":false,"categories":["Topic Modeling"],"details":null,"provenance":"openalex+arxiv","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv.","topics":[{"slug":"llm-architectures","name":"LLM Architectures & Pretraining","domain_slug":"ai"}],"cited_in_dossiers":[]}