# Multi-label Contrastive Predictive Coding

Preprint · Jiaming Song, Stefano Ermon

- Archive id: arxiv:2007.09852
- Published: 2020-07-20
- arXiv: https://arxiv.org/abs/2007.09852
- Citations: 30
- Source URL: https://arxiv.org/abs/2007.09852
- PDF: https://arxiv.org/pdf/2007.09852
- Provenance: arxiv

## Abstract

Variational mutual information (MI) estimators are widely used in unsupervised representation learning methods such as contrastive predictive coding (CPC). A lower bound on MI can be obtained from a multi-class classification problem, where a critic attempts to distinguish a positive sample drawn from the underlying joint distribution from $(m-1)$ negative samples drawn from a suitable proposal distribution. Using this approach, MI estimates are bounded above by $\\log m$, and could thus severely underestimate unless $m$ is very large. To overcome this limitation, we introduce a novel estimator based on a multi-label classification problem, where the critic needs to jointly identify multiple positive samples at the same time. We show that using the same amount of negative samples, multi-label CPC is able to exceed the $\\log m$ bound, while still being a valid lower bound of mutual information. We demonstrate that the proposed approach is able to lead to better mutual information estimation, gain empirical improvements in unsupervised representation learning, and beat a current state-of-the-art knowledge distillation method over 10 out of 13 tasks.

_Abstract reproduced under CC0-1.0 (arXiv descriptive metadata)._

Topics: [NeuroAI & Computational Neuroscience](https://space-kitty.com/topics/neuroai)

---
Space Kitty Archive · https://space-kitty.com · API: https://space-kitty.com/api/v1 · MCP: https://space-kitty.com/mcp · Premium: $0.20/request via x402 (Base Sepolia (testnet)). Metadata: arXiv (CC0), OpenAlex (CC0), Europe PMC (CC BY/CC0 works), ClinicalTrials.gov.