{"id":"arxiv:2407.01126","kind":"preprint","title":"Investigating the potential of Sparse Mixtures-of-Experts for multi-domain neural machine translation","authors":["Nadezhda Chirkova","Vassilina Nikoulina","Jean-Luc Meunier","Alexandre Bérard"],"author_count":4,"year":2024,"published":"2024-07-01","venue":null,"cited_by":0,"doi":null,"arxiv_id":"2407.01126","nct_id":null,"source_url":"https://arxiv.org/abs/2407.01126","archive_url":"https://space-kitty.com/papers/arxiv:2407.01126","updated":"2024-07-01","pmid":null,"pmcid":null,"openalex_id":"W4400341941","pdf_url":"https://arxiv.org/pdf/2407.01126","abstract":"We focus on multi-domain Neural Machine Translation, with the goal of developing efficient models which can handle data from various domains seen during training and are robust to domains unseen during training. We hypothesize that Sparse Mixture-of-Experts (SMoE) models are a good fit for this task, as they enable efficient model scaling, which helps to accommodate a variety of multi-domain data, and allow flexible sharing of parameters between domains, potentially enabling knowledge transfer between similar domains and limiting negative transfer. We conduct a series of experiments aimed at validating the utility of SMoE for the multi-domain scenario, and find that a straightforward width scaling of Transformer is a simpler and surprisingly more efficient approach in practice, and reaches the same performance level as SMoE. We also search for a better recipe for robustness of multi-domain systems, highlighting the importance of mixing-in a generic domain, i.e. Paracrawl, and introducing a simple technique, domain randomization.","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":null,"is_open_access":false,"retracted":false,"peer_reviewed":false,"categories":["cs.CL","cs.AI"],"details":null,"provenance":"arxiv","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv.","topics":[{"slug":"llm-architectures","name":"LLM Architectures & Pretraining","domain_slug":"ai"}],"cited_in_dossiers":[]}