Generative & Multimodal Models
Diffusion and flow-matching models, vision-language models, video generation and learned world models.
Most influential
- Denoising Diffusion Probabilistic Models
We present high quality image synthesis results using diffusion probabilistic models, a class of latent variable models inspired by considerations from nonequilibrium thermodynamics. Our best results are obtained by training on a weighted variational bound designed according to a novel connection between diffusion probabilistic models and denoising score mat…
Preprint8,714 citationsarXiv:2006.11239 - Diffusion Models in Vision: A Survey
Denoising diffusion models represent a recent emerging topic in computer vision, demonstrating remarkable results in the area of generative modeling. A diffusion model is a deep generative model that is based on two stages, a forward diffusion stage and a reverse diffusion stage. In the forward diffusion stage, the input data is gradually perturbed over seve…
Article1,909 citationsarXiv:2209.04747doi:10.1109/tpami.2023.3261988 - Denoising Diffusion Implicit Models
Denoising diffusion probabilistic models (DDPMs) have achieved high quality image generation without adversarial training, yet they require simulating a Markov chain for many steps to produce a sample. To accelerate sampling, we present denoising diffusion implicit models (DDIMs), a more efficient class of iterative implicit probabilistic models with the sam…
Preprint2,507 citationsarXiv:2010.02502 - Improved Denoising Diffusion Probabilistic Models
Denoising diffusion probabilistic models (DDPM) are a class of generative models which have recently been shown to produce excellent samples. We show that with a few simple modifications, DDPMs can also achieve competitive log-likelihoods while maintaining high sample quality. Additionally, we find that learning variances of the reverse diffusion process all…
Preprint1,550 citationsarXiv:2102.09672 - Diffusion models in medical imaging: A comprehensive surveyReview776 citationsdoi:10.1016/j.media.2023.102846
- MotionDiffuse: Text-Driven Human Motion Generation With Diffusion Model
Human motion modeling is important for many modern graphics applications, which typically require professional skills. In order to remove the skill barriers for laymen, recent motion generation methods can directly generate human motions conditioned on natural languages. However, it remains challenging to achieve diverse and fine-grained motion generation wi…
Article437 citationsarXiv:2208.15001doi:10.1109/tpami.2024.3355414 - DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Models
Diffusion probabilistic models (DPMs) have achieved impressive success in high-resolution image synthesis, especially in recent large-scale text-to-image generation applications. An essential technique for improving the sample quality of DPMs is guided sampling, which usually needs a large guidance scale to obtain the best sample quality. The commonly-used f…
Article280 citationsarXiv:2211.01095doi:10.1007/s11633-025-1562-4 - Restoring Vision in Adverse Weather Conditions With Patch-Based Denoising Diffusion Models
Image restoration under adverse weather conditions has been of significant interest for various computer vision applications. Recent successful methods rely on the current progress in deep neural network architectural designs (e.g., with vision transformers). Motivated by the recent progress achieved with state-of-the-art conditional generative models, we pr…
Article475 citationsarXiv:2207.14626doi:10.1109/tpami.2023.3238179 - Denoising diffusion probabilistic models for 3D medical image generation
Recent advances in computer vision have shown promising results in image generation. Diffusion probabilistic models have generated realistic images from textual input, as demonstrated by DALL-E 2, Imagen, and Stable Diffusion. However, their use in medicine, where imaging data typically comprises three-dimensional volumes, has not been systematically evaluat…
Abstract excerpt by Firas Khader et al., CC-BY, https://doi.org/10.1038/s41598-023-34341-2; truncated.
Article343 citationsdoi:10.1038/s41598-023-34341-2 - A Novel Data Augmentation Method Based on Denoising Diffusion Probabilistic Model for Fault Diagnosis Under Imbalanced DataArticle217 citationsdoi:10.1109/tii.2024.3366991
- A multimodal comparison of latent denoising diffusion probabilistic models and generative adversarial networks for medical image synthesis
The success of Deep Learning applications critically depends on the quality and scale of the underlying training data. Generative adversarial networks (GANs) can generate arbitrary large datasets, but diversity and fidelity are limited, which has recently been addressed by denoising diffusion probabilistic models (DDPMs) whose superiority has been demonstrat…
Article257 citationsarXiv:2212.07501doi:10.1038/s41598-023-39278-0 - Structured Denoising Diffusion Models in Discrete State-Spaces
Denoising diffusion probabilistic models (DDPMs) (Ho et al. 2020) have shown impressive results on image and waveform generation in continuous state spaces. Here, we introduce Discrete Denoising Diffusion Probabilistic Models (D3PMs), diffusion-like generative models for discrete data that generalize the multinomial diffusion model of Hoogeboom et al. 2021, …
Preprint303 citationsarXiv:2107.03006 - Protein structure generation via folding diffusion
The ability to computationally generate novel yet physically foldable protein structures could lead to new biological discoveries and new treatments targeting yet incurable diseases. Despite recent advances in protein structure prediction, directly generating diverse, novel protein structures from neural networks remains difficult. In this work, we present a…
Article193 citationsarXiv:2209.15611doi:10.1038/s41467-024-45051-2 - Vector Quantized Diffusion Model for Text-to-Image Synthesis
We present the vector quantized diffusion (VQ-Diffusion) model for text-to-image generation. This method is based on a vector quantized variational autoencoder (VQ-VAE) whose latent space is modeled by a conditional variant of the recently developed Denoising Diffusion Probabilistic Model (DDPM). We find that this latent-space method is well-suited for text-…
Article720 citationsarXiv:2111.14822doi:10.1109/cvpr52688.2022.01043 - Listen, Denoise, Action! Audio-Driven Motion Synthesis with Diffusion Models
Diffusion models have experienced a surge of interest as highly expressive yet efficiently trainable probabilistic models. We show that these models are an excellent fit for synthesising human motion that co-occurs with audio, e.g., dancing and co-speech gesticulation, since motion is complex and highly ambiguous given audio, calling for a probabilistic desc…
Article197 citationsarXiv:2211.09707doi:10.1145/3592458
Newest work
- Jailbreak Connectivity: Towards Diverse, Transferable, and Universal Multimodal Large Language Model JailbreakPreprintarXiv:2610.38091
- StepControl: Long-Horizon Human Motion Generation from Foot-Contact SchedulesPreprintarXiv:2610.38153
- REEL: Same Frames, Different Answers and Different CostsPreprintarXiv:2610.38158
- Complementing Text Based Moderation with Multimodal Grounding: Detecting Fraudulent Inducement Behaviors on Short Video PlatformsPreprintarXiv:2610.38185
- Progressive Coupling Method for Diffusion Posterior Sampling in PtychographyPreprintarXiv:2610.38207
- MEV: A Multi-Event Video Dataset for Long-Take GenerationPreprintarXiv:2610.38241
- NAGO: Noise-Aware Generative Operator via Flow MatchingPreprintarXiv:2610.38273
- PulmoWorld: An Asynchronous Multimodal Medical World Model for Longitudinal Pneumonia DynamicsPreprintarXiv:2610.38289
- Don't Ask the LLM: Visual Token Pruning with Spectral Structure and Semantic RelevancePreprintarXiv:2610.38303
- Aligning Misaligned Benchmarks for Single-View Novel View SynthesisPreprintarXiv:2610.38349