{"topic":"generative-multimodal","sort":"top","limit":25,"offset":0,"count":25,"papers":[{"id":"arxiv:2006.11239","kind":"preprint","title":"Denoising Diffusion Probabilistic Models","authors":["Jonathan Ho","Jain, Ajay","Pieter Abbeel"],"author_count":3,"year":2020,"published":"2020-06-19","venue":null,"cited_by":8714,"doi":null,"arxiv_id":"2006.11239","nct_id":null,"source_url":"http://arxiv.org/abs/2006.11239","archive_url":"https://space-kitty.com/papers/arxiv:2006.11239","updated":"2020-12-16","pmid":null,"pmcid":null,"openalex_id":"W3036167779","pdf_url":"https://arxiv.org/pdf/2006.11239","abstract":"We present high quality image synthesis results using diffusion probabilistic models, a class of latent variable models inspired by considerations from nonequilibrium thermodynamics. Our best results are obtained by training on a weighted variational bound designed according to a novel connection between diffusion probabilistic models and denoising score matching with Langevin dynamics, and our models naturally admit a progressive lossy decompression scheme that can be interpreted as a generalization of autoregressive decoding. On the unconditional CIFAR10 dataset, we obtain an Inception score of 9.46 and a state-of-the-art FID score of 3.17. On 256x256 LSUN, we obtain sample quality similar to ProgressiveGAN. Our implementation is available at https://github.com/hojonathanho/diffusion","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":null,"is_open_access":true,"peer_reviewed":false,"categories":["Epigenetics and DNA Methylation"],"details":null,"provenance":"openalex+arxiv","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv."},{"id":"arxiv:2209.04747","kind":"article","title":"Diffusion Models in Vision: A Survey","authors":["Florinel-Alin Croitoru","Vlad Hondru","Radu Tudor Ionescu","Mubarak Shah"],"author_count":4,"year":2023,"published":"2023-03-27","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","cited_by":1909,"doi":"10.1109/tpami.2023.3261988","arxiv_id":"2209.04747","nct_id":null,"source_url":"https://doi.org/10.1109/tpami.2023.3261988","archive_url":"https://space-kitty.com/papers/arxiv:2209.04747","updated":"2025-01-16","pmid":"37030794","pmcid":null,"openalex_id":"W4360884927","pdf_url":"https://arxiv.org/pdf/2209.04747","abstract":"Denoising diffusion models represent a recent emerging topic in computer vision, demonstrating remarkable results in the area of generative modeling. A diffusion model is a deep generative model that is based on two stages, a forward diffusion stage and a reverse diffusion stage. In the forward diffusion stage, the input data is gradually perturbed over several steps by adding Gaussian noise. In the reverse stage, a model is tasked at recovering the original input data by learning to gradually reverse the diffusion process, step by step. Diffusion models are widely appreciated for the quality and diversity of the generated samples, despite their known computational burdens, i.e. low speeds due to the high number of steps involved during sampling. In this survey, we provide a comprehensive review of articles on denoising diffusion models applied in vision, comprising both theoretical and practical contributions in the field. First, we identify and present three generic diffusion modeling frameworks, which are based on denoising diffusion probabilistic models, noise conditioned score networks, and stochastic differential equations. We further discuss the relations between diffusion models and other deep generative models, including variational auto-encoders, generative adversarial networks, energy-based models, autoregressive models and normalizing flows. Then, we introduce a multi-perspective categorization of diffusion models applied in computer vision. Finally, we illustrate the current limitations of diffusion models and envision some interesting directions for future research.","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":null,"is_open_access":true,"peer_reviewed":null,"categories":["Generative Adversarial Networks and Image Synthesis"],"details":null,"provenance":"openalex+arxiv","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv."},{"id":"arxiv:2010.02502","kind":"preprint","title":"Denoising Diffusion Implicit Models","authors":["Jiaming Song","Chenlin Meng","Stefano Ermon"],"author_count":3,"year":2020,"published":"2020-10-06","venue":null,"cited_by":2507,"doi":null,"arxiv_id":"2010.02502","nct_id":null,"source_url":"http://arxiv.org/abs/2010.02502","archive_url":"https://space-kitty.com/papers/arxiv:2010.02502","updated":"2022-10-05","pmid":null,"pmcid":null,"openalex_id":"W3092442149","pdf_url":"https://arxiv.org/pdf/2010.02502","abstract":"Denoising diffusion probabilistic models (DDPMs) have achieved high quality image generation without adversarial training, yet they require simulating a Markov chain for many steps to produce a sample. To accelerate sampling, we present denoising diffusion implicit models (DDIMs), a more efficient class of iterative implicit probabilistic models with the same training procedure as DDPMs. In DDPMs, the generative process is defined as the reverse of a Markovian diffusion process. We construct a class of non-Markovian diffusion processes that lead to the same training objective, but whose reverse process can be much faster to sample from. We empirically demonstrate that DDIMs can produce high quality samples $10 \\times$ to $50 \\times$ faster in terms of wall-clock time compared to DDPMs, allow us to trade off computation for sample quality, and can perform semantically meaningful image interpolation directly in the latent space.","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":null,"is_open_access":true,"peer_reviewed":false,"categories":["Generative Adversarial Networks and Image Synthesis"],"details":null,"provenance":"openalex+arxiv","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv."},{"id":"arxiv:2102.09672","kind":"preprint","title":"Improved Denoising Diffusion Probabilistic Models","authors":["Alex Nichol","Prafulla Dhariwal"],"author_count":2,"year":2021,"published":"2021-02-18","venue":null,"cited_by":1550,"doi":null,"arxiv_id":"2102.09672","nct_id":null,"source_url":"http://arxiv.org/abs/2102.09672","archive_url":"https://space-kitty.com/papers/arxiv:2102.09672","updated":"2021-02-18","pmid":null,"pmcid":null,"openalex_id":"W3122887982","pdf_url":"https://arxiv.org/pdf/2102.09672","abstract":"Denoising diffusion probabilistic models (DDPM) are a class of generative models which have recently been shown to produce excellent samples. We show that with a few simple modifications, DDPMs can also achieve competitive log-likelihoods while maintaining high sample quality. Additionally, we find that learning variances of the reverse diffusion process allows sampling with an order of magnitude fewer forward passes with a negligible difference in sample quality, which is important for the practical deployment of these models. We additionally use precision and recall to compare how well DDPMs and GANs cover the target distribution. Finally, we show that the sample quality and likelihood of these models scale smoothly with model capacity and training compute, making them easily scalable. We release our code at https://github.com/openai/improved-diffusion","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":null,"is_open_access":true,"peer_reviewed":false,"categories":["Generative Adversarial Networks and Image Synthesis"],"details":null,"provenance":"openalex+arxiv","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv."},{"id":"doi:10.1016/j.media.2023.102846","kind":"review","title":"Diffusion models in medical imaging: A comprehensive survey","authors":["Amirhossein Kazerouni","Ehsan Khodapanah Aghdam","Moein Heidari","Reza Azad","Mohsen Fayyaz","Ilker Hacihaliloglu","Dorit Merhof"],"author_count":7,"year":2023,"published":"2023-05-23","venue":"Medical Image Analysis","cited_by":776,"doi":"10.1016/j.media.2023.102846","arxiv_id":null,"nct_id":null,"source_url":"https://doi.org/10.1016/j.media.2023.102846","archive_url":"https://space-kitty.com/papers/doi:10.1016/j.media.2023.102846","updated":null,"pmid":"37295311","pmcid":null,"openalex_id":"W4377695098","pdf_url":null,"abstract":null,"abstract_license":null,"license":null,"is_open_access":false,"peer_reviewed":null,"categories":["Generative Adversarial Networks and Image Synthesis"],"details":null,"provenance":"openalex","attribution":"Metadata and citation counts: OpenAlex (CC0)."},{"id":"arxiv:2208.15001","kind":"article","title":"MotionDiffuse: Text-Driven Human Motion Generation With Diffusion Model","authors":["Mingyuan Zhang","Zhongang Cai","Liang Pan","Fangzhou Hong","Xinying Guo","Lei Yang","Ziwei Liu"],"author_count":7,"year":2024,"published":"2024-01-29","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","cited_by":437,"doi":"10.1109/tpami.2024.3355414","arxiv_id":"2208.15001","nct_id":null,"source_url":"https://doi.org/10.1109/tpami.2024.3355414","archive_url":"https://space-kitty.com/papers/arxiv:2208.15001","updated":"2022-08-31","pmid":"38285589","pmcid":null,"openalex_id":"W4391305822","pdf_url":"https://arxiv.org/pdf/2208.15001","abstract":"Human motion modeling is important for many modern graphics applications, which typically require professional skills. In order to remove the skill barriers for laymen, recent motion generation methods can directly generate human motions conditioned on natural languages. However, it remains challenging to achieve diverse and fine-grained motion generation with various text inputs. To address this problem, we propose MotionDiffuse, the first diffusion model-based text-driven motion generation framework, which demonstrates several desired properties over existing methods. 1) Probabilistic Mapping. Instead of a deterministic language-motion mapping, MotionDiffuse generates motions through a series of denoising steps in which variations are injected. 2) Realistic Synthesis. MotionDiffuse excels at modeling complicated data distribution and generating vivid motion sequences. 3) Multi-Level Manipulation. MotionDiffuse responds to fine-grained instructions on body parts, and arbitrary-length motion synthesis with time-varied text prompts. Our experiments show MotionDiffuse outperforms existing SoTA methods by convincing margins on text-driven motion generation and action-conditioned motion generation. A qualitative analysis further demonstrates MotionDiffuse's controllability for comprehensive motion generation. Homepage: https://mingyuan-zhang.github.io/projects/MotionDiffuse.html","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":null,"is_open_access":true,"peer_reviewed":null,"categories":["Human Motion and Animation"],"details":null,"provenance":"openalex+arxiv","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv."},{"id":"arxiv:2211.01095","kind":"article","title":"DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Models","authors":["Cheng Lu","Yuhao Zhou","Fan Bao","Jianfei Chen","Chongxuan Li","Jun Zhu"],"author_count":6,"year":2025,"published":"2025-06-22","venue":"Machine Intelligence Research","cited_by":280,"doi":"10.1007/s11633-025-1562-4","arxiv_id":"2211.01095","nct_id":null,"source_url":"https://doi.org/10.1007/s11633-025-1562-4","archive_url":"https://space-kitty.com/papers/arxiv:2211.01095","updated":"2025-05-19","pmid":null,"pmcid":null,"openalex_id":"W4411528087","pdf_url":"https://link.springer.com/content/pdf/10.1007/s11633-025-1562-4.pdf","abstract":"Diffusion probabilistic models (DPMs) have achieved impressive success in high-resolution image synthesis, especially in recent large-scale text-to-image generation applications. An essential technique for improving the sample quality of DPMs is guided sampling, which usually needs a large guidance scale to obtain the best sample quality. The commonly-used fast sampler for guided sampling is DDIM, a first-order diffusion ODE solver that generally needs 100 to 250 steps for high-quality samples. Although recent works propose dedicated high-order solvers and achieve a further speedup for sampling without guidance, their effectiveness for guided sampling has not been well-tested before. In this work, we demonstrate that previous high-order fast samplers suffer from instability issues, and they even become slower than DDIM when the guidance scale grows large. To further speed up guided sampling, we propose DPM-Solver++, a high-order solver for the guided sampling of DPMs. DPM-Solver++ solves the diffusion ODE with the data prediction model and adopts thresholding methods to keep the solution matches training data distribution. We further propose a multistep variant of DPM-Solver++ to address the instability issue by reducing the effective step size. Experiments show that DPM-Solver++ can generate high-quality samples within only 15 to 20 steps for guided sampling by pixel-space and latent-space DPMs.","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":"cc-by","is_open_access":true,"peer_reviewed":null,"categories":["Generative Adversarial Networks and Image Synthesis"],"details":null,"provenance":"openalex+arxiv","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv."},{"id":"arxiv:2207.14626","kind":"article","title":"Restoring Vision in Adverse Weather Conditions With Patch-Based Denoising Diffusion Models","authors":["Ozan Özdenizci","Robert A. Legenstein"],"author_count":2,"year":2023,"published":"2023-01-19","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","cited_by":475,"doi":"10.1109/tpami.2023.3238179","arxiv_id":"2207.14626","nct_id":null,"source_url":"https://doi.org/10.1109/tpami.2023.3238179","archive_url":"https://space-kitty.com/papers/arxiv:2207.14626","updated":"2022-11-21","pmid":"37021892","pmcid":null,"openalex_id":"W4317496742","pdf_url":"https://arxiv.org/pdf/2207.14626","abstract":"Image restoration under adverse weather conditions has been of significant interest for various computer vision applications. Recent successful methods rely on the current progress in deep neural network architectural designs (e.g., with vision transformers). Motivated by the recent progress achieved with state-of-the-art conditional generative models, we present a novel patch-based image restoration algorithm based on denoising diffusion probabilistic models. Our patch-based diffusion modeling approach enables size-agnostic image restoration by using a guided denoising process with smoothed noise estimates across overlapping patches during inference. We empirically evaluate our model on benchmark datasets for image desnowing, combined deraining and dehazing, and raindrop removal. We demonstrate our approach to achieve state-of-the-art performances on both weather-specific and multi-weather image restoration, and experimentally show strong generalization to real-world test images.","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":null,"is_open_access":true,"peer_reviewed":null,"categories":["Image Enhancement Techniques"],"details":null,"provenance":"openalex+arxiv","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv."},{"id":"doi:10.1038/s41598-023-34341-2","kind":"article","title":"Denoising diffusion probabilistic models for 3D medical image generation","authors":["Firas Khader","Gustav Anton Müller‐Franzes","Soroosh Tayebi Arasteh","Tianyu Han","Christoph Haarburger","Maximilian Franz Schulze-Hagen","Philipp Schad","Sandy Engelhardt","Bettina Baeßler","Sebastian Foersch","Johannes Stegmaier","Christiane Katharina Kuhl","Sven Nebelung","Jakob Nikolas Kather","Daniel Truhn"],"author_count":15,"year":2023,"published":"2023-05-05","venue":"Scientific Reports","cited_by":343,"doi":"10.1038/s41598-023-34341-2","arxiv_id":null,"nct_id":null,"source_url":"https://doi.org/10.1038/s41598-023-34341-2","archive_url":"https://space-kitty.com/papers/doi:10.1038/s41598-023-34341-2","updated":null,"pmid":"37147413","pmcid":null,"openalex_id":"W4372336638","pdf_url":"https://www.nature.com/articles/s41598-023-34341-2.pdf","abstract":"Recent advances in computer vision have shown promising results in image generation. Diffusion probabilistic models have generated realistic images from textual input, as demonstrated by DALL-E 2, Imagen, and Stable Diffusion. However, their use in medicine, where imaging data typically comprises three-dimensional volumes, has not been systematically evaluated. Synthetic images may play a crucial role in privacy-preserving artificial intelligence and can also be used to augment small datasets. We show that diffusion probabilistic models can synthesize high-quality medical data for magnetic resonance imaging (MRI) and computed tomography (CT). For quantitative evaluation, two radiologists rated the quality of the synthesized images regarding \"realistic image appearance\", \"anatomical correctness\", and \"consistency between slices\". Furthermore, we demonstrate that synthetic images can be used in self-supervised pre-training and improve the performance of breast segmentation models when data is scarce (Dice scores, 0.91 [without synthetic data], 0.95 [with synthetic data]).","abstract_license":"CC-BY (publisher license, via OpenAlex)","license":"cc-by","is_open_access":true,"peer_reviewed":null,"categories":["Generative Adversarial Networks and Image Synthesis"],"details":null,"provenance":"openalex","attribution":"Abstract of \"Denoising diffusion probabilistic models for 3D medical image generation\" by Firas Khader, Gustav Anton Müller‐Franzes, Soroosh Tayebi Arasteh et al., https://doi.org/10.1038/s41598-023-34341-2, used under its Creative Commons Attribution license (CC-BY); markup removed, otherwise unmodified."},{"id":"doi:10.1109/tii.2024.3366991","kind":"article","title":"A Novel Data Augmentation Method Based on Denoising Diffusion Probabilistic Model for Fault Diagnosis Under Imbalanced Data","authors":["Xiongyan Yang","Tianyi Ye","Xianfeng Yuan","Weijie Zhu","Xiaoxue Mei","Fengyu Zhou"],"author_count":6,"year":2024,"published":"2024-02-27","venue":"IEEE Transactions on Industrial Informatics","cited_by":217,"doi":"10.1109/tii.2024.3366991","arxiv_id":null,"nct_id":null,"source_url":"https://doi.org/10.1109/tii.2024.3366991","archive_url":"https://space-kitty.com/papers/doi:10.1109/tii.2024.3366991","updated":null,"pmid":null,"pmcid":null,"openalex_id":"W4392208038","pdf_url":null,"abstract":null,"abstract_license":null,"license":null,"is_open_access":false,"peer_reviewed":null,"categories":["Machine Fault Diagnosis Techniques"],"details":null,"provenance":"openalex","attribution":"Metadata and citation counts: OpenAlex (CC0)."},{"id":"arxiv:2212.07501","kind":"article","title":"A multimodal comparison of latent denoising diffusion probabilistic models and generative adversarial networks for medical image synthesis","authors":["Gustav Anton Müller‐Franzes","Jan Moritz Niehues","Firas Khader","Soroosh Tayebi Arasteh","Christoph Haarburger","Christiane Katharina Kuhl","Tianci Wang","Tianyu Han","Teresa Nolte","Sven Nebelung","Jakob Nikolas Kather","Daniel Truhn"],"author_count":12,"year":2023,"published":"2023-07-26","venue":"Scientific Reports","cited_by":257,"doi":"10.1038/s41598-023-39278-0","arxiv_id":"2212.07501","nct_id":null,"source_url":"https://doi.org/10.1038/s41598-023-39278-0","archive_url":"https://space-kitty.com/papers/arxiv:2212.07501","updated":"2022-12-14","pmid":"37495660","pmcid":null,"openalex_id":"W4385272245","pdf_url":"https://www.nature.com/articles/s41598-023-39278-0.pdf","abstract":"The success of Deep Learning applications critically depends on the quality and scale of the underlying training data. Generative adversarial networks (GANs) can generate arbitrary large datasets, but diversity and fidelity are limited, which has recently been addressed by denoising diffusion probabilistic models (DDPMs) whose superiority has been demonstrated on natural images. In this study, we propose Medfusion, a conditional latent DDPM for medical images. We compare our DDPM-based model against GAN-based models, which constitute the current state-of-the-art in the medical domain. Medfusion was trained and compared with (i) StyleGan-3 on n=101,442 images from the AIROGS challenge dataset to generate fundoscopies with and without glaucoma, (ii) ProGAN on n=191,027 from the CheXpert dataset to generate radiographs with and without cardiomegaly and (iii) wGAN on n=19,557 images from the CRCMS dataset to generate histopathological images with and without microsatellite stability. In the AIROGS, CRMCS, and CheXpert datasets, Medfusion achieved lower (=better) FID than the GANs (11.63 versus 20.43, 30.03 versus 49.26, and 17.28 versus 84.31). Also, fidelity (precision) and diversity (recall) were higher (=better) for Medfusion in all three datasets. Our study shows that DDPM are a superior alternative to GANs for image synthesis in the medical domain.","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":"cc-by","is_open_access":true,"peer_reviewed":null,"categories":["Generative Adversarial Networks and Image Synthesis"],"details":null,"provenance":"openalex+arxiv","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv."},{"id":"arxiv:2107.03006","kind":"preprint","title":"Structured Denoising Diffusion Models in Discrete State-Spaces","authors":["Jacob Austin","Johnson, Daniel D.","Jonathan Ho","Daniel Tarlow","Rianne van den Berg"],"author_count":5,"year":2021,"published":"2021-07-07","venue":null,"cited_by":303,"doi":null,"arxiv_id":"2107.03006","nct_id":null,"source_url":"http://arxiv.org/abs/2107.03006","archive_url":"https://space-kitty.com/papers/arxiv:2107.03006","updated":"2023-02-22","pmid":null,"pmcid":null,"openalex_id":"W3181640983","pdf_url":"https://arxiv.org/pdf/2107.03006","abstract":"Denoising diffusion probabilistic models (DDPMs) (Ho et al. 2020) have shown impressive results on image and waveform generation in continuous state spaces. Here, we introduce Discrete Denoising Diffusion Probabilistic Models (D3PMs), diffusion-like generative models for discrete data that generalize the multinomial diffusion model of Hoogeboom et al. 2021, by going beyond corruption processes with uniform transition probabilities. This includes corruption with transition matrices that mimic Gaussian kernels in continuous space, matrices based on nearest neighbors in embedding space, and matrices that introduce absorbing states. The third allows us to draw a connection between diffusion models and autoregressive and mask-based generative models. We show that the choice of transition matrix is an important design decision that leads to improved results in image and text domains. We also introduce a new loss function that combines the variational lower bound with an auxiliary cross entropy loss. For text, this model class achieves strong results on character-level text generation while scaling to large vocabularies on LM1B. On the image dataset CIFAR-10, our models approach the sample quality and exceed the log-likelihood of the continuous-space DDPM model.","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":null,"is_open_access":true,"peer_reviewed":false,"categories":["Generative Adversarial Networks and Image Synthesis"],"details":null,"provenance":"openalex+arxiv","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv."},{"id":"arxiv:2209.15611","kind":"article","title":"Protein structure generation via folding diffusion","authors":["Kevin Wu","Kevin Yang","Rianne van den Berg","Sarah Alamdari","James Zou","Alex Xijie Lu","Ava P. Amini"],"author_count":7,"year":2024,"published":"2024-02-05","venue":"Nature Communications","cited_by":193,"doi":"10.1038/s41467-024-45051-2","arxiv_id":"2209.15611","nct_id":null,"source_url":"https://doi.org/10.1038/s41467-024-45051-2","archive_url":"https://space-kitty.com/papers/arxiv:2209.15611","updated":"2022-11-24","pmid":"38316764","pmcid":null,"openalex_id":"W4391530977","pdf_url":"https://www.nature.com/articles/s41467-024-45051-2.pdf","abstract":"The ability to computationally generate novel yet physically foldable protein structures could lead to new biological discoveries and new treatments targeting yet incurable diseases. Despite recent advances in protein structure prediction, directly generating diverse, novel protein structures from neural networks remains difficult. In this work, we present a new diffusion-based generative model that designs protein backbone structures via a procedure that mirrors the native folding process. We describe protein backbone structure as a series of consecutive angles capturing the relative orientation of the constituent amino acid residues, and generate new structures by denoising from a random, unfolded state towards a stable folded structure. Not only does this mirror how proteins biologically twist into energetically favorable conformations, the inherent shift and rotational invariance of this representation crucially alleviates the need for complex equivariant networks. We train a denoising diffusion probabilistic model with a simple transformer backbone and demonstrate that our resulting model unconditionally generates highly realistic protein structures with complexity and structural patterns akin to those of naturally-occurring proteins. As a useful resource, we release the first open-source codebase and trained models for protein structure diffusion.","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":"cc-by","is_open_access":true,"peer_reviewed":null,"categories":["Protein Structure and Dynamics"],"details":null,"provenance":"openalex+arxiv","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv."},{"id":"arxiv:2111.14822","kind":"article","title":"Vector Quantized Diffusion Model for Text-to-Image Synthesis","authors":["Shuyang Gu","Dong Chen","Jianmin Bao","Fang Wen","Bo Zhang","Dongdong Chen","Lu Yuan","Baining Guo"],"author_count":8,"year":2021,"published":"2021-11-29","venue":"IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings","cited_by":720,"doi":"10.1109/cvpr52688.2022.01043","arxiv_id":"2111.14822","nct_id":null,"source_url":"https://arxiv.org/abs/2111.14822","archive_url":"https://space-kitty.com/papers/arxiv:2111.14822","updated":"2022-03-03","pmid":null,"pmcid":null,"openalex_id":"W4312388283","pdf_url":"https://arxiv.org/pdf/2111.14822","abstract":"We present the vector quantized diffusion (VQ-Diffusion) model for text-to-image generation. This method is based on a vector quantized variational autoencoder (VQ-VAE) whose latent space is modeled by a conditional variant of the recently developed Denoising Diffusion Probabilistic Model (DDPM). We find that this latent-space method is well-suited for text-to-image generation tasks because it not only eliminates the unidirectional bias with existing methods but also allows us to incorporate a mask-and-replace diffusion strategy to avoid the accumulation of errors, which is a serious problem with existing methods. Our experiments show that the VQ-Diffusion produces significantly better text-to-image generation results when compared with conventional autoregressive (AR) models with similar numbers of parameters. Compared with previous GAN-based text-to-image methods, our VQ-Diffusion can handle more complex scenes and improve the synthesized image quality by a large margin. Finally, we show that the image generation computation in our method can be made highly efficient by reparameterization. With traditional AR methods, the text-to-image generation time increases linearly with the output image resolution and hence is quite time consuming even for normal size images. The VQ-Diffusion allows us to achieve a better trade-off between quality and speed. Our experiments indicate that the VQ-Diffusion model with the reparameterization is fifteen times faster than traditional AR methods while achieving a better image quality.","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":null,"is_open_access":false,"peer_reviewed":null,"categories":["cs.CV","cs.LG"],"details":null,"provenance":"arxiv","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv."},{"id":"arxiv:2211.09707","kind":"article","title":"Listen, Denoise, Action! Audio-Driven Motion Synthesis with Diffusion Models","authors":["Simon Alexanderson","Rajmund Nagy","Jonas Beskow","Gustav Eje Henter"],"author_count":4,"year":2023,"published":"2023-07-26","venue":"ACM Transactions on Graphics","cited_by":197,"doi":"10.1145/3592458","arxiv_id":"2211.09707","nct_id":null,"source_url":"https://doi.org/10.1145/3592458","archive_url":"https://space-kitty.com/papers/arxiv:2211.09707","updated":"2023-05-16","pmid":null,"pmcid":null,"openalex_id":"W4377010269","pdf_url":"https://dl.acm.org/doi/pdf/10.1145/3592458","abstract":"Diffusion models have experienced a surge of interest as highly expressive yet efficiently trainable probabilistic models. We show that these models are an excellent fit for synthesising human motion that co-occurs with audio, e.g., dancing and co-speech gesticulation, since motion is complex and highly ambiguous given audio, calling for a probabilistic description. Specifically, we adapt the DiffWave architecture to model 3D pose sequences, putting Conformers in place of dilated convolutions for improved modelling power. We also demonstrate control over motion style, using classifier-free guidance to adjust the strength of the stylistic expression. Experiments on gesture and dance generation confirm that the proposed method achieves top-of-the-line motion quality, with distinctive styles whose expression can be made more or less pronounced. We also synthesise path-driven locomotion using the same model architecture. Finally, we generalise the guidance procedure to obtain product-of-expert ensembles of diffusion models and demonstrate how these may be used for, e.g., style interpolation, a contribution we believe is of independent interest. See https://www.speech.kth.se/research/listen-denoise-action/ for video examples, data, and code.","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":"cc-by-nc","is_open_access":true,"peer_reviewed":null,"categories":["Human Motion and Animation"],"details":null,"provenance":"openalex+arxiv","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv."},{"id":"arxiv:2105.06337","kind":"preprint","title":"Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech","authors":["Popov, Vadim","Ivan Vovk","Vladimir Gogoryan","Tasnima Sadekova","Mikhail Kudinov"],"author_count":5,"year":2021,"published":"2021-05-13","venue":null,"cited_by":251,"doi":null,"arxiv_id":"2105.06337","nct_id":null,"source_url":"http://arxiv.org/abs/2105.06337","archive_url":"https://space-kitty.com/papers/arxiv:2105.06337","updated":"2021-08-05","pmid":null,"pmcid":null,"openalex_id":"W3162673269","pdf_url":"https://arxiv.org/pdf/2105.06337","abstract":"Recently, denoising diffusion probabilistic models and generative score matching have shown high potential in modelling complex data distributions while stochastic calculus has provided a unified point of view on these techniques allowing for flexible inference schemes. In this paper we introduce Grad-TTS, a novel text-to-speech model with score-based decoder producing mel-spectrograms by gradually transforming noise predicted by encoder and aligned with text input by means of Monotonic Alignment Search. The framework of stochastic differential equations helps us to generalize conventional diffusion probabilistic models to the case of reconstructing data from noise with different parameters and allows to make this reconstruction flexible by explicitly controlling trade-off between sound quality and inference speed. Subjective human evaluation shows that Grad-TTS is competitive with state-of-the-art text-to-speech approaches in terms of Mean Opinion Score. We will make the code publicly available shortly.","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":null,"is_open_access":true,"peer_reviewed":false,"categories":["Speech Recognition and Synthesis"],"details":null,"provenance":"openalex+arxiv","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv."},{"id":"arxiv:2112.03126","kind":"preprint","title":"Label-Efficient Semantic Segmentation with Diffusion Models","authors":["Dmitry Baranchuk","Ivan Rubachev","Andrey Sergeevich Voynov","Valentin Khrulkov","Artem Babenko"],"author_count":5,"year":2021,"published":"2021-12-06","venue":null,"cited_by":237,"doi":null,"arxiv_id":"2112.03126","nct_id":null,"source_url":"http://arxiv.org/abs/2112.03126","archive_url":"https://space-kitty.com/papers/arxiv:2112.03126","updated":"2022-03-16","pmid":null,"pmcid":null,"openalex_id":"W4200632170","pdf_url":"https://arxiv.org/pdf/2112.03126","abstract":"Denoising diffusion probabilistic models have recently received much research attention since they outperform alternative approaches, such as GANs, and currently provide state-of-the-art generative performance. The superior performance of diffusion models has made them an appealing tool in several applications, including inpainting, super-resolution, and semantic editing. In this paper, we demonstrate that diffusion models can also serve as an instrument for semantic segmentation, especially in the setup when labeled data is scarce. In particular, for several pretrained diffusion models, we investigate the intermediate activations from the networks that perform the Markov step of the reverse diffusion process. We show that these activations effectively capture the semantic information from an input image and appear to be excellent pixel-level representations for the segmentation problem. Based on these observations, we describe a simple segmentation method, which can work even if only a few training images are provided. Our approach significantly outperforms the existing alternatives on several datasets for the same amount of human supervision.","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":null,"is_open_access":true,"peer_reviewed":false,"categories":["Domain Adaptation and Few-Shot Learning"],"details":null,"provenance":"openalex+arxiv","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv."},{"id":"arxiv:2203.09481","kind":"article","title":"Diffusion Probabilistic Modeling for Video Generation","authors":["Ruihan Yang","Prakhar Kumar Srivastava","Stephan Mandt"],"author_count":3,"year":2023,"published":"2023-10-20","venue":"Entropy","cited_by":166,"doi":"10.3390/e25101469","arxiv_id":"2203.09481","nct_id":null,"source_url":"https://doi.org/10.3390/e25101469","archive_url":"https://space-kitty.com/papers/arxiv:2203.09481","updated":"2022-12-08","pmid":"37895590","pmcid":null,"openalex_id":"W4221146061","pdf_url":"https://www.mdpi.com/1099-4300/25/10/1469/pdf?version=1697803890","abstract":"Denoising diffusion probabilistic models are a promising new class of generative models that mark a milestone in high-quality image generation. This paper showcases their ability to sequentially generate video, surpassing prior methods in perceptual and probabilistic forecasting metrics. We propose an autoregressive, end-to-end optimized video diffusion model inspired by recent advances in neural video compression. The model successively generates future frames by correcting a deterministic next-frame prediction using a stochastic residual generated by an inverse diffusion process. We compare this approach against five baselines on four datasets involving natural and simulation-based videos. We find significant improvements in terms of perceptual quality for all datasets. Furthermore, by introducing a scalable version of the Continuous Ranked Probability Score (CRPS) applicable to video, we show that our model also outperforms existing approaches in their probabilistic frame forecasting ability.","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":"cc-by","is_open_access":true,"peer_reviewed":null,"categories":["Generative Adversarial Networks and Image Synthesis"],"details":null,"provenance":"openalex+arxiv","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv."},{"id":"doi:10.1007/s00259-023-06417-8","kind":"article","title":"PET image denoising based on denoising diffusion probabilistic model","authors":["Kuang Gong","Keith A. Johnson","Georges El Fakhri","Quanzheng Li","Tinsu Pan"],"author_count":5,"year":2023,"published":"2023-10-03","venue":"European Journal of Nuclear Medicine and Molecular Imaging","cited_by":166,"doi":"10.1007/s00259-023-06417-8","arxiv_id":null,"nct_id":null,"source_url":"https://doi.org/10.1007/s00259-023-06417-8","archive_url":"https://space-kitty.com/papers/doi:10.1007/s00259-023-06417-8","updated":null,"pmid":"37787849","pmcid":null,"openalex_id":"W4387296387","pdf_url":"https://pmc.ncbi.nlm.nih.gov/articles/PMC10958486/pdf/nihms-1972295.pdf","abstract":null,"abstract_license":null,"license":null,"is_open_access":true,"peer_reviewed":null,"categories":["Image and Signal Denoising Methods"],"details":null,"provenance":"openalex","attribution":"Metadata and citation counts: OpenAlex (CC0)."},{"id":"arxiv:2202.09778","kind":"preprint","title":"Pseudo Numerical Methods for Diffusion Models on Manifolds","authors":["Luping Liu","Yi Ren","Zhijie Lin","Zhou Zhao"],"author_count":4,"year":2022,"published":"2022-02-20","venue":null,"cited_by":188,"doi":null,"arxiv_id":"2202.09778","nct_id":null,"source_url":"http://arxiv.org/abs/2202.09778","archive_url":"https://space-kitty.com/papers/arxiv:2202.09778","updated":"2022-10-31","pmid":null,"pmcid":null,"openalex_id":"W4226148126","pdf_url":"https://arxiv.org/pdf/2202.09778","abstract":"Denoising Diffusion Probabilistic Models (DDPMs) can generate high-quality samples such as image and audio samples. However, DDPMs require hundreds to thousands of iterations to produce final samples. Several prior works have successfully accelerated DDPMs through adjusting the variance schedule (e.g., Improved Denoising Diffusion Probabilistic Models) or the denoising equation (e.g., Denoising Diffusion Implicit Models (DDIMs)). However, these acceleration methods cannot maintain the quality of samples and even introduce new noise at a high speedup rate, which limit their practicability. To accelerate the inference process while keeping the sample quality, we provide a fresh perspective that DDPMs should be treated as solving differential equations on manifolds. Under such a perspective, we propose pseudo numerical methods for diffusion models (PNDMs). Specifically, we figure out how to solve differential equations on manifolds and show that DDIMs are simple cases of pseudo numerical methods. We change several classical numerical methods to corresponding pseudo numerical methods and find that the pseudo linear multi-step method is the best in most situations. According to our experiments, by directly using pre-trained models on Cifar10, CelebA and LSUN, PNDMs can generate higher quality synthetic images with only 50 steps compared with 1000-step DDIMs (20x speedup), significantly outperform DDIMs with 250 steps (by around 0.4 in FID) and have good generalization on different variance schedules. Our implementation is available at https://github.com/luping-liu/PNDM.","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":null,"is_open_access":true,"peer_reviewed":false,"categories":["Generative Adversarial Networks and Image Synthesis"],"details":null,"provenance":"openalex+arxiv","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv."},{"id":"arxiv:2205.11487","kind":"preprint","title":"Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding","authors":["Chitwan Saharia","William Chan","Saurabh Saxena","Lala Li","Jay Whang","Emily Denton","Seyed Kamyar Seyed Ghasemipour","Burcu Karagol Ayan","S. Sara Mahdavi","Rapha Gontijo Lopes","Tim Salimans","Jonathan Ho","David J Fleet","Mohammad Norouzi"],"author_count":14,"year":2022,"published":"2022-05-23","venue":null,"cited_by":2528,"doi":null,"arxiv_id":"2205.11487","nct_id":null,"source_url":"https://arxiv.org/abs/2205.11487","archive_url":"https://space-kitty.com/papers/arxiv:2205.11487","updated":"2022-05-23","pmid":null,"pmcid":null,"openalex_id":"W4281485151","pdf_url":"https://arxiv.org/pdf/2205.11487","abstract":"We present Imagen, a text-to-image diffusion model with an unprecedented degree of photorealism and a deep level of language understanding. Imagen builds on the power of large transformer language models in understanding text and hinges on the strength of diffusion models in high-fidelity image generation. Our key discovery is that generic large language models (e.g. T5), pretrained on text-only corpora, are surprisingly effective at encoding text for image synthesis: increasing the size of the language model in Imagen boosts both sample fidelity and image-text alignment much more than increasing the size of the image diffusion model. Imagen achieves a new state-of-the-art FID score of 7.27 on the COCO dataset, without ever training on COCO, and human raters find Imagen samples to be on par with the COCO data itself in image-text alignment. To assess text-to-image models in greater depth, we introduce DrawBench, a comprehensive and challenging benchmark for text-to-image models. With DrawBench, we compare Imagen with recent methods including VQ-GAN+CLIP, Latent Diffusion Models, and DALL-E 2, and find that human raters prefer Imagen over other models in side-by-side comparisons, both in terms of sample quality and image-text alignment. See https://imagen.research.google/ for an overview of the results.","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":null,"is_open_access":false,"peer_reviewed":false,"categories":["cs.CV","cs.LG"],"details":null,"provenance":"arxiv","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv."},{"id":"doi:10.1088/1361-6560/acca5c","kind":"article","title":"2D medical image synthesis using transformer-based denoising diffusion probabilistic model","authors":["Shaoyan Pan","Tonghe Wang","Richard L. J. Qiu","Marian Axente","Chih‐Wei Chang","Junbo Peng","Ashish Bharat Patel","Joseph W. Shelton","Sagar Anil Patel","Justin R. Roper","Xiaofeng Yang"],"author_count":11,"year":2023,"published":"2023-04-04","venue":"Physics in Medicine and Biology","cited_by":160,"doi":"10.1088/1361-6560/acca5c","arxiv_id":null,"nct_id":null,"source_url":"https://doi.org/10.1088/1361-6560/acca5c","archive_url":"https://space-kitty.com/papers/doi:10.1088/1361-6560/acca5c","updated":null,"pmid":"37015231","pmcid":null,"openalex_id":"W4362594575","pdf_url":null,"abstract":"Abstract Objective . Artificial intelligence (AI) methods have gained popularity in medical imaging research. The size and scope of the training image datasets needed for successful AI model deployment does not always have the desired scale. In this paper, we introduce a medical image synthesis framework aimed at addressing the challenge of limited training datasets for AI models. Approach . The proposed 2D image synthesis framework is based on a diffusion model using a Swin-transformer-based network. This model consists of a forward Gaussian noise process and a reverse process using the transformer-based diffusion model for denoising. Training data includes four image datasets: chest x-rays, heart MRI, pelvic CT, and abdomen CT. We evaluated the authenticity, quality, and diversity of the synthetic images using visual Turing assessments conducted by three medical physicists, and four quantitative evaluations: the Inception score (IS), Fréchet Inception Distance score (FID), feature similarity and diversity score (DS, indicating diversity similarity) between the synthetic and true images. To leverage the framework value for training AI models, we conducted COVID-19 classification tasks using real images, synthetic images, and mixtures of both images. Main results . Visual Turing assessments showed an average accuracy of 0.64 (accuracy converging to 50 % indicates a better realistic visual appearance of the synthetic images), sensitivity of 0.79, and specificity of 0.50. Average quantitative accuracy obtained from all datasets were IS = 2.28, FID = 37.27, FDS = 0.20, and DS = 0.86. For the COVID-19 classification task, the baseline network obtained an accuracy of 0.88 using a pure real dataset, 0.89 using a pure synthetic dataset, and 0.93 using a dataset mixed of real and synthetic data. Significance . A image synthesis framework was demonstrated for medical image synthesis, which can generate high-quality medical images of different imaging modalities with the purpose of supplementing existing training sets for AI model deployment. This method has potential applications in many data-driven medical imaging research.","abstract_license":"CC-BY (publisher license, via OpenAlex)","license":"cc-by","is_open_access":true,"peer_reviewed":null,"categories":["Generative Adversarial Networks and Image Synthesis"],"details":null,"provenance":"openalex","attribution":"Abstract of \"2D medical image synthesis using transformer-based denoising diffusion probabilistic model\" by Shaoyan Pan, Tonghe Wang, Richard L. J. Qiu et al., https://doi.org/10.1088/1361-6560/acca5c, used under its Creative Commons Attribution license (CC-BY); markup removed, otherwise unmodified."},{"id":"arxiv:2312.06662","kind":"article","title":"Photorealistic Video Generation with Diffusion Models","authors":["Agrim Gupta","Lijun Yu","Kihyuk Sohn","Xiuye Gu","Meera Hahn","Li Fei-Fei","Irfan Essa","Lu Jiang","José Lezama"],"author_count":9,"year":2023,"published":"2023-12-11","venue":"Lecture notes in computer science","cited_by":83,"doi":"10.1007/978-3-031-72986-7_23","arxiv_id":"2312.06662","nct_id":null,"source_url":"https://arxiv.org/abs/2312.06662","archive_url":"https://space-kitty.com/papers/arxiv:2312.06662","updated":"2023-12-11","pmid":null,"pmcid":null,"openalex_id":"W4403990754","pdf_url":"https://arxiv.org/pdf/2312.06662","abstract":"We present W.A.L.T, a transformer-based approach for photorealistic video generation via diffusion modeling. Our approach has two key design decisions. First, we use a causal encoder to jointly compress images and videos within a unified latent space, enabling training and generation across modalities. Second, for memory and training efficiency, we use a window attention architecture tailored for joint spatial and spatiotemporal generative modeling. Taken together these design decisions enable us to achieve state-of-the-art performance on established video (UCF-101 and Kinetics-600) and image (ImageNet) generation benchmarks without using classifier free guidance. Finally, we also train a cascade of three models for the task of text-to-video generation consisting of a base latent video diffusion model, and two video super-resolution diffusion models to generate videos of $512 \\times 896$ resolution at $8$ frames per second.","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":null,"is_open_access":false,"peer_reviewed":null,"categories":["cs.CV","cs.AI","cs.LG"],"details":null,"provenance":"arxiv","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv."},{"id":"arxiv:2305.19467","kind":"article","title":"Synthetic CT generation from MRI using 3D transformer‐based denoising diffusion model","authors":["Shaoyan Pan","Elham Abouei","Jacob Frank Wynne","Chih‐Wei Chang","Tonghe Wang","Richard L. J. Qiu","Yuheng Li","Junbo Peng","Justin R. Roper","Pretesh R Patel","David S. Yu","Hui Z. Mao","Xiaofeng Yang"],"author_count":13,"year":2023,"published":"2023-11-27","venue":"Medical Physics","cited_by":144,"doi":"10.1002/mp.16847","arxiv_id":"2305.19467","nct_id":null,"source_url":"https://doi.org/10.1002/mp.16847","archive_url":"https://space-kitty.com/papers/arxiv:2305.19467","updated":"2023-05-31","pmid":"38011588","pmcid":null,"openalex_id":"W4389070723","pdf_url":"https://pmc.ncbi.nlm.nih.gov/articles/PMC10994752/pdf/nihms-1947131.pdf","abstract":"Magnetic resonance imaging (MRI)-based synthetic computed tomography (sCT) simplifies radiation therapy treatment planning by eliminating the need for CT simulation and error-prone image registration, ultimately reducing patient radiation dose and setup uncertainty. We propose an MRI-to-CT transformer-based denoising diffusion probabilistic model (MC-DDPM) to transform MRI into high-quality sCT to facilitate radiation treatment planning. MC-DDPM implements diffusion processes with a shifted-window transformer network to generate sCT from MRI. The proposed model consists of two processes: a forward process which adds Gaussian noise to real CT scans, and a reverse process in which a shifted-window transformer V-net (Swin-Vnet) denoises the noisy CT scans conditioned on the MRI from the same patient to produce noise-free CT scans. With an optimally trained Swin-Vnet, the reverse diffusion process was used to generate sCT scans matching MRI anatomy. We evaluated the proposed method by generating sCT from MRI on a brain dataset and a prostate dataset. Qualitative evaluation was performed using the mean absolute error (MAE) of Hounsfield unit (HU), peak signal to noise ratio (PSNR), multi-scale Structure Similarity index (MS-SSIM) and normalized cross correlation (NCC) indexes between ground truth CTs and sCTs. MC-DDPM generated brain sCTs with state-of-the-art quantitative results with MAE 43.317 HU, PSNR 27.046 dB, SSIM 0.965, and NCC 0.983. For the prostate dataset, MC-DDPM achieved MAE 59.953 HU, PSNR 26.920 dB, SSIM 0.849, and NCC 0.948. In conclusion, we have developed and validated a novel approach for generating CT images from routine MRIs using a transformer-based DDPM. This model effectively captures the complex relationship between CT and MRI images, allowing for robust and high-quality synthetic CT (sCT) images to be generated in minutes.","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":null,"is_open_access":true,"peer_reviewed":null,"categories":["Advanced Radiotherapy Techniques"],"details":null,"provenance":"openalex+arxiv","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv."},{"id":"doi:10.1038/s44222-023-00114-9","kind":"review","title":"Diffusion models in bioinformatics and computational biology","authors":["Zhiye Guo","Jian Liu","Yanli Wang","Mengrui Chen","Duolin Wang","Dong Xu","Jianlin Jack Cheng"],"author_count":7,"year":2023,"published":"2023-10-27","venue":"Nature Reviews Bioengineering","cited_by":144,"doi":"10.1038/s44222-023-00114-9","arxiv_id":null,"nct_id":null,"source_url":"https://doi.org/10.1038/s44222-023-00114-9","archive_url":"https://space-kitty.com/papers/doi:10.1038/s44222-023-00114-9","updated":null,"pmid":"38576453","pmcid":null,"openalex_id":"W4388185854","pdf_url":"https://pmc.ncbi.nlm.nih.gov/articles/PMC10994218/pdf/nihms-1980877.pdf","abstract":null,"abstract_license":null,"license":null,"is_open_access":true,"peer_reviewed":null,"categories":["Generative Adversarial Networks and Image Synthesis"],"details":null,"provenance":"openalex","attribution":"Metadata and citation counts: OpenAlex (CC0)."}],"next":"https://space-kitty.com/api/v1/topics/generative-multimodal/papers?sort=top&limit=25&offset=25"}