On the Application of Diffusion Models for Simultaneous Denoising and Dereverberation
Adrian Meise, Tobias Cord-Landwehr, Reinhold Haeb-Umbach
Department of Communications Engineering, Paderborn University
Email: meise@nt.upb.de, cord@nt.upb.de, haeb@nt.upb.de
Abstract: Diffusion models have been shown to achieve natural-sounding enhancement of speech degraded by noise or reverberation. However, their simultaneous denoising and dereverberation capability has so far not been studied much, although this is arguably the most common scenario in a practical application. In this work, we investigate different approaches to enhance noisy and/or reverberant speech. We examine the cascaded application of models, each trained on only one of the distortions, and compare it with a single model, trained either solely on data that is both noisy and reverberated, or trained on data comprising subsets of purely noisy, of purely reverberated, and of noisy reverberant speech. Tests are performed both on artificially generated and real recordings of noisy and/or reverberant data. The results show that, when using the cascade of models, satisfactory results are only achieved if they are applied in the order of the dominating distortion. If only a single model is desired that can operate on all distortion scenarios, the best compromise appears to be a model trained on the aforementioned three subsets of degraded speech data.
EARS
Example 1 (p102)
Clean speech

EARS-WHAM
Observation
Denoise
Dereverb
Denoise → Dereverb
Dereverb → Denoise
Joint Model
Mixed Objective


EARS-Reverb
Observation
Denoise
Dereverb
Denoise → Dereverb
Dereverb → Denoise
Joint Model
Mixed Objective


EARS-WHAMR!
Observation
Denoise
Dereverb
Denoise → Dereverb
Dereverb → Denoise
Joint Model
Mixed Objective
Example 2 (p103)
Clean speech

EARS-WHAM
Observation
Denoise
Dereverb
Denoise → Dereverb
Dereverb → Denoise
Joint Model
Mixed Objective


EARS-Reverb
Observation
Denoise
Dereverb
Denoise → Dereverb
Dereverb → Denoise
Joint Model
Mixed Objective


EARS-WHAMR!
Observation
Denoise
Dereverb
Denoise → Dereverb
Dereverb → Denoise
Joint Model
Mixed Objective
Example 3 (p104)
Clean speech

EARS-WHAM
Observation
Denoise
Dereverb
Denoise → Dereverb
Dereverb → Denoise
Joint Model
Mixed Objective


EARS-Reverb
Observation
Denoise
Dereverb
Denoise → Dereverb
Dereverb → Denoise
Joint Model
Mixed Objective


EARS-WHAMR!
Observation
Denoise
Dereverb
Denoise → Dereverb
Dereverb → Denoise
Joint Model
Mixed Objective
Example 4 (p105)
Clean speech

EARS-WHAM
Observation
Denoise
Dereverb
Denoise → Dereverb
Dereverb → Denoise
Joint Model
Mixed Objective


EARS-Reverb
Observation
Denoise
Dereverb
Denoise → Dereverb
Dereverb → Denoise
Joint Model
Mixed Objective


EARS-WHAMR!
Observation
Denoise
Dereverb
Denoise → Dereverb
Dereverb → Denoise
Joint Model
Mixed Objective
Example 5 (p106)
Clean speech

EARS-WHAM
Observation
Denoise
Dereverb
Denoise → Dereverb
Dereverb → Denoise
Joint Model
Mixed Objective


EARS-Reverb
Observation
Denoise
Dereverb
Denoise → Dereverb
Dereverb → Denoise
Joint Model
Mixed Objective


EARS-WHAMR!
Observation
Denoise
Dereverb
Denoise → Dereverb
Dereverb → Denoise
Joint Model
Mixed Objective
Example 6 (p107)
Clean speech

EARS-WHAM
Observation
Denoise
Dereverb
Denoise → Dereverb
Dereverb → Denoise
Joint Model
Mixed Objective


EARS-Reverb
Observation
Denoise
Dereverb
Denoise → Dereverb
Dereverb → Denoise
Joint Model
Mixed Objective


EARS-WHAMR!
Observation
Denoise
Dereverb
Denoise → Dereverb
Dereverb → Denoise
Joint Model
Mixed Objective
REVERB
Near
Example 1
Observation
Denoise
Dereverb
Denoise → Dereverb
Dereverb → Denoise
Joint Model
Mixed Objective


Example 2
Observation
Denoise
Dereverb
Denoise → Dereverb
Dereverb → Denoise
Joint Model
Mixed Objective
Far
Example 1
Observation
Denoise
Dereverb
Denoise → Dereverb
Dereverb → Denoise
Joint Model
Mixed Objective


Example 2
Observation
Denoise
Dereverb
Denoise → Dereverb
Dereverb → Denoise
Joint Model
Mixed Objective
WSJ
Example 1
Clean speech

WSJ-CHiME3
Observation
Denoise
Dereverb
Denoise → Dereverb
Dereverb → Denoise
Joint Model
Mixed Objective


WSJ-Reverb
Observation
Denoise
Dereverb
Denoise → Dereverb
Dereverb → Denoise
Joint Model
Mixed Objective
Example 2
Clean speech

WSJ-CHiME3
Observation
Denoise
Dereverb
Denoise → Dereverb
Dereverb → Denoise
Joint Model
Mixed Objective


WSJ-Reverb
Observation
Denoise
Dereverb
Denoise → Dereverb
Dereverb → Denoise
Joint Model
Mixed Objective