DREAM-Talk: Diffusion-based Realistic Emotional Audio-driven Method for

Single Image Talking Face Generation

Chenxu Zhang1*,
Chao Wang1*,
Jianfeng Zhang1,
Hongyi Xu1,
Guoxian Song1,
You Xie1,
Linjie Luo1,
Yapeng Tian2,
Xiaohu Guo2,
Jiashi Feng1,
  1Bytedance Inc.   2The University of Texas at Dallas
arXiv Code

Video Results

▶ Qualitative Comparisons with MakeitTalk, SadTalker and EAMM

▶ Qualitative Comparisons with EVP



Applications

▶ More Characters

Results with Real Human Faces and AIGC-generated Faces.

▶ Multiple Emotions

Results with Various Emotions, Such as Anger, Happy, and Surprise.

▶ A Conversation Across Time and Space

Leonardo predominantly expresses anger, while Mona Lisa exhibits happiness.

▶ Generation of Different Languages

Multiple Language Support for Emotional Talking Faces Generation, Including Chinese, Japanese, French, German, etc.

BibTeX

@inproceedings{zhang2023dreamtalk,
    author = {Zhang, Chenxu and Wang, Chao and Zhang, Jianfeng and Xu, Hongyi and Song, Guoxian and Xie, You and Luo, Linjie and Tian, Yapeng and Guo, Xiaohu and Feng, Jiashi},
    title = {DREAM-Talk: Diffusion-based Realistic Emotional Audio-driven Method for Single Image Talking Face Generation},
    booktile = {arxiv},
    year = {2023}
}