LIP2: Lip-to-Speech Synthesis

Comparing multi-speaker (LRS3) vs single-speaker voice-converted (LRS3-VC-LJS) training with AV-HuBERT encoder + Flow decoder, evaluated with UnivNet vocoder on LRS3 test set (1321 samples).

Model Noise Scale WER ↓ SECS ↑ ESTOI ↑
LRS30.128.3%0.6650.370
VC-LJS0.128.5%0.8150.411
LRS31.0TBDTBDTBD
VC-LJS1.0TBDTBDTBD
Good Low WER Samples (0% error @ ns=0.1)
"I FORGIVE YOU AND I DO NOT HATE YOU"
8iSpNVj5KMw/00002 · 2.6s
Noise Scale 0.1
Ground Truth
LRS3
WER 0%SECS 0.70ESTOI 0.301
VC-LJS
WER 0%SECS 0.80ESTOI 0.311
Noise Scale 1.0
LRS3
VC-LJS
"I THINK WHAT THAT MEANS IS THAT PEOPLE JUST COULDN'T SEE WHAT WAS IN FRONT OF THEM"
rP7nmdDA1Fg/00006 · 4.0s
Noise Scale 0.1
Ground Truth
LRS3
WER 0%SECS 0.63ESTOI 0.321
VC-LJS
WER 0%SECS 0.91ESTOI 0.403
Noise Scale 1.0
LRS3
VC-LJS
"PROGRESSIVE MOVEMENTS ARE GROWING AND RESISTING WITH TREMENDOUS COURAGE"
JSSc7hYKstI/00008 · 4.9s
Noise Scale 0.1
Ground Truth
LRS3
WER 0%SECS 0.65ESTOI 0.399
VC-LJS
WER 0%SECS 0.89ESTOI 0.576
Noise Scale 1.0
LRS3
VC-LJS
"WE CAN CREATE A DECENTRALIZED DATABASE THAT HAS THE SAME EFFICIENCY OF A MONOPOLY"
RplnSVTzvnU/00003 · 5.7s
Noise Scale 0.1
Ground Truth
LRS3
WER 0%SECS 0.69ESTOI 0.472
VC-LJS
WER 0%SECS 0.91ESTOI 0.518
Noise Scale 1.0
LRS3
VC-LJS
Medium Moderate WER Samples (1-3 errors @ ns=0.1)
"BUT IT'S NOT ABOUT FIRE AND BRIMSTONE EITHER"
ROgFmb3oTLo/00005 · 2.7s
Noise Scale 0.1
Ground Truth
LRS3
WER 13%SECS 0.61ESTOI 0.492
VC-LJS
WER 25%SECS 0.88ESTOI 0.482
Noise Scale 1.0
LRS3
VC-LJS
"BUT THE SECOND OF THE TRANSFORMATIONS THE CLIMATE TRANSFORMATIONS WE HAVE TO DECIDE TO DO"
Gmai4zkKNcM/00006 · 5.2s
Noise Scale 0.1
Ground Truth
LRS3
WER 20%SECS 0.82ESTOI 0.508
VC-LJS
WER 33%SECS 0.82ESTOI 0.502
Noise Scale 1.0
LRS3
VC-LJS
"TEXTING HAS A 100 PERCENT OPEN RATE"
LiUClSItcy0/00001 · 3.2s
Noise Scale 0.1
Ground Truth
LRS3
WER 29%SECS 0.60ESTOI 0.391
VC-LJS
WER 43%SECS 0.88ESTOI 0.588
Noise Scale 1.0
LRS3
VC-LJS
Hard High WER Samples (>3 errors @ ns=0.1)
"SO WE NEED THE SOLUTIONS AND THESE PEOPLE PLAYING THE GAME THEY ARE TRYING OUT"
qYUmI5kGsYk/00001 · 6.0s
Noise Scale 0.1
Ground Truth
LRS3
WER 33%SECS 0.70ESTOI 0.303
VC-LJS
WER 33%SECS 0.88ESTOI 0.444
Noise Scale 1.0
LRS3
VC-LJS
"MORE TRUST IS NOT AN INTELLIGENT AIM IN THIS LIFE"
1PNX6MSdVsk/00005 · 3.7s
Noise Scale 0.1
Ground Truth
LRS3
WER 90%SECS 0.57ESTOI 0.294
VC-LJS
WER 90%SECS 0.81ESTOI 0.282
Noise Scale 1.0
LRS3
VC-LJS
"WE'RE GOING TO START PUTTING AN ENTIRE LAYER OF DIGITAL INFORMATION ON THE REAL WORLD"
H9ZOpQzjukY/00001 · 4.6s
Noise Scale 0.1
Ground Truth
LRS3
WER 40%SECS 0.74ESTOI 0.391
VC-LJS
WER 20%SECS 0.90ESTOI 0.410
Noise Scale 1.0
LRS3
VC-LJS
LIP2 Research · AV-HuBERT + Flow Decoder · UnivNet Vocoder · LRS3 Test Set (1321 samples)