Audio-visual speech synthesis

In many real-world situations, we communicate face-to-face. While speech, encoding language, seems to be the medium by which communication is primarily performed, face-to-face situations provide a considerable amount of what is frequently called nonverbal communication, and it is clear that people attribute significant value to this. In principle, everything in business could be conducted over the telephone, but in fact busy people will travel considerable distances just to engage in face-to-face commimication. 

These models offer a basic set of movements that far exceeds what a real face can achieve, so a research goal has been to find the natural patterns of movements which constitute an effective set of control parameters. A basic problem concerns where in the overall process the control should be manifested. While we may be able to build a realistic lip model that exhibits the correct degrees of freedom for the lips, the lips themselves are the outputs of other more-fundamental controls such as speaking, smiling and other actions governed by other facial movements. An attractive solution, then, is to use more-fundamental controls based on articulatory degrees of freedom. 

An alternative approach is to create models that mimic the actual biomechanical properties of the face, rather than simply rely on deformations of the grid [154], [354], [490], [491], [492]. This is the parallel approach to articulatory synthesis described in Section 13.4, and the pros and cons are just the same. While this in a sense is the proper and ultimate solution, the enormous complexities of the muscle movements involved make it a complex process. Furthermore, as with articulatory synthesis, there is no single solution as to how complex each muscle model should be approaches range from simple models to close mimicry. At present the computational requirements and 

---

Because of these difficulties, there is little engineering work in articulatory synthesis, but it is central in the other areas of speech production, articulator physiology and audio-visual or talking head synthesis. 

---