Gestures and Feet Can Tell Us how You Speak! On the Relationships between Voice and Body Language in VR Public Speeches

Effective communication hinges on the speaker's delivery, which comprises two non-verbal channels: vocal performance or speech prosody and body language. The two channels interact in complex ways with words – and with each other. And yet, there remains a significant gap in understanding how prosody is linked to or influenced by body language. Our research addresses this gap, investigating correlations of acoustic-prosodic features with the activity patterns of hands and feet. We conducted a study with 28 participants, tasked with delivering a 1-2 minute oral presentation. Using a VR headset, each participant presented in front of a virtual audience. The results not only confirmed expected correlations but also revealed novel insights: (1) Higher gesture rates correlated with more resonant and powerful voices; (2) increased steps per minute led to more silent pauses in speech and less pronounced acoustic indicators for jaw movements, indicating less clear articulation; (3) involvement of both hands resulted in higher pitch levels; (4) standing mainly on heels led to deeper pitch falls and more variable timbre. We discuss the implications of our results for practical speaking training and, more generally, the multi-channel construction of communicative meaning.