Restraining and encouraging the use of gestures: exploring the effects on speech

Humans speak and produce hand gestures. It has been shown that hand gestures are a key component of the process of speech production and that the two modalities interact at multiple levels (semantic, pragmatic, temporal and prosodic). The underlying mechanisms and functions of this interaction are still a debated question in the field. This dissertation aims to investigate some of these aspects by exploring the effects of restraining and encouraging the use of gestures on speech production in a narration task. Previous studies have shown that speech and gesture production are controlled by a common motor control system involving a neural network connected to Broca’s area; they are also interdependent at a biomechanical level. This suggests that speech acoustics should be interdependent to body movements/gestures in terms of motoric control and biomechanics as well as communicative and prosodic needs. However, the possible direct effects of both restraining and encouraging the use of gestures on the acoustic features of speech have not received much attention in previous studies. Previous research has investigated how the inability to gesture impacts speech production in terms of e.g., speech fluency, length, and content. However, the mixed results obtained do not allow to draw solid conclusions. Moreover, research is lacking as to how encouraging the use of gesture should affect speech in similar terms. Two empirical studies were carried out with 40 Italian speakers telling short narratives to a listener. In Study 1 (Chapter 2), speakers were restrained from gesturing by sitting on their hands; in Study 2 (Chapter 3) they were encouraged to make gestures while speaking. In both studies, the speech of the target narratives was assessed based on a set of acoustic, prosodic and textual features, and specifically by analyzing speech discourse length (number of words and discourse length in seconds), disfluencies (filled pauses, self-corrections, repetitions, insertions, interruptions), speech rate and acoustic properties (measures of F0 and intensity). Additional qualitative and quantitative analyses on the data collected for Studies 1 and 2 are reported in Chapter 4, which also discusses some aspects of the methods adopted in the studies. The results show that (1) the inability to gesture does not affect speech, which does not become significantly longer, more disfluent or monotonous; however, (2) enhancing the gesture stream by encouraging speakers to gesture can affect speech length and acoustics, as evidenced by an increase in F0 and intensity metrics. Moreover, as shown in Chapter 4, encouraging the use of gesture leads speakers (3) to produce more gestures and in a higher (more salient) gesture space; (4) to make use of more representational gestures and (5) to bodily enact characters and actions more often by using various multimodal cues. Overall, this thesis investigates the potential functions of gesture production in the speech planning and articulation phases and provides evidence that gesture production can enhance some prosodic features of speech in semi-spontaneous narratives. Also, it shows that the inability to gesture is not detrimental to fluent speech production and spoken prosody in narrative speech. Further investigations in this direction would contribute to shed light on if and how gesture and prosodic structures are jointly planned and produced.