UC Merced Proceedings of the Annual Meeting of the Cognitive Science Society Title Beat gestures facilitate speech production Permalink https://escholarship.org/uc/item/8pw1x0sx Journal Proceedings of the Annual Meeting of the Cognitive Science Society, 36(36) ISSN 1069-7977 Authors Lucero, Che Zaharchuk, Holly Casasanto, Daniel Publication Date 2014 Peer reviewed eScholarship.org Powered by the California Digital Library University of California Beat gestures facilitate speech production Ché Lucero Holly Zaharchuk Daniel Casasanto (chelucero@uchicago.edu) (hzaharchuk@uchicago.edu) (casasanto@uchicago.edu) Department of Psychology, University of Chicago 5848 S. University Ave, Chicago, IL 60637 USA Beat gestures are simple, stereotyped, and repetitive movements. They contrast, therefore, with iconic gestures, which are often novel, elaborated, and complex. These differences mean that beat gestures should be less cognitively taxing for the speaker. Simply moving the hands could raise the activation level of a sought-for word (Butterworth & Hadar 1989; Ravizza, 2003; Alibali & Hostetter 2007), irrespective of the form of the movement. Thus, beats might provide a benefit to the speaker at minimal cost. How can we test whether beat gestures facilitate speech production? Studies seeking evidence that gesturing benefits speech production have primarily relied on gestureprevention paradigms. Typically, participants perform a language production task while their gestures are restricted by sitting on their hands (e.g. Hostetter, Alibali, & Kita, 2007), holding an item (e.g. Frick-Horbury & Guttentag, 1998), having their hands restrained (e.g. Pine, Bird, & Kirk 2007), or are simply instructed not to gesture (e.g. Graham & Heywood, 1975). Performance in the gesture-prevented condition is then compared to a naturalistic condition in which no secondary task interfering with gesture is required. Performance on language production tasks usually worsens when gestures are prevented (but see Beattie & Coughlan 1999). Can data from gesture prevention studies provide evidence that gestures facilitate speech production? Not really. The only inference they can license is that preventing gestures can impair speaking. Gesture prevention conditions are generally less natural than free-gesture conditions, so any observed speech impairment could result from the presence of unnatural task demands, and not from the absence of gesture. More fundamentally, it is not possible even in principle to show that gesturing benefits speaking on the basis of gesture prevention; gestures must be produced, and their impact on speech measured. In order to test for a causal influence of gesture on speech production, a treatment condition in which gesturing is required must be compared with a control condition in which participants are allowed to behave naturally, gesturing or not gesturing at will. In gesture prevention paradigms, the relative naturalness of the conditions works in favor of the hypothesis that gesturing benefits speech, and is therefore a potential source of Type I error (i.e., false positive results). By contrast, when gestures are required the relative naturalness of the conditions works against the hypothesis that gesturing benefits speech. Abstract Does gesturing help speakers find the right words? According to several theories of speech-gesture relationships, iconic gestures should facilitate speech production, but beat gestures should not. Here we tested the effects of gesturing on word production in two experiments. Participants produced lowfrequency words from their definitions while instructed to perform beat gestures, iconic gestures, or while not given any instructions about gesturing (baseline condition). Compared to baseline, participants were faster to produce the target words while performing beat gestures, bimanually or with their left hand alone, but they were slower to produce the target words when instructed to perform iconic gestures. Results provide the first evidence that beat gestures can help speakers produce words. This benefit may arise from the fact that gestures are motor actions, rather than from any special properties of gestures, per se. Keywords: Speech production; Beat gesture; Iconic gesture. Introduction Does gesturing help speakers produce words? Studies addressing this question have tended to focus on iconic gestures, gestures that use the hands to depict some aspect of the referent of the words they accompany (McNeill, 1992). For example, drawing a circle in the air could help speakers produce the word “carousel,” either by crossmodal priming of the word’s form (Krauss, 1998), by helping speakers formulate a pre-verbal message (DeRuiter, 2000), by packaging their thoughts for speech (Alibali, Kita, & Young, 2000), or by helping them maintain a mental image of the word’s referent during lexical search (Wesp, Hesse, Keutmann, & Wheateon, 2001). Unlike iconic gestures, which are taken to be meaningful and believed to benefit speech production, beat gestures which often mark prosodic peaks in speech are taken to be meaningless (i.e., non-referential). They are not depictive, so they cannot contribute to word production in any of the ways that iconics are proposed to help. Some theorists have expressly denied any role for beat gestures in producing words, suggesting that they “do not seem to be involved in lexical search” or other stages of speech production (Krauss & Hadar, 1999). Yet, beat gestures are ubiquitous, they are often produced when speakers are searching for words, and they have been found to occur as often as iconic gestures during successful resolution of tip-of-the-tongue states (Beattie & Coughlan, 1999). We propose that beat gestures could facilitate word production. 898 Here we evaluated the effects of required gesturing on word production to test the long-standing proposal that iconic gestures facilitate speech, and also to test our proposal that beat gestures can help speakers find the right words. In Experiment 1, we assigned participants to perform iconic gestures and beat gestures during a word-naming task, and compared performance in these conditions to performance in a naturalistic control condition in which participants received no instructions about gesturing. To preview our results, only beat gestures facilitated word production relative to control. In Experiment 2, we compared word production during bimanual beats, righthand beats, and left-hand beats against word production in the naturalistic control condition, to investigate the mechanism by which beats facilitate word production. recorded their gestures as well as the computer screen. The IRB approved recording, and all participants consented to recording beforehand. Procedure Word definitions were presented to participants one at a time, in 3 blocks of 30 words each. Each block constituted one of the gesture-instruction conditions, and contained one word list. The assignment of word lists to conditions, and the order of the conditions, were counterbalanced across participants. The order of items within a wordlist was randomized for each participant. Participants were instructed to say aloud the target word that matched the definition. At the start of each trial a blank screen appeared for one second, after which a white fixation cross appeared for two seconds. A word definition in white text 15-point font replaced the fixation cross, and remained until the end of the trial. After eleven seconds, a hint appeared below the definition consisting of the first few letters of the word. After fifteen seconds from definition onset, the trial ended and a new trial began. Participants could hit the spacebar to end the trial if they produced a response before it timed out. In the Iconic Gesture condition, participants were asked to depict the word with their hands as they searched their memory for a word that matched the definition. In the Beat Gesture condition, participants were asked to perform a repetitive bimanual beat gesture. In the No Gesture Instruction condition, no reference to gesturing was made. Videos were played after the instructions to demonstrate the types of gestures required in the Iconic and Beat gesture conditions. For Iconic gestures, videos demonstrating a pantomime (i.e., hammering for the word “hammer”), a depictive gesture (i.e., tracing and arch in the air for the word “arch”), and a metaphoric gesture (i.e., a rightward sweep of the right hand for the word “future”) played, with the target words appearing at the top of the videos. For Beat gestures a video demonstrated a repeated, rhythmic bimanual palm-up-open hand gesture (cf. Müller, 2004). Before each block participants received written instructions, saw one trial demonstrated by the experimenter, and then performed one practice trial themselves. If the participant expressed confusion or performed inappropriately, the experimenter triggered two more practice trials. All participants displayed a good understanding of the task by the end of the practice trials. Experiment 1: Does gesturing help speakers produce words? Experiment 1 tested whether assigning participants to make beat gestures or iconic gestures facilitated their word production relative to a baseline condition in which gestures were neither required nor inhibited. According to previous theories (Krauss, 1998; DeRuiter, 2000; Alibali et al., 2000), gestures help speakers find words by virtue of a resemblance between the form of the gestures and the form of the words’ referents; therefore, only iconic gestures should help. Alternatively, if gestures help speakers simply through movement – which increases arousal or raises the activation level of the sought-for word – then both iconics and beats should be potentially helpful. On this proposal, beats could be even more helpful than iconics because producing beats is perhaps less cognitively taxing than producing iconics. Method Participants Participants (N=38) from the New School community in New York City participated for payment. Two participants were excluded prior to analysis, one for having severe difficulties with speech production and social interaction during greeting and consent,, and the other for not following instructions. Data from the remaining 36 participants were analyzed. Materials We created three lists of thirty word definitions. Each item defined a low-frequency, highly concrete and highly imageable word (e.g., tomahawk). Words and their norms were drawn from two published databases (Coltheart, 1981; Paivio, Yuille, & Madigan, 1968). The majority of words had a Thorndike-Lorge written frequency of less than 10 instances per million, concreteness ratings above 5, and imageability ratings above 5 on a scale of 1 to 7. Due to an error, two lists had one item that was displayed twice for the first 17 participants. When the error was discovered, the duplicate items were replaced. Only the first presentation of the duplicated item was analyzed. Materials were presented on a 27-inch iMac in 1024x768 resolution using a custom python script. A camcorder visible to the participants Results and Discussion Reaction Times (RTs) were defined as the latency between the appearance of a definition at the start of a trial and the successful production of the target word, disregarding any disfluencies (i.e., fillers, false starts). Only words produced before the hint was given were coded as successful. RT coding was done manually by one of the experimenters using the ELAN software package (Brugman & Russel, 2004). Sixty successful trials (twenty from each of the three conditions) were selected at random for recoding by a second coder, who was blind to the experimental 899 significantly slower than in the Beat condition (χ2(1)=25.3, p=.01). Whereas iconic gestures impaired the production of correct definitions relative the No Instruction baseline condition, beat gestures facilitated word production. The beneficial effect of beats, which cannot be attributed to a speed-accuracy tradeoff, was found despite the fact that the beat gesture condition was arguably less natural and more demanding (i.e., a dual-task condition) compared to the baseline condition. We tested for a Condition by Block interaction to confirm that the effect of gesturing on RTs did not depend on the order in which participants performed the three blocks (χ 2(4)<1, ns.). hypotheses. Inter-coder reliability was high (r=.92, p=.01). On approximately 1% of trials in the gesture conditions, participants did not produce a gesture. To be maximally conservative these trials were included, since their inclusion works against our ability to detect an effect of the gesture conditions. Analyses of both experiments used linear and logistic mixed-effects regressions with the lme4 package (R Core Team, 2013). We used likelihood ratio tests (LRTs) to test for fixed effects, with post-hoc contrasts performed on subsets of the data. Omnibus analyses of RT in Experiment 2 failed to converge when including random intercepts and Condition slopes for Subject and Item. We simplified the random effects structure of the omnibus analyses by dropping the correlation between the random intercept and random Condition slope for Subject (Barr, Levy, Scheepers, & Tily, 2013, p. 45). All other analyses included random intercepts and Condition slopes for Subject and Item. Experiment 2: How does beat gesturing help speakers produce words? Why did beat gesturing help speakers produce definitions in Experiment 1? We conducted Experiment 2 to evaluate three possibilities, all of which follow from Kinsbourne’s (1973) theory of activation overflow: when activity in one brain area is increased, activity in connectively “nearby” (often ipsilateral) areas may also be increased, even if these areas are functionally unrelated. Like performing other motor actions, performing beat gestures requires an increase in activity in the brain’s motor system, which could result in an increase in overall brain activity, thus raising the activation level of any brain areas involved in retrieving the sought-for word. We call this first possibility the Global Activation Hypothesis. Alternatively, because moving one hand selectively increases activation in the contralateral hemisphere, it is possible that activation in one hemisphere or the other, alone, could facilitate word production. Language is generally lateralized to the left hemisphere; perhaps beating with the right hand increases activity in left hemisphere language circuits. We call this second possibility the Left Hemisphere Activation Hypothesis. Finally, the right hemisphere has been implicated in processing difficult language (e.g., Yang, Edens, Simpson, & Krawczyk, 2009). Producing low-frequency words from their definitions could rely on right-hemisphere circuits for retrieving distant semantic associations (Goldstein, Revivo, Kreitler, & Metuki, 2010), and beat gesturing with the left hand could increase activity in these circuits. We call this third possibility the Right Hemisphere Activation Hypothesis. The Global Activation Hypothesis would be supported if bimanual gesturing were found to facilitate word production (compared to baseline) more than gesturing with either hand, alone. The Left Hemisphere Activation Hypothesis would be supported if beating with the right hand produced the fastest RTs, and the Right Hemisphere Activation Hypothesis would be supported if beating with the left hand produced the fastest RTs. Figure 1: Results of Experiment 1. Mean RTs to generate target words when participants were instructed to produce beat gestures (left), iconic gestures (right), or were given no gesture instructions (middle). Error bars show SEM corrected for within-subject comparisons (Morey, 2008). Accuracy The percentage of correct trials did not differ across conditions (Beat: 65% ±1%; No Instruction: 54% ±2%; Iconic: 65% ±2%; χ2(2)=1.1, p=.58). Errors are SEM corrected for within-subject comparisons (Morey, 2008). Reaction Times We analyzed RTs only for successful trials, defined as trials for which the participant produced the target word without first receiving a hint. RTs differed significantly across the three conditions (χ2(2)=22.1, p=.01; fig. 1). Target words were produced faster in the Beat condition than in the No-Instruction condition (χ2(1)=4.4, p=.04). By contrast, targets were produced slower in the Iconic condition than in the No-Instruction condition (χ2(1)=8.5, p=.01). RTs in the Iconic condition were also 900 Reaction Times RTs differed significantly across the four conditions (χ2(3)=97.1, p=.01). Target words were produced faster in the Bimanual beat condition than in the NoInstruction condition, replicating this novel finding from Experiment 1 (χ2(1)=4.7, p=.03). RTs in the Bimanual beat condition were also faster than in the Right-hand condition (χ2(1)=4.0, p=.04). RTs in the Left-hand beat condition were indistinguishable from RTs in the Bimanual condition (χ2(1)=.2, p=.68), and were marginally faster than in the No-Instruction condition (χ2(1)=2.8, p=.09) and the Righthand beat condition (χ2(1)=2.2, p=.13). RTs in the Righthand beat condition were indistinguishable from RTs in the No-Instruction condition (χ2(1)<1, ns.). No block by condition interaction was found (χ2(19)<1, ns.) To summarize, bimanual beats facilitated word production compared to both right-hand beats and the naturalistic control condition. The effect of left-hand beats was indistinguishable from the effect of bimanual beats. RTs to produce words in the left-hand beat condition were marginally faster than in both the right-hand beat and the control conditions. RTs in the right-hand beat condition were no faster than the naturalistic control condition. This pattern of results supports the Right Hemisphere hypothesis. Method Participants Participants (N=34) were recruited from the University of Chicago community.Two participants were left-handed. Two participants were excluded prior to analysis, one for showing signs of developmental abnormality, and the other due to a failure of the recording equipment. Data from the remaining 32 participants were analyzed. Materials We used the materials from Experiment 1, and added a new 30-item wordlist constructed with the same sources and criteria. Procedure The Experiment 2 procedure was similar to Experiment 1, with the following changes. There were four blocks, four conditions, and four wordlists. The four gesture instruction conditions were No Gesture Instruction, Bimanual Beat, Right Hand Beat, and Left Hand Beat. Counterbalancing was done with orthogonal Latin squares. This provided sixteen unique condition/wordlist orders, each of which was used twice across the thirty-two participants. General Discussion Does gesturing help speakers find the right words? When presented with a definition of a low-frequency word, participants instructed to perform beat gestures either bimanually (Experiments 1 & 2) or with their left hand (Experiment 2) produced the target word more quickly than when they had no gesture instructions. When instructed to produce iconic gestures, speakers were slower to produce words than when instructed to beat or when given no instructions to gesture (Experiment 1). These results provide no support for the proposal that iconic gestures help speakers find words, but provide the first evidence that beat gestures can facilitate word production, even when compared to a naturalistic baseline condition. Why did beat gestures help? The benefit of producing beat gestures appears to be driven by the left hand more than the right. This may have to do with activation of the right cerebral hemisphere. The right hemisphere is reported to be involved in abstract semantic processing (Beeman et al, 1994), creative thinking (Razumnikova, 2007), and understanding novel metaphors (Bottini et al, 1994). Right hemisphere involvement has also been implicated in the processing of definition-like sentences (Yang et al, 2009). Thus, left-hand beat gestures might help speakers produce target words given their definitions because moving the left hand increases neural activity in the right hemisphere. Beat gestures might cause speakers to find words more quickly simply because they are motor actions: not because they are gestures, per se. Bimanual tapping was found to help participants find words relative to baseline task in Figure 2: Results of Experiment 2. Mean RTs to generate target words when participants were instructed to produce beat gestures bimanually, with the left hand, with the right hand, or were given no gesture instructions. Error bars show SEM corrected for within-subject comparisons (Morey, 2008). Results and Discussion Accuracy The percentage of correct trials did not differ across conditions (Bimanual: 68% ±2%; Left: 69% ±2%; Right: 68% ±2%; No Instruction: 73% ±2%; χ2(3)=2.1, p=.56). Errors are SEM corrected for within-subject comparisons (Morey, 2008). 901 which movements were inhibited (Ravizza, 2003). Squeezing a ball with the left hand (but not the right) has been reported to increase divergent thinking and creativity (Goldstein, et al., 2010). Our results are compatible with these previous studies showing that simple repetitive motor actions can have cognitive benefits. access using the tip‐of‐the‐tongue phenomenon. British Journal of Psychology, 90(1), 35-56. Beeman, M., Friedman, R. B., Grafman, J., Perez, E., Diamond, S., & Lindsay, M. B. (1994). Summation priming and coarse semantic coding in the right hemisphere. Journal of Cognitive Neuroscience, 6(1), 2645. Bottini, G., Corcoran, R., Sterzi, R., Paulesu, E., Schenone, P., Scarpa, P., & Frith, D. (1994). The role of the right hemisphere in the interpretation of figurative aspects of language A positron emission tomography activation study. Brain, 117(6), 1241-1253. Bowden, E. M., & Beeman, M. J. (1998). Getting the right idea: Semantic activation in the right hemisphere may help solve insight problems. Psychological Science, 9(6), 435-440. Brugman, H., & Russel, A. (2004, May). Annotating Multimedia/Multi-modal Resources with ELAN. In LREC. Butterworth, B., & Hadar, U. (1989). Gesture, speech, and computational stages: A reply to McNeill. Psychological Review, 96, 168-174 Coltheart, M. (1981). The MRC psycholinguistic database. The Quarterly Journal of Experimental Psychology, 33(4), 497-505. De Ruiter, J. (2000). The production of gesture and speech. Language and Gesture (pp. 284–311). Cambridge: Cambridge University Press Hostetter, A. B., & Alibali, M. W. (2007). Raise your hand if you’re spatial: Relations between verbal and spatial skills and gesture production. Gesture, 7(1), 73-95. Hostetter, A. B., Alibali, M. W., & Kita, S. (2007). Does sitting on your hands make you bite your tongue? The effects of gesture prohibition on speech during motor descriptions. In Proceedings of the 29th annual meeting of the Cognitive Science Society (pp. 1097-1102). Mawhah, NJ: Erlbaum. Frick-Horbury, D., & Guttentag, R. E. (1998). The effects of restricting hand gesture production on lexical retrieval and free recall. The American Journal of Psychology, 4362. Goldstein, A., Revivo, K., Kreitler, M., & Metuki, N. (2010). Unilateral muscle contractions enhance creative thinking. Psychonomic Bulletin & Review, 17(6), 895899. Graham, J. A., & Heywood, S. (1975). The effects of elimination of hand gestures and of verbal codability on speech performance. European Journal of Social Psychology, 5(2), 189-195. Kinsbourne, M. (1973). The control of attention by interaction between the cerebral hemispheres. In S. Kornblum (Ed.), Attention and Performance IV. New York: Academic Press. Krauss, R. M. (1998). Why do we gesture when we speak?. Current Directions in Psychological Science, 7(2), 54-60. Krauss, R. M., & Hadar, U. (1999). The role of speechrelated arm/hand gestures in word retrieval. Gesture, speech, and sign, 93-116. Why didn’t iconic gestures help? Why did instructing participants to gesture iconically slow them down? Like gesture prevention tasks, gesture induction tasks impose a dual-task penalty. The finding that required iconic gestures slow speakers down should not be interpreted to mean that spontaneous iconic gestures necessarily have the same effect. Spontaneous iconic gestures may impose fewer costs on speakers than required gestures do. Several previous studies have been interpreted as showing that spontaneous iconic gestures benefit the speaker (Alibali et al, 2000; Krauss, 1998; DeRuiter, 2000; Wesp et al, 2001), as well as the listener (Yap, So, Yap, Tan, & Teoh, 2011). No previous study has compared speech production during iconic gesturing to production during a naturalistic baseline task during which gestures were neither required nor inhibited; as such, it remains an open question whether, and under what circumstances, iconic gestures help speakers speak. Conclusions Beat gestures – even artificially imposed beat gestures – can help speakers produce low-frequency words. This benefit appears to depend more on left-hand beats than on righthand beats, possibly because performing gestures (and presumably performing other motor actions) with the left hand potentiates right-hemisphere circuits involved in difficult language processing. There may be multiple mechanisms by which gestures of different types influence speech production. Gestures serve communicative functions, but more fundamentally, they are a kind of motor action. The effect of beat gestures on speech production may arise from broad principles of motor action and neural connectivity that are not peculiar to gesture. Acknowledgments Research was funded by a James S. McDonnell Foundation Scholar Award (#220020236; http://www.jsmf.org) to D.C. References Alibali, M. W., Kita, S., & Young, A. J. (2000). Gesture and the process of speech production: We think, therefore we gesture. Language and Cognitive Processes, 15(6), 593613. Barr, D. J., Levy, R., Scheepers, C., & Tily, H. J. (2013). Random effects structure for confirmatory hypothesis testing: Keep it maximal. Journal of Memory and Language, 68(3), 255-278. Beattie, G., & Coughlan, J. (1999). An experimental investigation of the role of iconic gestures in lexical 902 McNeill, D. (1992). Hand and mind: What gestures reveal about thought. University of Chicago Press. Morey, R. D. (2008). Confidence intervals from normalized data: A correction to Cousineau (2005). Tutorials in Quantitative Methods for Psychology, 4(2), 61-64. Müller, C. (2004). Forms and uses of the Palm Up Open Hand: A case of a gesture family. In C. Müller & R. Posner (Eds.), The semantics and pragmatics of everyday gestures, Berlin: Weidler Verlag. Paivio, A., Yuille, J. C., & Madigan, S. A. (1968). Concreteness, imagery, and meaningfulness values for 925 nouns. Journal of Experimental Psychology, 76(1p2), 1. Pine, K. J., Bird, H., & Kirk, E. (2007). The effects of prohibiting gestures on children's lexical retrieval ability. Developmental Science, 10(6), 747-754. R Core Team. (2013). R: A language and environment for statistical computing [Computer software manual]. Vienna, Austria. Available from http://www.Rproject.org/ Ravizza, S. (2003). Movement and lexical access: Do noniconic gestures aid in retrieval? Psychonomic Bulletin & Review, 10(3), 610-615. Razumnikova, O. M. (2007). Creativity related cortex activity in the remote associates task. Brain research bulletin, 73(1), 96-102. Wesp, R., Hesse, J., Keutmann, D., & Wheaton, K. (2001). Gestures maintain spatial imagery. The American journal of Psychology, 114, 591-600. Yang, F. G., Edens, J., Simpson, C., & Krawczyk, D. C. (2009). Differences in task demands influence the hemispheric lateralization and neural correlates of metaphor. Brain and Language, 111(2), 114-124. Yap, D. F., So, W. C., Melvin Yap, J. M., Tan, Y. Q., & Teoh, R. L. S. (2011). Iconic gestures prime words. Cognitive Science, 35(1), 171-183. 903