A possible way for describing hand gestures in talk shows
Article Sidebar
Main Article Content
Abstract
The main aim of our paper is to introduce a multimodal annotation schema which is suitable for multimodal corpus analysis on the levels of speech and hand gestures, furthermore, it describes the semantic relation between speech and hand gestures. In the first part of the present paper we delineate the annotation schema which is based on McNeill's (1992) and Kendon's (2004) definition of gesture, and it is founded on the gesture semantic taxonomy of Alibali et al. (2001) and Kong et al. (2015). The transcription of speech follows the conventions of conversation analysis by Jefferson (2004). The second part of this paper is a case study which is made for testing the applicability of the suggested annotation schema. The analysed corpus contains political debate shows and the case study focuses on verbal aggression.
Downloads
Article Details
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
References
Abuczki Ágnes 2011. A multimodális interakció szekvenciális elemzése. In Németh T. Enikő (szerk.) Ember-gép kapcsolat. A multimodális ember-gép kommunikáció modellezésének alapjai. (Segédkönyvek a nyelvészet tanulmányozásához 133) Budapest: Tinta Könyvkiadó. 119–144.
Alibali, Martha W. – Dana C. Heath – Heather J. Myers 2001. Effects of visibility between speaker and listener on gesture production: Some gestures are meant to be seen. Journal of Memory and Language 44/2:169–188. doi: https://doi.org/10.1006/jmla.2000.2752
Blache, Philippe – Roxane Bertrand – Emmanuel Bruno – Brigitte Bigi – Robert Espesser et al. 2010. Multimodal annotation of conversational data. Proceedings of the Fourth Linguistic Annotation Workshop. Uppsala: Association for Computational Linguistics. 186–191.
Bressem, Jana 2013. A linguistic perspective on the notation of form features in gestures. In Cornelia Müller – Alan Cienki – Ellen Fricke – Silva Ladewig – David McNeill – Sedinha Tessendorf (szerk.) Body – Language – Communication. An International Handbook on Multimodality in Human Interaction. Vol. 1. Berlin, Boston: De Gruyter. 1079–1098. doi: https://doi.org/10.1515/9783110261318.1079
Bressem, Jana – Silva H. Ladewig – Cornelia Müller 2013. Linguistic Annotation System for Gestures. In Cornelia Müller – Alan Cienki – Ellen Fricke – Silva Ladewig – David McNeill – Sedinha Tessendorf (szerk.) Body – Language – Communication. An International Handbook on Multimodality in Human Interaction. Vol. 1. Berlin, Boston: de Gruyter. 1098–1124. doi: https://doi.org/10.1515/9783110261318.1098
Brook, Rebecca – Maroš Servátka 2016. The anticipatory effect of nonverbal communication. Economics Letters 144:45–48. doi: https://doi.org/10.1016/j.econlet.2016.04.033
Ekman, Paul 1971. Universals and cultural differences in facial expressions of emotion. In James K. Cole (szerk.) Nebraska Symposium on Motivation. Vol. 19. Lincoln: University of Nebraska Press. 207–284.
Ekman, Paul – Wallace V. Friesen 1981. The repertoire of nonverbal behavior: Categories, origins, usage, and coding. In Adam Kendon – Thomas A. Sebeok – Jean Umiker-Sebeok (szerk.) Nonverbal Communication, Interaction, and Gesture. (Approach to Semiotics 41) Berlin, New York: Mouton Publishers. 57–106. doi: https://doi.org/10.1515/9783110880021.57
ELAN 2019. Version 5.8. Nijmegen: Max Planck Institute. https://archive.mpi.nl/tla/elan
Esposito, Anna – Alessandro Vinciarelli – Klára Vicsi – Catherine Pelachaud – Anton Nijholt (szerk.) 2011. Analysis of Verbal and Nonverbal Communication and Enactment. The Processing Issues: COST 2102 International Conference, Budapest, Hungary, September 7-10, 2010. (Lecture Notes in Computer Science 6800) Berlin, Heidelberg: Springer.
Fasola, Alexia – F. -Xavier Alario – Marion Tellier – Bernard Giusiano – Carlo Alberto Tassinari – Fabrice Bartolomei – Agnès Trébuchon 2020. A description of verbal and gestural communication during postictal aphasia. Epilepsy & Behavior 102:106646. doi: https://doi.org/10.1016/j.yebeh.2019.106646
Hall, Edward T. 1963. A system for the notation of proxemic behavior. American Anthropologist 65/5:1003–1026. doi: https://doi.org/10.1525/aa.1963.65.5.02a00020
Hogg, Michael A. – Graham M. Vaughan 2018. Language and Communication. In Michael A. Hogg – Graham M. Vaughan Social psychology. 8. kiadás. Pearson Education. 596–636.
Hunyadi László 2011a. A multimodális ember-gép kommunikáció modellezésének alapjai. In Németh T. Enikő (szerk.) Ember-gép kapcsolat. A multimodális ember-gép kommunikáció modellezésének alapjai. (Segédkönyvek a nyelvészet tanulmányozásához 133) Budapest: Tinta Könyvkiadó. 7–14.
Hunyadi László 2011b. A multimodális ember-gép kommunikáció technológiái. Elméleti modellezés és alkalmazás a beszédfeldolgozásban. In Németh T. Enikő (szerk.) Ember-gép kapcsolat. A multimodális ember-gép kommunikáció modellezésének alapjai. (Segédkönyvek a nyelvészet tanulmányozásához 133) Budapest: Tinta Könyvkiadó. 15–42.
Hunyadi László – Földesi András – Szekrényes István – Kiss Hermina – Abuczki Ágnes – Bódog Alexa 2012. Az ember-gép kommunikáció elméleti-technológiai modellje és nyelvtechnológiai vonatkozásai. In Prószéky Gábor – Váradi Tamás (szerk.) Általános Nyelvészeti Tanulmányok XXIV. Nyelvtechnológiai kutatások. Budapest: Akadémiai Kiadó. 265–309.
Ilie, Cornelia 2001. Semi-institutional discourse: The case of talk shows. Journal of Pragmatics 33/2:209–254. doi: https://doi.org/10.1016/S0378-2166(99)00133-2
Jefferson, Gail 2004. Glossary of transcript symbols with an introduction. In Gene H. Lerner (szerk.) Conversation Analysis: Studies from the First Generation. (Pragmatics & Beyond New Series 125) Amsterdam: John Benjamins Publishing Company. 13–31. doi: https://doi.org/10.1075/pbns.125.02jef
Kendon, Adam 1980. Gesticulation and speech: Two aspects of the process of utterance. In Mary Ritchie Key (szerk.) Nonverbal Communication and Language. Hague: Mouton. 207–277.
Kendon, Adam 2000. Language and gesture: unity or duality? In David McNeill (szerk.) Language and Gesture. (Language Culture and Cognition 2) Cambridge: Cambridge University Press. 47–63. doi: https://doi.org/10.1017/CBO9780511620850.004
Kendon, Adam 2004. Gesture. Visible Action as Utterance. Cambridge: Cambridge University Press
Kendon, Adam – Thomas A. Sebeok – Jean Umiker-Sebeok (szerk.) 1981. Nonverbal Communication, Interaction, and Gesture. (Approaches to Semiotics 41) Hague: De Gruyter Mouton.
Kirk, Lorraine – Michael Burton 1981. Physical versus semantic classification of nonverbal forms: A cross-cultural experiment. In Adam Kendon – Thomas A. Sebeok – Jean Umiker-Sebeok (szerk.) Nonverbal Communication, Interaction, and Gesture. (Approaches to Semiotics 41) Hague: De Gruyter Mouton. 459–482. doi: https://doi.org/10.1515/9783110880021.459
Kita, Sotaro – Asli Özyürek 2003. What does cross-linguistic variation in semantic coordination of speech and gesture reveal? Evidence for an interface representation of spatial thinking and speaking. Journal of Memory and Language 48/1:16–32. doi: https://doi.org/10.1016/S0749-596X(02)00505-3
Kong, Anthony Pak-Hin – Sam-Po Law – Connie Ching-Yin Kwan – Christy Lai – Vivian Lam 2015. A coding system with independent annotations of gesture forms and functions during verbal communication: Development of a Database of Speech and GEsture (DoSaGE). Journal of Nonverbal Behavior 39/1:93–111. doi: https://doi.org/10.1007/s10919-014-0200-6
Kostić, Aleksandra – Derek Chadee (szerk.) 2015. The Social Psychology of Nonverbal Communication. London: Palgrave Macmillan.
Ladewig, Silva H. – Jana Bressem 2013. A linguistic perspective on the notation of gesture phases. In Cornelia Müller – Alan Cienki – Ellen Fricke – Silva Ladewig – David McNeill – Sedinha Tessendorf (szerk.) Body – Language – Communication. An International Handbook on Multimodality in Human Interaction. Vol. 1. Berlin, Boston: de Gruyter. 1060–1079. doi: https://doi.org/10.1515/9783110261318.1060
Mather, Susan M. 2005. Ethnographic research on the use of visually based regulators for teachers and interpreters. In Melanie Metzger – Earl Fleetwood (szerk.) Attitudes, Innuendo, and Regulators: Challenges of Interpretation. Washington, DC: Gallaudet University Press. 131–161.
Matsumoto, David – Mark Frank – Hyi Hwang 2013. Nonverbal Communication: Science and Applications. London: Sage Publications.
McNeill, David 1992. Hand and Mind. What Gestures Reveal about Thought. Chicago: University of Chicago Press.
McNeill, David 2005. Gesture and Thought. Chicago: University of Chicago Press.
McNeill, David 2013. Gesture as a window onto mind and brain, and the relationship to linguistic relativity and ontogenesis. In Cornelia Müller – Alan Cienki – Ellen Fricke – Silva Ladewig – David McNeill – Sedinha Tessendorf (szerk.) Body – Language – Communication. An International Handbook on Multimodality in Human Interaction. Vol. 1. Berlin, Boston: de Gruyter. 28–54. doi: https://doi.org/10.1515/9783110261318.28
Müller, Cornelia – Silva H. Ladewig – Jana Bressem 2013. Gestures and speech from a linguistic perspective: A new field and its history. In Cornelia Müller – Alan Cienki – Ellen Fricke – Silva Ladewig – David McNeill – Sedinha Tessendorf (szerk.) Body – Language – Communication. An International Handbook on Multimodality in Human Interaction. Vol. 1. Berlin, Boston: de Gruyter. 55–81. doi: https://doi.org/10.1515/9783110261318.55
Németh T. Enikő 1989. Az emberi kommunikáció kutatásának néhány alapkérdése. Néprajz és Nyelvtudomány 33:43–56.
Németh T. Enikő 1994. Megnyilatkozás: típus–példány. Néprajz és Nyelvtudomány 35:69–101.
Patterson, Miles L. 1983. Nonverbal Behavior. A Functional Perspective. New York: Springer.
Patterson, Miles L. 1994. Interaction behavior and person perception: An integrative approach. Small Group Research 25/2:172–188. doi: https://doi.org/10.1177/1046496494252002
Ramseyer, Fabian – Wolfgang Tschacher 2010. Nonverbal synchrony or random coincidence? How to tell the difference. In Anna Esposito – Nick Campbell – Carl Vogel – Amir Hussain – Anton Nijholt (szerk.) Development of Multimodal Interfaces: Active Listening and Synchrony: Second COST 2102 International Training School, Dublin, Ireland, March 23–27, 2009. (Lecture Notes in Computer Science 5967) Berlin, Heidelberg: Springer. 182–196. doi: https://doi.org/10.1007/978-3-642-12397-9_15
Seyfeddinipur, Mandana 2006. Disfluency: Interrupting Speech and Gesture. (MPI Series in Psycholinguistics 39) Nijmegen: Max Planck Institute.
Szabó Éva 2020. A nyelvi agresszió megjelenése a politikai vitaműsorokban. Argumentum 16:31–43. doi: https://doi.org/10.34103/ARGUMENTUM/2020/3
Tóth Csilla 2011. Tekintetmintázatok és funkcióik a HuComTech-projekt céljainak tükrében. In Németh T. Enikő (szerk.) Ember-gép kapcsolat. A multimodális ember-gép kommunikáció modellezésének alapjai. (Segédkönyvek a nyelvészet tanulmányozásához 133) Budapest: Tinta Könyvkiadó. 101–118.