COMMUNICATION THROUGH DIAGRAMS: DIVISION OF LABOR BETWEEN GESTURES AND ARROWS A THESIS SUBMITTED TO THE GRADUATE SCHOOL OF INFORMATICS INSTITUTE OF THE MIDDLE EAST TECHNICAL UNIVERSITY BY MELDA COSKUN IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF MASTER OF SCIENCE IN THE DEPARTMENT OF COGNITIVE SCIENCE SEPTEMBER 2014 COMMUNICATION THROUGH DIAGRAMS: DIVISION OF LABOR BETWEEN GESTURES AND ARROWS Submitted by Melda CoĢkun in partial fulfillment of the requirements for the degree of Master of Science in Cognitive Science, Middle East Technical University by, Prof. Dr. Nazife Baykal Director, Informatics Institute Prof. Dr. Cem BozĢahin Head of Department, Cognitive Science Assist. Prof. Dr. Cengiz Acartürk Supervisor, Cognitive Science, METU Examining Committee Members: Prof. Dr. Deniz Zeyrek BozĢahin Cognitive Science, METU Assist. Prof. Dr. Cengiz Acartürk Cognitive Science, METU Assist. Prof. Dr. Murat Perit Çakır Cognitive Science, METU Assoc. Prof. Dr Annette Hohenberger Cognitive Science, METU Assist. Prof. Dr. Erol Özçelik Department of Computer Engineering, Atılım University Date: 03.09.2014 I hereby declare that all information in this document has been obtained and presented in accordance with academic rules and ethical conduct. I also declare that, as required by these rules and conduct, I have fully cited and referenced all material and results that are not original to this work. Name, Last name : Melda Coşkun Signature : iii ABSTRACT COMMUNICATION THROUGH DIAGRAMS: DIVISION OF LABOR BETWEEN GESTURES AND ARROWS Melda CoĢkun MSc., Department of Cognitive Science Supervisor: Assist. Prof. Dr. Cengiz Acartürk September 2014, 72 pages In this study, we aim at studying the relation between arrow production and gesture production in various multimedia teaching environments. Twenty-four participants, who were all experienced educators, were asked to teach a topic of their choice, as if he teaches to a learner audience. They taught the same topic in the same way in three experimental conditions: (i) Teaching on the board, (ii) Pen-and-paper teaching, (iii) Teaching by a tablet. All experiment sessions were videotaped. For the analysis, gestures and arrows were annotated. Arrows were classified into three groups: deictic arrows direct attention to the specific area, relational arrows connect two representations, and iconic arrows present motion, force, physical representations, and processes in a depiction. Gestures were also divided into three groups: deictic gestures point to entities, iconic gestures present picturable aspects of semantic content, and finally beat gestures are speech-related rhythmic hand movements. The results indicated that speakers produced more gestures and fewer arrows in the board session. This difference decreased in the paper session and reversed itself in the tablet session. Similar trends were also found in the relation between deictic and iconic gestures together compared to all arrows, and iconic gestures compared to iconic arrows. These results support the hypothesis that there is a trade-off between arrow and gesture production. These results also suggest that arrows and gestures are used for similar communicative purposes in different modalities. Keywords: Gesture Production, Arrow Production, Multimodal Communication, Diagrams iv ÖZ DAYAGRAMLAR ARACILIĞIYLA ĠLETĠġĠM: EL HAREKETLERĠ VE OKLAR ARASINDAKĠ ĠġBĠRLĠĞĠ Melda CoĢkun MSc., Department of Cognitive Science Supervisor: Yrd. Doç. Dr. Cengiz Acartürk Eylül 2014, 72 sayfa Bu çalıĢmada çeĢitli multimedia öğretme ortamlarında kullanılan oklar ile el hareketleri arasındaki iliĢkiyi incelemeyi amaçlıyoruz. Eğitim alanında deneyimli 24 katılımcıdan seçtikleri herhangi bir konuyu öğrenmek isteyen dinleyiciler varmıĢ gibi öğretmelerini istedik. Aynı konuyu üç farklı deney koĢulunda aynı Ģekilde anlattılar: (i) Tahtada, (ii) Kağıt-Kalem kullanarak, (iii) Tablet kullanarak. Tüm deney oturumlarının video kaydı alındı. Analiz kısmında, kullanılan el hareketleri ve oklar belirlendi. Oklar üç grupta sınıflandırıldı: iĢaret eden oklar dikkati belirli bir alana yönlendirirler, iliĢkisel oklar iki temsili iliĢkilendirirler ve ikonik oklar hareket, kuvvet, fizikle ilgili gösterimler ve iĢlemsel gösterimleri temsil ederler. El hareketleri de üç gruba ayrıldı: iĢaret eden el hareketleri varlıkları gösterirler, ikonik el hareketleri içeriğin resmedilebilir bileĢenlerini gösterirler ve son olarak vurgu el hareketleri konuĢmanın ritmiyle iliĢkili el hareketleridir. Sonuçlar gösterdi ki konuĢmacılar tahtada ders anlatıkları oturumda daha çok el hareketi ve daha az ok kullandılar. Bu fark kalem-kağıt kullanılan oturumda azaldı ve tablet kullanılan oturumda daha çok azaldı . Benzer eğilimler ayrıca iĢaret eden ve ikonik el hareketleri birlikte tüm oklarla karĢılaĢtırınca ve ikonik oklar ikonik el hareketleriyle karĢılaĢtırılınca da bulundu. Bu sonuçlar ok ve el hareketi kullanımı arasında bir iĢbirliği (takas) olduğu hipotezini destekliyor. Ayrıca bu sonuçlar ok ve el hareketlerinin aynı iletiĢimsel amaç için farklı tiplerde kullanıldığını öneriyor. Anahtar Kelimeler: Gesture Kullanımı, Ok Kullanımı, Farlı Tiplerin EtkileĢimi v To my family; mom and dad, my husband, Fırat Dilara and Barış vi