SIMULATING GAZE BEHAVIOR OF VIRTUAL CROWDS BY PREDICTING INTEREST POINTS a thesis submitted to the graduate school of engineering and science of bilkent university in partial fulfillment of the requirements for the degree of master of science in computer engineering By Umut A˘gıl July 2016 SIMULATING GAZE BEHAVIOR OF VIRTUAL CROWDS BY PREDICTING INTEREST POINTS By Umut A˘gıl July 2016 Wecertifythatwehavereadthisthesisandthatinouropinionitisfullyadequate, in scope and in quality, as a thesis for the degree of Master of Science. Ug˘ur Gu¨du¨kbay (Advisor) ¨ Ozgu¨r Ulusoy ˙ Veysi I¸sler Approved for the Graduate School of Engineering and Science: Levent Onural Director of the Graduate School ii ABSTRACT SIMULATING GAZE BEHAVIOR OF VIRTUAL CROWDS BY PREDICTING INTEREST POINTS Umut A˘gıl M.S. in Computer Engineering Advisor: U˘gur Gu¨du¨kbay July 2016 Creating realistic crowd behavior is one of the major goals in crowd simula- tions. Simulating gaze behavior and predicting interest points of virtual charac- ters play a significant role in creating believable scenes, however this aspect has not received much attention in the field. This study proposes a saliency model, which enables virtual agents to produce gaze behavior. The model measures the effects of distinct pre-defined saliency features that are implemented by examin- ing the state-of-the-art perception studies. When predicting an agent’s interest point, we compute the saliency scores by using a weighted sum function for other agents and environment objects in the field of view of the agent for each frame. Then we determine the most salient entity in the virtual scene according to the viewer agent by comparing the scores. We execute this process for each agent in the scene, thus agents gain a visual understanding about their environment. Be- sides, our model introduces new aspects to crowd perception, such as perceiving characters as groups of people, gaze copy phenomena and effects of agent velocity on attention. For evaluation, we compare the resulting saliency gaze model with real world crowd behavior in captured videos. In the experiments, we simulate the gaze behavior in real crowds. The results show that the proposed approach generates plausible gaze behaviors and is easily adaptable to varying scenarios for virtual crowds. Keywords: crowd simulation, saliency, gaze behavior, perception, interest point detection, gaze copy. iii ¨ OZET ˙ ˙ ˙ ILGI NOKTALARINI TAHMIN EDEREK SANAL ˙ ˙ KALABALIKLAR IC¸IN BAKIS¸ DAVRANIS¸I ˙ ¨ SIMULASYONU Umut A˘gıl Bilgisayar Mu¨hendisli˘gi, Yu¨ksek Lisans Tez Danı¸smanı: Ug˘ur Gu¨du¨kbay Temmuz 2016 Ger¸cekc¸i kalabalık insan grubu davranı¸sı yaratmak kalabalık simu¨lasyonlarının en temel amac¸larından birisi olarak yer alır. Sanal karakterlerin bakı¸s davranı¸sını simu¨le etmek ve ilgi ¸cekici noktaları tahmin etmek inandırıcı sahneler olu¸sturmada ¨onemli rol oynamaktadır. Fakat bu yakla¸sım bu zamana kadar bil- gisayar grafig˘i alanında yeteri kadar ilgi ¸cekmemi¸stir. Bu ¸calı¸sma sanal karakter- lerin bakı¸s davranı¸sı u¨retmesini sag˘layan bir dikkat ¸cekicilik modeli sunmaktadır. Model algı u¨zerine yapılmı¸s gu¨ncel ¸calı¸smaları ele alarak dikkat ¸cekici unsurlar belirler ve bu unsuların etkilerini o¨l¸cer. Bir karakterin ilgisini c¸eken noktayı tah- min ederken sahnede karakterin go¨ru¨¸s alanınında bulunan di˘ger karakterler ve nesnelerin dikkat ¸cekme skorları bir ag˘ırlıklı toplam fonksiyonu ile hesaplanır. Ardından bu skorlar kar¸sıla¸stırılarak izleyici karaktere g¨ore sahnedeki en dikkat ¸cekici varlık bulunur. Bu i¸slem sahnedeki her bir karakter i¸cin hesaplanır ve bo¨ylece tu¨m karakterler ¸cevreleri hakkında g¨orsel bir algıya sahip olurlar. Bunun ¨ yanında, modelimiz kalabalık algısına yeni bakı¸s ac¸ıları da getirmektedir. Ornek olarak, kalabalıkların gruplar halinde ele alınması, bakı¸s kopyalama, hızın dikkate ¨ etkisi verilebilir. Onerilen modelin deg˘erlendirmesi i¸cin elde etti˘gimiz dikkat ¸cekme modelini videolardan elde ettig˘imiz ger¸cek du¨nya kalabalık davranı¸s ve- rileri ile kar¸sıla¸stırdık. Hazırlanan senaryolarda, ger¸cek kalabalık davranı¸slarının simu¨lasyonunu yaptık. Elde edilen sonu¸clar modelin ger¸cek¸ci bakı¸s davranı¸sı sunan ve farklı senaryolara kolaylıkla uygulanabilir oldug˘unu go¨stermektedir. Anahtar s¨ozcu¨kler: kalabalık simu¨lasyonu, dikkat c¸ekme, bakı¸s davranı¸sı, algı, ilgi ¸cekici noktaların tespit edilmesi, bakı¸s kopyalama. iv Acknowledgement First and foremost, I would like to express my gratitude to my supervisor Prof. Dr. U˘gur Gu¨du¨kbay. Without his guidance and assistance, it would not be possible to complete this thesis. ¨ I also would like to thank to rest of my thesis committee, Prof. Dr. Ozgu¨r ˙ Ulusoy and Prof. Dr. Veysi I¸sler for evaluating this thesis. Besides, I am grateful to Dr. Erdal Yılmaz for his advice with the development of the framework and the evaluation of the model. Thanks to Tarık Yal¸cın and Sec¸kin Yal¸cın for their assistance on creating virtual scenes and preparing videos. I would like to thank to Onur Polat for his companionship during this process and for his help in capturing videos. ˙ I am grateful to my friends Volkan, Faruk, Emre, Kubilay, Gu¨lfem, Istemi, Olcay, Serkan, Yusuf and all the people who have shared their valuable time. Special thanks to the members of ”Motor Oil Co.”, who have always kept my spirit high, for all the goy they have given. Finally, I must express my sincere gratitude to my family, who have supported me throughout my years of study and through the entire thesis process. This accomplishment would not have been possible without their unconditional love. v Contents 1 Introduction 1 1.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 1.2 Research Questions . . . . . . . . . . . . . . . . . . . . . . . . . . 4 1.3 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 2 Background and Related Works 6 2.1 Saliency in Computer Graphics . . . . . . . . . . . . . . . . . . . 6 2.2 Crowd Simulations and Crowd Models . . . . . . . . . . . . . . . 7 2.3 Attention Models and Applications on Crowds . . . . . . . . . . . 9 2.4 Human Visual System and Gaze Behavior . . . . . . . . . . . . . 11 2.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 3 Simulating Interest Point Detection and Gaze Behavior 13 3.1 Interest Point Detection . . . . . . . . . . . . . . . . . . . . . . . 14 3.1.1 Attributes of Characters and Objects . . . . . . . . . . . . 15 vi CONTENTS vii 3.1.2 Saliency Score Computation Parameters . . . . . . . . . . 16 3.1.3 Saliency Scoring . . . . . . . . . . . . . . . . . . . . . . . . 17 3.2 Adjusting Gaze . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 3.2.1 Gaze Copy . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 3.2.2 Gaze Shifting Animation . . . . . . . . . . . . . . . . . . . 28 3.2.3 Gaze Duration . . . . . . . . . . . . . . . . . . . . . . . . 28 4 Implementation of the Framework 30 4.1 Graphics Models . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 4.2 Animation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 4.3 Architecture of the Simulation . . . . . . . . . . . . . . . . . . . . 32 5 Evaluation and Results 35 5.1 Scenario Simulations . . . . . . . . . . . . . . . . . . . . . . . . . 35 5.1.1 Scenario 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 5.1.2 Scenario 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 5.1.3 Scenario 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 5.1.4 Scenario 4 . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 5.1.5 Scenario 5 . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 5.1.6 Scenario 6 . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 CONTENTS viii 5.1.7 Scenario 7 . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 5.1.8 Scenarios 8 . . . . . . . . . . . . . . . . . . . . . . . . . . 42 5.1.9 Scenarios 9 . . . . . . . . . . . . . . . . . . . . . . . . . . 43 5.2 Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 6 Conclusion 48 Bibliography 50 List of Figures 1.1 Gaze behavior in popular computer games . . . . . . . . . . . . . 3 4.1 The relationships between agent components. . . . . . . . . . . . 32 4.2 Gaze decision logic. . . . . . . . . . . . . . . . . . . . . . . . . . . 33 5.1 Effects of proximity and periphery: (a) video, (b) simulation, and (c) simulation (weight=0). . . . . . . . . . . . . . . . . . . . . . . 36 5.2 Effects of agent velocity: (a) video, (b) simulation, and (c) simu- lation (weight=0). . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 5.3 Effects of agent attractiveness: (a) video, (b) simulation, and (c) simulation (agent’s attractiveness=0.4). . . . . . . . . . . . . . 38 5.4 Effects of distinctiveness, curiosity and height: (a) video, (b) simu- lation, (c) distinctiveness = 0, (d) curiosity=0, and (e) child height = 1.65. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 5.5 Still frames showing the effect of agent shyness: (a) video frames, (b) shyness = 0.25, (c) shyness = 0.7. . . . . . . . . . . . . . . . . 40 5.6 Effects of object distinctiveness: (a) video, (b) simulation, and (c) simulation (weight=0). . . . . . . . . . . . . . . . . . . . . . . 41 ix LIST OF FIGURES x 5.7 Still frames showing the effects of gaze copy: (a) video frames and (b) simulation frames. . . . . . . . . . . . . . . . . . . . . . . . . 42 5.8 Effects of periphery: first row: the real video, second row: the simulation result. . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 5.9 Still frames from a 40 second video: left column: the real video, right column: the simulation result. . . . . . . . . . . . . . . . . . 44 5.10 Graphs depicting (a) maximum frame computation times (msecs) and (b) frame rates (frames per second) for different number of agents. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 5.11 A crowd simulation with 100 agents. Gaze behavior scripts are (a) disabled and (b) enabled. . . . . . . . . . . . . . . . . . . . . . 47