0 1 _ P r e fa c e 0 Natural user interfaces 2 _ In tr o d u c tio n from all angles 0 3 _ T h e P r o d u c tio n An investigation of interaction methods 0 4 using depth sensing cameras _ C o n tr ib u tio A Medialogy project by n Michael Birkehøj Jensen 0 5 _ D is c u s s io n 0 6 _ R e fe r e n c e s ii Department of Architecture, Project title: Design and Media Technology Natural user interfaces from all angles Medialogy, 10th Semester Project Period: Abstract: 8-2-2011 / 31-5-2011 With the emergence of natural user interfaces a new task of design- ing meaning full interaction has come along. Would it make sense to Semester Theme: not have to get up to get the remote control, if you had to get up anyway to interact with an interface controlled by a camera... This Master Thesis project investigated the possibilities of natural interfaces using depth sensing camera technology to control a media center like interface. Supervisor: The project proposes new methods of interaction using only the body as a controller, which supposedly should perform better at all Mads Græsbøl Christensen angles to the interface. This means that the user would not have to get up, but would be able to control the interface right away from Projectgroup no.: the initial position. mea111041 To test the hypothesis, the project designs and implements a test interface set in the scene of a typical media center interface. The test By: users are presented with tasks to select different movies by moving the cursor to the right movie box cover and holding it there fore a Michael Birkehøj Jensen little while. This same task of selecting the right movie is iterated with different system configuration of user position relative to the inter- face difficulty and interaction method. The project found that there is a way for the proposed methods, however more accurate user tracking in a larger area is needed for the proposed interaction methods will be competitive to traditional remote control. Printed copies: .....2. Appendix: ......Included in the back of the report . . ... . . ... . . . . .CD can be found with the report Copyright©2011. This report and/or appended material may not be partly or completely published or copied without prior written approval from the authors. Neither may the contents be used for commercial purposes without this written approval of the author. III 01_Introduction| In0tr2o_dAuncatiloynsi s 0A3n_aDlyessisig n 0D4e_sIigmnp lemImenptlaetmioenn ta0ti5o_nT estT e0s6tin_gE valuEavtiaoluna ti0o7n_ RefReerefenrceensc es0 8_AAppppeennddiixx 01|Introduction 02|Analysis 03|Design 04|Implementation 05|Testing 06|Evaluation 07|References 08|Appendix iv Table of content 1. Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 2. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 2.1. Delimitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 2.2. Readers guide . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 3. Analysis. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 3.1. Interaction and natural interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 3.2. Interaction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10 3.2.1. Depth sensing cameras. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 3.2.2. Analysis of depth information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 3.3. Graphics and visualization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .12 3.4. Tools for implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .12 3.4.1. Damping and noise filtering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 3.5. OpenNI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .13 3.5.1. Production Nodes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 3.5.2. Production chains . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 3.5.3. Capabilities. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 3.5.4. Generating and Reading Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 3.5.5. Main Objects. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 3.5.6. Configuration Using XML . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 3.6. Conclusion of analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .19 4. Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 4.1. Interface design and graphics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .23 4.1.1. The test interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 4.1.2. Buttons and interactive elements. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 4.1.3. Graphics design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 4.2. Interaction method design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .30 4.2.1. Mouse or track pad . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 4.2.2. Hand direct. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 4.2.3. Line of sight . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 4.2.4. User reach area mapping. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 4.3. Software design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .34 4.4. Sound design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .34 4.5. Conclusion of design. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .34 v 01_Introduction| In0tr2o_dAuncatiloynsi s 0A3n_aDlyessisig n 0D4e_sIigmnp lemImenptlaetmioenn ta0ti5o_nT estT e0s6tin_gE valuEavtiaoluna ti0o7n_ RefReerefenrceensc es0 8_AAppppeennddiixx 01|Introduction 02|Analysis 03|Design 04|Implementation 05|Testing 06|Evaluation 07|References 08|Appendix 5. Implementation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 5.1. Hardware setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .39 5.2. Software implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .39 5.2.1. OpenNI and Nite. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 5.2.2. GUI implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 5.2.3. Positioning the test interface GUI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 5.2.4. Calculation and animation of wait timer bar . . . . . . . . . . . . . . . . . . . . . . . . 43 5.2.5. Consistency of the users experience. . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 5.2.6. Sound implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 5.2.7. Implementation of interaction methods . . . . . . . . . . . . . . . . . . . . . . . . . . 43 5.3. Conclusion of implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .46 6. Test. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 6.1. Test setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .49 6.1.1. Actual test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 6.1.2. Test database and data gathering. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 6.2. Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .51 6.3. Data processing analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .51 6.3.1. Initial data processing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52 6.3.2. Test material . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 6.3.3. Classifying data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 6.3.4. Qualitative results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64 7. Evaluation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67 8. References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71 vi Table of figures Figure 1. The concept of “Flow” [http://amandakatarina.files.wordpress. com/2010/04/1000px-challenge_vs_skill-svg4.png] . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 Figure 2. System overview of the Primesense depth camera technology [http://www. primesense.com/?p=514]. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 Figure 3. Line plane intersection - [http://en.wikipedia.org/wiki/Line-plane_intersection] . . . . 13 Figure 4. Xbox Zune user interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 Figure 5. Xbox kinect user interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 Figure 6. Xbox Zune user interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 Figure 7. First implementation of the test interface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 Figure 8. Wireframe interface mockup. 2 by 2 grid size. blue: button areas, yellow: main area.. 26 Figure 9. The three different states of the gui elements. Left: non-active icon, middle: acti- vated icon, right: activated icon with interaction time indicator.. . . . . . . . . . . . . . . . . . . . 27 Figure 10. The cursor designed for the test interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 Figure 11. Graphical visualization of the found user position, with five different areas of an- gles to the interface.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 Figure 12. Simple one line movie icon interface design. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 Figure 13. Large icons on the movie selector interface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 Figure 14. Small icons on the movie selector interface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 Figure 15. Direct hand position mapping concept. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 Figure 16. Concept setup of line of sight method. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 Figure 17. User reach area mapping interaction method concept diagram . . . . . . . . . . . . . . . . . . . . 33 Figure 18. Mapping of user reach area to interface screen area . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 Figure 19. Concept setup of line of sight method, camera tilted relative to screen.. . . . . . . . . . . . . 45 Figure 20. Raw data - time sample histogram - 1000 bins . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 Figure 21. Histogram - time data - all samples - threshold 25 second - 50 bins . . . . . . . . . . . . . . . . . 53 Figure 22. Relation between evaluators and information gain [http://www.useit.com/pa- pers/heuristic/heuristic_evaluation.html]. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54 Figure 23. Standard deviation of N number of test users (Time measurement). . . . . . . . . . . . . . . . . 54 Figure 24. Standard deviation of error rate according to number of expert test users . . . . . . . . . . 55 Figure 25. Average standard deviation of user error rate according to number of expert users (10,000 random iterations) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 Figure 26. Example of plotting data using PRTools - Success according to x- and y-position. . . 56 Figure 27. Error rate for each interaction method, all test users included . . . . . . . . . . . . . . . . . . . . . . 57 Figure 28. User position with respect to class (interaction method) . . . . . . . . . . . . . . . . . . . . . . . . . . . 58 Figure 29. Time statistics for each interaction method. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59 Figure 30. X-position statistics for each interaction method. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59 Figure 31. Y-position statistics for each interaction method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 Figure 32. 2D real world user position scatter plot with color indicated position difficulty. . . . . . 61 Figure 33. Bar chart - Error rate for all combination of position and interface difficulty (mouse) 62 Figure 34. Bar chart - Error rate for all combination of position and interface difficulty (line of sight). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 Figure 35. Bar chart - Error rate for all combination of position and interface difficulty (hand direct). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 Figure 36. Bar chart - Error rate for all combination of position and interface difficulty (user area mapping). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64 vII 01_Introduction| In0tr2o_dAuncatiloynsi s 0A3n_aDlyessisig n 0D4e_sIigmnp lemImenptlaetmioenn ta0ti5o_nT estT e0s6tin_gE valuEavtiaoluna ti0o7n_ RefReerefenrceensc es0 8_AAppppeennddiixx 01|Introduction 02|Analysis 03|Design 04|Implementation 05|Testing 06|Evaluation 07|References 08|Appendix Code examples Code example 1. OpenNi XML configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 Code example 2. Initialization function called once when the context is created . . . . . . . . . . . . . . . . . . . . . 40 Code example 3. Deconstructor for the OpenNIContext class. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 Code example 4. InitThread, responsible for setting up the connection to OpenNI and the Nite Middleware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 Code example 5. OpenNIContext class, Start()- and Update()-function. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 Code example 6. The OpenNI nodes used by the implementation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 Code example 7. C# - unity gui transformation matrix. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 Code example 8. C# - Unity gui coordinates. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 Code example 9. C# - Button area size calculations pseudo code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 Code example 10. Calculation of masking area effect for wait for interaction timer visualization . . . . . . . 43 Code example 11. Matlab - classifying data according to position difficulty . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 Code example 12. Matlab - scaling time according to error rate. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62 List of tables Table 1. http://www.primesense.com/?p=514 .............................................14 Table 2. Database design 1. for test data gathering.........................................44 Table 3. Database design 2. for test data gathering.........................................45 Table 4. Error correction test 1. Keyboard layout, test words and features ...................46 Table 5. Database design..................................................................47 Table 6. Error rate for all combinations of interface- and position difficulty..................55 viii Abbreviations and Definitions GUI: Graphical User Interface GUI layout: .. Sizing and positioning GUI elements to form a functional, visually attractive screen. NUI: Natural user interface IDE: Integrated Development Environment; a software development tool that includes at least an editor, a compiler and a debugger. lo-fi: Abstract, low level of detail, visually imperfect Hi-fi: High level of detail, visually elaborate, looking like real Mockup: A non-interactive, high-fidelity representation of a GUI OOP: “Object-oriented programming is a method of implementation in which programs are orga- nized as cooperative collections of objects, each of which represent an instance of some class, and whose classes are all members of a hierarchy of classes united via inheritance relationships.” Round-trip engineering: A functionality of software development tools that provides generation of models from source code and generation of source code from models; this way, existing source code can be converted into a model, be subjected to software engineering methods and then be converted back. UI: User Interface UML: Unified Modeling Language Wireframe interface: Computer-drawn, low-fidelity version of a GUI that is used in the early stages of GUI design. DADIU: ”Det Danske Akademi for Digital, Interaktiv Underholdning” - educates students in making computer games, and is an association of university and art schools throughout Denmark ECTS: European Credit Transfer and Accumulation System 3D: three-dimensional 2D: two-dimensional Ix 01_Introduction| In0tr2o_dAuncatiloynsi s 0A3n_aDlyessisig n 0D4e_sIigmnp lemImenptlaetmioenn ta0ti5o_nT estT e0s6tin_gE valuEavtiaoluna ti0o7n_ RefReerefenrceensc es0 8_AAppppeennddiixx 01|Introduction 02|Analysis 03|Design 04|Implementation 05|Testing 06|Evaluation 07|References 08|Appendix x