Programming by Voice: A Hands-Free Approach for Motorically Challenged Children Author: Amber (Krug) Wagner Advisor: Dr. Jeff Gray Committee Member: Dr. Eugene Syriani Committee Member: Dr. Randy Smith Committee Member: Dr. John Lusth Fall 2012 Department of Computer Science College of Engineering University of Alabama Abstract Computer Science (CS) educators frequently develop new methodologies, languages, and programming environments to teach novice programmers the fundamental concepts of CS. A recent trend has focused on new environments that reduce the initial challenges associated with the heavy syntax focus of traditional environments. There are numerous Initial Programming Environments (IPEs) available that have been created for student use that in some cases have fostered self-discovery and inquiry-based exploration. In this qualifier paper, three IPEs are surveyed: Scratch, Lego Mindstorms LabVIEW, and App Inventor. These three IPEs were selected due to the high number of students and teachers currently using them, as well as my personal experience with these environments through past teaching activities. The creative and graphical nature of these three IPEs have been shown to increase student and teacher interest in CS through adoption in classrooms worldwide at the K-12 levels, as well as interest in introductory university courses. An additional reason for selecting these environments for the focused study is that I have assisted in offering professional development workshops for teachers and student summer camps utilizing these environments. Although the graphical nature of these IPEs can be helpful for learning concepts in CS, a small group of students is being left out from learning experiences and engagement in CS. Graphical environments often require the use of both a mouse and keyboard, which motorically challenged users often are unable to operate. Based on research performed and presented in this paper, a Vocal User Interface (VUI) is a viable solution that offers a “Programming by Voice” (PBV) capability (i.e., a capability to describe a program without the use of a keyboard or mouse). There are two primary disadvantages with VUIs as a technology to address the limitations of motorically challenged users: 1) vocal strain can emerge for technical solutions that require a deep amount of vocal interactions, and 2) the process of integrating voice controls into a legacy application (e.g., an IPE) is a very time consuming process. There are existing vocal tools (e.g., the generic Vocal Joystick) that could be integrated into various applications; however, due to the needs of IPEs and the duration of IPE usage, the Vocal Joystick is not feasible due to the potential vocal strain, which is why a more command-driven approach may offer benefits for the PBV concept. The command-driven approach leads to a time-consuming process in terms of adapting legacy applications, particularly, if multiple applications (like the three IPEs previously mentioned) require specialized VUIs. Each environment has its own layout and its own commands; therefore, each application requires a different VUI. In order to create a more generic solution, Model- Driven Engineering (MDE) and Domain-Specific Languages (DSLs) can be applied to create a semi- automated process allowing a level of abstraction that captures the specific needs of each IPE. From the specification of each IPE, a customized VUI can be generated that integrates with the legacy application in a non-invasive manner. This paper presents background information on IPEs focusing on Scratch, LabVIEW, and App Inventor. Accessibility issues are discussed as well as previous work surrounding the concept of PBV. The possible application of MDE and DSL to support an integration solution is also introduced. -2- Table of Contents Abstract .................................................................................................................................................. 2 1. Introduction ........................................................................................................................................ 4 2. Motivation for Improving Accessibility ................................................................................................. 5 3. Computer Science Education for First Learners .................................................................................... 7 3.1 Learning Environments .................................................................................................................. 7 3.2 Accessibility Issues ....................................................................................................................... 13 4. Previous Work in Voice-Driven Control of Programming Environments .............................................. 14 4.1 Programming by Voice (PBV) ....................................................................................................... 15 4.2 Myna ........................................................................................................................................... 17 5. Summary of Future Research Plans .................................................................................................... 19 5.1 Myna ........................................................................................................................................... 19 5.2 Overview of Possible Future Contributions from Extending Myna ................................................. 20 6. Conclusion ......................................................................................................................................... 22 Acknowledgements ............................................................................................................................... 23 References ............................................................................................................................................ 23 -3- 1. Introduction According to the US Census Bureau [39], there were approximately 54,030,469 children ages six to seventeen in the US in 2010. Among those, 3,212,225 (5.9%) participated in the College Board’s Advanced Placement (AP) program. Furthermore, only 20,120 (0.037%) participated in AP Computer Science [9]. This low participation rate suggests that either students do not understand or are unaware of the opportunities in Computer Science, or schools are not offering Computer Science courses. Figure 1 references the trajectory of student participation in the AP Computer Science program over the past decade. A large increase in participation from 1991 to 2001 (146%) can be seen in the graph; however, the total number of students taking AP CS (A and AB) has declined since then. Because the numbers dropped primarily for the AB exam, the College Board discontinued the course in 2009, although the A exam is still offered. AP Computer Science 25,000 20,000 15,000 AP CS (Total) 10,000 AP CS A 5,000 AP CS AB 0 112345678901 900000000011 900000000000 122222222222 Figure 1. AP Computer Science participation, 1991-2011, provided by the College Board [9]. The Computer Science Teachers Association (CSTA) conducted two surveys: one in 2005 and an identical one in 2007 regarding High School Computer Science. In the analysis of the two surveys, Gal-Ezer and Stephenson [53] attribute the decrease in AP CS to three primary factors: 1. Rapidly changing technology; 2. Lack of student interest; and 3. Lack of staff support. While teachers cannot control the speed at which technology changes or the amount of staff support available, teachers can provide a pedagogically sound yet interesting curriculum to motivate students rather than discourage them. The current generation of students will not be impressed with the traditional “Hello World” program [21, 44] and pedagogical instruction techniques used with previous generations of students. The current generation of students has been playing video games with advanced graphics since they learned to walk, and the notion of internet accessibility is often viewed as a common utility. To increase the interest and attention of students, exciting new programs and technologies are being developed. Additionally, student engagement is often more successful when the context is driven by current topics that are of direct interest to students. For example, most teenagers are frequent users of smartphones, which -4- provides a unique context for engagement. The adoption rate of mobile computing among students age 12-17 in the US continues to grow and has been estimated at 23% for smartphone adoption (and 54% for general cell phone usage) [30]. Based on current US census results [40], this suggests that there are 4.8M middle and high school students with smartphones. Educators can take advantage of these devices as a springboard for motivating topics involving Computer Science [6, 9, 13, 25]. Mobile application development is not the only way to reach students, but based on the statistics, it is clearly an excellent starting point. The realization that programming is a difficult topic for a first-time learner is not novel and has been a challenge for CS educators since the inception of our discipline. In fact, Kelleher and Pausch [21] reviewed 81 programming environments starting from 1963, in which the goal of each environment was to simplify the code-writing process for beginners. The improvements in programming environments over the past 40 years is very evident and likely have made a contribution toward how children are learning to program while concurrently raising their interest in CS. While this is positive, it is important to note that programming environments require the use of a keyboard and mouse. The dexterity required for these peripheral devices causes a substantial group of children to be left out of the CS education process. My long-term research focus is to investigate a new approach that makes an impact by offering motorically challenged children the same opportunities as an unimpaired child. The various topic areas and technologies that inform and drive this long-term vision are the focus of this qualifier paper. This paper describes the context and motivation for this qualifier topic area in Section 2, followed by an explanation of initial CS learning environments and the accessibility issues associated with those environments in Section 3. Previous work is presented in Section 4, with a discussion including questions resulting from observations from the literature amid current needs for accessibility. The paper also provides a brief future research plan in Section 5, followed by concluding comments in Section 6. 2. Motivation for Improving Accessibility Based on the statistics for AP CS participation presented in the previous section, methodologies and tools that introduce new students to CS and programming must be improved. Papert [56] suggests that programming languages should satisfy three conditions: low-floor (simple to start using), high-ceiling (ability to increase difficulty and complexity over time), and wide-walls (ability to create various types of projects). The current approach instructors are using to meet these suggested requirements is to utilize an environment with a graphical interface that eliminates the concern of syntax (e.g., Scratch [33], Alice [2], Lego Mindstorms LabVIEW [24], and App Inventor [3]) – these are often called Initial Programming Environments (IPEs) because their focus is on education and pedagogical concerns, rather than the traditional focus of a development environment for a programmer. Based on observations and empirical studies [55], the use of IPEs offers numerous benefits, such as: removal of the syntactic details of a language, the fostering of creativity thus increasing engagement and interest, and the ability to start problem solving on day one. To address the concern over the challenges of learning the syntax of a programming language, many of the current IPEs have a form of “drag and drop” programming whereby a student connects program -5- constructs together in a manner that forces correct syntax (but may still produce logic errors). The mechanism for this capability is often through the usage of a mouse that requires a level of dexterity to connect the program statements together. However, because this new breed of IPEs requires motor functionality to operate a mouse and keyboard, those students who are motorically challenged are left out from these new learning experiences. The ACM code of ethics states, “[i]n a fair society, all individuals would have equal opportunity to participate in, or benefit from, the use of computer resources regardless of race, sex, religion, age, disability, national origin or other such similar factors” [50]. By not providing alternative means of access, users with disabilities are being denied learning opportunities that may allow them to explore career paths in computing. The driving motivation for the underlying theme of this qualifier paper is recognition that more children (or adults) should have the opportunity to learn about programming and CS using these new IPEs. Improving the diversity of a user base has several advantages, as noted by Kelleher and Pausch, who wrote that “[i]f the population of people creating software is more closely matched to the population using software, the software designed and released will probably better match [users’] needs” [21]. Many individuals suffer from a condition impairing motor functionality [29]. Of those with spinal cord injuries, 70% are unemployed [18]. If computer accessibility can be increased, new career opportunities may emerge that can help to reduce the unemployment rate among those with impaired motor function. Table 1 presents information collected from the National Institute of Neurological Disorders and Stroke [29], which lists four diagnoses impacting the motor function of children and/or young adults in the US. These individuals have the cognitive ability to understand the concepts of programming and CS, but they are unable to utilize the standard Windows Icon Mouse Pointer (WIMP) tools to interact with the computer. Diagnosis Number Impacted Age of Onset Spinal Cord Injury 250,000 56% occur between 16-30 Muscular Dystrophy 8,000(males 5-24 years of 400-600 males are born age) with MD each year Multiple Sclerosis 250,000-350,000 20-40 Cerebral Palsy 800,000 10,000 babies born with CP each year Table 1. List of a few major diagnoses impacting users’ abilities, particularly children or young adults [7, 39, 40, 41]. A possible solution to widen the gap of accessibility to IPEs is to use a form of a voice-driven interface to assist as an input modality for those with motor impairments. The concept of “Programming By Voice” (PBV) is the long-term focus that drives the topic of this qualifier paper. Voice-driven applications, such as those presented in [5, 10, 11, 18, 34, 43], primarily cater to specific applications, like the Internet [14, 38]. These applications are discussed and compared in Section 4. Scratch (discussed further in the next section) has a large number of users [33] and encourages a community of contributors where students can share and learn from each other. At the time of this writing, there are over 2.7M Scratch programs that have been shared among students and teachers. As an initial study to show the potential for extending an IPE to address accessibility needs, I have been leading the development of a prototype that explores the concept of PBV within the Scratch -6- environment [42]. This first effort is a fixed solution that is not extendable to other IPEs. It would be desirable to have a general approach that can assist in extending other IPEs with the concept of PBV. Kelleher and Pausch [21] described 81 programming environments that were developed over several decades. Although there are many environments that are no longer in use, CS educators and programming language researchers continue to develop better tools to teach programming. This qualifier paper is driven by the belief that the next step in IPE development will be to increase the usability [43] of all WIMP-based applications. However, incorporating voice-based control into an application is extremely time-consuming. The future research section of this qualifier (Section 5) outlines some ideas that point toward novel approaches to realize the goal of a generalized methodology for adding a VUI to legacy IPEs. 3. Computer Science Education for First Learners Instructors are beginning to utilize “drag and drop” IPEs for novice programmers. These tools are plentiful in number, but I have chosen three for the purposes of this research – Scratch, LabVIEW, and App Inventor. These three environments and their common ancestry are described below along with the accessibility issues introduced by the environments. 3.1 Learning Environments The College Board [9] indicates that high school CS enrollment is decreasing. The surveys conducted by the CSTA [53] highlight that student interest is an issue, which motivates the following question: Question 1: How can we improve initial programming instruction? Since 1963, researchers at universities have been developing initial programming environments to introduce programming concepts and problem solving with languages aimed at simplifying syntax [21]. Some environments (e.g., BASIC, Turing, and GNOME) were designed for college-level beginning programmers while others (e.g., Play, LogoBlocks, and Alice) were built for a younger target audience, such as elementary school children. The environments (e.g., Karel, GRAIL, and LegoSheets) reviewed by Kelleher and Pausch “tried to make programming accessible in three main ways namely, by simplifying the mechanics of programming, by providing support for learners and by providing students with motivation to learn to program” [21]. This subsection provides a brief history of the literature in this area of IPEs and compares and contrasts several popular existing environments. The 81 programming environments surveyed by Kelleher and Pausch date back to 1963 when Dartmouth College created BASIC (Beginners All-Purpose Symbolic Instruction Code) to help teach “non- science students about computing through programming.” BASIC was syntactically simpler than the more popular languages at the time (e.g., Fortran and Algol) [21]. The subtle simplification offered by BASIC was substantial in that it “sacrificed computer time for user time” [21]. Since the creation of BASIC, other universities have strived to design new languages with even simpler syntax such as GRAIL [49], created by Monash University in 1999. GRAIL was built on three principles: “maintain a consistent syntax; use terms that novice programmers are likely to be familiar with and avoid standard programming terms that have different meanings in English; and include only constructs that are fairly simple” [21]. -7- GRAIL and BASIC are languages where textual programs are coded, as are many others created between 1963 and 1999 (e.g., JJ, Pascal, and Smalltalk). Language developers and those working in the initial days of CS Education, such as Seymour Papert [56] and Alan Kay [57], realized very early that children were suitable subjects for learning how to program. In 1967, MIT created Logo [21], which is a Lisp-based language that was made more appropriate for children by removing most of the need to understand syntax. While Logo was designed to “allow children to explore a wide variety of topics from mathematics and science to language and music,” [21] it is most famous for the Logo turtle - a robotic turtle with a pen function that would illustrate the turtle’s movements around the screen, given by the child’s commands. Following from Logo’s goal of being more approachable by students, other graphical-driven environments were investigated for novice programmers, such as TORTIS, which was developed in 1976 by MIT’s Artificial Intelligence Lab. TORTIS, like many of its successors, was inspired by the Logo turtle in that children could programmatically control a robotic turtle to explore concepts of programming and even algorithm development. Many of the graphical languages were based on users creating rules, performing computations or I/O, or creating stories. With each new language development, more interesting and technical features became available, from simple environments such as Scratch [33], to more complicated environments such as Alice [2] by Carnegie Mellon University. Alice allows the creation of 3D interactive worlds simply by dragging and dropping program constructs into a syntax- directed editor, which removes the challenges of learning the details of syntax. Alice emerged as one of the most popular IPEs over the past decade among middle and high schools teachers, as well as many introductory courses within CS departments at the university level [54]. In addition to specific language environments, methodologies emerged that presented a common theme for exploration that are environment-independent, such as the Media Computation [17] approach that provides a context for understanding CS principles through manipulation of media files (e.g., changing properties of images or sound files). The following three learning environments are explained in more detail in the following subsections: Scratch [33], Lego Mindstorms LabVIEW [23], and Android App Inventor [3]. 3.1.1 Scratch MIT created Bongo in 1997, which allowed users to create their own video games and then share them with friends via the web [21]. MIT followed this same community-driven sharing model when creating Scratch, which was introduced in 2004 as an IPE that is simplified enough to teach to third graders, yet complex enough to teach to college freshmen or non-CS majors [58]. The University of California, Berkeley has adapted MIT’s version of Scratch to a more advanced version called Build Your Own Blocks (BYOB) [6] (recently renamed SNAP), which Berkeley (and the University of Alabama) teaches to freshmen and non-CS majors. The creators of Scratch at MIT intended for Scratch to be a “networked, media-rich programming environment designed to enhance the development of technological fluency at after-school centers in economically-disadvantaged communities” [27]. The Scratch community has emerged into a social network for its 350,678 project creators who have shared 2,732,346 projects for all 1,187,229 registered members to view [33, 59]. Kelleher and Pausch [21] observed that social environments such as networked applications provide more motivation for students to learn how to program, and Scratch has provided further evidence to support this observation [58]. Scratch is available in 13 languages and is visited daily from 16,798 cities (see Figure 2) [33, 59]. As an IPE, Scratch is reaching hundreds of thousands of students and educators. The projects that are shared within the -8- community serve to inspire other members in numerous ways (e.g., students share their culture, political, and religious views, students collaborate to create scripts other students can utilize, and students learn how to critique one another). Scratch is more than just a programming environment; it is a socially-driven creative tool that children can use to express themselves and explore their own creativity. Figure 2. The location and amount of Scratch users [33]. Scratch is heavily utilized throughout the world because of its ease of use and general availability (Scratch is freely available). Figure 3 is an image of the Scratch UI. All of the commands are “drag and drop” allowing the programmer to focus on logic and problem solving rather than syntax. Also, the programmer can develop his or her own images and import sounds, thereby fostering a creative environment that makes the underlying difficulty of the program transparent to the programmer. Figure 3. MIT's Scratch interface [33]. -9- 3.1.2 Lego Mindstorms Software The University of Colorado created LegoSheets in 1995 to work with the MIT Programmable Brick (“intelligent, built-in computer-controlled LEGO brick that lets a MINDSTORMS robot come alive and perform different operations” [60]). LegoSheets was primarily graphical and allowed users to progress slowly from controlling basic motor functionality of a robot to adding conditional statements in a program. In 1996, the MIT Media Lab developed LogoBlocks, which was “a graphical programming language designed for the Programmable Brick” [21]. With LogoBlocks, users could “drag and drop” code blocks to manipulate the brick and also allowed the user to learn about the creation and use of parameterized procedures to support generic reusable functionality. The successor of the programmable brick is the commercially available Lego Mindstorms LabVIEW environment developed by the MIT Media Lab in 1998. Since then, the programming environment has improved along with the hardware (e.g., the initial RCX Mindstorms robot used a very fragile infrared process for downloading programs, but the newer NXT platform supports Bluetooth and USB connections). The latest Mindstorms hardware is the NXT 2.0. A special programming template has been developed for National Instruments’ LabVIEW [51], which provides a graphical programming environment that is used by many K-12 schools. The software comes with 46 tutorials to assist educators and/or students in learning how to program and command the NXT robot. The tutorials demonstrate how to build the robot (this is helpful if specific sensors are needed) and how to build the blocks for the code [24]. Lego has created a social media forum for users called NXTLOG 2.0 where users can post their projects and share ideas [23], similar to Scratch. Figure 4 is a screenshot of the Mindstorms LabVIEW interface [24], which is similar to MIT’s original design. There is a block palette on the left, the program editor in the center, and the tutorials are on the right. Figure 4. Lego Mindstorms’ LabVIEW interface [24]. 3.1.3 Android App Inventor Google’s Android App Inventor [3, 45], now maintained by MIT’s Center for Mobile Learning, enables educators to take advantage of the interest that high school students have in smartphones. App -10-