Journal of Technology and Science Education INTEGRATING TECHNOLOGY IN STEM EDUCATION Priya Chacko, Sarah Appelbaum, Heejoo Kim, Jinhui Zhao, Jin Kim Montclare New York University Polytechnic School of Engineering United States [email protected], [email protected], [email protected], [email protected], [email protected] Received March 2014 Accepted January 2015 Abstract Students have access to the Internet at their fngertps via e-tablets and smart phones. However, the STEM felds are struggling to remain relevant in students’ lives outside the classroom. In an efort to improve high school science curricula and to keep students engaged in the classroom, we developed a technology-rich bioengineering summer program for high school students in grades 9-12. The program utlized touch screen technology in conjuncton with hands-on experiments and traditonal lecturing to create an entertaining, relevant, and efectve classroom experience. Keywords – Technology, Classrooms, Educaton, STEM. ---------- 1 INTRODUCTION In elementary school classrooms fve years ago, children were struggling with cursive – atemptng to make the graceful curves and connected leters that teachers claimed would be used in high school and postsecondary educaton (Wallace & Schomer, 1994). Although cursive came in handy 100 years ago when all legal documents were hand writen (Supon, 2009), the world has largely turned to new and progressive technologies – rendering those difcult cursive leters almost obsolete. Recently for English classes, teachers have created websites dedicated to their curriculum (Dunn, 2011), while students submit essays online to prevent plagiarism and seek out supplemental material to augment their course work (Baek & Freehling, 2007). In a world where technology is ever changing through innovaton, STEM classrooms appear to be lef behind (Pitler, 2011). The challenge for STEM educators in the coming years is to answer the age-old queston: How can students with very litle motvaton or interest in STEM be engaged in the classroom? A brief survey of what students are doing in their free tme points to one soluton to this problem: electronic devices. With four generatons of iPads out, new smart phones every month, and beter laptops every year, students are actvely engaging with hundreds of thousands of new touch screen applicatons (or apps) (Freierman, 2011). With such technologies available, students are less willing to sit in a classroom and atempt to decipher complex chemical formulas, equatons, or abstract concepts. Using a paperless classroom model, the program taught students about the fronters of bioengineering through a combinaton of lectures, classroom actvites, case studies, practcal laboratory exercises, and research techniques. Each week focused on a diferent disease, diabetes, cancer, and HIV/AIDS, and each student was expected to decide upon a topic of research. The students conducted and presented independent research projects at the end of the four weeks based on what they had learned. Journal of Technology and Science Educaton. Vol 5(1), 2015, pp 5 On-line ISSN 2013-6374 – Print-ISSN 2014-5349 DL: B-2000-2012 – htp://dx.doi.org/10.3926/jotse.124 Journal of Technology and Science Educaton – htp://dx.doi.org/10.3926/jotse.124 2 OBJECTIVES The overarching objectve of the research was to investgate how incorporatng “paperless” technology would beneft educaton and increase an interest in the STEM felds. By incorporatng a curriculum that focuses gradually on students learning independently rather than relying on textbooks and lecture-based learning traditonally utlized in STEM classrooms, students were to increase their understanding of topics covered as well as create an independent research project to present at the end of four weeks. 3 APPROACH For the last two years, two groups of students in 9th through 12th grade from various New York City schools partcipated in a paperless summer science program in which technology was fully integrated into science educaton. This program aimed to educate high school students about health related topics including diabetes, cancer and HIV/AIDS while encouraging them to conduct independent research. The four week program ran from July-August for a group of an average of 18 students (the number of students who partcipated in the program varied from year to year, but overall atendance never waivered from week to week). Each topic was introduced by a technical lecture. Once the students were given the background informaton, laboratory experiments were conducted in small groups. Aferwards, students and instructors discussed the results. Using what they learned from the lecture and lab, students were then given classroom actvites to complete based on the course curriculum (Table 1). For our curriculum, classroom actvity is defned as experiences involving students manipulatng their knowledge by partcipatng in discussions, creatng presentatons, assessing case studies, watching videos, and topic related games/actvites. At the end of every week, students were given an evaluaton of the module to determine whether or not they had increased their understanding in the STEM feld covered and their interest level in pursuing a STEM feld as a career afer high school. In this way, the modules could be evaluated by data provided by students. Week 1: Diabetes Monday Tuesday Wednesday Thursday Introducton to Lecture: Biology of Lecture: How to make Lecture: Chemistry of program diabetes a scientfc poster glucose and insulin Actvity: Creatng Lecture: Introducton to Actvity: Diabetes Lecture: Technology professional Epidemiology treasure hunt and diabetes PowerPoint slides Lecture: Introducton to Actvity: Diabetes Independent Lab: Diabetes epidemiology business case study presentaton drafing Lecture: Do’s and Lecture: Introducton to Actvity: Genetcs and don’ts of making Mock Presentatons diabetes diabetes science-related presentatons Week 2: Cancer Monday Tuesday Wednesday Thursday Actvity: Immortal Life Lecture: Lecture: Types of Actvity: Henrieta of Henrieta Lacks Computatonal cancer treatments Lacks Jeapordy game reading modeling Lecture: Biology of Actvity: Guest Lecture Lecture: Future of cancer Lab: DNA Restricton Industry Researcher cancer research Actvity: Causes of lab cancer Actvity: Immortal Life Independent Poster Independent Poster Lecture: Guest Speaker of Henrieta Lacks Drafing Drafing Academic Researcher medical ethics discussion Vol. 5(1), 2015, pp 6 Journal of Technology and Science Educaton – htp://dx.doi.org/10.3926/jotse.124 Week 3: HIV Monday Tuesday Wednesday Thursday Lecture: Intro to Lecture: Science of Lecture: Biochemistry epidemiological AIDS of HIV statstcs Internatonal AIDS Lecture: Technologies Actvity: Viewing “And Lab: HIV HLA lab vaccine initatve site to fght HIV the Band Played On” exercise visit Actvity: Paraphrasing Actvity: Post flm Actvity: AIDS Statstcs, Actvity: Spread of discussion, refectons Gapminder AIDS Week 4: Presentaton Monday Tuesday Wednesday Thursday Lecture: Intro to Drafing a business Project Week Independent project Independent project Plan for biomedical Independent project drafing drafing technology drafing Project Updates Project Updates Project Updates Project Updates Table 1. Lesson plan schedule for the Course The frst week of the program was focused on basic lessons on epidemiology and diabetes as well as an introducton on how to make science presentatons. The purpose of the frst week was to introduce the students to a concept that they were familiar with (diabetes) and to outline the expectatons of the summer program from an academic standpoint. In doing so, students understood the concept of a paperless classroom and self-directed learning. The second week focused on cancer in which the students read The Immortal Life of Henrieta Lacks by Rebecca Skloot (2010) and discussed the progress of biomedical research over the past 50 years as well as the ethical implicatons of patent consent. The second week helped reinforce the paperless classroom model of teaching. Students were able to discuss more complex topics like cancer while beginning to research the topics that they wanted to focus on by the end of the summer program. In additon to lectures on cancer biology, causes, and treatment, two guest lecturers also visited the classroom to talk about academic and industrial research. The frst lecturer was a professor at a university while the second was a practcing and researching physician at the NYU Langone Medical Center. The third week of the program was based on HIV/AIDS in which the science of AIDS, epidemiology, biochemistry and technologies to combat it were discussed. The students visited the Internatonal AIDS Vaccine Initatve (IAVI) site in order to establish the relatonship between what they were learning and academia/industry. The site visit allowed the students to apply their knowledge of new technologies to ensure the safe, efectve development of AIDS vaccines (the goal of IAVI). In order to understand the social and historical background of AIDS, the students viewed a movie ttled “And The Band Played On” (Spotswoode, 1993). This movie, based on a nonfcton book, discusses the discovery and batle against AIDS. Students used it to explore the impact of AIDS in the politcal, social and scientfc communites. The third week curriculum fnalized the paperless curriculum. The students utlized fewer lectures and more actvites, labs, and site visits to learn about HIV and AIDS. Utlizing the evaluaton of the modules (Figures 1-2), it was possible to see that the paperless classroom model was a success because students increased their understanding of the STEM topics while challenging themselves. Vol. 5(1), 2015, pp 7 Journal of Technology and Science Educaton – htp://dx.doi.org/10.3926/jotse.124 Figure 1. Percentage of students who responded “I Understand This Topic Beter Now” to evaluatons of each week of the program Figure 2. Students were asked “Has This Lesson Triggered Your Interest in Science For the Future” and their responses (taken from anonymous evaluatons) are charted over the four modules Vol. 5(1), 2015, pp 8 Journal of Technology and Science Educaton – htp://dx.doi.org/10.3926/jotse.124 The fnal week of the program was dedicated to the students using technology and resources provided to them (Table 2) to research the topics they chose at the beginning of the program. At the end, students presented to each other and then to a panel of third party judges who evaluated their presentatons. An overall theme of the program was to engage students in and outside the classroom using technology where they were constantly required to partcipate. The lessons learned from the classroom actvites, videos and online resources provided students the tools for independent study and successful self-learning. By the end of the summer, the students created independent research projects, culminatng with a formal oral presentaton of their research. At the beginning of the program, students seemed daunted by the sheer amount of informaton about to be presented to them. However, a combinaton of hands on actvites, interactve websites, and videos made the material digestble and engaging. The students were exposed to various methods of teaching the same material, reinforcing what they learned. In doing so, the paperless curriculum model provided an alternatve to the paper and pencil, lecture-based classrooms that are traditonally utlized in STEM classrooms. By self- directng the learning for the fourth week of the summer camp, students were able to perfect their research skills and strategies and produce a high-level independent research projects instead of relying on lecturers. Resource Descripton EBSCO Host (htp://www.ebscohost.com/) Research database service that contains online artcles from various academic journals available to view from a computer/iPad. PubMed (htp://www.ncbi.nlm.nih.gov/pubmed) Hosted by the NCBI, this search engine contains 22 million citatons for biomedical literature from various scientfc journals NYU Libraries (htp://library.nyu.edu/) Students were allowed access to the journals at Dibner Library on NYU-Poly’s campus. PhD Students at NYU-Poly Students were encouraged to talk to PhD Academic Researcher students from NYU-Poly if they were Industry Researcher interested in similar topics that the PhD candidate was researching. Google (book/scholar search) Students were encouraged to use Google htp://scholar.google.com/ Scholar and Google Books to research new htp://books.google.com/ and developing technologies in the scientfc community Table 2. Online and Resources Provided to Students for Independent Research Afer using all of these tools, if students had questons, wanted to share the informaton they learned, or wanted to explore more, they partcipated in an educatonal blog set up for the program. The blog was run internally using school servers and Google Docs, so it would be a safe online community only accessible by the students and teachers. Students were able to submit assignments online, post links relevant to classroom lessons, and host discussions about classroom material and extracurriculars. Students also stored their research on the blog in team folders so they could have access to the informaton regardless of where they were. Helpful videos and links to academic journal artcles were posted regularly. Links and questons meant to begin student discussions were posted and moderated by the teachers, but were largely contributed to by the students. None of the blog work was mandatory, but incentves (such as jokes and funny videos) were emailed to students who partcipated. In additon to the online tools, students were encouraged to remain in contact with the guest lecturers who visited the classroom. The students contacted instructonal mentors who were graduate students for more informaton about the topics covered and research questons. This access to scientsts on all levels (from graduate students to physicians/scientsts) allowed the students to directly engage with those working on the cutng edge of bioengineering in the feld of research. This combinaton of virtual and in person interactons emphasized the importance of actve engagement in the classroom. Vol. 5(1), 2015, pp 9 Journal of Technology and Science Educaton – htp://dx.doi.org/10.3926/jotse.124 4 RESULTS AND DISCUSSION The evaluaton for modules 1 and 2 on diabetes and HIV/AIDS respectvely resulted in 100% of the students understanding the topic (Figure 1). While 7% of the students expressed that the 3rd module on cancer did not improve their understanding of it, a 93% majority indicated that they gained insights into cancer (Figure 1). As there was much to cover for the cancer module due to the complexity in terms of types and pathways to cancer, the students suggested that the lesson plan be extended beyond the single week. For the fnal independent study/research project, all of the students as demonstrated by the 100% response improved their understanding of their selected research topic. In literature, the percentage of students (specifcally those in control groups) indicatng sustained interest were below 80% (Kim, Chacko, Zhao & Montclare, 2014), suggestng that the students in this program had a high level of positve response. This was also corroborated by the judges’ evaluatons of their presentatons (the grading rubrics can be examined in Tables 3-5). They commented that the students were able to not only efectvely artculate scientfc concepts and research studies but also respond to challenging questons during their presentaton. Group 1 Category Unsatsfactory Satsfactory Excellent Oral Communicaton (eye contact, etc) 1 2 3 4 5 6 7 Poster layout and design 1 2 3 4 5 6 7 Demonstrated depth of knowledge 1 2 3 4 5 6 7 Research practce (referencing, etc) 1 2 3 4 5 6 7 Ability to address questons 1 2 3 4 5 6 7 Teamwork (equal distributon of efort) 1 2 3 4 5 6 7 Table 3. Sample Judges Grading Rubric Exceeds standard (4) Meets standard (3) Emerging (2) Atempt made (1) Most of the content The content is All content throughout is accurate but generally accurate, Content confusing or the presentaton is Content accuracy there is one piece of but one piece of contains more than accurate. There are no informaton that informaton is one factual error. factual errors. seems inaccurate. clearly inaccurate. Most informaton is Some informaton is Informaton is organized in a clear, logically sequenced. There is no clear organized in a clear, Sequencing of logical way. One An occasional slide plan for the logical way. It is easy informaton slide or piece of or piece of organizaton of to antcipate the next informaton seems informaton seems informaton. slide. out of place. out of place. Project includes all Project is lacking Project is lacking material needed to Project is missing several key elements one or two key give a good more than two key and has elements. Project is Efectveness understanding of the elements. It is rarely inaccuracies. Project consistent with topic. The project is consistent with the is completely driving queston consistent with the driving queston. inconsistent with most of the tme. driving queston. driving queston. All graphics are All graphics are Several graphics are A few graphics are atractve (size and atractve but a few unatractve AND not atractve but Use of graphics colors) and support do not support the detract from the all support the topic the topic of the topic of the content of the of the presentaton. presentaton. presentaton. presentaton. Vol. 5(1), 2015, pp 10 Journal of Technology and Science Educaton – htp://dx.doi.org/10.3926/jotse.124 Exceeds standard (4) Meets standard (3) Emerging (2) Atempt made (1) Font formatng has Font formats (color, Font formats have been carefully Font formatng bold, italic) have been Text – font choice been carefully planned to makes it very carefully planned to & formatng planned to enhance complement the difcult to read the enhance readability readability. content. It may be a material. and content. litle hard to read. Presentaton has 1- Presentaton has 1- Presentaton has Presentaton has no Spelling & 2 misspellings, but 2 grammatcal more than 2 misspellings or grammar no grammatcal errors but no grammatcal and/or grammatcal errors. errors. misspellings. spelling errors. Group shares tasks Group shares tasks Group shares tasks Group ofen is not and all performed and performed and performs efectve in sharing Cooperaton responsibly all of the responsibly most of responsibly some of tasks and/or sharing tme. the tme. the tme. responsibility. Level 4 indicates competence in all standards/benchmarks and exceptonal performance in a few Level 3 indicates general competence in all standards/benchmarks Level 2 indicates general competence in most standards/benchmarks with difcultes in some Level 1 indicates difcultes in a majority of standards/benchmarks Table 4. Sample Powerpoint Slide Grading Rubric Motvatons for the students to enter the STEM felds also improved over the course of the program. While 86% of students responded afer the frst week of the program that the diabetes lesson inspired them to work or study in a related feld, 14% of students noted that STEM felds were not of interest to them at all (Figure 2). However, by the end of the program during the week on presentatons, 100% of the students indicated interest in STEM felds.. In fact, 86% of the students were motvated to pursue a STEM feld in the future. While the remaining 14% of students were unsure if the fnal lesson triggered their STEM interest, they enjoyed it. Overall, the program resulted in students giving science a chance –the 14% of students who initally were not interested in STEM felds changed their minds because they were engaged and had fun while doing the lessons. Utlizing videos, artcles, and websites on their laptops and/or iPads also resulted in students using their free tme to explore their specifc interests. Students were exposed to a virtual playground where they were allowed to safely frolic in informaton that would help them gain a stronger understanding of various scientfc topics. These tools encouraged students to conduct their own research by clicking from educatonal link to educatonal link. Thus, if a student began the day trying to learn more about diabetes, he or she could explore the diferent types of diabetes, the current and emerging detecton methods, and factors that complicate the disease within two or three hours without being lectured once! By creatng an interactve community outside of the classroom, students were encouraged to learn and reinforce what they were being taught. These approaches facilitated students to become self-motvated and engaged in the classroom. This required the students to use metacognitve skills, directng them to understand what they were studying and, most importantly, why they were studying it (Schraw, 1998). Vol. 5(1), 2015, pp 11 Journal of Technology and Science Educaton – htp://dx.doi.org/10.3926/jotse.124 Exceeds standard (4) Meets standard (3) Emerging (2) Atempt made (1) Subject Demonstrates Demonstrates Demonstrates some Demonstrates litle knowledge mastery of the topic accurate knowledge knowledge of the knowledge of the of the topic topic topic Organizaton and Organizes Organizes most Generally organizes Poorly organizes coherence informaton informaton and informaton, informaton and coherently and stays stays on the topic occasionally straying ofen strays from the on the topic from the topic topic Physical gestures Actvely engages the Usually engages the Occasionally Neglects to engage audience by making audience by making engages the the audience by and maintaining eye and maintaining eye audience by making rarely making and contact and using contact and using and maintaining eye maintaining eye movement (facial movement (facial contact and using contact or using expressions, posture, expressions, movement (facial movement (facial gestures) to focus posture, gestures) expressions, posture, expressions, posture, atenton and interest to focus atenton gestures) to focus gestures) to focus and interest atenton and atenton and interest interest Voice Always speaks Usually speaks Speaks Does not speak clearly/loudly clearly/loudly clearly/loudly clearly/loudly Language Uses appropriate Uses mostly Makes some errors Makes many conventons grammar and appropriate in grammar and grammatcal vocabulary grammar and vocabulary mistakes vocabulary Visual aids Creatvely uses a Uses a variety of Moderately Does not/inefectve variety of visual aids visual aids and/or inefectve use of use of some visual and/or other other methods of some visual aids aids and/or other methods of delivery delivery and/or other methods of delivery methods of delivery Appearance Thoroughly Generally Demonstrates Fails to demonstrate demonstrates demonstrates minimal appropriate appropriate appropriate understanding of appearance appearance appearance appropriate appearance Level 4 indicates competence in all standards/benchmarks and exceptonal performance in a few Level 3 indicates general competence in all standards/benchmarks Level 2 indicates general competence in most standards/benchmarks with difcultes in some Level 1 indicates difcultes in a majority of standards/benchmarks Table 5. Sample Presentaton Grading Rubric 5 CONCLUSIONS By integratng technology in STEM educaton, the program encouraged students to become self-motvated learners and researchers. For our summer program, we provided students the tools for independent research, study, and learning through a technology-rich lesson plan. The program was interactve to encourage independent exploraton and engagement through labs, lessons and exposure to scientsts. Using the methodology outlined in this paper, it is demonstratably possible for a bioengineering summer program to utlize a technology-rich science curriculum. More signifcantly, the students who partcipated in this program benefted from it. Due to the overwhelming increase in understanding from week to week (Figure 1), the paperless classroom model utlized for these summer programs can be deemed a success. Additonally, the paperless classroom model and the methodology utlized for this summer program also increased the percentage of students who desired to pursue a science-related feld in the future (Figure 2). Most importantly, the percentage of students who had decided that science was not for them decreased from 13% to 0% by the end of the program (Figure 2). The world is not what it was fve years ago – we have undergone a technological revoluton (Collins & Halverson, 2010). Most other facets of educaton – from standardized testng to Vol. 5(1), 2015, pp 12 Journal of Technology and Science Educaton – htp://dx.doi.org/10.3926/jotse.124 submission of assignments are electronic now; because the curriculum outlined in this paper was a success, shouldn’t more technology-rich science educaton be integrated in classrooms world wide? REFERENCES Baek, E., & Freehling S. (2007). Using Internet Communicaton Technologies by Low-Income High School Students in Completng Educatonal Tasks Inside and Outside the School Setng. Computers in the Schools, 24, 33-55. htp://dx.doi.org/10.1300/J025v24n01_04 Collins, A., & Halverson, R. (2010). The Second Educatonal Revoluton: Rethinking Educaton in the Age of Technology. Journal of Computer Assisted Learning, 26, 18-27. htp://dx.doi.org/10.1111/j.1365-2729.2009.00339.x Dunn, L.S. (2011). Making the Most of Your Class Website. Educatonal Leadership, 68(5), 60-62. Freierman, S. (2011, December 12). One Million Mobile Apps, And Countng at a Fast Pace. The New York Times, B3. Kim, H., Chacko, P., Zhao, J., & Montclare, J.K. (2014). Using Touch-Screen Technology, Apps, and Blogs To Engage and Sustain High School Students’ Interest in Chemistry Topics. J. Chem. Ed., 91, 1818-1822. htp://dx.doi.org/10.1021/ed500234z Pitler, H. (2011). Technology in the Classroom -- Is It or Is It Not Being Used? THE Journal, 38.6, pp. 42-44. Schraw, G. (1998). Promotng General Metacognitve Awareness. Instructonal Science, 26, 113-125. htp://dx.doi.org/10.1023/A:1003044231033 Skloot, R. (2010). The Immortal Life of Henrieta Lacks. New York, NY, USA: Crown. Spotswoode, R. (Director). (1993). And the Band Played On [Moton Picture]. Santa Monica, CA: Home Box Ofce. Supon, V. (2009). Cursive Writng: Are Its Last Days Approaching. Journal of Instructonal Psychology, 36(4), 357-359. Wallace, R.R. & Schomer, J.H. (1994). Simplifying Handwritng Instructon for the 21st Century. Educaton, 114(3), 413-417. Citaton: Chacko, P., Appelbaum, S., Kim, H., Zhao, J., & Montclare, J.K. (2015). Integratng technology in STEM educaton. Journal of Technology and Science Educaton (JOTSE), 5(1), 5-14. htp://dx.doi.org/10.3926/jotse.124 On-line ISSN: 2013-6374 – Print ISSN: 2014-5349 – DL: B-2000-2012 AUTHOR BIOGRAPHY Priya Chacko Priya Chacko graduated from the NYU School of Engineering in 2014. As a tutor and teacher's assistant, her passion has always been teaching. She currently works as a process engineer for Estee Lauder in Melville, NY where her interests include process optmizaton, reformulaton, and new products. Sarah Appelbaum Sarah Appelbaum graduated from NYU School of Engineering in 2015 with a Master's in Management of Technology and a Bachelor's in Chemical and Biomolecular Engineering which were completed concurrently. Sarah works as a Business Technology Analyst for Deloite Consultng where her work is focused on data analysis, technology implementaton and process efciency. She currently resides in Brooklyn, NY. Vol. 5(1), 2015, pp 13 Journal of Technology and Science Educaton – htp://dx.doi.org/10.3926/jotse.124 Heejoo Kim Heejoo Kim works as a full tme research assistant in NYU School of Engineering. She recieved a BS degree in Biomedical Science in 2013 from NYU Polytechnic School of Engineering. She worked as a mentor in Chem-Bio Technology Lab during 2011-2013 and helped developing lessons and modules based on touch-screen technology. She had helped 10th grade girls in Urban Assembly Insttute of Math and Science as a mentor and motvated students to pursue their studies in STEM feld. Jinhui Zhao Jinhui Zhao obtained her Bachelor of Science degree in Biomolecular Sciences with a concentraton in Chemistry from the NYU Polytechnic School of Engineering in 2012. While there, as a Dreyfus Fellow (2010- 2012) and American Chemical Society's Science Coach (2011-2012), she worked on novel teaching strategies using hands-on technology and helped develop the iPad App, LewisDots. She is currently a candidate for Doctor of Medicine at the State University of New York Downstate Medical Center. Jin Kim Montclare Jin Kim Montclare, PhD, is an Associate Professor in the Department of Chemical and Biomolecular Engineering at NYU Polytechnic School of Engineering. She runs a research group that specializes in synthetc biology with a focus on protein design. Since 2008, she has been involved in K-12 Educaton and inspiring the next generaton to pursue STEM felds. Published by OmniaScience (www.omniascience.com) Journal of Technology and Science Educaton, 2015 (www.jotse.org) Artcle's contents are provided on a Atributon-Non Commercial 3.0 Creatve commons license. Readers are allowed to copy, distribute and communicate artcle's contents, provided the author's and JOTSE journal's names are included. It must not be used for commercial purposes. To see the complete licence contents, please visit htp://creatvecommons.org/licenses/by-nc/3.0/es/ Vol. 5(1), 2015, pp 14