The Predictive Validity of the Abstract Reasoning Test and the Raven’s Advanced Progressive Matrices Test for the Academic Results of First Year Engineering Students. Julia Groves 458 533 Declaration: A research project submitted in partial fulfilment of the requirements for the degree of MA by coursework and Research Report in the field of Industrial Psychology in the Faculty if Humanities, University of Witwatersrand, Johannesburg, 14 February 2015. I declare that this research report is my own, unaided work. It has not been submitted before for any other degree or examination at this or any other university. ______________________ Julia Groves Date: Word Count: 25 992 1 Abstract This research project examined the predictive validity of the Abstract Reasoning Test and the Raven’s Advanced Progressive Matrices on the academic results of first year engineering students. Additionally, biographical variables were examined in order to assess their contribution to the student’s scores on the psychometric tests. This research is important as the engineering department were looking to combat the high failure rate amongst first year engineering students. The department was looking to use the ART and the Raven’s to foresee the subjects in which students would struggle, enabling them to prepare extra assistance in this regard. The sample was the 2013 and 2014 first year engineering students at the University of the Witwatersrand, Johannesburg (N=395). The analysis showed that the ART and Raven’s do not predict the academic results of engineering students in their first year of study. The academic results refer to the marks obtained in the first year subjects of Chemical and Metallurgical Engineering, Physics, Chemistry, Economics and Mathematics. However, the biographical variables (especially those of home language and race) play an important role in contributing to the scores achieved on both psychometric tests. 2 ACKNOWLEDGEMENTS: To my wonderful parents, Jacqui and Les Groves. I do not know where I would be without your many, and much needed, words of encouragement and enlightenment. A special thank you to my mum who has read this report many times and offered invaluable feedback. To Benjamin Deeb whose constant assistance and love supported me throughout my post-graduate struggles. To my supervisor, Dr Fiona Donald, on whom I relied consistently for guidance and input throughout my Honours and Masters years. 3 Chapter 1: Table of Contents 1. Table of Contents Chapter 1: Table of Contents .................................................................................................................. 4 Chapter 2: Introduction .......................................................................................................................... 5 Chapter 3: Theoretical and Conceptual Background ...................................................................... 6 3.1 Education in South Africa ........................................................................................................ 6 3.2 Economics and the skills shortage in South Africa .................................................................. 7 3.3 Understanding the concept of Validity ................................................................................. 13 3.4 Intelligence Testing and Engineering .................................................................................... 14 3.5 The psychometric tests in this study ..................................................................................... 17 3.6 Background of similar studies ............................................................................................... 20 3.7 Research Hypotheses/Questions .......................................................................................... 23 Chapter 4: Method ................................................................................................................................ 24 4.1 Overall Research Design ....................................................................................................... 24 Chapter 5: Results ................................................................................................................................. 37 5.1 Phase 1: Combined sample ................................................................................................... 37 5.2 Phase 2: Results separated into 2013 and 2014 first year students ..................................... 54 5.3 Phase 3: Analysing the differences in the psychometric tests using year as a covariate ..... 81 Chapter 6: Discussion ............................................................................................................................ 86 6.1 Phase One Discussion ........................................................................................................... 86 6.2 Phase Two Discussion ........................................................................................................... 89 6.3 Phase Three ........................................................................................................................... 92 6.4 Comparing the results to the literature and its corresponding implications ....................... 92 6.5. The limitations of the study .................................................................................................. 96 6.6. Directions for future research............................................................................................... 97 Chapter 7: Conclusion ........................................................................................................................... 99 Chapter 8: References ......................................................................................................................... 102 Chapter 9: Appendices ........................................................................................................................ 106 9.1. Appendix A: Consent Form ................................................................................................. 106 9.2. Appendix B: Letter for 2014 Participants ............................................................................ 107 9.3. Appendix C: Letter for 2013 Participants ............................................................................ 108 9.4. Appendix D .......................................................................................................................... 109 9.5. Appendix E .......................................................................................................................... 110 4 Chapter 2: Introduction 2. Introduction In South Africa there is a critical skills shortage which means that the professional sector lacks individuals with certain skill sets – such as engineers (Daniels, 2007). The skills shortage can be linked directly to the fact that there are many Grade 12 pupils who qualify for university acceptance who are products of a schooling system which does not place emphasis on skills such as mathematics, science and technology-based programmes (Zaaiman, van der Flier & Thijs, 2001, Christie, 1998). This inadequate schooling produces university candidates who struggle to achieve well in courses for which these skills are essential (such as engineering) (Zaaiman et al., 2001). These education problems are ones for which many universities are still struggling to find a solution (Zaaiman et al., 2001). The possibility of using psychometric and other tests which will highlight these inadequacies is an appealing one as it will allow lecturers to identify the weaknesses within the skill sets of the students, as well as pinpoint which students may require extra tutoring (Zaaiman et al., 2001, Schaap & Luwes, 2013). Most universities are finding that Grade 12 results are no longer a reliable predictor of what the students are capable and are searching for other measures which will add to and enhance the selection process (Schaap & Luwes, 2013). The engineering department at the University of the Witwatersrand has archival data regarding first year engineering students from 2013 and 2014 on the Abstract Reasoning Test and the Raven’s Advanced Progressive Matrices. The aim of this research report is to ascertain whether these tests have any predictive value with regard to mid-year and end of year academic results. If this occurs, the engineering department will be able to identify early in the year, which students may require extra tutorage, and possibly, which subjects are weak in terms of both individuals and the group as a whole. The variables in this study are the scores students achieved in both tests (the Abstract Reasoning Test and the Raven’s Advanced Progressive Matrices) as well as the mid-year and end of year results for their first year of tertiary education. 5 This research report contains a theoretical and conceptual background, a methodology, analysis and results, and discussion. The theoretical and conceptual background will give a more in-depth discussion regarding the education problem and skills shortage in South Africa, a brief overview of the tests to be used and the qualities they are designed to measure. The methodology will contain more information regarding the design and construct of the study, while the final two chapters will show and explain the results found. Chapter 3: Theoretical and Conceptual Background 3. Theoretical and Conceptual Background 3.1 Education in South Africa The problems within the schooling system can themselves be attributed to problems that arose during the Apartheid era and that have not yet been rectified (Christie, 1998). Most of the schools in which there are currently problems with education and learning are schools that were previously in black-sectioned areas where goals to excel in schooling were ignored by the Apartheid government (Christie, 1998). The academic results recorded for the National Senior Certificate (NSC) exams showed that in South Africa, 29.8% of Grade 12 pupils failed to pass their exams, while only 24.3% of learners qualified to attend university (Department of Basic Education, 2013). While the education programme has been gradually addressed from the 1980’s, the lack of education that preceded this period is generally blamed for the ‘skills shortage’ we have in South Africa (Chisholm, 1983). Chrisholm believes that the inequality in South Africa with the high unemployment and lack of education within the population will continue to work towards a lack of skills within our country (Chrisholm, 1983). Although Chrisholm’s article is several years old, it can be seen that what the author feared is indeed still a problem in our society with many of our population remaining uneducated and the schooling system much in need of help (Christie, 1998, Department of Basic Education, 2013). One’s level of education directly influences one’s ability to find employment (http://beta2.statssa.gov.za/). 97.3% of graduates are employed in the formal sector, whereas only 52.9% of 6 candidates who have less than a Grade 12 education level are thusly employed (http://beta2.statssa.gov.za/) There have been many attempts to reform the education system – most of which have failed. One such attempt, put into place to help the post-Apartheid education system was outcome-based education (OBE) (Jansen, 1998). The fundamental principles of this intervention were to furnish schools/teachers/principals with the outcomes that the students are expected to achieve (Jansen, 1998) However, there were a number of drawbacks to this system (Jansen, 1998). Firstly, it proved too difficult for educators to manage and apply to their pre-set curriculums (Jansen, 1998). Secondly, there was no evidence that changing a schooling process or outcome will have any impact on South Africa’s failing economy and skills shortage (Jansen, 1998). Thirdly, the outcomes that have been decided on, do not address values and the teaching of values to students which is so important in a society that is trying to shake off the monstrosities of Apartheid (Jansen, 1998). Some of the problems stemming from this absence of value-teaching include lack of authority being held by both principals and teachers, poor attendance on behalf of both students and teachers alike, demotivation and a low willingness to teach and be taught and poor school results (Christie, 1998). Often the issues from the surrounding community will spill over into the school environment with the children displaying problems with alcohol and drugs, affiliation with gangs, violent tendencies and criminal activities (Christie, 1998). All of these problems result in the emphasis not being placed on education which will have an impact on poor understanding, learning and final marks (Christie, 1998). 3.2 Economics and the skills shortage in South Africa One’s success in academics is intrinsically linked to one’s success in future life in terms of career choice and future opportunities (Laidra, Pullmann & Allik, 2006). It can be argued that a test used prior to acceptance into a university course, which would enable the department to separate those who should excel in the course, from those who might fail (and thus require extra help during their time at university), would be an important tool to ensure that universities are able to produce as many qualified people as possible for the working world. The need for qualified engineers in the South African context, particularly in terms of government projects such as 7 road infrastructure, is steadily increasing (Schaap & Luwes, 2013). There is also a growing world-wide need in the international infrastructure for engineers (Schaap & Luwes, 2013). As many qualified graduates (in various sectors) leave South Africa to work abroad, it is important to have as large a pool of engineers as possible in order to ensure that the need for engineers in the South African context is satisfied (Schaap & Luwes, 2013). It has been said that the most important economic question is how to increase growth of output for each individual (Romer, 2001). The output of an individual is what he/she is able to accomplish and provide for his/her organisation and country’s economy (Romer, 2001). A small increase of the growth rate will lead to a cumulative effect on the standard of living within the country (Romer, 2001). The twentieth century in the United States was a stage characterised by rapid technological growth which led to a high standard of living and, additionally, an education system which provided the fast-paced output of technology that was needed in this innovative period (Romer, 2001). It is believed that sustaining and improving this trend of growth in the United States revolves around the improvement and focus on the tertiary education of scientists and engineers (Romer, 2001). Romer’s article focuses on the importance of this growing trend and predicts that faster growth would be able to monetarily resolve any budget difficulties as well as provide resources for the many social problems we face (Romer, 2001). As such, it shows the economic benefits that one can expect from the training of a large base of engineers (Romer, 2001). South Africa is not as fortunate as the United States in terms of skilled professionals as SA suffers from a skills shortage (Daniels, 2007). The skills shortage can be understood through the premise that the demand for skills far exceeds the supply (Daniels, 2007). Labour supply is defined as the human capital who participate in the labour market, while labour demand refers to the organisations that employ the human capital (Daniels, 2007). Between these two are the arrangements that help form the demand and supply relationship including the universities that train the human capital/labour force (Daniels, 2007). When Daniels’ paper was researched, South Africa’s economy was in an upswing after Apartheid in which increasing skills amongst the labour force played an important part (Daniels, 2007). Currently, only 25% of the South African population are employed in skilled positions, with 46% in 8 semi-skilled positions and 29% in low-skilled occupations (http://beta2.statssa.gov.za/). In terms of the Government’s definition, skills are comprised both of qualifications and experience in the field (Daniels, 2007). A scarce skill is defined as “a scarcity of qualified and experienced people, currently or anticipated in the future, either (a) because such skilled people are not available, or (b) because they are available but do not meet employment criteria” (as cited in Daniels, 2007). Absolute scarcity of these skills is defined in (a) above, where people do not hold the skills that are required, whereas relative scarcity refers to (b) above (Daniels, 2007). Critical skills refer to specific skills within an occupation (Daniels, 2007). This can be divided into two groups: generic skills and particular occupational skills (Daniels, 2007). The former skills include double-loop learning, language, literacy and team- player skills whereas the latter are skills specific to the occupation in which the individual is working (Daniels, 2007). It is important to keep all of the above definitions in mind when contemplating the skills shortage (Daniels, 2007). Various changes between sectors in our economy can also contribute to skills shortages and structural unemployment (Daniels, 2007). An example of this between-sector change could be a decrease in employment in the primary sector and an increase in available jobs in the tertiary sector (Daniels, 2007). The primary sector comprises the agriculture, mining and petroleum industries, while the tertiary sector includes real estate, transport and finance (Coughlin & Segev, 1999). This will result in the problem of people in the primary sector being left without jobs while positions in the tertiary sector are waiting for people who are able to fill them (Daniels, 2007). Unfortunately, the people from the primary sector are not able to fill these open positions due to their lack of the required skills (Daniels, 2007). A within-sector change would mean changes within organisations that require employees to be re- trained or taught new skills. At an economy-wide level, this could result in a critical skills shortage (Daniels, 2007). This shows that South Africa is not only having to deal with the skills shortage as a result of Apartheid education, but also skills based changes due to reintegration with the international market (Daniels, 2007, Kingdon & Knight, 2005). The result of this is that the labour demand was not great enough to support the labour supply and as a result we have a mismatch between demand and supply (Daniels, 2007, Kingdon & Knight, 2005). 9 In an attempt to curb this mismatch and to produce more skilled workers (as there is greater opportunity and demand for them than unskilled labour) the Skills Development Act was passed (Daniels, 2007, Kingdon & Knight, 2005). This Act outlines what needs to happen for skills development to be changed and how organisations are expected to train workers (Daniels, 2007, Kingdon & Knight, 2005). This led to the Human Resources Development Strategy which tried to target three levels of skills development by focusing on the linking of schooling (and adult) education, human resource development (including tertiary education), demand-side dimensions (skills that are to be given by the employers) and finally, national systems of innovation, research and development (Daniels, 2007). However, the impact of this has not yet been seen in the engineering sector. In 2011, 9 287 students graduated as engineers in South Africa (Esterhuizen, 2013). This number fell short of the goal set by the Higher Education and Training Minister, Dr Blade Nzimande who was hoping to see 10 093 engineering science graduates (Esterhuizen, 2013). Dr Nzimande reported that interventions were being put into place in order to increase this number, but that changes would only be evident in 2015/2016 (Esterhuizen, 2013). In the mean time, only16% of enrolled students have graduated, below that of the international calculation of 24% (Seggie, 2012). The graph below displays the difference for the past number of years between the number of students enrolled in engineering courses, compared to the percentage that graduate as qualified engineers (Seggie, 2012). The graph shows the number of students who registered in a given year, as well as the number of students who graduated in that year (Seggie, 2012). So while it is not a direct comparison between the same students, it does still serve to illustrate the gap between the number of students a university will accept every year, compared to the number of engineers they are able to produce into the working world (Seggie, 2012). As discussed above, these figures should steadily increase throughout the upcoming years. 10