International Education Studies; Vol. 6, No. 6; 2013 ISSN 1913-9020 E-ISSN 1913-9039 Published by Canadian Center of Science and Education Direct Measurement and Evaluation for Mechanical Engineering Programme Outcomes: Impact on Continuous Improvement Mohd Faizal Mat Tahir1,2, Nor Kamaliana Khamis1,2, Zaliha Wahid1, Ahmad Kamal Ariffin Mohd Ihsan1,2, Jaharah Ab Ghani1,2, Mohd Anas Mohd Sabri1,2, Zainuddin Sajuri1, 2, Shahrum Abdullah1,2 & Abu Bakar Sulong1 1 Department of Mechanical Engineering and Materials, Faculty of Engineering and the Built Environment, Universiti Kebangsaan Malaysia, Selangor, Malaysia 2 Center for Automotive Research, Faculty of Engineering and the Built Environment, Universiti Kebangsaan Malaysia, Selangor, Malaysia Correspondence: Mohd Faizal Mat Tahir, Department of Mechanical Engineering and Materials, Faculty of Engineering and the Built Environment, Universiti Kebangsaan Malaysia, 43600 UKM Bangi, Selangor, Malaysia. Tel: 60-389-216-516. E-mail: [email protected] Received: February 7, 2013 Accepted: March 27, 2013 Online Published: May 30, 2013 doi:10.5539/ies.v6n6p161 URL: http://dx.doi.org/10.5539/ies.v6n6p161 Abstract Universiti Kebangsaan Malaysia (UKM) is a research university that continuously undergoes an audit and accreditation process for the management of its courses. The Faculty of Engineering and the Built Environment (FKAB) is subjected to such processes, one of them is the auditing conducted by the Engineering Accreditation Council (EAC), which gives recognition to engineering programmes in Malaysia. The criteria that have been set by the EAC requires the faculty to measure and evaluate graduates based directly on the course outcomes (CO) and programme outcomes (POs) to ensure that they are able to achieve the programme and course objectives. This paper discusses a method developed for the measurement and evaluation of POs which is a direct measurement for assessing the courses offered in the mechanical engineering programme. Assessment templates are developed using a Microsoft Excel based on the data presented on the grade coordination meetings conducted at the end of each semester. The data used for this study were semester 2 2010/2011 session, and semester 1 session 2011/2012. Based on the results obtained, the majority of the courses achieves more than 80% for course level monitoring for their individual PO achievement except for five subjects. For programme level monitoring, from the average POs marks, all POs were scored more than 80%, which indicates that the Mechanical Programs offered by the department are satisfied with the department standard. Keywords: accreditation, EAC, programme outcome 1. Introduction All programmes in institutes of higher learning and universities are required to implement an outcome-based education curriculum set by the Ministry of Higher Education to ensure quality graduates are produced (Shahrir et al., 2008). The Faculty of Engineering and the Built Environment (FKAB) has initiated the concept of Outcome-Based Education (OBE) in 2005, particularly because the FKAB has to ensure that programmes are recognized by authorized accreditation bodies. The Engineering Accreditation Council (EAC) is one of Malaysia’s accreditation bodies and is responsible for ensuring quality engineering programmes at local universities. The EAC has set certain standards to be achieved, emphasizing the measurement of student achievement through the OBE and in a manner that is consistent with the requirements of the ministry (Siti Aminah et al., 2011). Programme Outcome can be measured by a variety of methods either on questionnaires, grading courses, observation of student achievement, portfolio analysis and interviews. All of these methods can be divided into two approaches, namely the direct or indirect measurement (Breslow, 2007). Allen (2008) explains that the direct measurement methods to assess the students' work in the form of creative assignments and systematically can be assessed based on rubric. While the indirect measurement methods can assess students' perception of learning and teaching environment. The result of this method is often combined with the direct method results. According Wojtczak (2002), Programme Outcome measurement is useful to assess student achievement to meet 161 www.ccsenet.org/ies International Education Studies Vol. 6, No. 6; 2013 the needs of external examiners and also can provide the information for the purpose of improving the curriculum, as well as to document the success or otherwise for a programme. Based on the OBE approach, the measurement of student achievement upon completion of the course is a requirement to determine the ability of the students based on their cumulative grade point average measurement (CGPA) for each semester. Many of researcher assess the Programme Outcome and Course Outcome based on their student achievement (Ishak et al.,2011, Nurina Anuar et al,.2011, Juridah Johari et al, 2010, Hamimi Fadziati et al,.2010, Mohd Faizal et al, 2012) . OBE also specifies that the assessment results for each course [course outcomes (COs)] will have to depend on what has been stated in the programme handbook. In addition, CO measurement is directly dependent on the achievement of students in each assessment determined in each course as scores from quizzes, tutorials, laboratories, and examinations. Thus, to ensure that the end result is achieved with quality assured, teaching and learning plans should be designed and provided by the course instructor at the beginning of each semester (Rozeha et al., 2007). Each course offered throughout the mechanical engineering programme has its own COs to be achieved by the students upon course completion. Every course is designed to relate the results and contribute to the achievement of programme outcomes (POs). A detailed syllabus for each course as well as the respective results of the course and its relationship with the PO can be seen in FKAB undergraduate handbook for each academic session. Figure 1 shows the process flow of the presentation and evaluation of student performance and achievement of COs as practiced in the department. KKKM2XX4 KKKM3XX5 Course Delivery •LKecKtuKrFe1XX4 •Lecture •Lecture •LKeKctKurMe4XX3 •LKeKctKurQe2XX4 •PBL •Lab •Lab •PBL •PBL •PBL •PBL •••DEM-ielseecautrisnnsgiinogn ••DE-ilsecaursnsiinogn ••DE-ilsecaursnsiinogn •••DEM-ielseecautrisnnsgiinogn •••DEM-eilseecautrisnnsgiinogn •Meeting •Meeting AsCseosusrmsee nt Assignment Quiz Project Mid Sem Final Exam Assignment Quiz Project Mid Sem Final Exam Lab Assignment Quiz Project Mid Sem Final Exam Lab Assignment Quiz Project Mid Sem Final Exam Assignment Quiz Project Mid Sem Final Exam CO Assessment O1 O2 O3 O4 O5 O6 O1 O2 O3 O4 O5 O6 O1 O2 O3 O4 O5 O6 O1 O2 O3 O4 O5 O6 O1 O2 O3 O4 O5 O6 Course level C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C PO Assessment Programme level PO1 PO2 PO3 PO4 PO5 PO6 PO7 PO8 PO9 PEO Assessment Department level PEO1 PEO2 PEO3 PEO4 PEO5 2-4 year cycle Figure 1. Process of delivery and assessment of student performance and achievement as practiced in the department Figure 1 shows that every course has specific delivery and evaluation methods for student achievement. Each element of assessment must be performed, and the course instructor must ensure the measurement results of the course evaluation achieve a predetermined threshold at the departmental level (EAC Document JKMB 2012). Table 1. List of POs of the mechanical engineering and manufacturing engineering programmes List of Programme Outcomes PO1: Ability to apply knowledge of mathematics, science, and engineering PO2: Ability to identify, formulates, solve, and improve engineering problems using techniques, skills, and modern engineering tools necessary for engineering practice PO3: Ability to design a component, system, or process to meet desired needs PO4: Understanding of professional and ethical responsibility of knowledge of environment and contemporary issues 162 www.ccsenet.org/ies International Education Studies Vol. 6, No. 6; 2013 PO5: Ability to understand and apply in-depth knowledge of one or more area of specialization in mechanical engineering/manufacturing PO6: Ability to design and conduct experiments as well as to analyse and interpret data PO7: Ability to communicate and to function effectively in a team PO8: Recognition of the need for, and ability to tap into lifelong learning PO9: Knowledgeable in project management, administration, business acumen, and entrepreneurship In the course assessment, each CO contribution to that particular course is mapped to the relevant PO. This enables the evaluation and its contribution toward achieving the PO for the four years of study. Table 1 shows the list of POs for the mechanical engineering programme beginning in semester 2010/2011 (FKAB Undergraduate Handbook 2011). For each programme, such as the mechanical engineering programme for the second semester of 2010/2011, approximately 20 courses are offered including optional elective subjects. The general role of the programme coordinator is to monitor whether every course offered achieves the COs and PO according to prescribed standards. This paper discusses a template developed to help the coordinator monitor the performance of each subject offered within a semester. The huge number of courses makes it difficult for the programme coordinator to conduct an overall monitoring. However, this template aids in the process of monitoring the expected PO performance of each subject and prevents any course from being overlooked. 2. Methodology The measurement procedure was developed using Microsoft Excel format based on the standards established by the department. The template contains the course code column representing the entire departmental programme as well as the percentage fields pertaining to the achievement of outcomes for each course, as shown in Table 2. Coordinator for each course has to determine his/her CO relationship to POs as stated in the Undergraduate Handbook. They have to define the percentage workload or assessment for every COs towards the particular POs marks. The relationship between POs and COs are static, but the percentage contribution from each COs are varies depending on the instructional planning. Figure 2 shows an example of the weightage for each CO to the PO for subject KM 1914. The correlation between each PO and CO for each course is outlined in the FKAB undergraduate handbook for each semester. For instance, PO3 consist of 50% assessment marks from CO4 and C06 respectively. PO scores were obtained from the scores that have been determined during the grade coordination meetings using scoring templates developed by previous researchers (Shahrir, 2007). Every student’s assessment marks such as from examination, tutorial or quiz will contribute to their COs marks depending on the weight distribution between the assessments. Then, the individual POs marks will generate based on the individual COs marks and the weight dissemination between each CO for the particular PO. Figure 2. Sample entry weights that determine the relationship between COs and POs 163 www.ccsenet.org/ies International Education Studies Vol. 6, No. 6; 2013 Figure 3. Final percentages for student achievement of POs for the course KM 1914 For programme monitoring at the individual level, the department has set that every student should get more than 50 marks to be considered archiving that particular PO. Next, from the individual PO marks achieved by each student, the percentage scoring for the PO measurement of each course measure was calculated based on the total percentage of students who achieve at least 50 marks over the total number of students taking the course. Figure 3 shows the final marks achieved by the course at the end of the programme. The scores for each course will then be transferred to a monitoring template to determine the PO achievement for the whole programme. For programme monitoring at the course level,the department has designated that a PO is achieved if the percentage of students who scored at least 50 marks in a subject is 80%. Any subject that fails to achieve 80% will be marked in yellow, as shown in Table 2, to alert the course lecturer and programme coordinator about the PO-related achievement of the course so that further actions could be taken. However, to measure the PO for programme monitoring at the programme level, the average of these data, excluding the POs for elective subjects, is calculated. In addition, to obtain a more accurate record, the average PO for selected subjects is also calculated. 3. Results and Discussion The department offers two programmes: mechanical engineering and manufacturing engineering. To accomplish the self-EAC 2012 report, the department requires an approach for monitoring every course for each semester and for both programmes for 2010/2011 and for semester 1 2011/2012. However, in this paper, only the mechanical engineering programme is discussed with a focus on departmental compulsory courses and faculty, excluding compulsory university courses. Table 2. Percentage of students who scored a minimum of 50 marks for specific courses for the mechanical engineering programme for semester 2 2010/2011 PO MARKS CODE PO1 PO2 PO3 PO4 PO5 PO6 PO7 PO8 PO9 KKKM1024 100 100 98.5 - 100 - - - - KKKF1121 - - - 100 - - - - - KKKM1324 60.3 - - - - 91.2 - - - KKKM1344 95.7 88.4 - - - - - - - KKKJ2324 70 90 - - 70 - 98.33 - - KKKJ2344 91.8 96.7 - - 98.4 95.1 98.4 - - KKKJ2724 88.1 90.5 - - - 90.5 90.5 - - KKKJ3124 73.3 83.3 75.0 - 100.0 71.7 95.0 - - KKKJ3164 93.3 95.0 - - 88.3 90.0 - 100.0 - KKKJ3344 100 100 - - 100 100 - - - KKKJ3944 96.7 100.0 100.0 - 75.4 - 98.4 - - 164 www.ccsenet.org/ies International Education Studies Vol. 6, No. 6; 2013 KKKJ4023 100.0 98.3 98.3 100.0 100.0 100.0 100.0 100.0 - KKKJ4703 96.6 96.6 - 96.6 96.6 - 96.6 96.6 96.6 KKKJ4543 87.2 82.1 - - 82.1 87.2 - - - KKKJ4373 93.3 100.0 100.0 - 100.0 - - - - KKKJ4363 86.7 90.0 86.7 - - 86.7 - - - KKKJ4513 92 - 92 88 88 92 92 92 88 KKKJ4523 100 100 - - - - 100 - - KKKJ4413 - 84.4 90.6 - 84.4 90.6 90.6 - - KKKJ4163 95.5 100 - 81.82 - 100 100 - - Table 3. Percentage of students who scored a minimum of 50 marks for specific courses for the mechanical engineering programme for semester 1 2011/2012 PO MARKS CODE PO1 PO2 PO3 PO4 PO5 PO6 PO7 PO8 PO9 KKKF1111 - - - - - - 100 - - KKKM1513 60.3 - - - 74.4 98.7 98.7 - - KKKM1914 98.4 98.4 98.4 - 98.4 - - - - KKKM2013 100 - - - - 100 - - - KKKM2314 97.1 - - - 95.7 98.6 98.6 - - KKKM2114 87.3 96.8 - - 81 93.7 - - - KKKM2513 95.6 100 - - - 82.6 - - - KKKF3283 - 98.8 - 98.8 - - 98.8 98.8 - KKKJ3143 95.2 97.6 92.9 - 97.6 92.9 92.9 - - KKKJ3333 100 100 - - 100 100 100 - - KKKL3004 91.7 85.0 - - 96.7 - 85.0 - - KKKJ3933 81.8 100 100 - 81.8 - 100 - - KKKJ4353 95.2 96.8 - - 96.8 96.8 - - - KKKJ4953 96.6 100 100 - - - 100 100 100 KKKJ4013 100 100 100 100 100 100 100 100 - KKKJ4573 83.3 88.9 - - 94.4 - - - - KKKJ4553 77.8 77.8 - - 77.8 - - - - KKKJ4364 100 93.8 100 - - 100 - - - KKKJ4143 100 100 100 100 100 100 - - 94.4 PO assessments are correlated to the CO evaluation based on measurements that have been specified in the delivery and evaluation of the course at the beginning of each semester. Lecturers for courses provide a certain percentage of each element of evaluation (e.g., assignments, examinations, etc.) that contribute to the COs. Percentage weighting may be found in the teaching files for each semester. The percentage of students who obtained a minimum score of 50 marks is derived from the grading template and then transferred to the monitoring template. This type of monitoring can provide an overview of each subject offered to ensure the quality of PO delivery. Tables 2 and 3 show the PO performance of individual courses offered by the mechanical engineering programme over two consecutive semesters. The listed courses were offered in the specified semesters. For almost all courses, 80% of students met the required standards by scoring a minimum of 50 marks for individual 165 www.ccsenet.org/ies International Education Studies Vol. 6, No. 6; 2013 PO. However, five subjects, KKKM1324, KKKM2324, KKKJ3124, KKKM1513, and KKKJ4553, failed to meet the departmental standards. The small marks on POs, was related to the average score grade achieved by the student. If the average marks were low, it also indicates a low POs marks. All the marks were depended with the course assessment and low average may because of a lot of factors such as the type of assessment of the course, level of question being asked, the attitude of the students, the successfully of course delivery and much more. The PO achievement results will be discussed during the meeting of the quality unit (UP3) each semester (six-month cycle) after grading is completed for the course. The department, through the UP3 committee, identifies the causes of low PO achievement and informs the respective lecturers so that corrective action could be taken. Every course subject need to report their POs and COs course achievement in the course file for the department and faculty record. They also need to discuss the reason behind their low POs and COs marks (if applicable) and the action taken to counter back the deficit or the improvement can be made in the future. Changes and improvements in the delivery and assessment methods for the course are undertaken as part of a continuous quality improvement process for the programme in JKMB. For programme level monitoring, a measurement of each PO for every course can be conducted. However, selecting certain core courses that represent the measurement of a certain PO is more appropriate. The choice of courses is determined by the department based on the courses’ importance in terms of contribution to a particular PO. For example, to represent PO1, the courses considered are the KM1344, KM2114, KJ3124, KJ3933, and KJ4953, whereas for PO9, KJ4703 and KJ4953 are considered. For each PO, at least two courses are selected to provide a cumulative measurement. Figure 4 shows the average percentage of student achievement for each PO based on a particular subject or all subjects, excluding elective courses. PO is measured based on the CO’s the relationship with and contribution to the PO (CO attainment directly measured for each course). From figure 4, it shows that all POs exceeded the target of 80%. PO1 recorded the lowest achievement at 86.9% for selected subjects and 90.1% for all all subjects. Figure 4. Percentage of student achievement for each PO (Achievement for 50 marks and above) There are quite difficult to indicate student PO achievement beside the direct measurement from academic assessment and indirect measurement which is student perception. However, high student PO achievement can be demonstrated indirectly through the active participation of students in numerous national and international competitions such as the Perodua Eco-Challenge (2009, 2011, and 2012), Shell Eco-Marathon Asia (2010, 2011, and 2012), Proton Green Mobility Challenge 2012, and Malaysian Autodesk Design Competition (2010 and 2011). 4. Conclusion All POs have generally scored over 80% which are satisfied for programme level monitoring. The results are based on the theoretical aspect (class lectures) and the practical aspect (exposure through participation in 166 www.ccsenet.org/ies International Education Studies Vol. 6, No. 6; 2013 competitions and association. However, for course level monitoring, a number of courses can further enhance their PO achievement. This comprehensive monitoring system ensures that each course’s PO achievement is determined by the department. The department can identify the elements of underachievement and then inform concerned lecturers to take action for the purpose of continuous quality improvement, as intended for all EAC report documentations. References Breslow, L. (2007). Methods of Measuring Learning Outcomes and Value Added. Teaching and Learning Laboratory, Massachusetts Institute of Technology. Retrieved from http://tll.mit.edu/sites/default/files/guidelines/a-e-tools-methods-of-measuring-learning-outcomes-grid-2.pdf Fakulti Kejuruteraan, & dan Alam Bina. (2010). Universiti Kebangsaan Malaysia, Panduan Prasiswazah Fakulti Kejuruteraan dan Alam Bina, Sesi Akademik 2010–2011, Fakulti Kejuruteraan dan Alam Bina, UKM. Hamimi Fadziati, A. Wahab, Afida Ayob, Wan Mimi Diyana W. Zaki, Hafizah Husain, & Aini Hussain. Proses Pengukuran dan Penilaian Secara Langsung Hasil Program. Kongres Pengajaran dan Pembelajaran UKM 2012,13-15 Disember 2010, (Seminar Pendidikan Kejuruteraan & Alam Bina PeKA 2010, pp. 226-235). Ishak Arshad, Siti Fatin Mohd Razali, & Zawawi Samba. (2012). Pengukuran Hasil Pembelajaran Program Melalui Peperiksaan Bagi Progam Kajuruteraan Awam dan Struktur. Kongres Pengajaran dan Pembelajaran UKM 2012,18-20 Disember 2011 (Seminar Pendidikan Kejuruteraan & Alam Bina PeKA 2011, pp. 81-86). Juridah Johari, Jaafar Sahari, Dzuraidah Abd Wahab, Shahrum Abdullah, Shahrir Abdullah, Mohd Zaidi Omar, Norhamidi Muhamad. Indeks Kesukaran Peperiksaan dan hubung Kait dengan Pencapaian Hasil Program. Kongres Pengajaran dan Pembelajaran UKM 2012,13-15 Disember 2010, (Seminar Pendidikan Kejuruteraan & Alam Bina PeKA 2010, pp. 126-136). Laporan Kendiri Engineering Accreditation Council (EAC). (2012). Jabatan Kejuruteraan Mekanik dan Bahan. M. J. Allen. (2008). Strategies for Direct and Indirect Assessment of Student Learning. SACS-COC Summer Institute. Retrieved from http://www.lmu.edu/Assets/Academic+Affairs+Division/ Assessment+and+Data+Analysis/Christine$!27s+Folder/Selecting+an+Assessment+Measure/Strategies+for +Direct+and+Indirect+Assesssment+of+Student+Learning.pdf Mohd Faizal Mat Tahir, Zaliha Wahid, Zainuddin Sajuri, Shahrum Abdullah, Jaharah A Ghani, Ahmad Kamal Ariffin Mohd Ihsan, … Nor Kamaliana Khamis. (2012). Pengukuran dan Penilaian Secara Langsung Bagi Hasil Pembelajaran Kursus di Jabatan Kejuruteraan Mekanik dan Bahan: Kesan Terhadap Penambahbaikan Berterusan. Kongres Pengajaran dan Pembelajaran UKM 2012,15-18 Disember 2012, (Seminar Pendidikan Kejuruteraan & Alam Bina PeKA 2012, pp. 289-298). Nurina Anuar, Siti Rozaimah Sheikh Abdullah, & Abu Bakar Mohammad. (2012). Pengukuran Hasil Pembelajaran Program Dalam Perlaksanaan Kursus Projek Ilmiah di Jabatan Kejuruteraan Kimia dan Proses UKM. Kongres Pengajaran dan Pembelajaran UKM 2012, 18-20 Disember 2011 (Seminar Pendidikan Kejuruteraan & Alam Bina PeKA 2011, pp. 117-124). Rozeha A. R., Azami Z., & Mohd Saidfuddin M. (2007). Application of Rasch Measurement in Evaluation of Learning Outcomes: A case study in electrical engineering. Regional Conference on Engineering Mathematics, Mechanics, Manufacturing & Architecture (EM3ARC) 2007. Shahrir Abdullah, Riza Atiq Abullah O.K Rahmat, Azami Zaharim, Norhamidi Muhamad, Baba Md Deros, Norhisyam Tan Kofli … Che Husna Azhari, (2008). Implementation Continual Review of Programme Education Objective and Outcome for OBE Curriculum Baased Stakeholders’ Input, Proceeding of the 7th WSEAS International Conference on Education & Education Technology (EDU 08), Vince Italy 21–23 November 2008, pp. 218–223. Shahrir Abdullah. (2007). Panduan Penggunaan Templet Pengredan Excel (Kemahiran Kompetensi), Fakulti Kejuruteraan, Universiti Kebangsaan Malaysia. Siti Aminah Odsman, Othman Jaafar, Wan Hamidon Wan Badaruzzaman, & Riza Atiq Abullah O. K. Rahmat. (2011). Course Outcomes (COs) Evaluation for civil engineering design II course. Kongress pengajaran dan pembelajaran UKM. Wojtczak, A. (2001). Evaluation of learning outcomes: Assessment methods and measurement instruments working review. IIME Glossary of Medical Education Terms. Retrieved from http://www.iime.org/documents/elo.htm 167