Schleicher, Andreas (2016). Challenges for PISA. RELIEVE, 22(1), art. M13. DOI: http://dx.doi.org/10.7203/relieve.22.1.8429 Revista ELectrónica de Investigación e-Journal of Educational Research, y EValuación Educativa Assessment and Evaluation ISSN: 1134-4032 Challenges for PISA Desafíos para PISA Schleicher, Andreas OECD Abstract The OECD Programme for International Student Assessment (PISA) provides a framework in which over 80 countries collaborate to build advanced global metrics to assess the knowledge, skills and character attributes of the students. The design of assessments poses major conceptual Reception Date and technical challenges, as successful learning. Beyond a sound conceptual foundation, PISA 2016 April 3 needed to fulfil a range of sometimes competing demands. The school administrators, policymakers, and teachers need to be able to use this assessment information to determine how Approval Date to create better opportunities for student learning. Assessments should also provide productive 2016 June 14 feedback, at appropriate levels of detail, to fuel improvement and accountability decisions at Publication Date: each level of the education systems. In fact, also to assess of reading, mathematics and science 2016 June 14 as key foundation skills, PISA is now progressively incorporating also some of the broader cognitive, social and emotional competencies. Keywords: PISA; assessment; assessment of educational systems; education; education policy; competences Resumen El Programa para la Evaluación Internacional de Alumnos (PISA) proporciona un marco en el que más de 80 países colaboran para construir métricas globales avanzadas para explorar los conocimientos, habilidades y atributos del estudiantado. El diseño de las evaluaciones plantea importantes desafíos conceptuales y técnicos, como el éxito del aprendizaje. Más allá de una base Fecha de recepción conceptual sólida, PISA necesita cumplir una serie de exigencias a veces contradictorios. Los/as 3 Abril 2016 administradores de escuelas, autoridades normativas y los docentes deben ser capaces de utilizar Fecha de aprobació esta información de la evaluación para determinar cómo crear mejores oportunidades para el 14 Junio 2016 aprendizaje de los estudiantes. Las evaluaciones también deben proporcionar retroalimentación productiva, con niveles apropiados de detalle, para potenciar las decisiones de mejora y rendición Fecha de publicació de cuentas, en todos los niveles educativos. De hecho, además de evaluar la lectura, las 14 Junio 2016 matemáticas y la ciencia como competencias clave, PISA va incorporando progresivamente también algunas de las competencias cognitivas más amplias, sociales y emocionales. Palabras clave: PISA; Evaluación; evaluación de sistemas educativos; política pducativa; competencias The demands on modern education systems and need to learn from the extraordinary, and are evolving fast. In the past, education was sometimes we make mistakes along the way. about teaching people something. Now, it’s And it will often be the mistakes and failures, about making sure that students develop a when properly understood, that create the reliable compass and the navigation skills to context for learning and growth. A generation find their own way through an increasingly ago, teachers could expect that what they uncertain, volatile and ambiguous world. taught would last for a lifetime of their students. Today, schools need to prepare These days, we no longer know exactly students for more rapid economic and social how things will unfold, often we are surprised Autor de contacto / Corresponding author Schleicher, Andreas. Directorate for Education and Skills, OECD. 2, rue André Pascal 75775 Paris Cedex 16 (France). ([email protected]). │1 Schleicher, Andreas (2016). Challenges for PISA. RELIEVE, 22(1), art. M13. DOI: http://dx.doi.org/10.7203/relieve.22.1.8429 change than ever before, for jobs that have not will not gain the imaginative skills to connect yet been created, to use technologies that have the dots where the next invention will come not yet been invented, and to solve social from. problems that we don’t yet know will arise. The world is also no longer divided into How do we foster motivated, engaged specialists and generalist. Specialists learners who are prepared to conquer the generally have deep skills and narrow scope, unforeseen challenges of tomorrow, not to giving them expertise that is recognized by speak of those of today? The dilemma for peers but not valued outside their domain. educators is that routine cognitive skills, the Generalists have broad scope but shallow skills that are easiest to teach and easiest to skills. What counts increasingly are the test, are also the skills that are easiest to versatilists who are able to apply depth of digitize, automate and outsource. There is no skill to a progressively widening scope of question that state-of-the-art knowledge and situations and experiences, gaining new skills in a discipline will always remain competencies, building relationships, and important. Innovative or creative people assuming new roles. They are capable not generally have specialized skills in a field of only of constantly adapting but also of knowledge or a practice. And as much as constantly learning and growing, of ‘learning to learn’ skills are important, we positioning themselves and repositioning always learn by learning something. themselves in a fast changing world. However, educational success is no longer Perhaps most importantly, in today’s mainly about reproducing content knowledge, schools, students typically learn individually but about extrapolating from what we know and at the end of the school year, we certify and applying that knowledge in novel their individual achievements. But the more situations. Put simply, the world no longer interdependent the world becomes, the more rewards people just for what they know – we rely on great collaborators and Weibo or Google knows everything – but for orchestrators who are able to join others in what they can do with what they know. life, work and citizenship. Innovation, too, is Because that is the main differentiator today, now rarely the product of individuals working education is becoming more about ways of in isolation but an outcome of how we thinking, involving creativity, critical mobilize, share and link knowledge. Schools thinking, problem-solving and decision- need to prepare students for a world in which making; about ways of working, including many people need to collaborate with people communication and collaboration; about tools of diverse cultural origins, and appreciate for working, including the capacity to different ideas, perspectives and values; a recognize and exploit the potential of new world in which people need to decide how to technologies; and, last but not least, about the trust and collaborate across such differences; social and emotional skills that help people and a world in which their lives will be live and work together (Schleicher, 2012). affected by issues that transcend national Conventionally our approach to problems boundaries. Expressed differently, schools was breaking them down into manageable bits need to drive a shift from a world where and pieces, and then to teach students the knowledge that is stacked up somewhere techniques to solve them. But today we create depreciating rapidly in value towards a world value by synthesizing the disparate bits. This in which the enriching power of is about curiosity, open-mindedness, making communication and collaborative flows is connections between ideas that previously increasing (Schleicher, 2015). seemed unrelated, which requires being In many schools around the world, familiar with and receptive to knowledge in teachers are trying to help students develop other fields than our own. If we spend our such kinds of knowledge, skills and character whole life in a silo of a single discipline, we attributes. But education systems are often RELIEVE │2 Schleicher, Andreas (2016). Challenges for PISA. RELIEVE, 22(1), art. M13. DOI: http://dx.doi.org/10.7203/relieve.22.1.8429 still struggling with reflecting these in the Beyond a sound conceptual foundation, tests and assessments that are used to validate PISA needed to fulfil a range of sometimes what students know and can do. Indeed, the competing demands. Most obviously, vast majority of tasks on many conventional assessments need to be fair, technically sound tests can now be solved with the help of a and valid for purpose. They also need to build smartphone. If we want to know whether on a range of methods to ensure adequate students are smarter than a smartphone, we measurement of intended constructs and need to build more advanced tests and measures of different grain size to serve assessments. decision-making needs at different levels of the education system. Assessments should PISA project also provide productive feedback, at appropriate levels of detail, to fuel The OECD Programme for International improvement and accountability decisions at Student Assessment (PISA) provides a each level of the education systems. Teachers framework in which over 80 countries need to be able to understand what the collaborate to build advanced global metrics assessment reveals about students’ thinking. to assess the knowledge, skills and character And school administrators, policymakers, and attributes that matter for student success and teachers need to be able to use this assessment that are essential for full participation in information to determine how to create better modern societies. opportunities for student learning. PISA faces The design of assessments poses major the added challenge of ensuring that the conceptual and technical challenges, as outcomes are valid across the cultural, successful learning is as much about the national and linguistic boundaries over which process as it is about facts and figures. The they extend and that the target populations design of PISA began with the establishment from which the samples in the participating of a conceptual framework which extended countries are drawn are comparable from: the development of a working (Schleicher, 2014). definition for the assessment areas to be PISA began its assessments in 2000 with assessed and the description of the an in-depth assessment of reading literacy assumptions that underlay that definition; an skills, defined as understanding, using, examination of how to organize sets of tasks reflecting on and engaging with written texts. constructed in order to report to policy- This concept deliberately went beyond the makers and researchers on performance in traditional notion of decoding information and each assessment area; the identification of a literal interpretation of what is written, and set of key characteristics to be taken into encompassed a range of situations in which account when assessment tasks are people read, the different ways written texts constructed in ways that meaningfully reflect are presented through different media, and the learning progressions; the operationalization variety of ways that readers approach and use of the set of key characteristics to be used in texts, from the functional and finite, such as test construction; and the validation of the finding a particular piece of practical variables, and assessment of the contribution information, to the deep and far-reaching, which each made to the understanding of task such as understanding other ways of doing, difficulty. The framework then provided the thinking and being. Since 2000, the PISA foundation for the design of tasks that can be reading assessments have evolved used to generate informative student considerably, reflecting in large parts the responses, the coding/valuing of those changes in the nature of reading with the responses, the delivery of the tasks and the advent of digital technologies. In the past, gathering of the responses, and the modelling teachers could tell students to look of the responses with respect to the constructs information they were missing up in an to be assessed. encyclopaedia, and to rely on that information RELIEVE │3 Schleicher, Andreas (2016). Challenges for PISA. RELIEVE, 22(1), art. M13. DOI: http://dx.doi.org/10.7203/relieve.22.1.8429 generally being accurate and true. Nowadays, Interpreting involves determining whether digital texts require students to manage non- and how the results of mathematical work are linear information structures, to build their related to the original problem and judging own mental representation of information as their adequacy. (3) Representation entails they find their way through hypertext on the selecting, interpreting, translating between internet, and to deal with ambiguity and to and using a variety of representations to interpret and resolve conflicting information capture a situation, interact with a problem, or which they find somewhere on the web. present one’s work. (4) Reasoning and Indeed, the more content knowledge digital argument is required throughout the different technologies allow student to search and stages and activities associated with access, the more important becomes the mathematical literacy. This capability capacity to make sense out of this content, involves thought processes rooted in logic and the capacity of students to question or that explore and link problem elements so as seek to improve the accepted knowledge and to be able to make inferences from them, practices of their time (OECD, 1999). check a justification that is given, or provide a justification of statements or solutions to In 2003, the focus of PISA turned to problems. (5) Devising strategies for solving assessing mathematics, defined as students’ problems is characterised as selecting or capacity to formulate, employ and interpret devising a plan or strategy to use mathematics mathematics in a variety of contexts. The to solve problems arising from a task or assessment was about reasoning context, and guiding and monitoring its mathematically and using mathematical implementation. (6) Using symbolic, formal concepts, procedures, facts, and tools to and technical language and operations describe, explain and predict phenomena. involves understanding, interpreting, Students were also asked to demonstrate that manipulating and making use of symbolic and they could recognise the role that arithmetic expressions and operations, using mathematics plays in the world and to make formal constructs based on definitions, rules the well-founded judgements and decisions and formal systems, and using algorithms needed by constructive, engaged and with these entities. Finally, (7) using reflective citizens. This approach asserts the mathematical tools involves knowing about importance of mathematics for full and being able to use various tools (physical participation in society and it stipulates that or digital) that may assist mathematical this importance arises from the way in which activity, and knowing about the limitations of mathematics can be used to describe, explain such tools (OECD, 2003). and predict phenomena of many types. PISA has established a set of seven fundamental In 2006, the focus of PISA shifted to mathematical capabilities that underpin science, defined as the ability of students to performance in the PISA mathematics engage with science-related issues, and with assessments: (1) Communication is both the ideas of science, as a reflective citizen. To receptive and expressive. Reading, decoding do well on the PISA science test, students and interpreting statements, questions, tasks need to be willing to engage in reasoned or objects enables the individual to form a discourse about science and technology. This mental model of the situation. Later, the requires the competencies to (1) explain problem-solver may need to present or phenomena scientifically (which implies the explain the solution. (2) Mathematising ability to recognise, offer and evaluate involves moving between the real world and explanations for a range of natural and the mathematical world. It has two parts: technological phenomena), (2) evaluate and formulating and interpreting. Formulating a design scientific enquiry (which implies the problem as a mathematical problem can ability to describe and appraise scientific include structuring, conceptualising, making investigations and propose ways of addressing assumptions and/or constructing a model. questions scientifically) and (3) to interpret RELIEVE │4 Schleicher, Andreas (2016). Challenges for PISA. RELIEVE, 22(1), art. M13. DOI: http://dx.doi.org/10.7203/relieve.22.1.8429 data and evidence scientifically (which maintaining shared understanding; (2) taking implies the ability to analyse and evaluate appropriate actions to solve problems; and (3) data, claims and arguments in a variety of establishing and maintaining team representations and draw appropriate organisation. To facilitate this, individual scientific conclusion). Explaining scientific students are required to interact and and technological phenomena demands a collaborate with computer-generated team knowledge of the content of science. The member(s) in controlled situations to solve a second and third competencies, however, particular problem. This process necessitates require more than a knowledge of what we students determining their own role and know. Rather, they depend on an responsibilities in regards to other agents, understanding of how scientific knowledge is monitoring aspects of group organisation, and established and the degree of confidence with facilitating adjustments and changes that are which it is held. Recognising and identifying needed when communication breaks down, the features that characterise scientific when new obstacles appear, or when enquiry requires a knowledge of the opportunities for performance optimisation procedures that are the foundation of the arise. diverse methods and practices used to Taking this further, countries are currently establish scientific knowledge – referred to collaborating to develop an assessment of here as procedural knowledge. Finally, the global competence to be delivered in 2018 competencies require epistemic knowledge – and defined as the capability and disposition an understanding of the rationale for the to act and interact appropriately and common practices of scientific enquiry, the effectively, both individually and status of the knowledge claims that are collaboratively, when participating in an generated, and the meaning of foundational interconnected, interdependent and diverse terms such as theory, hypothesis and data world. The assessment is built around four (OECD, 2006). key dimensions: (1) Communication and relationship management – which refers to the Challenges for PISA project willingness and capability to adapt one’s While continuing with the assessment of communication and behaviour in order to reading, mathematics and science as key interact appropriately and effectively with foundation skills, PISA is now progressively others holding diverse perspectives and in incorporating also some of the broader different contexts. (2) Knowledge of and cognitive, social and emotional competencies interest in global developments, challenges discussed above. The assessment of social and trends – which refers to a learner’s competencies became a priority in 2015. As interest in and knowledge of cultures, major noted before, young individuals entering into issues, events and phenomena in the world, as the workforce and public life need the skills well as the learner’s ability to understand their and attitudes to collaborate and effectively global significance and their implications for solve problems, increasingly in situations adapting appropriately and effectively to where members of the group are learning, working, and living situations with geographically dispersed, working in different others holding diverse perspectives and in time zones, and connected through different contexts. (3) Openness and technology. Societies expect them to have the flexibility – which refers to being receptive to capacity to resolve problems and provide and understanding of new ideas, people and solutions collaboratively through the pooling situations, as well as to differing perspectives of knowledge, skills, and effort. As a first and practices. It also refers to the ability to step, PISA introduced in 2015 an assessment seek out and understand new and differing of collaborative problem solving skills, which perspectives and experiences and assesses students according to three core appropriately and effectively adapt one’s competencies: (1) establishing and thinking, behaviours and actions to learning, RELIEVE │5 Schleicher, Andreas (2016). Challenges for PISA. RELIEVE, 22(1), art. M13. DOI: http://dx.doi.org/10.7203/relieve.22.1.8429 working, and living situations that involve and Problem Solving knowledge and skills. others holding diverse perspectives and in Paris: OECD. Retrieved from different contexts. (4) Emotional strength and http://www.oecd.org/edu/school/programmef resilience – which refers to the ability, orinternationalstudentassessmentpisa/336948 developing the coping mechanisms and 81.pdf resilience, to deal appropriately with the OECD. (2006). Assessing Scientific, Reading ambiguity, changes, and challenges that these and Mathematical Literacy. A Framework different perspectives and experiences can for PISA 2006. Paris: OECD. Retrieved from present. http://www.oecdilibrary.org/docserver/downl As our societies evolve further, the oad/9806031e.pdf?expires=1465551043&id= countries participating in PISA will continue id&accname=guest&checksum=02E5A7F7B to collaborate to define and measure the 73F1CCFA0DA6E8336A2F3D8 knowledge, skills and character attributes that Schleicher, A. (2012). Preparing Teachers and will help individuals to keep abreast of these Developing School Leaders for the 21st changes and to meet rising demands. Every Century: Lessons from around the World. three years, the results from PISA provide a Paris: OECD Publishing. doi: powerful tool that countries can use to http://dx.doi.org/10.1787/9789264xxxxxx-en develop, review and fine-tune their curricular standards and education policies with the aim Schleicher, A. (2014). Equity, Excellence and to provide the best education possible for all Inclusiveness in Education Policy Lessons of their students. The OECD stands ready to from Around the World. Paris: OECD support countries in this challenging and Publishing. Retrieved from crucial endeavour. https://www.ucy.ac.cy/equality/documents/A rticles-Material/OECD_2014_Report.pdf References Schleicher, A. (2015). Schools for 21st- OECD. (1999). Measuring student knowledge Century Learners: Strong Leaders, Confident and skills. A new Framework for assessment. Teachers, Innovative Approaches, Paris: OECD. Retrieved from International Summit on the Teaching http://www.oecd.org/edu/school/programmef Profession. Paris: OECD Publishing. doi: orinternationalstudentassessmentpisa/336939 http://dx.doi.org/10.1787/9789264231191-en 97.pdf OECD. (2003). The PISA 2003. Assessment Framework. Mathematics, Reading, Science Author / Autor To know more / Saber más Schleicher, Andreas ([email protected]). Director for Education and Skills, and Special Advisor on Education Policy to the Secretary-General at the Organisation for Economic Co-operation and Development (OECD) in Paris. the work of the Directorate includes the Programme for International Student Assessment (PISA), the OECD Survey of Adult Skills (PIAAC), the OECD Teaching and Learning International Survey (TALIS), and the development and analysis of benchmarks on the performance of education systems (INES). Before joining the OECD, Mr. Schleicher was Director for Analysis at the International Association for Educational Achievement (IEA). He studied Physics in Germany and received a degree in Mathematics and Statistics in Australia. His postal address is: Directorate for Education and SkillsOECD 2, rue André Pascal 75775 Paris Cedex 16 (France) RELIEVE │6 Schleicher, Andreas (2016). Challenges for PISA. RELIEVE, 22(1), art. M13. DOI: http://dx.doi.org/10.7203/relieve.22.1.8429 Revista ELectrónica de Investigación y EValuación Educativa E-Journal of Educational Research, Assessment and Evaluation [ISSN: 1134-4032] © Copyright, RELIEVE. Reproduction and distribution of this articles it is authorized if the content is no modified and their origin is indicated (RELIEVE Journal, volume, number and electronic address of the document). © Copyright, RELIEVE. Se autoriza la reproducción y distribución de este artículo siempre que no se modifique el contenido y se indique su origen (RELIEVE, volumen, número y dirección electrónica del documento). RELIEVE │7