From Kenya to India to the United States, world leaders are realizing the global learning crisis that threatens to rob millions of children of the fundamental human right to education, and the knowledge and skills required for well-being and prosperity in the 21st century. Well before Education for All (EFA) was endorsed in 1990, educators recognized that providing access to schooling without also ensuring student learning is an empty promise. With a new set of global development goals on the post-2015 horizon, what can the education community do now to catalyze a shift in global focus and investment from universal access to ensuring access plus improving learning opportunities and outcomes worldwide?
Beginning with the End in Mind
To help answer this question, the Learning Metrics Task Force (LMTF) has undertaken an 18-month-long process of research, global consultation and consensus-building on learning measurement with education stakeholders around the world. Taking a cue from effectiveness guru Stephen Covey, the LMTF process is designed to force the education community to “begin with the end in mind.” Accordingly, Phase I of the project sought to identify the learning end-goal by answering the question, what do all children and youth need to learn in order to succeed in the 21st century?
Considering recommendations from a working group of experts chaired by Seamus Hegarty— visiting professor at the University of Warwick and former chair of the International Association for the Evaluation of Educational Achievement (IEA)—the task force decided in its first in-person meeting in September 2012 that indeed there were important competencies that all children and youth should master no matter where they live in the world. The first report from the task force, Toward Universal Learning: What Every Child Should Learn, presents a broad, holistic framework of seven learning domains, with various competencies in each, as the aspiration for all children and youth across the globe. The seven domains are:
• Physical well-being
• Social and emotional
• Culture and the arts
• Literacy and communication
• Learning approaches and cognition
• Numeracy and mathematics
• Science and technology
After establishing what children should learn, Phase II of the project works back to the preceding step: how will we know whether learning is occurring under each of the seven domains? More specifically, how can we measure and track progress in learning at the global and national levels? The Measures and Methods Working Group of 60 experts in education and assessment, chaired by César Guadalupe, Associate Researcher at Universidad del Pacífico in Perú, is currently working to provide technical guidance to help the task force address this question.
A Meeting in Dubai
On February 20-21, the working group presented its recommendations on this topic to the task force at an in-person meeting hosted by Dubai Cares. Representatives of the education community convened from all over the globe to focus their collective expertise on the issue of learning measurement. Among the 44 attendees were representatives of low, middle and high-income countries; stakeholders from Africa, Asia, Europe, the Middle East, North and South America, and Oceania; key U.N. and multilateral agencies; regional bodies; teacher organizations; civil society; and bilateral donor agencies. Not in attendance but still heard were the voices of the more than 500 individuals from 49 countries who submitted comments on a previous draft through public consultation.
With so many diverse constituencies and perspectives at the table, task force members engaged in vigorous debate over the two days. Here we highlight a few of the most contentious topics, and then outline the decisions that emerged from the meeting.
Key Debates
Scope of Measurement and Unintended Consequences
In formulating its recommendations, the task force faced the challenge of striking a delicate balance: how to communicate the importance of all seven learning domains presented in the Phase I framework, while also identifying just a small set of measures for tracking at the global level.
Some members worried that pulling out just a few domains for global measurement might signal to policymakers and education systems that the other domains are less important. This might in turn have the unintended consequence of limiting diversity in national curricula or driving donor funding toward narrow learning goals. Others argued that the task force must limit its recommendations to no more than a handful of clearly articulated goals, or risk having noneducationalists do the job and potentially leave learning out of the post-2015 agenda altogether.
In the end, the task force agreed that it was necessary to identify a small number of measures for tracking at the global level and selected six specific domains of measurement. To guard against unintended consequences, the task force proposed a new global measure that would track the breadth of learning opportunities young people received; namely, are children and youth being given the opportunity to learn across all seven learning domains? The task force also emphasized the need to operationalize the global domains of measurement while simultaneously helping to build measurement capacity at the national level.
International Comparability and Statistical Rigor
Assessments such as PIRLS, TIMSS and PISA are regarded as the gold standard for internationally comparable learning assessment. However, a country can spend up to $250,000 to participate, plus the costs to administer the tests, which can be substantially more. On the other hand, some national assessments have high levels of statistical rigor and provide information that is often more relevant to the individual country context.
The following questions regarding international comparability were hotly debated, both within the working group and at the task force meeting:
- Should global learning goals be measured in an internationally comparable way, that is, using the same measures everywhere? Or is it also valuable to use a common tool globally that would allow some basic information to be compared globally but would mainly enable countries to track their own progress over time?
- Do all countries have the interest or the resources to participate in the international assessments currently available? Drawing a parallel to the pharmaceutical industry, should the education community make available “generic” (as opposed to branded) test items as a global public good?
- How do large-scale international assessments impact policy and practice to improve learning outcomes? Is the information generated from assessment more useful to high-income countries? How can international assessments provide information beyond “static league tables”?
Task force members finally agreed that international comparability was important in some areas, such as reading comprehension and mathematics, but measuring learning using a common tool to compare progress over time, or equity in learning outcomes, could also be useful in tracking progress toward global goals.
Measuring Learning in School and Out of School
Some task force members argued it is pragmatic to focus measurement efforts in schools not only because it is economical, but also because the primary aim of large-scale assessments is to influence education policy and school systems. And after all, isn’t the goal to get all children into school? But other members were concerned that a continued focus on measuring learning within schools, where the majority of current assessment efforts are focused already, would lead to further exclusion of out-of-school children and youth. With an estimated 61 million primary-age-children out of school, clearly the access agenda is unfinished. Still others pointed out that given current trends and the proliferation of learning technologies, learning might take place in a wider range of contexts in the future. How can the task force’s recommendations allow for these innovations?
The task force decided that access indicators (on enrollment and completion) should be paired closely with learning indicators to maintain a focus on getting children into school while also improving learning outcomes. Furthermore, the task force promotes a broad definition of schooling that allows for a range of intentional learning contexts (e.g., job-embedded learning, nonformal programs, distance learning), beyond the walls of the traditional school building.
Accounting for a Diversity of Contexts and Learning Levels
The task force recognized that while some (mostly high and upper-middle-income) countries participate in rigorous national and internationally comparable assessments, others have a very limited “culture of evaluation” and therefore limited information on how well their education systems are functioning. Another challenge is how to account for existing learning levels in countries where a large proportion of learners would score below the lowest internationally benchmarked levels. Expanding internationally comparable tests to these contexts would not give policymakers, educators or the general public information about what the problem is or how to fix it.
There was general consensus among task force members that building on internationally comparable assessments was advisable in some contexts, but may not be a good fit for countries where vast numbers of children are unable to read in the language of the test. In these environments, additional tools would be necessary to capture all learning levels.
Task Force Decisions
After hours of debate and deliberation on these topics, both in small break-out sessions and as a whole group, the task force came to consensus on the following set of decisions.
The task force has identified six domains of measurement that represent important learning opportunities for children and youth to enable them to become effective global citizens:
1.) Access to and Completion of Learning Opportunities
Rationale:Children and youth must access, and most importantly complete their education.This domain addresses the unfinished access agenda for out-of-school children and youth and emphasizes the importance of tracking completion, which currently is not done systematically. It also allows for a broad definition of schooling, including any intentional learning programs, whether formal, nonformal or virtual. Evidence shows that the skills and knowledge needed for global citizenship are rarely learned outside of intentional learning activities.
2.) Exposure to a Breadth of Learning Opportunities Across all Seven Domains
Rationale: Children and youth should have a breadth of learning opportunities that, at a minimum, covers the seven learning domains. It is expected that an even broader set of competencies is necessary at the national and local levels; however, the task force recommends that the breadth of learning that education systems offer, at least across these seven domains, be tracked globally.
3.) Early Childhood Experiences that Result in Readiness for Primary School
Rationale: The early childhood years are critical to later learning and development. Entry to primary school is a key milestone in a child's learning trajectory, and measuring school readiness can help drive improvements in preprimary education, health, family services, etc. Because of the varying rates at which young children develop, a holistic measure across multiple domains is the best way to capture learning at this stage. School readiness is broadly defined and typically includes aspects of learning related to at least five of the seven domains: physical well-being, social and emotional, literacy and communication, learning approaches and cognition, and numeracy and mathematics.
4.) The Ability to Read and Understand a Variety of Texts
Rationale: Children and youth must be able to communicate in their mother tongue and in the primary language of instruction. Foundational reading skills necessary for learning to read are critical for functioning in modern society, in addition to the ability to comprehend and analyze complex texts through a variety of media. This domain encompasses both primary and lower secondary levels.
5.) The Ability to Use Numbers and Apply this Knowledge to Real-Life Situations
Rationale: Children must be able to count and understand mathematical concepts both to make informed economic choices and to pursue advanced learning in such disciplines as science, engineering, economics, research, technology, etc. This domain also encompasses both the primary and lower secondary levels.
6.) An Adaptable, Flexible Skill Set to Meet the Demands of the 21st Century
Rationale: Beyond literacy and numeracy, children and youth need a variety of skills across the seven learning domains to succeed in the 21st century. Administered in lower secondary, this domain of measurement might cover multiple competencies such as environmental awareness, collaborative problem solving, information communications technology digital literacy, and social responsibility.
Information for these domains of measurement would be collected using internationally comparable assessments in some cases, such as reading comprehension and mathematics, and using alternative assessments for others. Data collected against these domains of measurement should describe average achievement levels in addition to progress over time and equity across groups (girls/boys, urban/rural and wealth levels, at a minimum).
The task force also recognizes that a system of global measurement will only be effective in improving learning outcomes if there is a simultaneous focus on improving capacity within countries to conduct and use learning assessments. The Measures and Methods Working Group proposed the establishment of a neutral international body, which may be an expansion of an existing agency, to operationalize these recommendations. In Dubai, the task force agreed to explore the feasibility of such an advisory group to track progress against the global domains of measurement and help build national capacity for measuring learning. In Phase III of the LMTF project, the Implementation Working Group will investigate this idea further and conduct an analysis of how to best translate these recommendations into practice.
Next Steps and Phase III
The next steps in the process include the April release of the task force’s final recommendations from the meeting in Dubai. A more detailed technical report on the Phase II process and results will follow.
The third and final working group on implementation will convene from March though August 2013. Led by Dzingai Mutumbuka—chair of the Association for the Development of Education in Africa (ADEA) and former minister of education of Zimbabwe—the Implementation Working Group will consider, among other things:
- The feasibility of setting up a neutral international advisory group on global learning measurement.
- How governments can convene stakeholders to improve learning measurement across the seven learning domains.
- What resources and financing would be needed to implement task force recommendations.
- How to operationalize the two proposed domains of measurement for which there are currently no global measures (i.e., breadth of learning opportunities and 21st century knowledge and skills).
In July 2013, the task force will meet for a third time in person to hear the working group’s proposal.
Finally, a word on project scope. As originally conceived, the job of the Learning Metrics Task Force is to build consensus around global aspiration for learning and the measurement of learning outcomes. The task force recognizes that this is only one small piece of the larger quality puzzle, albeit an important one. Assessments alone will not improve the quality of instruction or learning environments; rather, they provide a better understanding of outcomes to enable policymakers and educators to develop strategies for improving learning, while taking into account many other factors. In response to multiple requests for the task force to make recommendations regarding inputs, such as curriculum, instruction, etc., discussions are now underway to explore whether there should be an LMTF Phase IV to expand the project scope beyond learning metrics.
To receive regular updates on the task force’s progress, send your name and email address to LearningMetrics@brookings.edu. Or visit brookings.edu/learningmetrics for more information.
Authors
- Rebecca Winthrop
- Mari Soliván
- Kate Anderson Simons