THIRD INTERNATIONAL MATHEMATICS AND SCIENCE STUDY 

Introduction to the U.S.. National Research Center Report

This is the first in what we expect will be a series of newsletters published by the U.S.. National Center. This issue introduces the U.S.. National Research Center and the Third International Mathematics and Science Study. In future issues we will report on progress of the study and write about some of the findings as data become available and papers are written. This newsletter will report, briefly, on topics related to the study which we hope will be of interest to our readers.

We also hope to hear from our readers, so if you have questions, want additional information about a topic addressed in the newsletter or the study in general, please feel free to contact us. You can write or call Gilbert Valverde, Assistant Director - TIMSS, 115 Erickson Hall, Michigan State University, East Lansing, Michigan 48824-1034, telephone 517-353-7755, fax 517-336-1727, E-Mail, 21547gav@msu.edu. If you have suggestions of articles you would like to see in future newsletters or if you or someone you know did not receive this newsletter directly, but would like to be on our mailing list, please send your name and address along with your request to Jacqueline Babcock, same address as above or E-Mail: jbabcock@msu.edu.

We also are interested in making presentations to professional groups or associations who might be interested in being introduced to TIMSS or receiving updates on the project. If your organization is interested in hearing more about TIMSS and would like a representative from our office to present to your group, please let us know.

What is TIMSS?

The Third International Mathematics and Science Study (TIMSS) will be the most important international study of educational outcomes in the 1990ís. Sponsored by the International Association for the Evaluation of Educational Achievement (IEA), TIMSS will be conducted in approximately 50 countries from all parts of the world.

TIMSS is taking place in two phases 1991-1995 and 1996-1999. The strengths of the study come from enhancing the traditional survey of achievement to include a focus on instructional practices, as well as recognizing curricular influences on student learning. Analyses will focus on the curriculum as it is intended, as it is actually delivered in the classroom, and as it is learned by students.

TIMSS will also focus on societal contexts at the national, community, and individual student levels, and will explore studentsí attitudes and opinions. Nine-year-olds, thirteen-year-olds, and students in their last year of secondary school comprise the target populations. Mathematics and science are seen as essential parts of every young personís education, necessary for preparing them for integration within an increasingly technological work place.

TIMSS will seek to identify variables associated with high levels of achievement in mathematics and science, and will endeavor to explain characteristics that influence educational performance. The study will provide a wealth of information about the teaching and learning of mathematics and science, including, for example, the following:

*comprehensive documentation of current national and international data about curricula, teaching practices, and student achievement and attitudes;

*an assessment of the range and impact of alternative curricular offerings, teaching practices, and administrative arrangements on learning;

*an identification of what is possible in the teaching of mathematics and science.

TIMSS signals a new standard of quality and sophistication in international surveys of education, and will provide an unparalleled source of information for those seeking to improve educational policy and practice. The study will be of interest to a wide range of audiences including parents, academics, educators, policy-makers, researchers and politicians.

Projected Project Dates:

1993: Completion of instrument development; 1994: Full field trials; 1994-1995: Main data collection - Phase I; 1998 Main data collection - Phase II.

(The above article was excerpted from the ìProject Overviewî and TIMSS brochure produced by the ICC in Vancouver, Canada.)

The U.S. National Research Center

IEA project structure revolves around the International Study Committee which consists of three groups: an internationally composed Steering Committee, an International Coordinating Center (located at University of British Columbia, Vancouver, Canada under the direction of Dr. David Robitaille), and National Research Centers located in each of the participating countries. The National Science Foundation in conjunction with the National Center for Education Statistics has awarded a grant to establish the United States National Research Center for the Third International Mathematics and Science Study at Michigan State University under the direction of William H. Schmidt. The U.S. Center, however, includes not only Michigan State University, but also the National Center for Education Statistics and the National Science Foundation.

The purpose of the National Research Center is to assure:

1. those persons in the United Stated with the best qualifications and greatest experience in evaluation of mathematics and science performance and in international assessments are involved in the design of the study and have full opportunity for analysis of results;

2. the responsibilities of the United States government agencies which support the study are realized in fully effective implementation of the study;

3. the United States TIMSS project is efficiently integrated with the international TIMSS study.

The U.S. National Research Center is composed of three parts: a Steering Committee with responsibility for advice on planning, design and analysis; a National Research Coordination Committee with responsibility for oversight of study operations and a contractor to conduct the study.

Responsibilities of the National Project Coordinator are to assist in forming, chairing and arranging staff support for the U.S. National Steering Committee, to chair the National Research Coordinating Committee, to maintain continuing contacts with the International Coordinating Center and committees for the conduct of TIMSS and to supervise staff to assure logistical support and project design analysis functions are carried out.

U.S.. National Steering Committee

The U.S. National Steering Committee includes experts in many fields from higher education institutions, government and business, across the United States. The committee is an advisory committee and has met three times in Washington, D.C. (October 1991 and March and October 1992). The next meeting will take place in May 1993. Members include: Dr. Gordon Ambach, Executive Director, Council of Chief State School Officers; Dr. Deborah Ball, Associate Professor, Michigan State University; Dr. Leigh Burstein, Professor, Graduate School of Education, University of California-Los Angeles; Dr. Audrey Champagne, University at Albany-SUNY; Dr. Jewel Plummer Cobb, California State, Los Angeles, Dr. David Cohen, Professor and Director of the Institute for Research on Teaching and Learning Across the Life Span, Michigan State University; Dr. John Dossey, Distinguished Professor of Mathematics, Illinois State University; Dr. Emerson Elliott, National Center Education Statistics; Dr. Sheldon Glashow, Higgings Professor of Physics and Mellon Professor of the Sciences, Harvard University; Dr. George E. Hall, President, Slater Hall Information Products; Dr. Larry Hedges, Professor, Department of Education, University of Chicago; Professor Henry Heikkinen, Department of Chemistry and Biochemistry, University of Northern Colorado; Dr. Jeremy Kilpatrick, Regents Professor of Mathematics Education, University of Georgia; Dr. Mary Lindquist, Fuller E. Callaway Professor of Mathematics Education, Columbus College, and President of the National Council of Teachers of Mathematics; Dr. Marcia C. Linn, Professor and co-director of the Center for Research on Evaluation, Standards, and Student Testing, University of California-Berkeley; Dr. Robert L. Linn, Professor of Education, University of Colorado; Dr. Paul Sally, Professor of Mathematics, University of Chicago; Dr. Richard Shavelson, Dean, Graduate School of Education and Professor, Department of Education, University of California-Santa Barbara; Dr. Bruce Spencer, Professor, Department of Statistics and School of Education and Social Policy, Northwestern University; Dr. Elizabeth Stage, Executive Director, California Science Project, University of California-Berkeley; Dr. Harold Stevenson, Professor of Psychology, University of Michigan; Dr. James Taylor, Yankelovich, Clancy, Shulman, Westport, CT; Dr. Kenneth Travers, Office of Studies, Evaluation and Dissemination, National Science Foundation; and Dr. Paul H. Williams, Professor, Department of Plant Pathology, University of Wisconsin. The committee is chaired by Dr. William Schmidt, Professor, Michigan State University and National Project Coordinator of the U.S. National Research Center-TIMSS.

National Research Coordinating Committee

This group meets regularly in Washington, D.C. Members are: Dr. Jeanne Griffith, Associate Commissioner for Data Development, National Center for Education Statistics; Dr. Eugene Owen, National Center for Education Statistics; Dr. Lois Peak, National Center for Education Statistics; Dr. Larry Suter, Office of Studies and Program Assessment, National Science Foundation. The committee is chaired by Dr. William Schmidt, Michigan State University.

Introducing.....!

The United States National Research Center for the Third International Mathematics and Science Study (TIMSS) is located at Michigan State University in conjunction with NSF and NCES in Washington, D.C. Persons working on the project in East Lansing include:

William H. Schmidt, Professor and Executive Director of the U.S. National Research Center (National Project Coordinator). Schmidt also serves on the International Steering Committee for IEA, sponsor of TIMSS.

Gilbert A. Valverde, Assistant Director and Senior Research Associate, is ABD in comparative education at University of Chicago.

Senta Raizen, is the science subject matter consultant to the project. Raizen is Co-Director of The NETWORK, The National Center for Improving Science Education located in Washington, D.C.

Curtis McKnight, Associate Professor at University of Oklahoma is the mathematics subject matter consultant.

Edward Britton, Research Coordinator, with The NETWORK in Washington holds an Ed.D. from Florida State University and assists the project in science.

Leland Cogan, Research Associate, is ABD in Educational Psychology at Michigan State University.

Jacqueline E. Babcock, Executive Administrative Assistant, holds a Masters in Public Administration from Western Michigan University.

John Radford, Administrative Assistant for Budgeting and Financial Planning, holds an Accounting degree from Michigan State University.

Ann Jenkins, U. S. National Center Secretary, Nicole Gilmer and Margarita Bauza, university students, are responsible for keeping the U.S.. National Center office operating efficiently.

Doctoral level research assistants working on various phases of TIMSS in the U.S.. Center include: Norhayati Abdmukti, Louis Brandon, Jimin Cho, Virginia Keen, Mary Kino, Charles Rop, and Margaret Savage.

Thank You!

A project as large and diverse as the Third International Mathematics and Science Study involves a cast of a thousand people. We have been fortunate to have the cooperation and support of many individuals and organizations and we plan to take the opportunity to recognize some of these people in each newsletter. 

We have relied heavily on many members of the Council of Chief State School Officers (CCSSO). Gordon Ambach has provided valuable guidance and support throughout the project. Ramsay Selden and Edward Roeber have assisted us in contacting mathematics and science supervisors in the 50 states, many of whom we have talked with by telephone and others who have taken the time to respond to our questionnaires. Charles Allen, Lansing, MI; Charles A. Reeves, Tallahassee, FL; Bill Hopkins, Austin, TX; and Fernand J. Prevost, Concord, NH; mathematics supervisors and William Spooner, Raleigh, NC; Richard Clark, St. Paul, MN; and John C. Cairns, Dover, DE, science supervisors spent a day in Washington, D.C. meeting with Gilbert Valverde of the National Research Center last spring to assist with completion of one of the national participation questionnaires required by the International Coordination Center for each country participating in the study. This effort was financially supported by the National Center for Education Statistics. 

We wish to also acknowledge the assistance of persons at the Mathematical Sciences Education Board (MSEB), notably Kenneth Hoffman and Linda Rosen. At the National Academy of Sciences we also wish to acknowledge the advice of Dorothy Gilford, Executive Director of BICSE, and Elizabeth Stage (Director), Henry Heikkinen, and David Florio from Critique and Consensus, National Science Education Standards Project, National Research Council. Thank you also to the Board on International Comparative Studies in Education (BICSE) Board currently chaired by Marshall Smith, Dean of the College of Education at Stanford University. 

We have also had the help of many members of National Council of Teachers of Mathematics (NCTM) and National Association of Science Teachers (NAST). 

The following school district administrators and students must also be thanked for their willingness to assist us in the piloting of achievement test items which occurred in late November 1992: Terrance Coburn, Curriculum Supervisor, Oakland Intermediate School District; Rita Traynor, Assistant Superintendent for Instruction, Novi Community Schools; and John Radke, Testing and Curriculum Coordinator, East Detroit Public Schools, all located in Michigan. We also want to offer our appreciation to the following schools who assisted in piloting of open ended achievement items during spring of 1992 : Lansing School District and Flint Public Schools in Michigan.
 
 

RESEARCH NOTES 

CURRICULUM ANALYSIS: TOPIC TRACE MAPPING

International curriculum analysis is important to researchers, policy makers and educators. It provides a rich context for understanding and explaining international differences in student achievement and attitudes that will be uncovered in TIMSS. It will also enable a comparison of international curricular intentions through a common framework that is of interest in itself. The curriculum analysis should help policy makers in different countries make informed decisions about the content, form, selection and use of curricular documents, describe and direct reforms of curricula and assist them to critically question underlying assumptions embedded in their own curricula.

Results of the curriculum analysis will provide information on cross-national variations in curricular content, providing insight into what students at various grade levels are exposed to across the world. This will assist us in gaining insight into the curriculum development process and will document changes influenced by the introduction of technology into classrooms. This form of investigation will contribute greatly to new knowledge about the structure, design, and use of textbooks in countries and will provide information necessary for the interpretation of teaching practices and student outcomes associated with TIMSS. We expect to gain insight into factors which may influence the relationship such as streaming, degree of subject emphasis, content sequencing, and degree of national control of the curriculum.

In a world that many see as interdependent, the curriculum analysis will provide insight into an international range of topics intended for instruction, the range of initial entry and terminating points for topics, the range of sequencing and the range of topic foci. It will enable us to draw a picture of what mathematical and scientific knowledge is most valued world-wide and what mathematical and scientific concepts, skills, and values are common internationally. It should provide empirical insight into what many term world class standards of mathematics and science instruction.

Design of the Procedures

Intended Curriculum

The intended curriculum is the definition through official documents (curriculum guides) and semi-official documents (textbooks) of what a country expects to be taught or has as learning goals for pupils in its educational system. This intention does not characterize what is necessarily delivered but it does represent the goals of the system at least from the national perspective. We recognize that such learning goals can be specified at other levels in the system including region, school and teacher but the focus here is on the national specification of such goals.

In some countries this intention is very formal and is explicitly stated in official publications of the government while in others the intention is less formally defined and must be inferred from semi-official documents such as nationally used textbooks. A case somewhere in between is where the "national intention" is formally defined but for regions within a country.

The intended curriculum of interest in TIMSS is that defined for the primary, lower secondary and upper secondary portions of the educational system, this is to say, the pre-university segment. Thus in mathematics and science the intended curriculum would specify the educational goals for each grade level in the system from the beginning of schooling in that country to the end of secondary schooling.

For different countries, the level of detail in the specification of the learning goals might vary appreciably in official documents such as curriculum guides, but textbooks on the other hand provide very detailed statements of curricular intent in almost all cases. It is for this reason that our definition of the intended curriculum includes statements of learning goals derived both from curriculum guides and textbooks.

The measurement of the intended curriculum would ideally include an analysis of the relevant documents (a combination of textbooks and curriculum guides idiosyncratic to a country's educational system) from all relevant grade levels. This exhaustive characterization would allow a fairly complete measurement of the learning goals for a country.

This process, however, is further complicated by the fact that many countries differentiate the learning goals for various subgroups of students who are placed in streams or tracks. Thus the specification of intent is different at various grade levels in the educational system depending on the group of students to which one is referring. This implies that the tracing of educational intent is complex and can have multiple branches each specified by a unique curriculum guide and/or a unique set of textbooks.

Curriculum Measurement

As a result, the complete measurement of the learning goals for a country would necessitate the analysis of the curriculum guide and all textbooks available at each grade level and for each track or stream present at that grade level. To conduct such an analysis on a cross-national basis is neither practical nor economically feasible. There is simply too much detail, too many textbooks and too many grade levels to do this in any practical way.

The measurement task thus becomes one of devising a procedure that: 1) balances the need for depth in the specification of learning goals at the focal grade levels at which achievement data will also be collected in TIMSS (Population 1, Population 2, and Population 3b) and the need for breadth in the specification of learning goals across the entire pre-university educational system, 2) rests on a sampling plan idiosyncratic to each country that selects a representative sample of curriculum guides and textbooks that derives from the nature of that educational system, 3) provides a general context in which to interpret the specification of intent as found in curriculum guides and textbooks, i.e., a characterization of the nature of mathematics and science education and 4) characterizes the educational system and how it is organized as well as to characterize the decision making process by which the learning goals are specified.

The measurement procedures, especially those underlying (1), (2) and (3) rest on the existence of a common curriculum framework onto which the intended curriculum of each country can be mapped. Such a framework has been developed and a description is found elsewhere.

Using this framework the first phase of the curriculum measurement process is comprised of four parts which together are designed to provide a detailed characterization of the learning goals. The four parts are: (1) the expert questionnaires, (2) the document analysis of curriculum guides and textbooks (3) the in-depth topic trace mapping and (4) the modified topic trace mapping.

The second phase of the curriculum measurement process is comprised of questions asked on the participation questionnaires of all countries relative to the nature of the educational system and the locus of decisions affecting the curriculum. The synthesis of the data describing the learning goals together with the data describing country level system organization and curricular decision making structures will permit an extensive characterization of aspects of the intended curriculum.

Depth or Breadth

The first issue is one of depth versus breadth. TIMSS has as its primary focus the characterization of achievement patterns, learning goals, teacher qualifications and instructional organization at four focal points including nine year olds, 13 year olds and end of secondary schooling for those students focusing on mathematics and science and those not having this focus in their instructional program. It would naturally follow that the measurement of the intended curriculum or the learning goals specified at the national level should focus at these grade levels as well.

Document analysis. The document analysis of textbooks and curriculum guides is designed to serve this purpose. It produces a rather detailed description of learning goals but only at the focal grade levels. This procedure is designed to be complete and exhaustive at least in terms of the materials that are analyzed and in terms of coverage of all topics in the frameworks.

The strength of such an analysis is its completeness and its depth at those focal grade levels but its weakness is that by looking only at selected points during the schooling of a pupil there is no sense of continuity regarding topic coverage. We only are able to characterize the presence or absence of a topic at each of the focal points. Its absence, for example, represents an ambiguity since we don't know if it was covered at other grade levels but not included at the focal grades or if it is not covered in the curriculum at all.

On the other hand the specification of the topic at a focal grade only tells us of its presence at that level and not whether this is the grade level at which it was first introduced or if this is the grade level at which instruction on this topic is concluded. Further, it might be a point of concentration for the coverage of the topic or only one of numerous grades in which it is covered.

Topic trace mapping. Such information can only be provided if a topic is traced through the entire educational system or at least, given the focus of TIMSS, from the beginning of schooling to the end of secondary. The resulting map would allow a more complete characterization of coverage regarding the topic that is being traced but unless done for all topics in the mathematics and science frameworks it is narrow in its description of the curriculum.

The possibility of doing such a topic trace mapping for all topics in the frameworks would depend on the amount of detail required as a part of the mapping procedures. One type of topic trace mapping would focus only on the characterization of points at which instruction is begun, finalized and concentrated on and not require an extensive amount of documentation nor be based on the content analysis of documents associated with the curriculum. In such an instance, doing this for all topics defined by the two frameworks would be possible.

The weakness inherent in such an approach is that it is not based on detailed and exacting document analysis procedures but more on expert opinions (of those completing the trace mapping and their knowledge of the country's intended curriculum) and it is based on a more cursory examination of official and semi-official documents. Also, the focus of the procedures is on each topic treated in an isolated fashion even though many of the topics are likely related in terms of their coverage.

Modified topic trace mapping. This approach, however, is practical and does provide "longitudinal" information of the coverage of all topics in the frameworks. The validity and reliability of the information is, however, not as good as it would be under more demanding procedures and in-depth information regarding coverage is not available. This approach describes the modified topic trace mapping procedures to which we will return in the next section.

In-depth topic trace mapping. The same logic of topic trace mapping can be applied but with the additional focus of providing an in-depth perspective of the coverage of the topic. In other words the focus is not only to provide information on the initial and final points and on a point (or points) of concentration but to characterize the introduction of different subparts of a topic together with their milestones and how the sub-parts fit together in the coverage of the topic as a whole.

This implies a more complete characterization of coverage involving multiple initial points and milestones. The amount of documentation necessary to support this approach certainly is more involved. In order to accomplish this the procedures would of necessity involve a content analysis of documents that provide the detail necessary to support such complex and intricate detail.

The strength of this approach is clearly one of depth and careful analysis but it represents a set of procedures which clearly cannot be done for all topics in the frameworks at least not in a practical sense or from a point of view of feasibility. Such procedures we label as in-depth topic trace mapping and they will be applied to only six topics in each of mathematics and science. The twelve topics were chosen because international coverage of these topics is broad and they are of interest to the various mathematics and scientific education communities.

A Composite Portrayal

We believe that a good characterization of the pupil learning goals of a country depends on the combination of the data derived from each of the three procedures described above. The modified topic trace mapping (MTTM) provides broad information regarding all topics in the frameworks at least in terms of cross-grade level coverage but more detailed and more well documented information about cross-grade level coverage must be derived from the in-depth topic trace mapping (ITTM) but it is available only on a small number of topics. The document analysis procedure (DA) provides the most complete, detailed and well documented information regarding all topics in the frameworks but only at the focal point grade levels.

The combination of procedures provides both a breadth in terms of all topics treated in an isolated sense across all grade levels but with little supporting detail (MTTM) and in terms of all topics at the three focal pints (DA) and a depth in terms of a limited number of topics across all grade levels (ITTM) and in terms of topic coverage at the three focal points representing all topics in the frameworks in a very detailed fashion (DA).

The U.S.. National Center is responsible for completion of the U.S.. curriculum analysis as well as quality control and the analysis of the international curriculum data from each country. Staff are currently receiving Phase I data which is a trial run for countries, verifying that procedures are being followed correctly, providing feedback to countries and authorizing them to proceed with completion of the curriculum analysis process. When all data has been received the U.S.. Center will be responsible for data analysis for the international curriculum analysis.

Survey of Mathematics and Science Opportunity (SMSO)

Bill Schmidt is project director of the SMSO project, a research and development grant from the National Science Foundation in conjunction with the National Center for Education Statistics related to the Third International Mathematics and Science Study (TIMSS). During 1991-1992 the SMSO project developed and piloted questionnaires for TIMSS related to teacher beliefs and background, student opportunities to learn and classroom practices. These questionnaires were piloted in the United States, Norway, France, Spain and Japan. Three international meetings were held during 1992, in East Lansing, Michigan in February, in Tokyo at the National Institute for Educational Research (NIER) in June and in Madrid in December. SMSO was also instrumental in the creation of curriculum frameworks for mathematics and science to be used in TIMSS. SMSO personnel also developed materials for curriculum analysis training sessions held in The Hague, Prague, Hong Kong, East Lansing, Oslo, Miami and Frascati, Italy. Trainers were Curtis McKnight from University of Oklahoma, Ted Britton from The NETWORK in Washington, D.C. and William Schmidt from Michigan State University. The SMSO project is currently ready to go into the field in France, Japan, Spain, Switzerland and the U.S.. with questionnaires related to the second phase of the project, the measurement of Opportunity to Learn. The group hopes to meet in May to begin writing of a book of case studies and what was learned from the first phase of the study, Instructional Practices.

The SMSO project wishes to acknowledge the cooperation of several school districts who have allowed us to observe in classrooms and have the participation of students and teachers for completion of questionnaires and logs. During the 1991-1992 school year these included: Flint Public Schools, Lansing School District and Ithaca Public Schools in Michigan; and schools in Oslo, Norway; Paris, France,; Madrid, Spain; and Tokyo, Japan. School districts currently participating in the Opportunity to Learn phase of the study are Diocese of Lansing, Lansing School District, Lansing Christian Schools in Michigan; Montgomery County Schools in Maryland; and other locations in Paris, France; Bern, Switzerland; Madrid, Spain; and Tokyo, Japan.
 
 

This newsletter is published by the TIMSS U.S.. National Research Center at Michigan State University The newsletter is edited by Jacqueline E. Babcock.