THIRD INTERNATIONAL MATHEMATICS AND SCIENCE STUDY 

I was extremely saddened over the death of my colleague, Leigh Burstein. Leigh was a member of the U.S. Steering Committee for TIMSS and a member of the SMSO team but most importantly, he was my friend. His contributions to this study were innumerable and his loss is simply not possible to put into words. He was in Annapolis attending a meeting of the U.S. Steering Committee for TIMSS at the time of his death. My life was enriched and changed by his presence and I miss him terribly. A special tribute is planned at AERA in the spring. Donations may be made in Leigh's memory to the Leigh Burstein Memorial Fund, UCLA Foundation, 405 Hilgard Avenue, Los Angeles, CA 90024-1521. 

Bill Schmidt 

U.S. Center News

The U.S. National Research Center was host to two international visitors. Dr. Nancy Law arrived at Michigan State University in mid August on sabbatical from Hong Kong University. She worked on the curriculum analysis for TIMSS in Hong Kong and assisted the U.S. Center with plans for the analysis and publication of the international curriculum analysis. Dr. Law also worked with the Hong Kong data base while in East Lansing. She was joined by her son, Yu Kay who attended Middle School in the East Lansing Public Schools. They departed East Lansing on October 19.

Katsuhiko Shimizu, Doctor of Education arrived on October 9 for two weeks. He is a researcher with the National Institute for Educational Research. He is a member of the SMSO team and also is involved with the main TIMS study in Japan and the videotaping portion of the study which is taking place currently. Dr. Shimizu worked with Leland Cogan in the U.S. Center to complete the data and translations for opportunity to learn and instructional practices. Dr. Shimizu works
 

under the direction of Dr. Toshio Sawada of NIER and returned to Tokyo on October 21.
 
 
 
 
 

Upcoming Events 

October 31-November 4 - 7th World Conference for TIMSS National Research Coordinators meets in Salzburg, Austria 

November 4-7 - Free-Response Item Coding Committee meets in Salzburg, Austria 

November 16 - TIMSS U.S. Coordinating Committee meets in Washington, D.C. 

November 30-December 1 - Consultant Round Table meetings on TIMSS in Washington, D.C. 

December 2 - TIMSS U.S. Coordinating Committee meets in Arlington, VA 

December 15-16 - U.S. Steering Committee meets in Baltimore 

March 1995 - Supervisor training for TIMSS data collection will be held in mid-March 1995, and student sampling and assessments for Populations 1 and 2 will begin in late March. 

April-May 1995 - Assessments for Population 3 students will be conducted 

Update on Main Study in U.S.

Sample selection for the Main Study data collection phase of U.S. TIMSS was completed in June 1994. The sample includes 33 states, about 350 school districts, and 670 public and non-public, elementary and secondary schools.

The first steps in the data collection process were initiated in August 1994 when letters announcing TIMSS and requesting support for the study were sent to state test directors in the sampled states. Later in the month, a 4-day training session was held at Westat, Inc. for Field Managers assigned to the TIMSS project. Immediately after the training, the Field Managers began telephoning each state to answer questions and recruit participation. As each state agrees to participate, this process is followed by a series of letters and telephone contacts with sampled districts and schools to gain cooperation at the local level. School Principals are asked if they wish to name a School Coordinator, and the School Coordinator, in turn, is asked to set a TIMSS assessment date for next spring. These arrangements will be confirmed in writing (with copies to the School Principal and the School District) from Westat before the end of the calendar year.

School Coordinators will receive a reminder letter in February-March 1995 along with several parental information and consent letters for their use in their schools. Supervisor training for TIMSS data collection will be held in mid-March 1995, and student sampling and assessments for Populations 1 and 2 will begin in late March. Assessments for Population 3 students will be conducted during the second half of the field data collection period -- from about mid-April through early

May.

Contributed by Lucy Gray, Westat

Data Collection Initiated For Case Studies and Videotape Study

As reported in an earlier newsletter, U.S. TIMSS includes two innovative components -- Case Studies of education policy issues, and a Videotape Study of classroom instructional practices. These will be conducted by the U.S. in three countries -- the United States, Germany and Japan. Both studies were launched late this summer with training sessions for field data collectors. Training for Case Study researchers was held in mid-August, and videographer training was conducted in September 1994.

Data collection for the Case Studies was initiated in early October in both Japan and Germany and is also underway in the U.S. as of mid-October. One primary site and two secondary sites are being studied in each country -- with the bulk of the researchers' time to be spent in the primary site in each country and 2-3 weeks in each secondary site to confirm findings from the primary site. The Case Studies are included in U.S. TIMSS to provide in-depth information about the beliefs, attitudes, and practices of students, parents, and teachers in four topic areas: teacher working conditions, implementation of standards, dealing with student ability differences, and the role of school in adolescents' lives. The Case Studies employ an ethnographic methodology that relies on the interaction of experienced researchers with families and teachers in each of the countries and on observation in homes and schools. The rich descriptive information obtained will complement and amplify the assessment and background data gathered in the Main TIMSS Study and will help to identify factors that may underlie cross-national differences in academic achievement.

Videotape Study data collection is also being initiated during the month of October in all three countries. This involves the filming of one class session in about 100 eighth grade mathematics classrooms in the U.S. and Germany and in 51 classrooms in Japan; these classrooms are sub samples of the eighth grade classrooms selected for the Main TIMSS Study. The Videotape Study is designed to provide a rich source of information about practices of classroom mathematics instruction in the three countries and to produce contextual background supplementary to the statistical indicators that result from the Main TIMSS Study. The videotapes will be transferred to CD-ROM format, linked in a multimedia database with transcriptions/translations of the speech used in the tapes, coded, and analyzed. This is the first large-scale study to collect video records of classroom instruction in the mathematics classrooms of different countries.

Contributed by Lucy Gray, Westat

Survey of Mathematics and Science Opportunity (SMSO) Activity Report

An SMSO working meeting was convened in Switzerland July 13-15. Representatives from Norway, Spain, Switzerland, and the United States examined the data collected in the SMSO Opportunity to Learn (OTL) study last year. Preliminary analyses generated considerable interest and confidence in the OTL instrument to accurately capture what teachers do in

the classroom in a way that can be significantly related to students' performance. Reasoning from the data, a consideration of the subsequent modifications that have been made in the OTL instrumentation which will be

used in the TIMSS main study resulted in an increased optimism for the OTL to meaningfully and significantly predict student performance on the TIMSS assessment.

In addition to the OTL discussion, representatives also discussed preliminary data from the TIMSS curriculum analysis and how these two aspects may be related and combined to provide a broader perspective on students' opportunity to learn. The great interest in the SMSO OTL and Instructional Practices research was noted and a number of ideas and plans for writing and publishing the research were considered. Work on the Instructional Practices book that had been accomplished since the SMSO meeting in January was further refined and additional responsibilities outlined. As a result of the consultations in Switzerland, SMSO representatives will be extended the option of coming to the SMSO research center where they may focus upon their SMSO writing tasks free from the distraction and interruption of their normal responsibilities. Consequently, the SMSO Center has had the privilege of hosting Katsuhiko Shimizu from Japan for two weeks in October while he has worked on the Japanese OTL and Instructional Practices data.

Contributed by Leland Cogan, U.S. National Research Center, Michigan State University

Bill Schmidt is project director of the SMSO project, a research and development grant from the National Science Foundation in conjunction with the National Center for Education Statistics related to the Third International Mathematics and Science Study (TIMSS). The SMSO project developed and piloted questionnaires for TIMSS related to teacher beliefs and background, student opportunities to learn and classroom practices.

U.S. Steering Committee Proceedings

The Steering Committee for U.S. TIMSS met at Historic Inns of Annapolis July 7-8. Members of the committee were given an update on the progress of the international curriculum analysis and participated in a discussion of what should appear in upcoming publications and reports. A presentation was made on communication and dissemination plans for the study when results become available. Also discussed were issues related to population 3 in the study, achievement tests and pilot results, State TIMSS and the videotaping and case study pieces of the study. Westat reported on the field test and recommendations they had made and the committee was asked to review the revised questionnaires for teachers and students and send further suggestions to Bill Schmidt before the final versions were completed. (These questionnaires are now complete and will be in the field in the Southern Hemisphere fall 1994.)

Education in the U.S. and Participation in TIMSS

Gaining a scientific understanding of which structural features of an educational system are central to systemic reform efforts and how the outcomes of a system may change in response to modifications of these aspects is vital to the future of educational reform in the United States. Awareness of how these structures may be varied is an essential first step to gaining such an understanding. Understanding how these structures can be altered and what the probable outcomes are, is greatly enhanced through comparisons across different systems. In short, to better understand how to change or reform a system of education is to measure it against others.

Meaningful measurement of educational systems requires a comprehensive conceptual framework and a corresponding array of measures designed to relate the various parts of the system to each other and with

their outcomes. If the technologies for the elaboration and evaluation of systemic reform are to move forward methodologies are needed to collect data rigorously in a consistent fashion and on a significant number of educational systems. This makes meaningful comparisons possible, resulting in a deeper understanding of an existing system as well as potential alternatives.

Since the early 1980's the public has become aware of the critically poor quality of pre-college education in the United States. With the 1983 publication by the National Commission on Excellence in Education of A Nation at Risk, a decade of policy formulation and its supporting research has addressed these shortcomings.

The results of these policy initiatives and supporting research have been an emerging movement towards content-driven systemic change and the formulation of national educational goals. At present, these movements are in a critical phase. The work performed at the U.S. National Research Center for TIMSS offers significant opportunities for valuable inputs to help inform change during this critical time.

Two current policy initiatives are important to understanding the usefulness of TIMSS to U.S. educational policy interests: (1) the National Educational Goals and (2) content-driven systemic school reform. A central theme of both of these initiatives is curriculum, understood as a body of challenging content and performance expectations embodying the goals or intentions of the educational enterprise. A second theme is the delivery of curriculum; the instructional practices and activities designed to provide each student with educational opportunities for learning experiences that produce the outcomes found in systemic intentions. A third theme is the assessment of outcomes against reasonable standards both national and cross-national.

The TIMSS is strategically positioned both to test and to advance the theoretical underpinnings of systemic reform by providing empirical analyses as well as practical quantitative methodologies.

The work already accomplished by TIMSS includes the curriculum analysis component and the methodologies to gather data on curricular implementation. To provide the array of tools and benchmarks needed to fully inform U.S. systemic reform in mathematics and the sciences, further work is needed to refine the analytic tools for comparisons and to perform analyses with these tools that will provide a finer-grained, more useful set of benchmarks. These results must also be reported in ways useful to those responsible for assessing systemic reforms, setting national goals and measuring their attainment and setting policies that guide the change of U.S. science and mathematics education at the pre-college level.

Our purpose at the U.S. Center is to perform work that not only flows out of the comparative education tradition but that also will enable the TIMSS data to address central policy questions of interest to the United States and to disseminate the knowledge thus gained to the interested national audiences: policy makers, researchers, educators, business and community leaders, and parents.

TIMSS and Content-driven Systemic School Reform

A major set of research and policy initiatives aimed at improving education in our country constitutes the movement for content-driven systemic school reform. The effort has in large part been led by NSF. A considerable body of conceptual and research work has been contributed to support this type of reform (see for example: Clune, 1993; O'Day and Smith, 1993). Not only has NSF contributed to the theory of systemic reform but they have also committed funds to major projects known as State Systemic Initiatives (SSI).

Theoreticians of content-driven systemic reform have formulated their position with reference to educational systems abroad. For example, O'Day and Smith (1993) state that: "When fully implemented, this model of content-driven systemic reform would be a uniquely American adaptation of the educational policies and structures of many of the world's highly developed nations (p. 252)." Clune (1993) also mentions that a possible result of content-driven systemic school reform would be a centralized system of governance and curricula "resembling the national curricula and educational ministries of countries that have high levels of student achievement" (p. 233).

The central tenant of systemic educational policy advocates is that it must be driven by curriculum. It is stated that ambitious curricula must be formulated and that appropriate mechanisms be designed to implement

these curricula so that students may have the opportunity to attain high levels of achievement. Content-based reform holds that a core curriculum provides a basis for determining which resources are necessary to ensure that students are provided the opportunities required for high levels of achievement. Thus, the curriculum would directly impact teacher training and certification, school course offerings, instructional resources and structures of educational governance and accountability. In fact, it is held that contrasting the curriculum with educational practices in the classroom is required to accurately monitor how students are provided with adequate opportunity to learn, and are one of the evaluation responsibilities of a reformed system.

The TIMSS is perfectly suited to test and to advance systemic reform theory and policy. In fact, it will provide one of the most important tests of many of its most basic tenants by:

ï Identifying how curricula are formulated and what policies for their implementation exist.

ï Determining the content of such diverse curricula.

ï Determining the influence of curricula on teacher training.

ï Studying the array of instructional practices used around the world to teach the curricula.

ï Describing how the world's textbooks are designed to teach the curricula underlying those textbooks.

ï Determining the extent to which students actually have the intended opportunities to learn.

ï Testing for differences between standardized and/or centralized and differentiated or decentralized educational systems in terms of opportunities to learn and student attainment.

In all of these cases, the information gathered provides not only a picture of the current state of education cross-nationally, but also provides understandings applicable to sub-national reforms (such as U.S. curriculum reform, and SSI activities). Systemic reform recognizes that there may be no one single model but that these are likely to vary from state to state depending on the local political and economic realities. The TIMSS model and associated methodologies are ideally designed to address this since they are used to collect data on intended, implemented and attained curricula in each participating country and do so within a framework that integrates these data through a comprehensive model of educational systems. These data will be particularly suited for exploring relationships between components of each country's educational system in terms of content objectives for mathematics and the sciences, and thus will provide data able to inform content driven systemic reform in this country.

TIMSS and The National Educational Goals

In 1989 six national educational goals were set out at the Education Summit in Virginia. These goals were and are intended to direct the national resolve to improve our nations schools by the end of this century. These goals have since passed into law as the Goals 2000: Educate America Act (H.R. 1804 and S. R. 846, 1993). Goals three and four address the content of schooling and expected achievement by establishing that students must demonstrate competency in challenging subject matter.

Additionally, the National Educational Goals Panel was created to monitor progress toward the goals. This panel has established that monitoring progress on Goal 4 requires them to actively promote research that clarifies current standards in mathematics and science around the world . This legislation also provides for the development of criteria for certifying voluntary content and student performance standards. These criteria are meant to address the extent to which standards are internationally competitive. The legislation also provides for developing voluntary opportunity to learn standards.

Data from the TIMSS can be used to contextualize and understand the national goals better by:

ï Clearly describing the contents, performance expectations and perspectives of the mathematics and science curricula of approximately 50 nations, thereby providing data that facilitates setting benchmarks.

ï Developing and refining methodologies for characterizing intended and implemented contents in mathematics and science curricula.

ï Assessing the mastery of science and mathematics attained by the students in participating nations thus establishing benchmarks for assessing realized goals in terms of learning outcomes.

It is only through the use of international data that we can hope to monitor our nation's quest to attain goals three and four. It is only through the use of these data that we can, to paraphrase Carroll Campbell Jr., Governor of South Carolina, "measure ourselves against the world's best." The National Educational Goals are expressly formulated in terms of global competition in recognition of the fact that the U.S. educational system is related to global economic competitiveness and cannot hope to fulfill its mission if it merely measures improvement against its own previous accomplishments.

Data and methods from the TIMSS can be used for this important purpose. In fact, it is the only study currently in a position to do this. By looking at the educational systems of the world we challenge our own conceptions, gain new and objective insights into education in our own country, and are thus empowered with fresh vision with which to formulate effective educational policy and new tools to monitor the effect of these new policies.

Contributed by Gilbert Valverde, U.S. National Research Center, Michigan State University

International Curriculum Analysis

The Curriculum Analysis has been a significant aspect of the Third International Mathematics and Science Study since its inception. It represents one of the most momentous innovations introduced in the area of large-scale international comparative educational research. As such, the design of the procedures has required the efforts of many individuals.

The development of the methodologies for this data collection and analysis is closely linked to other SMSO developmental work for the TIMSS. In particular, the coordination of the formulation of the TIMSS curriculum frameworks was of central importance.

As of 27 May 1994, data from a total of 252 science curriculum guides, 333 science textbooks, 229 mathematics curriculum guides and 277 mathematics

textbooks have been received, pre-processed, entered, cleaned and stored in the data base. Also, countries have submitted all of the original source documents (textbooks and curriculum guides) for both the 1,091 DA documents and more than 1,200 ITTM documents with units, blocks and segments clearly marked in them.

The electronic data base occupies 2,130,000,000 bytes of computer space. However, when expanding the data for analysis they take up much more space.

The International Reports

The TIMSS Curriculum analysis is designed to provide a characterization of the intended curriculum in each of the countries participating in the study. The intended curriculum is understood to be what a nation expects to be taught or has as learning goals for the pupils in its educational system. For different countries, the level of detail in the specification of the learning goals varies substantially in official documents such as curriculum guides. On the other hand, textbooks usually are exceptionally detailed and can be regarded as statements of curricular intent in most countries. Thus, the TIMSS Curriculum Analysis defines the intended curriculum to include learning goals stated in curriculum guides and inferred from the contents of textbooks.

The TIMSS Curriculum Analysis represents an innovative effort at characterizing international educational opportunities. Each country, through the Document Analysis procedures, has submitted data from page-by-page inventories of the science and mathematics textbooks most widely used for students in selected grades and of key national and regional curriculum guides, characterizing document segments using the various aspects of TIMSS mathematics and science curriculum frameworks. The within-country coded characterizations of aspects of textbooks and curriculum guides are being analyzed separately and in combination as are other important features of textbooks and curriculum guides.

These activities compose the first attempt at international characterization of the intended curriculum to include in-depth content analysis of documents as well as surveys of curriculum experts. To

educational researchers and policy makers across the world, the TIMSS Curriculum Analysis is regarded as being at least as important as the TIMSS achievement test results.

As the research team charged with the design of the Curriculum Analysis procedures, the entry and cleaning of the data and their analysis, the international activities being performed at the SMSO for the remainder of 1994 will be concentrated on the analyses and writing of the first two volume international reports. These first volumes, one for science and one for mathematics, will concentrate on the analysis of similarities and differences between curricular coverage and sequencing. This is to say, they will focus on common elements in terms of content, performance expectations and perspectives across all TIMSS countries and will also examine regional, cultural and individual country differences.

In 1995 major effort will be devoted to the analyses and production of two additional two volume sets (also by discipline). One set of volumes will be devoted to studies of characteristics of textbooks and guides as embodiments of the intended curriculum, characterizing form and structure. The other set will study the relationship of structure and form for textbooks, curriculum guides and educational systems, and intended content and performance expectations.

Contributed by Gilbert Valverde, U.S. National Research Center, Michigan State University

Presentations

Bill Schmidt was an invited speaker at the CRESST Conference, Getting Assessment Right! held on the campus of UCLA September 12. He participated on a panel related to Opportunity to Learn and described the experience and knowledge gained through the SMSO OTL study.

Bill Schmidt presented a major talk at the National Science Foundation in Arlington, VA on September 14, 1994. His talk was on preliminary results from the international curriculum analysis.

Bill Schmidt was interviewed for an upcoming article for Education Week.

Questionnaire Meeting

Bill Schmidt chaired a meeting involving representatives of several countries in Hamburg, Germany June 1-3. The purpose of the meeting was to make final recommendations for the context questionnaires for teachers and students that will be used in TIMSS during 1994-1995. Countries represented included France, Germany, Ireland, the Netherlands, and England. The meeting was hosted by Neville Postlethwaite.

We want to hear from our readers, so if you have questions, want additional information about a topic addressed in the newsletters or the study in general, please feel free to contact us. You can write or call Gilbert Valverde, Assistant Director - TIMSS, 457 Erickson Hall, Michigan State University, East Lansing, Michigan 48824-1034, telephone 517-353-7755, fax 517-432-1727, or E-Mail, valverde@msu.edu. 

If you have suggestions of articles you would like to see in future newsletters or if you or someone you know did not receive this newsletter directly, but would like to be on our mailing list, please send your name and address along with your request to Jacqueline Babcock, 464 Erickson Hall, Michigan State University, East Lansing, Michigan 48824-1034, telephone 517-353-7755, fax 517-432-1727, or E-Mail: jbabcock@msu.edu. 

We also are available to make presentations to professional groups or associations who might be interested in being introduced to TIMSS or receiving updates on the project. If your organization is interested in hearing more about TIMSS and would like a representative from our office to present to your group, please let us know. 

U.S. National Steering Committee Members: 

Dr. Gordon Ambach, Executive Director, Council of Chief State School Officers; Dr. Deborah Ball, Associate Professor, Michigan State University; Dr. Audrey Champagne, University at Albany-SUNY; Dr. Jewel Plummer Cobb, California State, Los Angeles; Dr. David Cohen, Professor, The University of Michigan; Dr. John Dossey, Distinguished Professor of Mathematics, Illinois State University; Dr. Emerson Elliott, National Center Education Statistics; Dr. Sheldon Glashow, Higgings Professor of Physics and Mellon Professor of the Sciences, Harvard University; Dr. George E. Hall, President, Slater Hall Information Products; Dr. Larry Hedges, Professor, Department of Education, University of Chicago; Professor Henry Heikkinen, Department of Chemistry and Biochemistry, University of Northern Colorado; Dr. Jeremy Kilpatrick, Regents Professor of Mathematics Education, University of Georgia; Dr. Mary Lindquist, Fuller E. Callaway Professor of Mathematics Education, Columbus College, and President of the National Council of Teachers of Mathematics; Dr. Marcia C. Linn, Professor and Director, Instructional Technology Program, University of California-Berkeley; Dr. Robert L. Linn, Professor and Co-director of the Center for Research on Evaluation, Standards, and Student Testing, University of Colorado; Dr. Paul Sally, Professor of Mathematics, University of Chicago; Dr. Richard Shavelson, Professor, Department of Education, University of California-Santa Barbara; Dr. Bruce Spencer, Professor, Department of Statistics and School of Education and Social Policy, Northwestern University; Dr. Elizabeth Stage, New Standards Project, Co-Director for Science, University of California, Office of the President; Dr. James Taylor, Hill and Knowlton, NYC; Dr. Kenneth Travers, University of Illinois, and Dr. Paul H. Williams, Professor, Department of Plant Pathology, University of Wisconsin. The committee is chaired by Dr. William Schmidt, Professor, Michigan State University and National Project Coordinator of the U.S. National Research Center-TIMSS. 

National Research Coordinating Committee 

Dr. Jeanne Griffith, Associate Commissioner for Data Development, National Center for Education Statistics; Dr. Eugene Owen, National Center for Education Statistics; Dr. Lois Peak, National Center for Education Statistics; Dr. Larry Suter, Office of Studies and Program Assessment, National Science Foundation. The committee is chaired by Dr. William Schmidt, Michigan State University. 

This newsletter is published by the TIMSS U.S. National Research Center located at Michigan State University. The newsletter is edited by Jacqueline E. Babcock.