THIRD INTERNATIONAL MATHEMATICS AND SCIENCE STUDY 

Survey of Mathematics and Science Opportunity (SMSO) Activity Report

Bill Schmidt is project director of the SMSO project, a research and development grant from the National Science Foundation in conjunction with the National Center for Education Statistics related to the Third International Mathematics and Science Study (TIMSS). The SMSO project developed and piloted questionnaires for TIMSS related to teacher beliefs and background, student opportunities to learn and classroom practices.

Reports:

Two major SMSO papers have been issued as International Coordinating Center publications (University of British Columbia, Vancouver, Canada): TEST BLUEPRINTS: A Description of the TIMSS' Achievement Test Content Design, written by Curtis McKnight, William H. Schmidt and Senta Raizen, and TIMSS EDUCATIONAL OPPORTUNITY MODEL: Detailed Instrumentation And Indices Development, written by William H. Schmidt. The TEST BLUEPRINTS paper presents the specifications for the TIMSS' student test. It details the testing time and number of items to be allocated for specific framework categories and the various question types which will be used. The EDUCATIONAL OPPORTUNITY paper describes how the data from the context questionnaires will be used to address specific research questions. It also details some proposed analyses and the development of key indices and typologies.

Context Questionnaires:

The U.S. Center continues to direct development and production of the context questionnaires. A three-day meeting was held in May to produce versions of the Student, School, and Teacher Questionnaires for international piloting. Leigh Burstein, Edward Britton, Leland Cogan, Curtis McKnight, Judith Torney-Purta
 
 

and Gilbert Valverde participated in these sessions which were directed by William H. Schmidt. The questionnaires were subsequently formatted by Trevor Williams at A.C.E.R. in Australia with Schmidt's assistance. The final pilot forms were submitted to the International Coordinating Center for use in the October 1993 Teacher and School Questionnaire Pilot. The revised version of the Student Questionnaire was used in the March 1994 International Field Trial.

Meeting in Paris, France:

Representatives from the six SMSO countries (France, Japan, Norway, Spain, Switzerland, and the U.S.) held an international working meeting focusing on classroom observations of teachersí instructional practices January 10-14, 1994 in Paris. Participants read, discussed and analyzed summaries of over 100 classroom observations that had been conducted in the six SMSO countries. Beginning with a list of ìsurprisesî these educators had in reading accounts of what goes on in classrooms in other countries, participants formed a list of seven issues or themes descriptive of the areas in which differences in instructional practices among the countries appeared. The seven issues or themes that emerged from discussions of the classroom observations include the characteristic pedagogical flow, structure of the lesson, use of homework/in-class work, subject matter content, classroom management, teacher/student interactions, and instructional resources and approaches. Plans were developed to complete analyses of the observational data and to create drafts of various parts of a book on Instructional Practices which will present the insights gained from these investigations of classroom practices in the six countries.

A summer meeting is being planned for SMSO representatives to complete analyses for the Opportunity to Learn pilot which was conducted in over 75 classrooms in France, Japan, Spain, Switzerland, and the United States. The Opportunity to Learn (OTL) pilot gathered information from teachers and students on studentsí opportunity to learn various topics as represented in the TIMSS Curriculum Frameworks. This pilot was designed to address issues of validation of teachersí indications of studentsí opportunity to learn which can provide a rich and valuable context for understanding studentsí performance on achievement assessment measures. The summer meeting will also provide an opportunity to discuss drafts of the planned Instructional Practices book. Plans are to complete the Instructional Practices book by the end of this year.

Contributed by Leland Cogan, SMSO Coordinator

National Research Center News

We have received notification that an award for the period of 9/1/92-2/28/94 with an extension through 2/95 has been awarded to William Schmidt by the National Science Foundation in support of the U.S. National Research Center at Michigan State University. The award is in cooperation with the National Center for Education Statistics.

Leonard Bianchi has been hired in a full time position as a Senior Research Associate in the U.S. National Research Center at Michigan State University effective February 1, 1994. Dr. Bianchi holds a Ph.D. from Michigan State University, College of Education and was most recently employed by Pelavin in Washington, D.C. Len was head data analyst for the Second International Mathematics Study (SIMS) and he has also taught in the doctoral sequence of statistics courses in the Department of Counseling, Educational Psychology and Special Education at Michigan State University. He will serve as an analyst working with the curriculum analysis data which has been sent to the U.S. Research Center from approximately 50 countries participating in TIMSS.

Contract Award For Collection of U.S. TIMSS Data

In late September 1993, the National Center for Education Statistics awarded a contract for the data collection for the U.S. portion of TIMSS to Westat, Inc. in Rockville, Maryland. The contract also involves several subcontractors: Educational Testing Service (ETS), National Computer Systems (NCS), Dr. Harold W. Stevenson of the University of Michigan, and Dr. James W. Stigler of the University of California at Los Angeles. Each subcontractor will operate a specific portion of the contract under the overall direction of Westat. At the National Center for Education Statistics, Lois Peak, TIMSS Project Officer, will be the government project officer for all contractor activities.

The contract will continue through December 1997. During this period, the following data collection activities will be performed:

TIMSS FIELD TEST AND MAIN STUDY DATA COLLECTIONS: Westat will be responsible for production of questionnaires and assessments, receipt and processing of completed instruments for the spring 1994 field test and spring 1995 main data collections. They will receive support from National Computer Systems in these activities. Educational Testing Service will also assist by reviewing instruments and materials and collaborating with Westat in data analysis activities.

STATE-LEVEL TIMSS: States who desire to compare the achievement of their students to that of other nations participating in TIMSS will have an opportunity to contract with Westat to participate as a "mini-nation" in the TIMSS data collection.

CASE STUDIES OF EDUCATION POLICY ISSUES IN THE U.S., JAPAN, AND GERMANY: Year-long case studies in ministries, schools, and homes in three countries will supplement the data from the main study data base by providing additional comparative data about four topics relating to opportunity to learn issues: teacher working conditions, the implementation of standards, how ability differences are dealt with, and the role of school in adolescents' lives. Case study activities will be directed by Harold Stevenson.

VIDEOTAPES OF CLASSROOM INSTRUCTION IN THE U.S., JAPAN, AND GERMANY: Videotapes will be used to provide observations of mathematics instruction in a subsample of 100 eighth grade classrooms participating in TIMSS. These observations will provide both quantitative data and qualitative descriptions of how instruction occurs in these countries. The videotape observation project will be directed by James Stigler.

Contributed by Lois Peak, NCES, and Lucy Gray, Westat

Invitation To Participate In State TIMSS

State TIMSS is an opportunity for states to participate as if they were a nation in the TIMSS project by administering the full TIMSS assessments and background questionnaires to a random sample of students in their states.

Key features of State TIMSS are:

- Assessment scores collected through State TIMSS can be compared reliably to international TIMSS scores.

- Data will include both mathematics and science scores.

- States will have the choice of assessing their 3rd-4th, 7th-8th, and/or 12th grade students.

- States will receive internationally comparable item-level data, as well as aggregated scores, on how their students perform on both the mathematics and science assessments.

- States will receive internationally comparable student,

teacher, and school data from TIMSS background questionnaires.

- Reports are scheduled to be available in December 1996.

- States will be responsible for funding their own participation in State TIMSS.

The State TIMSS program is now available for participation, and details of the design options and costs can be obtained from Westat. Please contact Westat to be included on the mailing list and to receive periodic updates on the status of State TIMSS.

Ms. Sandra Rieder, Westat, Inc.

1650 Research Blvd.

Rockville, Maryland 20850

TEL: (301)-738-3616 - FAX: (301)-294-2038

Contributed by Lois Peak, NCES, and Lucy Gray, Westat

U.S. Steering Committee Proceedings

The U.S. Steering Committee met twice during 1993 in May and November. Both meetings were held at the St. James Hotel in Washington, D.C. Items under discussion included: Performance Testing, Achievement Test, Sampling. William Schmidt also presented preliminary curriculum analysis data.

U.S. National Steering Committee Members: 

Dr. Gordon Ambach, Executive Director, Council of Chief State School Officers; Dr. Deborah Ball, Associate Professor, Michigan State University; Dr. Leigh Burstein, Professor, Graduate School of Education, University of California-Los Angeles; Dr. Audrey Champagne, University at Albany-SUNY; Dr. Jewel Plummer Cobb, California State, Los Angeles; Dr. David Cohen, Professor, The University of Michigan; Dr. John Dossey, Distinguished Professor of Mathematics, Illinois State University, currently on assignment at United States Military Academy, Westpoint, New York; Dr. Emerson Elliott, National Center Education Statistics; Dr. Sheldon Glashow, Higgings Professor of Physics and Mellon Professor of the Sciences, Harvard University; Dr. George E. Hall, President, Slater Hall Information Products; Dr. Larry Hedges, Professor, Department of Education, University of Chicago; Professor Henry Heikkinen, Department of Chemistry and Biochemistry, University of Northern Colorado; Dr. Jeremy Kilpatrick, Regents Professor of Mathematics Education, University of Georgia; Dr. Mary Lindquist, Fuller E. Callaway Professor of Mathematics Education, Columbus College, and President of the National Council of Teachers of Mathematics; Dr. Marcia C. Linn, Professor and Director, Instructional Technology Program, University of California-Berkeley; Dr. Robert L. Linn, Professor and co-director of the Center for Research on Evaluation, Standards, and Student Testing, University of Colorado; Dr. Paul Sally, Professor of Mathematics, University of Chicago; Dr. Richard Shavelson, Graduate School of Education and Professor, Department of Education, University of California-Santa Barbara; Dr. Bruce Spencer, Professor, Department of Statistics and School of Education and Social Policy, Northwestern University; Dr. Elizabeth Stage, Executive Director, California Science Project, University of California-Berkeley; Dr. Harold Stevenson, Professor of Psychology, The University of Michigan; Dr. James Taylor, Yankelovich, Clancy, Shulman, Westport, CT; Dr. Kenneth Travers, University of Illinois, and Dr. Paul H. Williams, Professor, Department of Plant Pathology, University of Wisconsin. The committee is chaired by Dr. William Schmidt, Professor, Michigan State University and National Project Coordinator of the U.S. National Research Center-TIMSS.

Publications

The International Coordinating Council for TIMSS has announced the first publication in a series of anticipated monographs. Curriculum Frameworks for Mathematics and Science is authored by William Schmidt, Senta Raizen, Curtis McKnight and Edward Britton of the U.S. National Center and David Robitaille and Cynthia Nicole from University of British Columbia. It describes the frameworks that will be used to examine mathematics and science curricula in the fifty countries that are taking part in the project. The monographs will be of interest to researchers and scholars specializing in mathematics and science education, policy-makers, and educational administrators. Copies may be ordered from Pacific Educational Press, Faculty of Education, University of British Columbia, Vancouver, B.C. V6T1Z4, Telephone: (604) 822-5385, Fax: (604) 822-6603 for $16.95. (Paper, 104 pp., appendices, ISBN 0-88865-090-6).

William Schmidt has received notice of the publication of "TIMSS curriculum analysis: topic trace mapping," in Prospects, Vol. XXII, No. 3, 1992 (83) published by UNESCO.

Gilbert Valverde authored a chapter entitled "United States of America: System of Education" for the Second Edition of the International Encyclopedia of Education.

International Curriculum Analysis

During the past year, Phase 1 quality control activities were completed. A team of graduate students, undergraduate students and senior staff members have been responsible for cleaning and assembling the data for the international curriculum analysis. This team is entrusted with contrasting data submissions with electronic files and identifying and solving problems in data collection and data entry.

In October of 1993, an international meeting was held at the U.S. National Research Center in East Lansing. Attendees included: Ted Britton and Senta Raizen (The NETWORK, Inc., Washington, D.C.), Vlado Burhan (Slovakia), Leigh Burstein (UCLA), Geoffrey Howson (England), Peter Labudde (Switzerland), Curtis McKnight (University of Oklahoma), Graham Orpwood (Canada), Miriam Reiner (Israel), David Robitaille (University of British Colombia, International Coordinating Center), David Wiley (Germany), Richard Wolfe (OISSE, Toronto), Theo Wubbels (The Netherlands), and Ken Travers (University of Illinois). Participants form the U.S. National Research Center were Leland Cogan, Richard Houang, and Gilbert Valverde. The meeting was chaired by Bill Schmidt. The purpose of the meeting was to examine initial data analysis results and to develop strategies for data analyses and reporting as well as additional data collection for the TIMSS curriculum analysis.

International Fall Context Questionnaires Pilot

TIMSS conducted a pilot of the Teacher and School questionnaires in all participating countries during October, November and December 1993. The development of these questionnaires has been directed by the U.S. National/SMSO Center. The Center was also responsible for writing the Pilot Manual, Administration Manual and the Data Code books for the international pilot. The Teacher Questionnaires were to be piloted to a convenience sample of 50 Population 1 teachers and 50 Population 2 teachers. The appropriate version of the School questionnaire was piloted among 25 principals representing all the populations at which a country is participating--i.e., in the U.S., Populations 1, 2 and 3. The time required for respondents to complete questionnaires and other information from the pilot will be used to revise the questionnaires for the main study. The U.S. Center is coordinating all aspects of the pilot and is coordinating revision of the questionnaires following the pilot.

U.S. Context Questionnaires Pilot

In the U.S. over 140 schools were contacted to secure the principal and teacher volunteers needed to complete the pilot questionnaires. Most of those contacted indicated a willingness to participate and have returned their completed questionnaires. With a few questionnaires still needing to be returned, the U. S. Center has currently received more than 25 Population 1 Teacher Questionnaires, 25 Population 2 Mathematics Teacher

Questionnaires, 25 Population 2 Science Teacher Questionnaires, and over 40 School Questionnaires.

Context Questionnaires Data

Analysis of data from the recent pilot of the Teacher and School Questionnaires is proceeding at the SMSO Research Center at Michigan State University under the direction of William Schmidt. Data files from the almost 20 countries that have submitted data are currently being cleaned and readied for analysis. In addition to the quantitative data which represents the major part of the questionnaire data, large data bases are being created containing the free responses of questionnaire respondents and the evaluative comments made by NRCs regarding specific items on the questionnaires and the pilot in general. These databases will be carefully analyzed and will guide the next round of questionnaire revision. Reports on the pilot are scheduled to be sent out sometime in April to those who participated. Revision of the questionnaires on the basis of the pilot is scheduled to be completed in April as well. Once these revisions have been made, the questionnaires will again be circulated among NRCs for review.

Contributed by Leland Cogan

TIMSS Field Test

The TIMSS field test is currently underway and is being

conducted in the United States by Westat, Inc. and its

subcontractors, under contract to NCES. Westat field staff were trained for TIMSS on February 19 and 20, 1994.

The field test for Populations 1 and 2 was begun on February 28, and data collection was completed on March 10. A total of 29 schools and 102 classrooms participated in seven states. About half the schools had third and fourth graders tested and the other had seventh and eighth graders tested. In most of these schools, four classrooms were tested, two at the upper grade and two at the lower grade for each Population. The data collection materials used included: the 8 two-part, pencil-and-paper assessment booklets provided by ICC/IEA for each population; and the U.S. versions of the student and teacher background questionnaires provided by the U.S. National Research Center at Michigan State University.

Other TIMSS field test activities will be conducted in the U.S. between mid-March and mid-May 1994. Performance Assessments are being scheduled for late March to mid-April. These will be conducted with groups of eight students in each of eight schools for Population 1 and for Population 2.

The field test data collection for twelfth grade students is planned for April 25 through May 6. This will involve pencil/paper assessment of 50-60 twelfth grade students in each of 20 schools. In each school, plans call for testing three groups of about 20 students each -- "generalists", mathematics specialists, and physics specialists. Processing of the field test will help in revising the TIMSS assessments and questionnaires, and administration procedures in preparation for the main study in spring 1995.

Contributed by Lois Peak, NCES, and Nancy Gray, Westat

Presentations:

U.S. National Research Center personnel have been actively involved in informing a wide range of the public about TIMSS. The study is of interest to education researchers, mathematics and science educators at all levels, policy makers, and parents. This newsletter is one way of getting the word out, but we also have been involved in many formal presentations to organizations and groups which have not previously been reported in this publication. We will keep you informed about upcoming presentations in future issues of this newsletter. We also wish to encourage any groups who would like additional information or to request a presentation to contact us. To give you an idea of the wide range of topics involved in the study following are some prior papers which have been presented:

March 1993 Comparative and International Education Society:

William Schmidt, Gilbert Valverde, Curtis McKnight, and Ted Britton (all associated with the U.S. National Research Center) presented papers in a panel entitled "New Trends in the Cross National Comparison of Curricula," in Kingston, Jamaica.

April 1993 AERA:

International Curriculum Analysis: A New Approach, Chaired by Raizen. Presenters: Schmidt, Britton, McKnight, Valverde, Robitaille.

Design Issues For The Third International Mathematics And Science Study, Chaired by Robitaille. Presenters: Foy, Schmidt, Wiley, Wolfe, Beaton, Horvitz.

Measuring Learning Opportunities and Instructional Practices in Mathematics and Science in Large-Scale Surveys: Examining the State of the Art, Moderated by Raizen. Panel: Burstein, Schmidt, Smithson, Blank, Valverde.

April 1993 National Association for Research in Science Teaching:

TIMSS: A Report on the Development of Teacher Questionnaires from Five Countries, presented both SMSO and OTL work, panel with Schmidt, Jorde and Britton.

June 1993 National Conference on Large Scale Assessment:

Senta Raizen, William Schmidt, Jeanne Griffith and Larry Suter (all associated with the U.S. National Research Center) were participants in a panel chaired by Gordon Ambach, CCSSO, entitled "What Can States Learn from the Third International Mathematics and Science Study (TIMSS) in Albuquerque.

CIES:

Several presentations were made by the U.S. National Research Center for TIMSS staff at the Midwest Regional Meetings of the Comparative and International Education Society held in East Lansing October 29-30, 1993. These were: "The development of a model for the international study of learning opportunities," William H. Schmidt; "The use of observational research in the design of items for the cross-national measurement of lesson structure," Gilbert A. Valverde; "Development and validation of instruments for assessing opportunities to learn," Leland Cogan and "An empirical analysis of intended curricula for a cross-national study of mathematics and science," a forum for discussion of preliminary results of curriculum analysis component of TIMSS presented by Richard Houang, Gilbert Valverde and Virginia Keen.

Gilbert Valverde presented "International Curriculum Analysis: Preliminary Results from the Third International Mathematics and Science Study" at the 1994 Annual Conference of the Comparative and International Education Society in San Diego March 22, 1994.

AAAS:

William Schmidt presented two papers at the national meeting of the American Association for the Advancement of Science, February 18-23, 1994 in San Francisco. The first paper was titled "International Mathematics and Science Curricula," and was part of panel on The Third International and Science Study (TIMSS) organized by Dorothy Gilford and Daniel Horvitz and also featuring Albert E. Beaton, Study Director for TIMSS and David Wiley and Richard Wolfe who are affiliated with the U.S. National Research Center. The second paper was "International Mathematics and Science Curricula," part of a panel on National and International Tests: How Good is the Science, organized by Audrey Champagne (who is on the TIMSS U.S. Steering Committee) and also featuring Elizabeth Stage who is on the U.S. Steering Committee for TIMSS.

Upcoming Presentations:

We urge you to publicize and attend one of the upcoming sessions related to TIMSS at one of the following conferences:

NARST - National Association for Research in Science Teaching, Anaheim, CA - March 26-29, 1994:

"International Study of Opportunities to Learn, Including Textbooks and Instructional Practice," Special Session, March 27, 2:45 PM, Medallion Room.

NSTA - Anaheim, CA - March 30-April 2, 1994:

"Analysis of Science Curricula from Over Forty Countries: Launching the Third International Mathematics and Science Study (TIMSS)", March 31, 2:00 PM, Hilton, Avila Room.

AERA - American Education Research Association - New Orleans, LA - April 4-8, 1994:

"Cross-National Measurement of Opportunities to Learn in Math and Science," April 5, 8:15 AM, Sheraton Ballroom E.

"What in the World Are World Class Standards?" April 5, 12:25 PM, Sheraton, Bayside B.

"International Curriculum Analysis: Results from the TIMSS," April 6, 8:15 AM, Sheraton Ballroom D.

"Measurement of Opportunity to Learn Elementary and Secondary Science and Mathematics: Recent Experience with Methods, Validity, and Interpretation," April 7, 10:35 AM, Marriott, LaGalerie 5.

CCSSO - National Conference on Large-Scale Assessment - Albuquerque, NM - June 1994:

Refer to programs for information when they become available.

We want to hear from our readers, so if you have questions, want additional information about a topic addressed in the newsletters or the study in general, please feel free to contact us. You can write or call Gilbert Valverde, Assistant Director - TIMSS, 457 Erickson Hall, Michigan State University, East Lansing, Michigan 48824-1034, telephone 517-353-7755, fax 517-336-1727, or E-Mail, valverde@msu.edu. 

If you have suggestions of articles you would like to see in future newsletters or if you or someone you know did not receive this newsletter directly, but would like to be on our mailing list, please send your name and address along with your request to Jacqueline Babcock, 464 Erickson Hall, Michigan State University, East Lansing, Michigan 48824-1034, telephone 517-353-7755, fax 517-336-1727, or E-Mail: jbabcock@msu.edu. 

We also are available to make presentations to professional groups or associations who might be interested in being introduced to TIMSS or receiving updates on the project. If your organization is interested in hearing more about TIMSS and would like a representative from our office to present to your group, please let us know. 

National Research Coordinating Committee

Dr. Jeanne Griffith, Associate Commissioner for Data Development, National Center for Education Statistics; Dr. Eugene Owen, National Center for Education Statistics; Dr. Lois Peak, National Center for Education Statistics; Dr. Larry Suter, Office of Studies and Program Assessment, National Science Foundation. The committee is chaired by Dr. William Schmidt, Michigan State University.

This newsletter is published by the TIMSS U.S.. National Research Center located at Michigan State University. The newsletter is edited by Jacqueline E. Babcock.