Session Information
09 SES 07 B, Issues in Test Construction and Validation
Paper Session
Contribution
A major finding from recent large-scale assessments on student achievement is that all around the world a remarkable proportion of students is only barely able to read. This does not only hold true at the end of primary school but also at the end of secondary school (Mullis, Martin, Kennedy, & Foy, 2007; OECD, 2010). Against this background, one might be concerned that students usually fill out the background questionnaires self-administered and individually in large-scale assessments. The present study, therefore, investigates whether the quality of scales can be improved by reading questionnaires aloud to the students.
The importance of language has been studied in the context of student assessments and survey research. But even though survey experts repeatedly emphasize the importance of the question wording and suggest using simple syntax and avoiding unfamiliar terms (e.g. Bradburn, Sudman, & Wansink, 2004), many questionnaires are quite demanding. The field of test development faces similar challenges. Here, test developers suggested to make linguistically demanding tests more accessible by reading them aloud. Experimental studies suggest that particularly poor readers can benefit from this test administration policy (e.g. Meloy, Deville, & Frisbie, 2002; Randall & Engelhard, 2010; Wolf, Kim, Kao, & Rivera, 2009). In other words, students’ test responses were not affected by the linguistic complexity when the tests were read to them and the factorial structure became one-dimensional. Consequently, the test results are more valid measures of the construct being assessed. With respect to background questionnaires, however, there is no research on the consequences of reading them aloud to students for the data quality.
The main aim of the current study is to estimate the effect of reading background questionnaires aloud on the data quality of scales. We focus on the factorial validity of the responses to a complex scale. Therefore, we conducted an experiment where we compare students’ responses when we read the questionnaires aloud (treatment group) with the responses of students that read them on their own (control group).
The study is part of the National Educational Panel Study (NEPS) in Germany (Blossfeld, , Rossbach & von Maurice, 2011). The project analyzes educational processes from early childhood to late adulthood. The main purpose is to collect longitudinal data of educational processes, decisions, competences, and returns of education throughout the life course. In our study we invited about 50 schools from the lower secondary track in four German states to participate. In most German states tracking starts after grade 4 and students from this track generally perform lowest in large-scale tests (Naumann, Artelt, Schneider & Stanat, 2010). The four states represent different geographical parts in Germany from rural and urban areas. About half of the schools accepted to participate with all in all 664 grade 5 students (323 girls, 341 boys, Mage=12.3 years, SDage=0.9 years) from 27 schools.
Method
Expected Outcomes
References
Blossfeld, H.-P., Rossbach, H. G., & von Maurice, J. (2011). Education as a Lifelong Process: The German National Educational Panel Study (NEPS). Zeitschrift für Erziehungswissenschaft (Sonderheft). 14. Wiesbaden: VS Verlag für Sozialwissenschaften. Crawford, L., & Tindal, G. (2004). Effects of a Read-Aloud Modification on a Standardized Reading Test. Exceptionality, 12(2), 89-106. DiStefano, C., & Motl, R. W. (2006). Further investigating method effects associated with negatively worded items on self-report surveys. Structural Equation Modeling, 13(3), 440-464. Marsh, H. W. (1996). Positive and negative global self-esteem: A substantively meaningful distinction or artifactors? Journal of Personality and Social Psychology, 70(4), 810-819. Meloy, L. L., Deville, C., & Frisbie, D. A. (2002). The Effect of a Read Aloud Accommodation on Test Scores of Students With and Without a Learning Disability in Reading. Remedial and Special Education, 23(4), 248–255. Naumann, J., Artelt, C., Schneider, W., & Stanat, P. (2010). Lesekompetenz von PISA 2000 bis PISA 2009 [Reading Literacy from PISA 2000 to PISA 2009]. In E. Klieme, C. Artelt, J. Hartig, N. Jude, O. Köller, M. Prenzel, W. Schneider & P. Stanat (Eds.), PISA 2009. Bilanz nach einem Jahrzehnt [PISA 2009. Review after one decade] (pp. 23-71). Münster: Waxmann. Randall, J., & Engelhard, G. (2010). Performance of Students With and Without Disabilities Under Modified Conditions. Using Resource Guides and Read-Aloud Test Modifications on a High-Stakes Reading Test. The Journal or Special Education, 44(2), 79-93. Wolf, M. K., Kim, J., Kao, J. C., & Rivera, N. M. (2009). Examining the effectiveness and validity of glossary and read-aloud accommodations for English language learners in a math assessment (CRESST Report 766). Los Angeles: University of California, National Center for Research on Evaluation, Standards, and Student Testing (CRESST).
Search the ECER Programme
- Search for keywords and phrases in "Text Search"
- Restrict in which part of the abstracts to search in "Where to search"
- Search for authors and in the respective field.
- For planning your conference attendance you may want to use the conference app, which will be issued some weeks before the conference
- If you are a session chair, best look up your chairing duties in the conference system (Conftool) or the app.