Quality Improvement in VET through Large Scale Assessment of Vocational Competence
Inspired by large scale assessment like PISA, TIMMS and PIAAC, German scholars made a feasibility study of a “PISA-VET” or VET-LSA (large scale assessment, Baethge, 2006) piloted in several European countries covering four different trades: electricians, car-mechatronics, social and health care, business and administration. One of the objectives of this initiative was to strengthen the content-specific descriptions of the European Qualification Frameworks’ occupational profiles as common measurement dimensions comparing profiles and learning outcomes at the end of VET (Baethge, 2010). It came to an end in 2009, and different evaluation reports have highlighted the challenges of constructing a large scale assessment instrument for international comparisons of vocational competence at the end of different VET-structures (Olsen, 2009; Baethge & Arends, 2009).
The authors of this proposal participated both in the Norwegian piloting of VET-LSA and are engaged in subsequent activities by conducting the MECVET-project (Measuring Competence Development in Vocational Education and Training). In a Norwegian context, MECVET replicates and validates another instrument for large scale assessment within the framework of the German KOMET-project (coordinated by IBB, University of Bremen; Rauner et al, 2009). This initiative toned down the (ex ante) comparative ambitions of VET-LSA and gave higher priority to the construction of a diagnostic tool that would support competence development at different educational levels (Rauner et al, 2009). Given the later objective, one may distinguish between competence assessment models that underlie large scale comparisons and those that have a local validity (Leigh et al., 2007). These two alternatives vary on a number of parameters – from designs that typically have focus on structural aspects at a national level supporting decisions about input factors that contribute to intended outcomes guided by principles of comparability, independence of tasks, large samples and psychometric approaches. This contrasts with the local assessment models in schools or companies where process improvement and learning are fore-fronted, widening the scope for self-assessment and judgmental variation in the factors to be included and the composition of tasks. As a tentative “middle ground” we have outlined a sectoral assessment approach (Blings & Spöttl, 2008) where “sectoral” is less defined in terms of geographical criteria than in terms of vocational/trade specificities. It relies on substantial models of competence development in a cluster of trades (for example health and social services) that may be characterized by specific work process structures (Fischer & Rauner, 2002). Decisions about contextual factors need to take into account comparability and important contingencies. It also incorporates the notion of a “Beruf” that expands the domain to be assessed in relation to the performance of specific tasks (Fischer et al., 2015; Shavelson, 2010). The model is a mid-stage summary of our validation of the KOMET-model in Norwegian VET and serves as a conceptual template for further explorations. Methodologically it also represents a “break down” (a moment for meta-reflection, Winograd & Flores, 1986) of our large scale assessment design that encountered unanticipated challenges necessitating a multi-case qualitative strategy to research (Stake, 2005). The research question was about the feasibility of a large scale assessment design for Norwegian VET.
The aim of the MECVET project was to conduct a feasibility study of the diagnostic instrument based on the KOMET-model in Norwegian VET. A multiple research design (Robson & McCartan, 2015) was chosen that combined principles of assessment tool development, contextual surveys and quality case-based studies of the instrumentation and implementation processes. The following elements were essential to overall research design:
• Instrument development. Test items (tasks) were translated from German and validated for a Norwegian work context by experts in three vocations; industrial mechanics, electricians and health care workers. Some changes were made and piloted on a subgoup of vocational students.
• Implementation to “try out” the diagnostic tool followed a longitudinal logic with one cohort tested in 2013 as students in their second year of vocational upper secondary school (N=149 or about 50 in each vocation), in 2014 as 1. year apprentices (N=147) – and in 2015 as 2. year apprentices (N=162). Although the same tasks were administered all three years, issues of test-retest were not encountered since we did not have the opportunity to track individuals through the whole period. The Norwegian VET-system is a sequential dual system with 2 years in upper secondary school followed by 2 years of apprenticeship in companies.
• All participants answered a web-based questionnaire on task motivation, vocational commitment and the learning environments of schools/ work places.
• The setup for rater training sessions, scoring rubrics etc. was similar to that of the German KOMET-project. However we collected video-material of both task-construction sessions and rater training in order to have a qualitative perspective on variation in the experts’ argumentation in the different contexts – in line with the sectoral approach outlined above.
• All test and questionnaire scores were processed by SPSS and subject to regression analyses with different measures of vocational competence as dependent variable whereas selected parts of the video-material have been transcribed and organized by qualitative analysis software, AtlasTi, and subject to thematic analysis (Braun & Clarke, 2006). A linguistic analysis of trade-specific literacy in the written solutions from students was done in order to deepen our understanding of effects of test formats on the scores.
Given that the MECVET project is a feasibility study intended to validate and “try out” in Norwegian VET a model and instrument for large scale assessment, the preliminary outcomes are primarily methodological. The outcomes strengthen our belief that a sectoral approach mentioned above is fruitful for further inquiries and later efforts to construct similar diagnostic tools for the measurement of competence development in VET.
• The KOMET-model leans on several theoretical assumptions that may be low in construct validity when the competence of very different vocations like industrial mechanics and health care workers is to be measured. Concepts like “work process knowledge” familiar to German scholars in the debate on VET-reforms (Fischer& Rauner, 2002) and included in the diagnostic tool, may not be salient to Norwegian raters.
• The written format of the test seems to favour the health care workers reflecting both school-experience with essay-based examinations and literacy proficiency.
• The implementation of the instrument in Norwegian VET was only possible if standard procedures as for the number of participants, the test-administration and selection of raters were modified, thus lowering its usefulness for international (ex post) comparisons.
• The most prominent substantial finding from the pilot was the marked contrasts between the three vocations – in absolute terms and on the different dimensions.
• The scores indicated a clear progression during the three years of training for the health care workers but only minor improvement for the electricians. The industrial mechanics ranked in between. Further analyses will clarify the factors that influence these results.
• School-industry networks seem to establish local/regional occupational differences and learning pathways for students and apprentices, but these preliminary observations need more detailed analyses.
Baethge, M. (2010): Ein europäisches Berufsbildungs-PISA als politisches und methodisches Projekt. Im Münk, D. & Schelten, A. (Eds) Kompetenzermittlung für die Berufsbildung. Verfahren, Probleme und Perspektiven im nationalen, europäischen und internationalen Raum. Bielefeld : Bertelsmann BIBB. S. 19-36
Baethge, M. et al (2006): PISA-VET. A feasibility study. Stuttgart: Franz Steiner.
Baethge, M. & Arends, L. (2009): Feasability Study VET-LSA. A comparative analysis of occupational profiles and VET programmes in 8 European countries. Vocational Training Research volume 8, Bielefeld: Bertelsmann Verlag
Blings, J. & Spöttl, G. (2008): "Ways toward a European vocational education and training space: a “bottom-up” approach", Journal of European Industrial Training, Vol. 32 No 2/3 pp. 157 - 170
Braun, V., & Clarke, V. (2006): Using thematic analysis in psychology. Qualitative research in psychology, 3(2), 77-101.
Fischer, M.; Rauner, F. (2002): The implications of work process knowledge for vocational education and training. In Boreham, N.C.; Samurçay, R. & Fischer, M. Work process knowledge. London: Routledge, p. 160-170.
Fischer, M., Rauner, F. & Zhao, Z. (Eds.) (2015): Kompetenzdiagnostik in der beruflichen Bildung. Methoden zum Erfassen und Entwickeln beruflicher Kompetenz: COMET auf dem Prüfstand. Berlin: Lit Verlag.
Leigh, I.W. et al (2007): Assessment Models. Professional Psychology: Research and Practice, Vol 38. No 5, pp 463-473
Olsen, O.J. (2009): Feasability Study VET-LSA. National report from Norway. Department of Sociology, University of Bergen, Bergen May 13th 2009
Rauner, F.; Haasler, B.; Heinemann, L. & Grollmann, P. (2009): Messen beruflicher Kompetenzen, Band 1: Grundlagen und Konzeption des KOMET-Projektes, Berlin LIT Verlag.
Robson, C. & McCartan, K. (2015): Real World Research. 4th Edition. NY: Wiley.
Shavelson, R. J. (2010): On the measurement of competency Empirical research in vocational education and training 2, 1, pp. 41-63
Stake, R.E. (2005): Multiple Case Study Analysis. NY: The Guilford Press.
Winograd, T., & Flores, F. (1986): Understanding computers and cognition: A new foundation for design. Norwood, NJ: Ablex.