Study Of Competence-Based Assessment Systems For Apprentices and Rater's Judgment
Research questions, objectives and theoretical framework|
This paper is based on my - project, HELVYRD (Competence based assessment for learning within the framework of vocational didactics) is part of the Norwegian research project MECVET (Measuring Competence in Vocational Education and Training) that will develop a model-based instrument for measuring vocational competence where the original prototype is the German KOMET (see Rauner et al, 2013). The focus of my study is to compare the judgments of raters that assess the test tasks in MECVET with judgments made in the ratings of trade examination and written exams of apprentices (electricians) in the Norwegian VET-system. What are the criteria they use, and to what extent are their judgment holistic and have vocational validity?
This study will compare the raters’ judgments within each assessment system and between the systems. The reason for comparing these settings is that they differ in terms high stake (examinations) and low stake (research study) and the type of format (“pen and pencil” essay-based tasks, short-questions/answers and performance tasks). How does holistic and vocation-based judgment vary across these settings?
Why to pick several systems is to look at this in a more holistic way and look at the similarities and differences in each system and related to one trade. How do this affect the raters and the way they assess?
The HELVYRD-project brings together different theoretical frameworks and areas of research. It is addressing issues that are at the core of assessment theory (Black et al, 2000) for example the referencing of competence and criterion based assessment (Wolf, 1995, 2001), the possibility of holistic judgment, in competence based VET education (Dobsen, 2010), task-assignment construction for high level of vocational validity (Haasler & Erpenbeck, 2008; Johnson, 2008), the institutionalization of model-based assessment systems compared with curriculum-based assessment systems and the combination of different assessment systems (Baartman et al, 2007, Gipps, 1999).
The German KOMET assessement instrument is based on a model from which are derived 8 dimensions or criteria (functionality, presentation, effectiveness, sustainability, business and work process, environmental, creativity, social acceptability) and for each task assignment a solution space is developed by experts in the trade. Thus the raters are trained to adhere to these guidelines whereas raters in the Norwegian trade examinations are assumed to have a focus on official competence goals, trade-specific criteria and the work contexts of the apprentices. In the study the main focus will be on comparison on the judgmental sources of the raters in these contrasting contexts since this objective is important for a validation of the model based KOMET-instrument in relation to an authentic institution like the regular trade examination. The HELVYRD project will provide data for international comparisons since the KOMET instrument has been piloted in Germany, China, South Africa (Rauner, 2012). In parts of Germany it may be integrated in the VET-system since it accords with recent changes rooted in the Lernfeld-reform (Fischer, 2011). The HELVYRD-project is also relevant for our understanding of competence-based examinations in VET. Although some studies have been done in this field, the research on the judgmental basis of raters is almost non-existant in the international VET-literature (MacLean & Wilson, 2009).
The research design in HELVYRD needs to be explorative given the lack of studies addressing the issues above. My project is connected to the MECVET-project (2012-2016) that is carrying out an longitudinal study of the KOMET instrument adapted to Norwegian VET in three trades (nurses assistant, industrial mechanics and electrician). Various data sources have been used in the HELVYRD project. (1) video/audio tapes from several workshops with trade experts, rater training sessions and documentary material and test solutions/questionnaires with pupils and apprentices in Norway. (2) data from observations and interviews with the MECVET-raters (more specific on last year apprentices) and the member of the board that are involved in the construction of the tasks used for trade examinations and raters of the examination. The sampling frame includes the written theoretical examination (last year apprentices) and one practical examination (last year apprentices). In each of the three assessment systems 4 raters were interviewed twice – of which one session was a stimulated recall review of 6 up to 8 task assignments representing different grade levels. In some cases elements of think aloud was used. The data has been analysed by thematic analysis used the computerbased qualitative analysis program Atlas TI.
The MECVET-project qualitative data and my own data collection amongst the MECVET raters, will be used in a comparison with the trade examination systems in Norway.
At the present my analysis has focus on different categories. Regarding those three systems the findings shows several similarities and differences related to framework, content, learning outcomes and competence, methods and tools, evaluation and in the end learning conditions.
The paper will discuss these experiences in relation to the issue of model-based and curriculum-based assessment systems. During the spring 2015 all the data-collection was done and during fall 2015 and spring 2016 the work on data analyzing will continue and the results will be ready for presentation.
Baartman, L.K.J. et al (2007) Evaluating assessment quality in competence-based education: A qualitative comparison of two frameworks. Educational research review, 2, 114-129.
Black, P. (2000). Research and the Development of Educational Assessment. Oxford Review of Education. Vol.26, Nos. 3&4, pp407-419.
Dobsen, S (2010). Nasjonale og internasjonale utfordringer innen elev- og lærlingvurdering. I
Dobsen, S. & Engh, R. r. (2010). Vurdering for læring i fag: Høyskoleforlaget.
Fischer, A. (2011) Das Lernfeldkonzept als Forschungsanlass und Diskursthema in der Berufs- und Wirtschaftspädagogik – Leuphana Notizen. bwp@ Spezial 5, 1-16
Gipps, C. (1999). Chapter 10: Socio-cultural aspects on assessment. Review of Research in
Education, Vol. 24, s. 355‐392.
Haasler, B. & Erpenbeck, J (2008): Assessing vocational competences. In Rauner, F. &
MacLean, R. (2008) Handbook of technical and vocational education and training research.
Johnson, M. (2008) Assessing at the borderline: Judging a vocationally related portfolio holistically. Issues in educational research, 18, 1, 26-44
Maclean, R. & D. Wilson (Eds.) International handbook of education for the changing world of work. Springer Science+Business Media B.V.
Rauner, F. et al (2012): Project Report Comet-Pilot Test South Africa, i:BB Universität Bremen
Rauner, F. et al (2013). Competence Development and Assessment in TVET (COMET).
Theoretical Framwork and Empirical results: Springer Science and Business Media.
Wolf, A. (1995). Competence-based assessment. Buckingham: Open University Press.
Wolf, A. (2001) Competence-based assessment. In Raven, J. & J. Stephenson (eds) Competence in the learning society. NY: Peter Lang