Author(s):Leo van Waveren (presenting)

Conference:ECER 2017

Network:02. Vocational Education and Training (VETNET)


Session Information

02 SES 08 A, Assessing outcomes of VET

Paper Session



Chair:Harm Biemans


Empirical study on the outcome of VET in the technical domain between Germany and Switzerland

Studies with an international scope are more commonly concerned with the systemic conditions than with empirical results of vocational training.  The German Federal Ministry of Education and Research sponsored from 2011 to 2015 projects to gain insights into the structure of professional competences in different demanding occupations with skill profiles relevant to current challenges in the job market in Germany. The field of work of electronics technicians for automation technology (ETAT) is closely linked to the newly emerging challenges of Industry 4.0, for example the integration of assembly lines and the need to reduce downtimes in production due to malfunction. The aim of the project KOKO EA at the University of Stuttgart was to provide a reliable instrument to assess the competences of ETAT at the end of their vocational education.

Throughout the domain of technical occupations a multidimensionality of professional competence is well established, separating content knowledge and its application (Nickolaus & Seeber, 2013), usually understood as (analytical) problem solving. 

For a number of professions e.g. automotive mechatronics (Schmidt, Nickolaus & Weber, 2014) and electronics technicians for energy and building services (Geißel, 2008) a multidimensional structure of content knowledge related to content domains at the end of vocational training proved to be the best fitting model. Consistent with the findings from other occupations within the technical domain van Waveren and Nickolaus (2015) documented a three-dimensional model drawing on content domains as appropriate for ETATs content knowledge at the end of their vocational training.

Discrepancies between intended and achieved competences in VET were documented in the ULME studies (Lehmann & Seeber, 2007) and have been repeatedly found in a broad number of occupations in the technical domain e.g. automotive mechatronics (Gschwendtner, 2008) and construction workers (Petsch, Norwig & Nickolaus, 2015). In the field of electronics technicians the trainees for automation technology are comparatively strong in the cognitive domain. Therefore a curricular valid paper-pencil test was used to assess the content knowledge at the end of VET. Using the approach by Beaton and Allen (1992) achievement scales were applied seperatly for each subdomain. Despite an above average IQ of 108 and a strong school background (roughly 70 % attended an intermediate, 25 % an upper secondary school) only a comparatively small percentage of the participants (roughly 10 %) achieve results similar to the requirements set in the curricula, while, depending on the subdomain, 50 – 60 % failed to achieve results equivalent to the level put forth in the curricula, when their content knowledge was assessed via a paper-pencil test (Walker et al., 2016).

In the Feasibility Study VET-LSA (Baethge & Arends, 2009) for a number of occupations a general overlap between the VET curricula in different countries was found. Among these was the commercial sector in Germany and Switzerland. In a comparison focussed on commercial competence Holtsch, Rohr-Mentele, Wenger, Eberle, and Shavelson (2016) demonstrated, that on general terms using an instrument developed for the German curriculum yields valid testscores in the respective training in Switzerland – although adaptation was needed.

Also in the Feasibility Study VET-LSA similarities between the German “Elektroniker für Automatisierungstechnik” and their Swiss counterpart “Automatiker (EFZ)“ are outlined. It was concluded that they share enough overlap between their curricula to warrant an empirical comparison regarding the training outcome.

In this paper results from a first pilot study are presented, comparing the outcome of VET for the vocation of electronics technicians in both countries.


A paper-pencil-test (using a balanced incomplete block design was constructed with assistance from vocational trainers, teachers and members of the examination board of the Chamber of Industry and Commerce (PAL) for the German curriculum. After a pilot study with 202 participants the revised instrument was accepted by the experts of the field. The final test contained 119 items and was grouped into nine different booklets. Testing time was set to 120 minutes. The tests were conducted in vocational schools (with the exception of two occasions in which the testing took place within regular working hours of the company acting as employer for the trainees). The CFT-20R was used to assess basic cognitive abilities.

1011 electronics technicians for automation technology at the end of vocational training from nine federal states and 33 different schools in Germany were tested. 105 trainees from five different schools at the end of their vocational training were tested as their counterpart in Switzerland. Time constraints at the vocational schools allowed for assessing basic cognitive abilities for roughly half of the participants (506 in Germany, 95 in Switzerland).

Building on the findings from Germany's ETAT at the end of their training, a three-dimensional model based on content domains was used in IRT modelling for content knowledge via ConQuest. Results between the samples were compared based on the EAP-estimators, Hedge’s g* was used to assess the effect size of differences between the groups.

Drawing on the achievements scales used by van Waveren and Nickolaus (2015) to assess the outcome of Germany’s training for ETAT’s are compared to those of Swiss “Automatiker” at the end of their training to offer a qualitative comparison of the outcome of VET in both countries.

Expected Outcomes

The various European countries employ very different systems in the vocational training. When considering an international large scale assessment (similar to PISA) findings on comparable occupations may serve as a framework a potential VET-LSA can draw upon.

The tests’ EAP-PV reliability for all three dimensions is acceptable (.72), a structural equation modelling including basic cognitive competences shows strong correlations between the subdimensions (.74 to .83).

This research project provided insights into the achieved competences at the end of vocational training for the occupation of electronics technicians in two countries. Despite the inherent character of a pilot study Hedge's g* gives evidence for medium effects (.44/.21/.71) in all three subdimensions of content knowledge in favour of the Swiss sample.

Consequently the Swiss participants were placed on average on higher levels in all three subdimension of content knowledge, when the achievement scales were applied.

In sum the main goal of this study, establishing, whether a curricular valid paper-pencil-test for ETAT from Germany can be used to assess content knowledge for their Swiss counterpart can be considered to be met.

Furthermore, this study suggests that despite the instrument being developed for the German curriculum, the differences observed in the sample were in favour of the Swiss participants.


Baethge, M., & Arends, L. (2009). Feasibility Study VET-LSA: A comparative analysis of occupational profiles and VET programmes in 8 European countries – International report. Vocational Training Research volume 8. Retrieved from The Federal Ministry of Education and Research (BMBF) website:

Beaton, A. E., & Allen, N. L. (1992). Interpreting Scales Through Scale Anchoring. Journal of Educational Statistics, 17(2), 191–204. doi:10.2307/1165169

Geißel, B. (2008). Ein Kompetenzmodell für die elektrotechnische Grundbildung: Kriteriumsorientierte Interpretation von Leistungsdaten. In R. Nickolaus (Ed.), Diskussion Berufsbildung: Vol. 9. Didaktik der gewerblich-technischen Berufsbildung. Konzeptionelle Entwürfe und empirische Befunde (pp. 121–141). Baltmannsweiler: Schneider Verlag Hohengehren.

Gschwendtner, T. (2008). Ein Kompetenzmodell für die kraftfahrzeugtechnische Grundbildung. In R. Nickolaus (Ed.), Diskussion Berufsbildung: Vol. 9. Didaktik der gewerblich-technischen Berufsbildung. Konzeptionelle Entwürfe und empirische Befunde (pp. 103–119). Baltmannsweiler: Schneider Verlag Hohengehren.

Holtsch, D., Rohr-Mentele, S., Wenger, E., Eberle, F., & Shavelson, R. J. (2016). Challenges of a cross-national computer-based test adaptation. Empirical Research in Vocational Education and Training, 8(1), 51. doi:10.1186/s40461-016-0043-y

Lehmann, R., & Seeber, S. (Eds.). (2007). Untersuchung von Leistungen, Motivation und Einstellungen der Schülerinnen und Schüler in den Abschlussklassen der Berufsschulen: ULME III. Hamburg.

Nickolaus, R., & Seeber, S. (2013). Berufliche Kompetenzen: Modellierungen und diagnostische Verfahren. In A. Frey, U. Lissmann, & B. Schwarz (Eds.), Beltz Pädagogik. Handbuch Berufspädagogische Diagnostik (1st ed., pp. 155–180). Weinheim: Beltz.

Petsch, C., Norwig, K., & Nickolaus, R. (2015). Berufsfachliche Kompetenzen in der Grundstufe Bautechnik - Strukturen, erreichte Niveaus und relevante Einflussfaktoren. In A.
Rausch, J. Warwas, J. Seifried, & E. Wuttke (Eds.), Konzepte und Ergebnisse ausgewählter Forschungsfelder der beruflichen Bildung. Festschrift zum 65. Geburtstag von Detlef Sembill. Baltmannsweiler: Schneider Verlag Hohengehren.

Schmidt, T., Nickolaus, R., & Weber, W. (2014). Modellierung und Entwicklung des fachsystematischen und handlungsbezogenen Fachwissens von Kfz-Mechatronikern. Zeitschrift für Berufs- und Wirtschaftspädagogik, 122(4), 549–574.

van Waveren, L., & Nickolaus, R. (2015). Struktur- und Niveaumodell des Fachwissens bei Elektronikern für Automatisierungstechnik. Journal of Technical Education, 3(2), 62–91.

Walker, F., Link, N., van Waveren, L., Hedrich, M., Geißel, B., & Nickolaus, R. (2016). Berufsfachliche Kompetenzen von Elektronikern für Automatisierungstechnik – Kompetenzdimensionen, Messverfahren und erzielte Leistungen. In K. Beck, M. Landenberger, & F. Oser (Eds.), „Wirtschaft – Beruf – Ethik“. Technologiebasierte Kompetenzmessung in der beruflichen Bildung – Resultate aus dem Forschungsprogramm ASCOT. Bielefeld: W. Bertelsmann Verlag.

This proposal is part of a master or doctoral thesis.

Author Information

Leo van Waveren (presenting)
Technical University Kaiserslautern
Department of Mechanical and Process Engineering