Mixed methods design – beyond the prototypes. Experiences from a Research Study of 65 Individual High School ICT Development Projects.
Author(s):
Hans-Peter Degn (presenting / submitting) Helle Mathiasen (presenting) Mette Brinch Thomsen
Conference:
ECER 2015
Network:
Emerging Researchers' Group (for presentation at Emerging Researchers' Conference)
Format:
Paper

Session Information

ERG SES G 02, ICT and Education

Paper Session

Time:
2015-09-08
09:00-10:30
Room:
303. [Main]
Chair:
Shosh Leshem

Contribution

The overall purpose of this paper is to present and discuss a mixed methods design fit to address the challenges of embracing a variety of individual and quite diverse development projects within a single research methods design, given that this methods design must be adequate for both extracting generalizable findings as well as identifying the relevant individual characteristics of the development projects. The empirical basis is 65 very different, individual development projects, each in their own way experimenting with different ways to implement ICT in high school teaching and learning activities. Each development project was scheduled for one year, and the overall research study covered four periods of different projects.

The proportions and organization of the projects were very diverse, ranging from single class projects, through groups of classes or complete schools, to network projects among groups of schools. In total, about 5,400 students and 550 teachers were involved. The content of the individual development projects were at least as diverse as their proportions and organization; focusing on different pedagogical elements, using a variety of ICT tools and in different ways, contexts and with different purposes.

 

A quantitative approach was needed to be able to extract generalizable findings from across the many – very diverse – development projects. Without this approach, the overall research study might be unable depict general tendencies and draw the necessary overall conclusions across the variety of development projects. However, given the diverse individual characteristics of the development projects, a quantitative approach can only “scratch the surface” and shed light on a limited number of aspects: the most common characteristics that all the development projects can reasonably be examined for. Therefore, concurrently a more flexible, qualitative approach was needed as well to address and scrutinize the individual characteristics of the projects. In the actual case, this called for observations of learning environments and semi-structured interviews with teachers and students involved in the projects.

However, this combination of approaches not only allows us to benefit from both generalizable quantitative and flexible qualitative data gathering. Inherent are also questions such as how to combine these approaches? How to balance the design between exploration and confirmation? Will the research study be best off by combining the approaches simultaneously or sequentially one after the other? Furthermore, how should the research methods design take into consideration the possibilities for ongoing refinement given by the fact that the research study consists of successive periods?

 

Furthermore, by illustrating and discussing potential benefits from different mixed methods prototypical approaches, we will discuss the implications of implementing the methodical ingredients simultaneously vs. sequentially and also the benefits of iterative/multiphase revisions. 

 

Our mixed methods research design draws on a theoretical framework combining quantitative methodology (e.g. Groves 2009, Vaus 2005) and qualitative ditto (e.g. Beatty 2007, Kvale 1999). Quantitatively, the focus is on questionnaires, whereas the qualitative focus is on observations and semi-structured interviews.

These methodological frameworks are combined with inspiration from, among others, Creswell & Clark (2011). Cresswell & Clark present six prototypical versions of mixed methods design (Creswell 2011: 68-72). According to these prototypes, the design of our study relates to both the convergent parallel design and the exploratory sequential design. Given the fact that the national study was conducted over four consecutive periods, a third prototype is also at stake; the multiphase design.

Method

The diversity of the projects and the fact that they were development projects inevitably called for an exploratory approach. A natural first stage in the methods design was observations of how the individual development projects were carried out and implemented within the respective learning environments, followed by qualitative interviews with a handful of students and a handful of teachers, separately. These opening qualitative data gatherings were conducted midway through the project periods. Based on the findings, the following/concluding data gathering from the projects were designed. The final data gatherings were scheduled at the completion of the project periods. One component was two quantitative questionnaires distributed to all involved teachers and students respectively; in total approx. 5,400 students and 550 teachers. As the design of these questionnaires were based on the findings of the initial observations and interviews our mixed methods approach relates to the exploratory sequential design prototype; building the second, quantitative phase with the aim to generalize (selected) initial, qualitative findings. However, our final data gatherings from the projects also included another round of qualitative semi-structured interviews. In this respect our methods design relates to the convergent parallel design prototype; prioritizing the qualitative and quantitative data from the final round of data gathering equally, conducting them simultaneously, and combining their results during the overall interpretation. Furthermore, given the iterative element of the overall research study covering four consecutive periods of projects, our research design featured extensive elements of multiphase design prototype as well, “combining both sequential and concurrent strands over a period of time” (Creswell 2011: 72). Our mixed methods design is not as “clear cut” as laid out in the prototypical descriptions referred to. Furthermore, what we have articulated as quantitative is not merely so; when designing the questionnaires we also drew upon qualitative methods. To ensure reliability and validity the phrasing of the questionnaires were tested on groups of teachers/students respectively. The combined test consisted of observations during the filling in of the questionnaire combined with “thinking-aloud” tests and follow-up “probing” tests (cf. Beatty 2007). Based on the conducted, qualitative tests, revised versions of the questionnaires were distributed. All together this illustrates our overall approach: Combining complementary qualitative and quantitative techniques and methods unprejudiced, selecting the methodical tool most adequate in the given situation and continuously enhancing the methods design (and thereby the strength of the collected data) by simultaneous, sequential and iterative/multiphase processes.

Expected Outcomes

The impacts on the refinement of the research design stemming from the sequential and the multiphase elements were most evident within/across the two first periods. This goes especially for the questionnaires. Given the experimental format of the development projects scrutinized, it would have been difficult to develop useful questionnaires in the first place, had we not been able to build on the findings from the initial, exploratory, qualitative data gatherings midway through the period. Without these, it would be difficult to foresee which common characteristics of the projects to address – in which ways and wording – in the questionnaires. The questionnaires for the second period were revised both on the basis of our experiences from the first period (multiphase revision) as well as by inspiration from the opening observations and interviews of the second period (sequential development/adjustment). Similar structural procedures were followed in the last two periods, though the alterations became increasingly fewer period by period. The potential benefits from sequential and multiphase developments and enhancements of the research design are obvious. However, it is important also to bear in mind the rivaling consideration of being able to compare findings across, in this case, all four periods. Each revision might at the same time sacrifice an opportunity to perform longitudinal comparisons across the periods of the overall study. Therefore, each revision should be considered carefully before implementation. Well-considered yet prejudiced combinations of qualitative and quantitative methods and techniques, utilizing their respective, complementary strengths, carefully selecting the methodical tool most adequate in the given situation, and continuously enhancing the methods design by sequential and iterative/multiphase processes, potentially provide a research yield regarding both quality, relevance, depth, broadness, and generalizability that cannot be obtained within a simple, one-stage research design focusing on either qualitative or quantitative methodology.

References

Beatty, P. C. & G. B. Willis (2007). Research Synthesis: The Practice of Cognitive Interviewing in Public Opinion Quarterly, vol. 71, issue 2, pp. 287-311. Bryman, A. (2008). Social Research Methods, 3rd edition. Oxford: Oxford University Press. Creswell, J. W. & V. L. P. Clark (2011). Designing and Conducting Mixed Methods Research, 2nd edition. Thousand Oaks, California: SAGE. Teddlie, C. & A. Tashakkori (2009). Foundations of Mixed Methods Research. Integrating Quantitative and Qualitative Approacheds in the Social and Behavioral Sciences. Thousand Oaks, California: SAGE. Groves, R. M. et al. (2009). Survey Methodology, 2nd edition. Hoboken, New Jersey: John Wiley & Sons, Inc. Kvale, S. (1999). InterView. Copenhagen: Hans Reitzels Forlag. Vaus, D. de (2005). Surveys in Social Research, 5th edition. Oxon: Routledge.

Author Information

Hans-Peter Degn (presenting / submitting)
Aarhus University
Aesthetics and Communication
Aarhus N
Helle Mathiasen (presenting)
University of Copenhagen
Mette Brinch Thomsen
Aarhus University, Denmark

Update Modus of this Database

The current conference programme can be browsed in the conference management system (conftool) and, closer to the conference, in the conference app.
This database will be updated with the conference data after ECER. 

Search the ECER Programme

  • Search for keywords and phrases in "Text Search"
  • Restrict in which part of the abstracts to search in "Where to search"
  • Search for authors and in the respective field.
  • For planning your conference attendance, please use the conference app, which will be issued some weeks before the conference and the conference agenda provided in conftool.
  • If you are a session chair, best look up your chairing duties in the conference system (Conftool) or the app.