Advancing Fully Integrated Mixed Methods and Quantitative Methods
The topics of papers here include: fully integrated mixed methods research designs in engineering education, mixing strategies during analysis, positionality, and systems view of enrollment/retention
Mixed Methods Research in Engineering Education
Exemplars of Integration in Engineering Education’s Use of Mixed Methods Research** (2020)
This theory paper explores ways in which the engineering education community can achieve more comprehensive integration in mixed methods designs. We searched for exemplars in the Journal of Engineering Education, the European Journal of Engineering Education, and the Australasian Journal of Engineering Education using “mixed-method” and “mixed methods” as keywords – resulting in 60 viable articles. Using Creamer’s Mixed Methods Evaluation Rubric (MMER), we found exemplar articles demonstrating considerable proficiency across one or more of the rubric’s four criteria: (1) amount of mixing, (2) interpretive comprehensiveness, (3) transparency and (4) methodological foundation. Using the exemplars as discussion points, we advocate for: (1) the use of mixing during analysis to increase the interplay between the different strands of inquiry (amount of mixing), (2) framing results using the study’s constructs, research questions, or perspectives to avoid siloing the study’s approaches (interpretive
comprehensiveness), (3) using methods flowcharts to communicate design features (transparency), and (4) drawing from a more extensive body of methodological literature to justify design decisions (methodological foundation).
This theory paper explores ways in which the engineering education community can achieve more comprehensive integration in mixed methods designs. We searched for exemplars in the Journal of Engineering Education, the European Journal of Engineering Education, and the Australasian Journal of Engineering Education using “mixed-method” and “mixed methods” as keywords – resulting in 60 viable articles. Using Creamer’s Mixed Methods Evaluation Rubric (MMER), we found exemplar articles demonstrating considerable proficiency across one or more of the rubric’s four criteria: (1) amount of mixing, (2) interpretive comprehensiveness, (3) transparency and (4) methodological foundation. Using the exemplars as discussion points, we advocate for: (1) the use of mixing during analysis to increase the interplay between the different strands of inquiry (amount of mixing), (2) framing results using the study’s constructs, research questions, or perspectives to avoid siloing the study’s approaches (interpretive
comprehensiveness), (3) using methods flowcharts to communicate design features (transparency), and (4) drawing from a more extensive body of methodological literature to justify design decisions (methodological foundation).
Exemplars of Integration in Engineering Education’s Use of Mixed Methods Research | |
File Size: | 681 kb |
File Type: |
Fully Integrated Mixed Methods Research Designs in Engineering Education
Characterizing Students’ Global Competence Development Paths Through a Global Engineering Program** (2018)
Global competence is increasingly recognized as an important skill for engineering students to develop in preparation for their entrance into the engineering workforce. A variety of global engineering programs have been developed to achieve this goal, and several studies have assessed the outcomes of such programs. To date, literature on global engineering programs has emphasized program overviews and assessment of student learning outcomes. Although outcomes-based assessment is important for the continuous improvement of such programs, recent critiques of global education research suggest that another perspective is missing from the literature. Few studies explore student conceptions of their global programs and how students may experience the same program in different ways. Understanding variation in students’ experiences is important to developing effective global programs, particularly as educators seek to improve the diversity of such programs. To address this need, our study piloted a fully-integrated complementarity mixed-methods approach to identify and characterize unique student paths through a single global engineering program.
ERRATUM: There is a reporting error in Tables 5 and 6. The cluster center for Cluster 2 is (0.48,0.33,-0.09,0.48), not (0.88,0.73,0.41,0.8). The number of students in Cluster 2 in the Europe track is 10, not 2. My apologies.
Global competence is increasingly recognized as an important skill for engineering students to develop in preparation for their entrance into the engineering workforce. A variety of global engineering programs have been developed to achieve this goal, and several studies have assessed the outcomes of such programs. To date, literature on global engineering programs has emphasized program overviews and assessment of student learning outcomes. Although outcomes-based assessment is important for the continuous improvement of such programs, recent critiques of global education research suggest that another perspective is missing from the literature. Few studies explore student conceptions of their global programs and how students may experience the same program in different ways. Understanding variation in students’ experiences is important to developing effective global programs, particularly as educators seek to improve the diversity of such programs. To address this need, our study piloted a fully-integrated complementarity mixed-methods approach to identify and characterize unique student paths through a single global engineering program.
ERRATUM: There is a reporting error in Tables 5 and 6. The cluster center for Cluster 2 is (0.48,0.33,-0.09,0.48), not (0.88,0.73,0.41,0.8). The number of students in Cluster 2 in the Europe track is 10, not 2. My apologies.
Characterizing Students’ Global Competence Development Paths Through a Global Engineering Program | |
File Size: | 651 kb |
File Type: |
Positionality
Positionality: The Stories of Self that Impact Others** (2019)
This initial work in progress paper explores a discussion of positionality from two doctoral candidate researchers in engineering education. Initiated and guided by Culture, and Researcher Positionality: Working Through Dangers Seen, Unseen, and Unforeseen (Milner, 2007), this paper will present a starting point of dialogue and self-exploration from both qualitative and quantitative research perspectives. Engineering education researchers task themselves with being innovative towards the production of knowledge that is intended to improve and serve current practices and other ways of thinking, knowing, and doing. However, what can be said of the internal conversation that either does or does not occur within and outside of the bounds of reflexivity in research? This work in progress paper will provide an initial and non-exhaustive presentation of literature related to qualitative and quantitative considerations in positionality, describe the initial process and thoughts of two researcher's conversation of key incidents related to positionality over time in their graduate careers, barriers and supports to exploring positionality, and how this exploration influences their respective research. The personal exploration presented here is intended to serve as a starting point to ongoing reflexive work for each graduate researcher as a means of continuous development and meant to enhance research practices.
This initial work in progress paper explores a discussion of positionality from two doctoral candidate researchers in engineering education. Initiated and guided by Culture, and Researcher Positionality: Working Through Dangers Seen, Unseen, and Unforeseen (Milner, 2007), this paper will present a starting point of dialogue and self-exploration from both qualitative and quantitative research perspectives. Engineering education researchers task themselves with being innovative towards the production of knowledge that is intended to improve and serve current practices and other ways of thinking, knowing, and doing. However, what can be said of the internal conversation that either does or does not occur within and outside of the bounds of reflexivity in research? This work in progress paper will provide an initial and non-exhaustive presentation of literature related to qualitative and quantitative considerations in positionality, describe the initial process and thoughts of two researcher's conversation of key incidents related to positionality over time in their graduate careers, barriers and supports to exploring positionality, and how this exploration influences their respective research. The personal exploration presented here is intended to serve as a starting point to ongoing reflexive work for each graduate researcher as a means of continuous development and meant to enhance research practices.
Positionality: The Stories of Self that Impact Others | |
File Size: | 406 kb |
File Type: |
Systems View of Enrollment/Retention
Visualization and Analysis of Student Enrollment Patterns in Foundational Engineering Courses (2019)
The literature in engineering education and higher education has examined the implications of course-taking patterns on student development and success. However, little work has analyzed the trajectories of students who need to retake courses in the curriculum, especially those deemed to be fundamental to a student's program of study, or sequences of courses. Sequence analysis in R was used to leverage historical transcript data from institutional research at a large, public, Land-Grant university to visualize student trajectories within individual courses-with attention to those who re-enrolled in courses-and the pathways students took through a sequence of courses. This investigation considered students enrolled in introductory mechanics courses that are foundational for several engineering majors: Statics, Dynamics, and Strength of Materials (also called Mechanics of Deformable Bodies). This paper presents alluvial diagrams of the course-taking sequences and transition matrices for the different possible grades received upon subsequent attempts for the Mechanics core courses to demonstrate how visualizing students' paths through sequences of classes by leveraging institutional data can identify patterns that might warrant programs to reconsider their curricular policies. (Pre-print from International Journal of Engineering Education)
Citation: Reeping, D., Knight, D., Grohs, J., & Case, S. (2019). The Visualization and Analysis of Course-Taking Patterns in Foundational Engineering Science and Mechanics Courses. International Journal of Engineering Education. 35(1A), 142-155.
The literature in engineering education and higher education has examined the implications of course-taking patterns on student development and success. However, little work has analyzed the trajectories of students who need to retake courses in the curriculum, especially those deemed to be fundamental to a student's program of study, or sequences of courses. Sequence analysis in R was used to leverage historical transcript data from institutional research at a large, public, Land-Grant university to visualize student trajectories within individual courses-with attention to those who re-enrolled in courses-and the pathways students took through a sequence of courses. This investigation considered students enrolled in introductory mechanics courses that are foundational for several engineering majors: Statics, Dynamics, and Strength of Materials (also called Mechanics of Deformable Bodies). This paper presents alluvial diagrams of the course-taking sequences and transition matrices for the different possible grades received upon subsequent attempts for the Mechanics core courses to demonstrate how visualizing students' paths through sequences of classes by leveraging institutional data can identify patterns that might warrant programs to reconsider their curricular policies. (Pre-print from International Journal of Engineering Education)
Citation: Reeping, D., Knight, D., Grohs, J., & Case, S. (2019). The Visualization and Analysis of Course-Taking Patterns in Foundational Engineering Science and Mechanics Courses. International Journal of Engineering Education. 35(1A), 142-155.
Visualization and Analysis of Student Enrollment Patterns in Foundational Engineering Courses | |
File Size: | 605 kb |
File Type: |
Transfer Student Information Asymmetries
The Transfer between community colleges and four-year institutions has become more common as student mobility increases, especially for engineering. Institutions have coped with this inter-institutional movement by establishing agreements with each other that facilitate pathways between programs. The forward-facing policies and guides to inform students on such pathways are known to be complicated, leading to the development of web-based tools like Transferology to help students navigate the system. Despite these advances, transfer students continue to struggle in their information search.
The purpose of this dissertation was to describe the extent to which information for engineering transfer students is scattered across multiple web-based sources and written in a manner not conducive to understanding. I used a fully integrated mixed methods design to create narratives capturing the interactions between the more quantitative idea of scatter using network analysis and the more qualitative aspect of language-use using visually based two-cycle coding across 38 U.S. four-year institutions. All data was readily available online, which were transformed and combined using several mixing strategies to form integrated stories of information asymmetries.
The resulting narratives characterized the information design for transfer students as a messy web of loosely connected structures with language that complicates understanding. Moreover, institutional websites describing transfer processes use communication strategies similar to private companies writing online privacy policies. In light of the themes of information asymmetries, opportunities for supporting transfer were highlighted. This work will be of interest to those interested in engineering transfer student experiences and pathways. Also, those looking to implement fully integrated mixed methods approaches or make extensive use of existing data, especially mixing during analysis, will see strategies applicable in their own work.
My dissertation can be found here: https://vtechworks.lib.vt.edu/handle/10919/95944
Errata:
(1) It has come to my attention that Table 5 on page 61 has the incorrect %sample statistics. These values were exported from R into the document, so it is simply a misprint. The interpretation and results stated in the remainder of the document are still valid. The correct values are given below.
Strata % Sample
Large Private 10.5
Large Public 57.9
Medium Private 10.5
Medium Public 7.9
Small Private 10.5
Small Public 2.6
(2) The Bazely (2012) reference on p. 10 and p. 194 is a typo. The correct spelling of her name is Bazeley, as given in her other citations.
(3) The qualitative conclusion in Figure 52 on p. 176 should read "Institutional websites" not "Institutions websites."
(4) The third last line on p. 177 should read: "D and E. I draw from Wyner et al.’s (2016) Transfer Playbook, a report from the"
(5) The third last line on p. 183 should read: "the data collection to drawing inferences. This work used all of Creamer’s (2018a) mixing strategies"
COPYRIGHT NOTICE: Papers on this page are shared in accordance with the copyright holder. The following statements are from the copyright holders. Note that year in parentheses is the appropriate year to replace the YEAR placeholder in the copyright statement.
** © (YEAR) American Society for Engineering Education. Other scholars may excerpt or quote from these materials with the same citation. When excerpting or quoting from Conference Proceedings, authors should, in addition to noting the ASEE copyright, list all the original authors and their institutions and name the host city of the conference. - Last updated April 1, 2015
** © (YEAR) American Society for Engineering Education. Other scholars may excerpt or quote from these materials with the same citation. When excerpting or quoting from Conference Proceedings, authors should, in addition to noting the ASEE copyright, list all the original authors and their institutions and name the host city of the conference. - Last updated April 1, 2015