Research Competencies
Evidence of Professional Competency in Research
My emerging philosophy of research is the result of my exposure to a variety of research approaches and methods in my classes and research groups, at conferences, and in my independent reading. As I formulate the goals of my research agenda, it is becoming increasingly clear to me that I must consider how those goals cohere with my nascent philosophy. The nature of those goals and the types of claims that I hope to make will guide me in the selection of appropriate research methods.
During my years as a technologist in higher education, I developed many software applications to address work-related problems and/or to improve work processes and products. I would like to continue in this design mode by addressing complex problems related to learning and teaching, but I also want to take my work further by evaluating its effectiveness and formulating new theories and design principles. This type of inquiry—which goes under various names, including design-based research (Barab & Squire, 2004; Wang & Hannafin, 2005), design and development research (Richey and Klein, 2007), and formative research (Reigeluth & Frick, 1999)—combines qualitative and quantitative methods and contributes to both theory and practice. The result should be a desired change in the world as well as a better understanding of how that change was accomplished and how it can be adapted for other situations. The nature of this approach is in the pragmatic tradition of Dewey, Peirce, and others. I think this kind of approach to research is socially responsible and will enable me to integrate better my research goals with my goals for teaching and service.
Research Competencies I Will Need
Given my intended approach to research, I will need experience in selecting and using both quantitative and qualitative methods of research. I understand that researchers can be both enabled and constrained by their training and knowledge. Because my responsibilities as a faculty member will include advising future teachers, designers, and researchers, I think it is important for me to have a broad knowledge of inquiry methods. I am by training and temperament more comfortable with words and images than with numbers, so I need to ensure that I acquire a solid foundation in quantitative methods of analysis. At the same time, much of my research will require observation, interviews, and other qualitative methods. Because of my interest in design research, I need to become more familiar with the inquiry methods favored by designers in other fields such as product design, interaction design, and experience design. Ultimately, to achieve my goal of becoming an innovative researcher, I hope to synthesize my knowledge of design inquiry and emerging technologies to develop new methods for gathering, analyzing, and interpreting data to assess and understand learners’ needs, achievements, and experiences.
Beyond expanding my knowledge about inquiry methods, I need more experience in writing research proposals and designing and conducting research studies. Specifically, I need more practice in identifying research problems; reviewing the related literature; formulating research questions; and collecting, analyzing, and interpreting data. These are fundamental competencies, and I need to master them in order to achieve my goals.
In order to pursue my research agenda, I will need funding, which means I need to develop grant-writing skills. These skills include finding appropriate funding opportunities, critically reading requests for proposals to understand the goals and requirements of the funders, developing timelines and budgets for projects, identifying and recruiting collaborators, and writing grant proposals that include appropriate research designs, methodologies, and evaluation strategies. If I can be successful in this area, I will be more desirable as a candidate for a faculty position and eventually for tenure and promotion.
I also need to disseminate the results of my research, which involves writing about my work, getting published in respected publications, and presenting in appropriate venues. Therefore, I need to hone my scholarly writing and presentation skills. In addition, I need to be able to critique the work of others, a skill which is also necessary in order to be an effective teacher and which overlaps with my service goal of editing scholarly publications.
Research Competencies I Have Acquired
Evidence of competencies in conducting research
My first set of research competencies are related to my knowledge about methods of inquiry. In addition to Y520 (Strategies for Educational Inquiry) and Y502 (Intermediate Statistics Applied to Education), I took Y604 (Multivariate Analysis) in the spring semester of 2008 to develop a better understanding of methods for analyzing quantitative data. I also took Y611 (Qualitative Methods) for an overview of the methods for collecting and analyzing qualitative data. In addition, as part of my doctoral minor in HCI-Design I took I502 (Special Topics in HCI: Experience Design) and I543 (HCI Design and Evaluation Methods), in which I learned about and practiced design research methods such as cultural probes, bodystorming, card sorting, and focus groups.
- Exploratory Factor Analysis: "Factors influencing achievement in mathematics literacy"
I conducted an exploratory factor analysis of data from the Third International Mathematics and Science Study (TIMSS) that addresses mathematics literacy. Analyzing these data give me practice in formulating a rationale for factor selection, conducting a principle axis factor analysis, and interpreting the meaning of the results.
- Matched-samples t test: "SimTIE-Math: A simulation-game on technology integration for mathematics learning"
As part of the formative evaluation of a game, I used a matched-samples (or paired) t test to see whether the difference between the means of pre-test and post-test scores was significant. This is reported in the paper “SimTIE-Math: A Simulation-Game on Technology Integration for Mathematics Learning” (see below in "Evidence of competencies in scholarly writing and presentation" for a full description of this paper).
- Independent-samples t test and effect size: "Assessment of strategies within the Diffusion Simulation Game"
I was second author of this study and my primary responsibility was data analysis. We divided the sample into two groups: successful game sessions (in which 22 adopters were achieved) and unsuccessful game sessions (in which fewer than 16 adopters were achieved). Rather than using SPSS, I used MS Excel and wrote the formulae for calculating t and Cohen's d, which helped me to develop a deeper understanding of these statistical methods (see below in "Evidence of competencies in scholarly writing and presentation" for a full description of this paper).
- Interviews, observation, artifact analysis: "What's age got to do with it? How older students, their families, and the institutions
perceive the experience of being a non-traditional graduate student"
For this paper, my colleagues and I used a variety of qualitative methods of inquiry. Data collection was conducted by me and two other students in Y611, but we each wrote our own final report. I gained a greater appreciation for how difficult and time-consuming qualitative methods can be. However, this experience will make be better prepared to use these kinds of methods in future research.
- "Rhythmning: The Art of the Beat"
In the spring semester of 2009, I enrolled in I502 (Experience Design) with my minor advisor, Dr. Jeffrey Bardzell. The focus of this course was on experience design, which I think will be useful in general when designing learning experiences and particularly applicable to the design of virtual learning environments such as simulations. In this class, I worked with two other designers to research, create, and test prototypes for an interactive museum exhibit to teach people about drumming and rhythm. Design inquiry methods we employed included prototyping to understand different disabilities, brainstorming, sketching, bodystorming, and rapid prototyping for usability testing. I documented our process on video and edited it for our final deliverable.
Evidence of competencies in writing grant proposals
My second set of research competencies are related to writing grant proposals. I have made significant progress toward my research competencies in grant proposal writing by assisting Dr. Frick with several proposals. Unfortunately none of these was funded, but the experience of writing them has prepared me to write my own proposals. I have acquired experience in finding and understanding requests for proposals, identifying and recruiting collaborators, developing timelines and budgets, and writing reviews of relevant literature.
- NSF I-TEST: "MAPSAT Curriculum Dashboard: Connecting the dots for STEM Careers"
One research project for which we sought funding was the application of MAPSAT to curriculum mapping in the form of a MAPSAT Dashboard. One of the challenges with MAPSAT is helping people to understand what it is and how it can be used, so I designed some visual representations to show what such a dashboard might look like and what it would mean in terms of curriculum mapping. These representations were used in grant proposals that we submitted to the W. T. Grant Foundation and NSF I-TEST. I also worked closely with Dr. Frick to develop a three-year plan for developing the MAPSAT Dashboard, which was included in the NSF I-TEST proposal. See "MAPSAT Curriculum Dashboard: Connecting the dots for STEM Careers," figures 1, 2, and 3.
- NSF Advanced Learning Technologies: "SimEd-Math: Modeling Differentiated Instruction in Mathematics"
Another project in which I participated in grant writing was “SimEd-Math: Modeling Differentiated Instruction in Mathematics” for the NSF Advanced Learning Technologies grant. I contributed the section “Research on Simulations, Games, and Learning” (pp. 5-7) and the section “How is SimEd-Math Different from Other Existing Simulations for Preservice Teachers” (pp. 11-12). I also worked closely with Dr. Frick to develop a three-year project plan and budget that were used in that proposal, and I created a visualization of the project timeline (Figure 3 in the proposal).
Evidence of competencies in scholarly writing and presentation
I have been a productive scholar during my time in the IST program, especially in the past two years as I have collaborated (and continue to collaborate) with various faculty and fellow students. I have co-authored several studies and have presented extensively at national conferences.
- Paper: "SimTIE-Math: A simulation-game on technology integration for mathematics learning"
I was first author of this design-based research which describes the design and development of a paper prototype for a digital game. We examined artifacts that resulted from the design process thus far, including design documents, iterations of game components and rules, e-mail exchanged by the designers, notes from meetings, reports from playtesting sessions, and other related documents. We used these data to construct a narrative of the design experience that focused on how instructional design theories guided the process and informed decisions about the game’s goals, rules, mechanics, and aesthetics. This paper was published in the AECT Proceedings but I am writing a new version which I plan to submit to The Journal of Applied Instructional Design.
- Paper: "Assessment of strategies within the Diffusion Simulation Game"
I was second author of this paper that describes a study of historical game-play data from the Diffusion Simulation Game (DSG) and my primary responsibility was data analysis. I formatted and analyzed the data and wrote the analysis. I applied MAPSAT’s analysis of patterns in time (APT) methodology to examine patterns of choices made by players. By analyzing patterns that lead to success in the game, we attempted to determine the game’s fidelity to theories of the diffusion of innovation. The results informed the redesign of the DSG and demonstrate how APT can be used during the design process for formative evaluation of an instructional product. I also wrote the literature review on "Fidelity" and edited several iterations of the paper, which has currently received favorable reviews and will likely be published in the journal Simulation and Gaming.
- Paper: "Creating video annotated discussions: An asynchronous alternative"
I was second author of this invited article which was published in the first issue of The International Journal of Designs for Learning. I wrote a program to collect data (student annotations of YouTube videos) and I wrote a description of this process for the "paper" which is actually a Flash piece that presents the design case using video, audio, and animation.
- Literature Review: "The Assessment of Learning in Instructional Games and Simulations"
I wrote a review of the literature on games/simulations for R711. Writing this paper has been an important part of my growth as a scholar as it has given me insight into past and current research on games/simulations in the field of IST. I have used parts and versions of this review in grant proposals and research articles.
- Literature Review: "Data Collection and Analysis of Interaction Trails"
This paper examines historical and current methods of collecting and analyzing data generated by users' interactions with and in hypermediated environments. It will be part of the literature review for my dissertation.
- Presentations:
As indicated on my CV, I have been very active in presenting my work. For the past several years I have led or been a member of multiple presentations at AECT on SimTIE, MAPSAT, and various studies related to the DSG.
I have also presented twice at the Games+Learning+Society (GLS) Conference in Madison, WI. In July of 2008, I presented a poster on SimTIE at the GLS Conference. The acceptance rate for this conference was only 37%. My poster presentation won first prize in the design area.
In October of 2010 I presented work-in-progress at the Meaningful Play conference. This on-going study is based on "The Dossier Game," which I designed and implemented at the 2010 IST Conference.
Evidence of competencies in critiquing scholarly writing
I have several examples that indicate my progress toward competency in the critique of scholarly work.
- "Critique of 'Exploring the Efficacy of the Cook School District Simulation'"
I wrote this paper for R690 (Application of Research Methods to IST Issues). I learned about the Cook School District simulation through my research on simulations designed for pre-service teachers. The purpose of the study was to test the effectiveness of the simulation in enabling teacher candidates to practice connecting teaching and learning and to prepare for evidence-based assessment of their teaching. I found problems with the sample selection and the self-assignment for treatment.
- "The Application of Statistical Methods in a Study of Technology Integration"
I wrote this paper for Y502 (Intermediate Statistics Applied to Education). The purpose of the study was to determine whether the use of specialized software to create cognitive organizers increased the learning of students as compared with traditional textbook instruction. I found that the authors violated the assumption of independence of observations that is required for analysis of variance (ANOVA).
- In the spring semester of 2008, Dr. Frick invited me to provide independent critiques of two articles that he had been asked to review, one submitted to Tech Trends and one submitted to Asia Pacific Education Review. I wrote a one-page critique of each article. I found this to be valuable, real-world practice that increased my confidence in critiquing scholarly work that has been submitted for publication.
- "Critique of Bumptop and Siftables"
To demonstrate the increasing breadth of my critical abilities, I am also including a critique of interactions that I wrote in I502 (Experience Design). As a designer, it is important to consider the felt experience of an interaction by examining its experiential qualities, thereby cultivating a capacity to critique. In this paper, I draw upon my experiences to critically explore the interactive qualities of a software program and a “smart” toy.
Research Competencies I Need to Acquire
I need to continue conducting research, especially studies with more experimental designs to enhance my experience with quantitative methods.
Even though editing a scholarly publication is more of a long-term goal, I am also seeking an opportunity to serve as a reviewer or assistant editor.