A Survey of the methodologies used in literacy research: A content analysis study

Introduction and background

Research methodology forms an integral part of research. It is important not merely for its procedural necessity, but more for what it can contribute epistemologically to research studies. Psychological caution about lack of sufficient knowledge of methodology is vividly captured in Kaplan’s law of instrument: “Give a small boy a hammer, and he will find that everything he encounters needs pounding” (Kaplan, 1964, p.28). It speaks volume of the detrimental potential of limited method knowledge: What we are able to use may affect what we can understand and its corresponding actions. It also points to the need for exploration of new methods and intelligent application of existing methods.

Literacy researchers have been paying attention to the research methodology (Duke & Mallette, 2004; Guzzetti, Anders, & Neuman, 1999). Major literacy research publication outlets have been open to methodology exploration and critiques. For example, Handbook of Reading Research has a separate section reviewing literacy research methods in Volume I (1984) and Volume III (2000) publications, including two chapters on empirical studies and ethnographic studies in the first volume and more broadly conceived research methods in ten chapters in the third volume. Major literacy research journals such as Reading Research Quarterly and Journal of Literacy Research (formerly Journal of Reading Behaviors) have consistently published critiques and commentaries on methodological applications in literacy research (e.g., Lysynchuk, Pressley, d’Ailly, Smith, & Cake, 1989; Ridegeway, Dunston, & Qian, 1993; Troia, 1999 ). Likewise Literacy Research Association’s (formerly National Reading Conference) yearbook has occasionally included methodological discussions (Authors, 1997; Beach, et al, 2009; Dunston, et al, 2012). Synthetic introduction of Literacy research methods has been published also in book form, showing an acute awareness of the importance of research methods for researchers doing literacy research (Duke & Mallette, 2004, 2011; Karmil, Mosenthal, Pearson, & Barr, 2002). New books on new literacy methods, including methods with literacy research in digital technology and visual arts situations are coming out (Albers, Holbrook, & Flint, 2013).

These methodological efforts in literacy research clearly show the realization of the literacy research community for the need of new methods and a better understanding and application of existing methodology (e. g., Lysynchuk, Pressley, d’Ailly, Smith, & Cake, 1989). They focus on both a micro level of method applications and a macro level of method summary and elucidation. At the micro level, the majority of the critical efforts on literacy research method applications are intended to directly address substantive literacy issues such as comprehension education, phonological instruction, or vocabulary teaching. Along this line of research, topic-related methodology analyses have been continuously pursued by researchers to enhance our exploration and consequently understanding of the topics at hand (e.g., Lysynchuk, Pressley, d’Ailly, Smith, & Cake, 1989; Ridegeway, Dunston, & Qian, 1993; Troia, 1999). This body of research has shed new light on our research of the substantive topics under study by bringing forth valuable insight into the methods employed in research studies of the topics. Methodologically, this body of research has pointed out some commonly-detected weaknesses in conceiving, conducting, and reporting the studies. For example, Troia (1999) has highlighted the serious and less serious flaws in experimental methods used in phonological awareness intervention studies that could undermine the studies themselves.

Macro level attention to the methods is mainly captured in literacy method books and handbook and yearbook chapters. For example, Handbook of Reading Research Volume III (Karmil, Mosenthal, Pearson, & Barr, 2000) contains ten chapters that review newly adopted research methods in literacy research. The methods covered show a spectrum of new scholarship in literacy research: teacher action research (Baumann & Duffy-Hester), program evaluation studies (Pigott & Barr), historical research (Monaghan & Hartman), narrative inquiry (Alvermann), critical approaches (Siegel & Fernandez), ethnographic studies (Florio-Ruane & McVee), protocol analysis (Afflerbach), single-subject experiment study (Neuman & McCormick), socio-cultural approaches (Gee), and synthesis research (Shanahan). Nuke and Mallette’s (2004) book has included 14 method chapters that summarize and highlight the various methodologies used in literacy research, some of which were fairly new and innovative in literacy research at the time of its publication such as formative design method (Reinking & Bradley, 2004).

Both the micro and macro level bodies of method knowledge provide methodological support for carrying out literacy research that contributes epistemologically to our study subject: literacy learning and instruction. However, they have different emphases: micro level examinations focusing on critiquing what has been done and pointing out pitfalls in existing studies; macro level summaries, on the other hand, setting their eyes on introducing methods and their conventional or accepted standards. They are also done differently. Micro level method studies resemble a reality check, while macro level chapters and books are to provide guidelines and paths for future studies.

Yet, both types have their own limitations. Reality check studies are usually narrowly focusing on one method dealing with one substantive issue in literacy research, thus making them less interesting to researchers seeking methodological assistance beyond a particular method. Furthermore, they include in their critiques studies on the topic beyond literacy research outlets. Inclusion of studies outside literacy research publication outlets would naturally introduce editorial preferences that may cater to a different audience than the literacy research community. On the other hand, method books and chapters may not be current or comprehensive enough to capture the methodological innovations and applications in the field of literacy research, contributing to a delay in fully appreciating and utilizing the innovations in literacy research methods (Sadoski, 2012).

The present proposal intends to address some limitations as we mentioned above. The study will focus on a comprehensive and synthetic reality check of the methods used in the field of literacy research. Specifically, I propose to examine the scope and depth of the research methods employed in the following leading literacy research publications (Reading Research Quarterly, Journal of Literacy research, Literacy Instruction and Research, Scientific Studies of Reading, Yearbooks of Literacy Research Association) in the past 10 years. Specific attention will be paid to both the methods and the underpinning theoretical paradigms.

The following research questions will be asked:

1. What are the research methods employed in the past decade of literacy studies in these leading literacy publications?

2. Does the field in the past decade show trends or patterns of methods application as indicated in these publications?

3. How do method applications of the past decade in the literacy field as indicated by these publications conform to the research conventions?

Methods

Selection criteria and justifications. The proposed study would use a purposive sampling method (Krippendorf, 2004) to include only literacy research studies in these major literacy research publications for the past decade. The following criteria are used for selecting the appropriate research. First, I will only focus on the studies that are conventionally accepted as studies, both qualitative and quantitative studies. They should involve basic design elements: sampling, subjects (or texts in content analysis), procedures, data sources, analysis procedures, results reporting and interpretations. This would exclude editorial comments, opinion papers, book reviews, and research briefings and summaries that don’t explicitly lay out the methods. Second, I would limit the inclusion of research studies for the past ten years, from 2001 to present. I believe such a time frame gives us the needed consistency in methodological applications in the field, as well as a reliable understanding of any worthwhile emerging trends in methodology. Third, any studies by the literacy researchers published beyond these publications of concern would not be included due to their purported different audience.

Since the purpose of the study is to survey the literacy research field to ferry out the research methods used in research studies, I believe such limited range of inclusion would have several advantages. First, these publications are the most likely leading literacy research outlets for literacy educators and researchers in getting their works published and obtaining their own updated information about the literacy research in the field. The selection of these publications for the present study would ensure that the methods reviewed and reflected in the studies in these journals would capture a fair picture of the extant methods used in the field. I am keenly aware of the fact that many of the literacy studies are published outside the limit of these major literacy research publications. However, the present selection focus would sufficiently represent the methodological endorsements in literacy publications by the literacy researchers.

Second, unlike specific issue/topic-related methods review studies (e. g., Lysynchuk et al, 1989) which intend to investigate the methodological issues associated with substantive subject such as comprehension, this is to bring out the method uses in the field of literacy research in general. I deem it proper to seek a fair representation of the method use within the present scope instead of being exhaustive of all other literacy research studies published in other journals that would have different audience and are undoubtedly influenced and governed by editorial policies not particularly tailoring to literacy researcher audience.

Third, such concentrated effort of scanning the methodological applications in literacy research within the boundary of these leading literacy research publications would offer a better platform for understanding the methodological issues that are under the purview of the literacy researchers. This would procedurally and resource-wise compensate for the unspecificity of the number of methods within the selected publications by offering a feasible in-depth examination of methodological issues in association with these unspecified number of methods included.

Fourth, the present selection scope would allow the implications from the present study to be directly applicable to the literacy research field both in terms of offering literacy researchers recommendations and choices method-wise in conducting their research and feeding back to the editorial boards of these publications for methodological suggestions in screening future studies and introduction of new perspectives and methods. More importantly, it should be an important step towards continuous building of a productive and strong community where ongoing academic discourses are made possible by mutual understanding of the methodological principles underlying various methods, and where academic open-mindedness continuously moves the field toward a better understanding of the literacy issues at hand and those still emerging.

Procedures. The study would adopt a qualitative content analysis approach (Krippendorff, 2004). Samples would include all research studies in these literacy publications for the past ten years. They will be physically copied and printed out for further reading with regard to their methods and theories. Specifically, these last two sections mentioned would serve as the recording units for our ensuing analysis. To maximize the relevancy of these recording units, I would, whenever warranted, pay special attention to the contexts in which these recording units are situated. Contexts would include more substantive aspects of research studies such as problems, results, interpretations and conclusions.

Data analysis. The analysis would include two phases. The first phase starts with the initial categorization of identified methods into traditionally defined methodological groups: quantitative, qualitative, mixed, and others. Further group stratification will be applied to them using either available methods categories identified by Duke and Mallette (2004), Abers, Holbrook, & Flint (2013) and in Handbook of Reading Research (2000) or newer methods that need explorations beyond literacy field. This first phase analysis should help answer the first and second research questions. The third research question will be answered through analyzing data in the second phase as described below.

After this first phase classifications of the recorded units, a second phase of analysis will start by focusing on the design and its components of any research studies that are the traditional key elements in educational research including sampling, subjects, materials, data sources, analysis procedures, descriptive information. At this stage of analysis, two levels of analysis (or two different units of analysis) will be applied: Designs, as a first level of unit of analysis, will be examined for their conformity to the research reporting conventions (Creswell, 2008; Creswell & Clark, 2010), and then each component, as a further level of unit of analysis, within the designs will be scrutinized for their rigor characteristics with inferences to external and internal validity issues (e.g., Authors, 1997; Duke & Mallette, 2004; Lysynchuk, et al, 1989; Ridegeway, Dunston, & Qian, 1993; Troia, 1999). A comprehensive component list for each type of designs will be established for this phase of analysis. Each component’s rigor captured in the methods will be examined within the tradition of that study design. Criteria for the rigor will be based on the traditional and accepted practice in the educational research field (Creswell, 2008; Duke & Mallette, 2004). A coding sheet for each type of studies will be set up for that purpose (Authors, 1997). Since it is an immensely comprehensive content analysis study, I expect that the criteria for each type of research studies would be completed by the time the second phase of the analysis actually starts. To give an example of such coding sheet, I am attaching a coding sheet I used to examine the methodological issues for content analysis email studies (see Table One in the appendix). Similar coding sheets with different emphasis will be developed.

To ensure the reliability of the coding, I would use two conventionally adopted strategies: using inter-coder indexes and situating methodological issues in the referential framework of theories and substantive issues. An outside rater will be trained after the first round of coding is finished by the researcher to use the developed codes to spot check via randomized selection of coded units to ensure the quality of the coding. Any possible emerging pattern from the data would be scrutinized further with reference to substantive issues of the individual research studies involved to ensure ecological validity that could be lacking when methods are examined exclusively as separate entities.

Implications and educational significance

The study has implications for both the literacy research community as a whole and graduate research courses at my own college. First, it would contribute to the knowledge base of the literacy research methods by providing a needed comprehensive survey of applied methods as well as a methodology guide to the researchers in the field. As a field, we need to be continuously informed methodologically to carry out literacy research that advances our knowledge of literacy learning and instruction (Sadoski, 2012). Second, the culminating research courses I have been teaching at College of Staten Island could benefit from the present study. Since I am conducting a content analysis research study as they are required to do in my two classes, students would get a real sense as to how to conduct a content analysis study. In addition, the results of the present study would be eventually made into a book of methodology on literacy research that could be used by my future students and others similarly interested in doing literacy research.

Timeline

Project phases Activities Time periods
Beginning preparation Identifying the relevant articles, making copies of the papers, categorizing them into different types of studies. Starting setting up rubrics for each identified literacy research method based on the conventionally accepted or described methodology principles Starting around June of 2013, continuing over the whole summer, ending at the end of September.
Data analysis Phase One Further classifying different types of studies according to the specific methods used, examining the distributions of different types, and exploring the trends and patterns of methodological applications. Continuing on the rubrics. October of 2013 to November of 2013
Data analysis Phase Two Focusing on the rigor of method application for each study; using and modifying the rubrics, examining the rigor of methodology application; training a research assistant for this phase of analysis. December of 2013 to the beginning of April of 2014
Writing up Writing up report to the PSC-CUNY grant committee; writing a paper on the first phase of the study. Mid April of 2014 to the end of May

The results of the study would be distributed in several formats. At national and international conferences such as Literacy Research Association (LRA), and American Educational Research Association (AERA), the results will be presented. Manuscripts based on the study will be written and submitted to major literacy journals for publication consideration. I also plan to devote the next year (2014 to 2015 School year) to getting together the chapters for a book on literacy research methodology.

Reference

Albers, P., Holbrook, T., & Flint, A. (Eds.). (2013). New Methods of Literacy Research. New York: Routledge.

Authors. (1997). Content analysis in email research: A methodological review.

Beach, R., Enciso, P. Harste, J., Jenkins, C., Raina, S. A., Rogers, R., Short, K. G., Sung, Y. K., Wilson, M., & Yenika-Agbaw, V. (2009). Exploring the “critical” in critical content analysis of children’s literature. In K. M. Leander, D. W. Rowe, D. K. Dickinson, M. K. Hundley, R. T. Jimenez, & V. J. Risko (Eds.), 58th Yearbook of the National Reading Conference, (pp. 129-143). Oak Creek, WI: National Reading Conference, Inc.

Creswell, J. W. (2008). Educational research: Planning, conducting, and evaluating quantitative and qualitative research. Upper Saddle River, NJ: Pearson Publishers.

Creswell, J. W., & Clark, V. P. (2010). Designing and conducting mixed methods research. Thousand Oaks, CA: Sage Publications, Inc.

Duke, N. K. & Mallette, M. H. (Eds.)(2004). Literacy research methodologies. New York: Guilford Press.

Dunston, P., Fullerton, S. K., Headley, K. & Stecker, P. M. (2012). 61st Yearbook of the Literacy Research Association. Oak Creek, WI: Literacy Research Association.

Kamil, M. L., Mosenthal, P. B., Pearson, P. D., & Barr, R. (Eds.). (2002). Methods of Literacy Research. Mahwah, NJ: Lawrence Erlbaum Associates.

Kaplan, A. (1964). The Conduct of Inquiry: Methodology for Behavioral Science. San Francisco: Chandler Publishing Co.

Lynsynchuk, L. M., Pressley, M., d’Ailly, H., Smith, M., & Cake, H. (1989). A methodological analysis of experimental studies of comprehension strategy instruction. Reading Research Quarterly, 24, 458-470.

Reinking, D. & Bradley, B. A. (2004). Connecting research and practice using formative and design experiments. In N. K. Duke and M. H. Mallette, (eds.) Literacy research methodologies (pp. 149-169). New York: Guilford Press.

Ridgeway, V. G., Dunston, P. J., Qian, G. (1993). A methodological analysis of teaching and learning strategy research at the secondary school level. Reading Research Quarterly, 28, 334-349.

Sadoski, M. (2012). Research methods. In P. Dunston, S. K. Fullerton, C.C. Bates, K. Headley, and P. M. Stecker (Eds.), 61st yearbook of the Literacy Research Association (pp. 379-380). Oak Creek, WI: Literacy Research Association.

Toria, G. A. (1999). Phonological awareness intervention study: A critical review of the experimental method. Reading Research Quarterly, 34, 28-52.

Table One: A sample of coding sheet for Content Analysis studies

Content analysis coding/check list for each of the studies

What’s reported What’s missing Comments
Study One Data collection
Sample size
Naturalistic/contrived
Unit of analysis
Reliability
External Validity
Internal validity
Theory or theoretical framework used