Discussion of Risko, et al.

The level of rigor and scrutiny described in this study is impressive, daunting, and inspiring. When I imagine something that I've written under this level of scrutiny, I shudder. Every time I thought they had taken their analysis of the studies as far as they could, they would back up and come at them a different way. Each decision that they made seems to be arrived at through careful and logical evaluation of the needs of the study. I would have liked for them to say, "and here's how we screwed up..." just to make them appear a little more human.

Of course, I'm also so grateful that they (a) did this analysis, and (b) published their procedures in such incredible detail. I have the feeling I'll be using Risko, et al., as a model for research for years to come.

Interestingly, the researchers note the lack of historical context provided in the studies they reviewed. I am interested in the progression of ideas through time and in contextualizing the work of individual researchers. I would be interested in doing some kind of paper that examined the relationships between different researchers' work, or perhaps tracing the movement of ideas across time.

Part 3 - Risko, et al.

Part 3 - Risko, V., Roller, C., Cummins, C., Bean, R., Block, C., Anders, P., Flood, J. (2008). A Critical Analysis of Research on Reading Teacher Education

Method for Data Analysis

Most of the studies were qualitative. The researchers applied "an inductive paradigmatic analysis process (Polikinhorne, 1995)" to analyze the remaining 82 studies.

They first categorized the studies by classifying details, events, and situations - producing 7 categories. Next, they identified the primary foci and reduced the categories to four:

1. beliefs
2. knowledge
3. pedagogy
4. program research

Next, they produced a detailed analysis of the layers of logic in each study. Finally, in contrast to their microanalysis, they produced a macroanalysis that produced a coding matrix of patterns within the studies.

Limitations

1. US only
2. Although their categories for analysis are supported by the data and methods, there may be other equally valid ways of organizing the data.
3. All research focused on prospective teachers.

Part 2 - Risko, et al.

Part 2 - Risko, V., Roller, C., Cummins, C., Bean, R., Block, C., Anders, P., Flood, J. (2008). A Critical Analysis of Research on Reading Teacher Education

Method for Lit Review

Risko, et al., evaluated the methodology and research design of each study in relation to its research questions. For this, they cite the "logics in use" concept from Kaplan (1964) and Howe and Eisenhart (1990).

Parameters for inclusion:

1. Published, empirical, peer-reviewed
2. Pub between 1990-2006
3. Pre-service K12 Reading teacher focus
4. In the US (this was added after finding only 3 from outside the US)

Search procedure:

1. Electronic search of databases (ERIC, InfoTrac, ISI Web, PsycINFO)
2. Manual search of annual conference yearbooks (NRC, CRA, ARF)
3. Manual search of recent journals not yet included in databases
4. Bibliographies of other reviews (Risko didn't list this source in the search procedure, but did list it as a source of studies included)

Sources of 233 selected studies:

• 79 electronic searches
• 53 NRC
• 24 ARF
• 40 CRA
• 8 manual journal search
• 29 from bibliographies of other reviews

The researchers established criteria to assess the quality of the studies, based on the work of others and in consultation with other experts. In addition to the 3 standards, they identified 7 criteria which could be used as a model for assessing quality of research. The standards are:

• Standard 1 - Clear and coherent argument and reasoning, connecting theory to research.
  
• Standard 2 - Clear disclosure of rigorous methodology.
• Standard 3 - Findings match the question and are supported by the data.

Coding for Inclusion

The researchers established a rigorous procedure for evaluating the 233 studies. They first carefully established inter-rater reliability. Then they went through several iterations of evaluating studies on a 3 point scale, rating each based on the 7 criteria. Those that met all 7 criteria were rated 3. The rest were eliminated.

They then coded each study for (1) level of explicitness, i.e. explicit instruction, modeling, etc. and (2) effect strength, examining what percentage of participants exhibited an effect.

Part 1 - Risko, et al.

Risko, V., Roller, C., Cummins, C., Bean, R., Block, C., Anders, P., Flood, J. (2008). A Critical Analysis of Research on Reading Teacher Education

Risko, et al., is a lit review of reading teacher education studies. I'm reading this primarily to study it as a model of how to conduct a lit review. Starting from a pool of 298 studies, they selected 82 to include.

------

• Began by reading lit reviews for "learning to teach" and reading teacher preparation. This helped them understand the characteristics of the field and set broad goals.
  
• Set the following objectives: 
  

1. identify criteria and quality indicators for evaluating research
2. conduct comprehensive search and analysis of empirical research meeting those criteria
3. analyze theoretical arguments and practical issues in the studies.
4. identify future research objectives

The researchers sought to include a variety of theoretical perspectives. They grouped studies into five theoretical categories:

1. positivist/behavioral - This represents a transmission model of teaching, an additive process. The theory was dominant in the 1970s. Characterized by lists of discrete, observable skills and behaviors to be mastered.
2. cognitive - Importance placed on connecting learning with prior knowledge and beliefs. Piaget and Bartlett's schemata.
3. constructivism - Focus on conditions for learning, problem-solving, inquiry, and collaboration. Shift away from transmission model.
4. socio-cultural - Focuses on individuals relationship to social context, "multiple forms of interaction with others".
5. critical theories - Focus on how literacy educators can address social justice issues, and on equitable learning conditions.

Researchers attempted to maintain "objectivity" by meeting and developing "shared and negotiated understandings of the papers"... "collaborative interpretive analysis and critique". They acknowledge that their personal histories influenced the meaning that they constructed.