Method for Lit Review
Risko, et al., evaluated the methodology and research design of each study in relation to its research questions. For this, they cite the "logics in use" concept from Kaplan (1964) and Howe and Eisenhart (1990).
Parameters for inclusion:
- Published, empirical, peer-reviewed
- Pub between 1990-2006
- Pre-service K12 Reading teacher focus
- In the US (this was added after finding only 3 from outside the US)
Search procedure:
- Electronic search of databases (ERIC, InfoTrac, ISI Web, PsycINFO)
- Manual search of annual conference yearbooks (NRC, CRA, ARF)
- Manual search of recent journals not yet included in databases
- Bibliographies of other reviews (Risko didn't list this source in the search procedure, but did list it as a source of studies included)
Sources of 233 selected studies:
- 79 electronic searches
- 53 NRC
- 24 ARF
- 40 CRA
- 8 manual journal search
- 29 from bibliographies of other reviews
The researchers established criteria to assess the quality of the studies, based on the work of others and in consultation with other experts. In addition to the 3 standards, they identified 7 criteria which could be used as a model for assessing quality of research. The standards are:
- Standard 1 - Clear and coherent argument and reasoning, connecting theory to research.
- Standard 2 - Clear disclosure of rigorous methodology.
- Standard 3 - Findings match the question and are supported by the data.
Coding for Inclusion
The researchers established a rigorous procedure for evaluating the 233 studies. They first carefully established inter-rater reliability. Then they went through several iterations of evaluating studies on a 3 point scale, rating each based on the 7 criteria. Those that met all 7 criteria were rated 3. The rest were eliminated.
They then coded each study for (1) level of explicitness, i.e. explicit instruction, modeling, etc. and (2) effect strength, examining what percentage of participants exhibited an effect.
No comments:
Post a Comment