More Thoughts on Formative Design

In Formative Design, the researcher works hand in hand with the classroom teacher to design the study and values the practitioner's knowledge and expertise in every stage of the research. 

Someone asked the question last night about generalizability... something along the lines of, "how do you make what you learn through formative design generalizable to other populations? Do you follow up with experimental or quasi-experimental research?"

This got me thinking about what's really going on. The implication in that question is that the variability that is necessarily introduced when the researcher and the instructor are collaborating during formative design represents the introduction of confounding variables. Each teacher may be implementing things a little bit (or a lot) differently, or even the researcher's presence may create a condition that is impacting student outcomes and can't be recreated in typical classroom settings. But you can't do without, because one of the values of formative design is that interventions shouldn't just be dropped down on practitioners, that teachers should be involved.

Maybe the real intervention, the one that has the most impact on student learning, is treating teachers like professionals and giving them control over what is happening in their classrooms. Maybe simply having a university researcher sitting down and saying, "I value your expertise. Let's work together to design an intervention that will work with your students," maybe that's what really works. Maybe it doesn't matter nearly as much what actual procedure or technique you are using with the students. 

That certainly is different than saying that interventions fail because of issues with fidelity of implementation. Instead, it's saying that interventions fail when teachers are not engaged as professionals in solving problems in their classrooms. I'd love to see a large scale study in which the treatment is formative design research on an intervention and the control is a fidelity model of implementing an intervention. Maybe you should even vary the intervention being implemented, because since the intervention itself may not be the most critical factor.

Formative design

Dr. Dennis visited our class and talked to us about formative design. I like the focus on valuing practitioner knowledge. The pragmatic epistemology also appeals to me.

Formative design differs from action research in its focus on utilizing and developing theory. Differs from experimental and quasi-experimental in that you can modify what you are doing based on what you see happening. It's not quite the same as Mixed Methods.

Part of pragmatism says that even conflicting ideas can be informative. How do conflicting theories come together to inform classroom practice? Praxis is an important concept in formative design.

This approach emphasizes close alignment between theory, research, and (classroom) practice, and it's iterative.

Your approach should be skepticism, not romanticism or advocacy. Your goal isn't to prove your theory is right. You have to balance your beliefs with what is actually working in the classroom.

This is a methodology in its infancy, so researchers working in this area must be explicit in all stages of the research.

Timeline for Formative Experiments

Phase 1: Preliminary phase - recruit schools and teachers, meet with stakeholders, discuss goals, obligations, responsibilities; negotiate plans.

Phase 2: Gather demographic data. Use "ethnographic" methods to create a thick description of the classroom, school, and community.

Phase 3: Gathering baseline data. Establish where participants are in relation to the pedagogical goal prior to implementing an intervention.

Phase 4: Implement the intervention; gather data.

Phase 5: Post assessment to provide comparison to baseline.

Phase 6: Consolidate findings and write it up.

For information on formative design and related issues, see:

Bauman, J., Ware, D., & Edwards, E. (2007). "Bumping into spicy, tasty words that catch your tongue": A formative experiment on vocabulary instruction. The Reading Teacher, 61(2), 108-122.

Johnson, R. B., & Onwuegbazie, A. J. (2004). Mixed methods research: A research paradigm whose time has come. Educational Researcher, 33(7), 14-26.

Reinking, D., & Bradley, B. A. (2008). Formative and design experiments: Approaches to language and literacy research. New York: Teachers College Press.

Reinking, D., & Watkins, J. (2000). A formative experiment investigating the use of multimedia book reviews to increase elementary students' independent reading. Reading Research Quarterly, 35(3), 384-419.

Ivey, G., & Broaddus, K. (2007). A formative experiment investigating literacy engagement among adolescent Latina/o students just beginning to read, write, and speak English. Reading Research Quarterly, 42(4), 512-545.

Dennis, D. (n.d.) Co-constructing the implementation of multi-level texts in middle school science classrooms. Unpublished proposal for research.

Vellutino & Schatschneider 2004

Vellutino, F., & Schatschneider, C. (2004). Experimental and quasi-experimental design in literacy research. In (chapter 7) N. Duke, & M. Mallette, editors. Literacy Research Methodologies. New York: Guilfords.

(aka Duke Ch. 7)

These authors did an outstanding job of summarizing and contextualizing the issues and parameters surrounding experimental and quasi-experimental design. I agree with Melissa that this chapter should be required reading in Stats. I think the difference here is in perspective. These writers place statistical concepts and procedures firmly within the context of literacy research. In Stats texts, I think the presentation of the material concentrates on the details without the context - and that's probably reasonable since we are learning concepts and procedures that are common to many disciplines and our classmates are not all interested in literacy research. However, that is a good argument for a quantitative statistical measurement class taught specifically for literacy researchers within the Reading program.

Vellutino & Schatschneider propose evaluation of experiments and quasi-experiments based on the adequacy of the hypothetical counterfactual and how well the design addresses concerns about internal validity, external validity, statistical conclusion validity, and construct validity. They give examples of how one might evaluate research using their own studies as exemplars. They make it clear that all studies have weaknesses and flaws and that understanding the flaws is important to understanding how to connect and apply research beyond the contexts of the original studies.

I think that this chapter (and its references) will be useful for evaluating existing research and designing new studies.

Class notes, Reading Research

Be open to the expertise of others, partners in research. Don't let ego get in the way. Your methodological decisions are dictated by your research questions and your orientation toward the data.

Kinch says, the higher the proposition density, the harder it is to understand a text.

Protocol analysis is inductive, a window into the cognitive process. You are making inferences to construct a model the cognitive process, a model of a reader's thoughts.

Subjects are trained in protocol analysis before data collection with similar task using different material. During training, reinforce process talk, not product talk. By doing this, you are skewing the verbal protocol toward the type of data you are looking for. If you don't train subjects, you can get a lot of unusable data. When you do, you run the risk of not finding things that you aren't looking for.

Subjects have to be engaging in goal-directed behavior so that you can collect data; they have to be trying to do something, not just read without a specific purpose.

The researcher's verbal prompts need to be on topic. Talking about making a hamburger while cleaning a carburetor will disrupt the cognitive process. In protocol analysis, you talk about the task that you are doing; it doesn't disrupt, it just slows down cognitive processing.

Just enough categories to account for the data. Start broad, with many categories, then check back with the data to narrow.

Next week, historical analysis.
Harvey Graff looks at what literacy does to cultures. He sees literacy's outside impact on culture, separate from culture. Doug Hartman and Jennifer Monahan research the history of reading. Also, content analysis and document analysis. Jim King and Norm Stahl work with oral histories. We will be looking at USF's collection of educator oral histories on iTunes U. Lastly, we will be looking at how one situates the history of reading for course delivery.

Gutiérrez & Orellana 2006, ELL Genres of Difference

Gutiérrez, K. & Orellana, M. (2006). The "Problem" of English language learners: Constructing genres of difference. Research in the Teaching of English, vol. 40, no. 4, 502-507.

Essay describing the limitations of many studies of ELL populations that result from methodology common to these studies. The authors describe these studies as a genre, with common characteristics, from participant selection to methods of analysis, typical of its members.

This might be an interesting article for AM's current project.

This article does not relate to genre in children's writing or literature.

Genre Research Process So Far

For this lit review, I'm researching two main bodies of work:

1. genre as it pertains to literature and writing in school, and
   
2. genre as it pertains to films and filmmaking

To begin a search for published research in the first category, I manually searched article titles for the past ten years (1999-2008) of Reading Research Quarterly (RRQ) for references to genre. I then searched the same time period in Research in the Teaching of English (RTE). I chose the ten year span because I wanted to capture the most recent research on this topic. I plan to review the references in the articles captured through this search to identify seminal pieces from before the ten year period. RRQ and RTE are top tier education research journals and should contain references to other work that will pertain.

To begin a search on the second category, I spoke to the research librarian for education at the USF library. We tried several searches, refining down to a search using Google Scholar (Advanced: contains "genre" AND exact phrase "film studies", 1999-2008). One of the articles produced from that search is Mittell 2001, published in Cinema Journal. That article referenced a book, Film/Genre by Rick Altman, which Mittell identifies as the key work done in examining genre as a discursive practice. Mittell also refers to Jane Feuer's essay in Channels of Discourse as the "most comprehensive discussion of television genre theory".


From here, I will move forward from Mittell 2001 to see what subsequent works cite him and return to the original search results. I also may do a title search, similar to those done for RRQ and RTE, in Cinema Journal and another journal suggested by the earlier search results. I also have to continue tracking down resources for the literature/writing side of the lit review.

House 2007 Computers and Reading Achievement

House, J. (2007). Relationships between computer use and reading achievement of elementary-school students: Results from the PIRLS 2001. International Journal of Instructional Media, vol. 34 (4).

PIRLS is the Progress in International Reading Study conducted  in 2001. House has published a series of articles analyzing data from the PIRLS. This article examines statistical relationships between a student's use of computers and performance on the PIRLS reading comprehension test.

My first question is about the PIRLS test itself. There isn't enough detail in this article to know how valid and reliable the PIRLS is, but I want to know how they defined reading comprehension and how they constructed the instrument used to measure the students. What were the reading passages and where did they come from? House mentions the selection process for the schools and classrooms, but I wonder if different regions of the country were represented. I don't know that it would make a difference in the results, but I am curious if it was a consideration.

So, the reading comprehension of 3,385 students was assessed using a test booklet containing two reading blocks, and, presumably, questions about the text. The students were also asked where and how much they use computers. House used several statistical procedures (most of which are currently way beyond me) to compare the data.

The results were mixed. Students who use computers more frequently at home also scored higher on the test. However, students who used the computer for writing, scored lower on the test. House notes several studies with results that are consistent with his and several studies with results that are inconsistent with his results.

I think that these studies are likely measuring different things. There are a number of factors that differentiate the quality of computer use that are not considered in this study. The statements about computer use on the assessment were: "I use the computer at home."; "I use the computer at school."; "I use the computer in some other place." That doesn't give a meaningful level of information about what is happening. "Using a computer" is far too general an activity to be meaningful for research. If I use an oven for three hours and you use an oven for three hours, we're not necessarily going to end up with the same cake.

My second concern is about a presumption of causation. House didn't go there, but others easily could. "Oh, using a computer more at home improves reading ability. So to help kids read, they should use the computer at home." No, no, no, no. What if that correlation is just that? Maybe a higher level of home computer use just signifies a more affluent home, and a more affluent home means more access to early text experiences, or a higher level of parental academic achievement, and one of those things causes the reading comprehension gains. If that's the case, just increasing home computer use won't help reading achievement at all. It would just cost a lot of money and could end up depleting resources that could be used to impact reading ability.

Discussion of Risko, et al.

The level of rigor and scrutiny described in this study is impressive, daunting, and inspiring. When I imagine something that I've written under this level of scrutiny, I shudder. Every time I thought they had taken their analysis of the studies as far as they could, they would back up and come at them a different way. Each decision that they made seems to be arrived at through careful and logical evaluation of the needs of the study. I would have liked for them to say, "and here's how we screwed up..." just to make them appear a little more human.

Of course, I'm also so grateful that they (a) did this analysis, and (b) published their procedures in such incredible detail. I have the feeling I'll be using Risko, et al., as a model for research for years to come.

Interestingly, the researchers note the lack of historical context provided in the studies they reviewed. I am interested in the progression of ideas through time and in contextualizing the work of individual researchers. I would be interested in doing some kind of paper that examined the relationships between different researchers' work, or perhaps tracing the movement of ideas across time.

Part 3 - Risko, et al.

Part 3 - Risko, V., Roller, C., Cummins, C., Bean, R., Block, C., Anders, P., Flood, J. (2008). A Critical Analysis of Research on Reading Teacher Education

Method for Data Analysis

Most of the studies were qualitative. The researchers applied "an inductive paradigmatic analysis process (Polikinhorne, 1995)" to analyze the remaining 82 studies.

They first categorized the studies by classifying details, events, and situations - producing 7 categories. Next, they identified the primary foci and reduced the categories to four:

1. beliefs
2. knowledge
3. pedagogy
4. program research

Next, they produced a detailed analysis of the layers of logic in each study. Finally, in contrast to their microanalysis, they produced a macroanalysis that produced a coding matrix of patterns within the studies.

Limitations

1. US only
2. Although their categories for analysis are supported by the data and methods, there may be other equally valid ways of organizing the data.
3. All research focused on prospective teachers.

Part 2 - Risko, et al.

Part 2 - Risko, V., Roller, C., Cummins, C., Bean, R., Block, C., Anders, P., Flood, J. (2008). A Critical Analysis of Research on Reading Teacher Education

Method for Lit Review

Risko, et al., evaluated the methodology and research design of each study in relation to its research questions. For this, they cite the "logics in use" concept from Kaplan (1964) and Howe and Eisenhart (1990).

Parameters for inclusion:

1. Published, empirical, peer-reviewed
2. Pub between 1990-2006
3. Pre-service K12 Reading teacher focus
4. In the US (this was added after finding only 3 from outside the US)

Search procedure:

1. Electronic search of databases (ERIC, InfoTrac, ISI Web, PsycINFO)
2. Manual search of annual conference yearbooks (NRC, CRA, ARF)
3. Manual search of recent journals not yet included in databases
4. Bibliographies of other reviews (Risko didn't list this source in the search procedure, but did list it as a source of studies included)

Sources of 233 selected studies:

• 79 electronic searches
• 53 NRC
• 24 ARF
• 40 CRA
• 8 manual journal search
• 29 from bibliographies of other reviews

The researchers established criteria to assess the quality of the studies, based on the work of others and in consultation with other experts. In addition to the 3 standards, they identified 7 criteria which could be used as a model for assessing quality of research. The standards are:

• Standard 1 - Clear and coherent argument and reasoning, connecting theory to research.
  
• Standard 2 - Clear disclosure of rigorous methodology.
• Standard 3 - Findings match the question and are supported by the data.

Coding for Inclusion

The researchers established a rigorous procedure for evaluating the 233 studies. They first carefully established inter-rater reliability. Then they went through several iterations of evaluating studies on a 3 point scale, rating each based on the 7 criteria. Those that met all 7 criteria were rated 3. The rest were eliminated.

They then coded each study for (1) level of explicitness, i.e. explicit instruction, modeling, etc. and (2) effect strength, examining what percentage of participants exhibited an effect.

Part 1 - Risko, et al.

Risko, V., Roller, C., Cummins, C., Bean, R., Block, C., Anders, P., Flood, J. (2008). A Critical Analysis of Research on Reading Teacher Education

Risko, et al., is a lit review of reading teacher education studies. I'm reading this primarily to study it as a model of how to conduct a lit review. Starting from a pool of 298 studies, they selected 82 to include.

------

• Began by reading lit reviews for "learning to teach" and reading teacher preparation. This helped them understand the characteristics of the field and set broad goals.
  
• Set the following objectives: 
  

1. identify criteria and quality indicators for evaluating research
2. conduct comprehensive search and analysis of empirical research meeting those criteria
3. analyze theoretical arguments and practical issues in the studies.
4. identify future research objectives

The researchers sought to include a variety of theoretical perspectives. They grouped studies into five theoretical categories:

1. positivist/behavioral - This represents a transmission model of teaching, an additive process. The theory was dominant in the 1970s. Characterized by lists of discrete, observable skills and behaviors to be mastered.
2. cognitive - Importance placed on connecting learning with prior knowledge and beliefs. Piaget and Bartlett's schemata.
3. constructivism - Focus on conditions for learning, problem-solving, inquiry, and collaboration. Shift away from transmission model.
4. socio-cultural - Focuses on individuals relationship to social context, "multiple forms of interaction with others".
5. critical theories - Focus on how literacy educators can address social justice issues, and on equitable learning conditions.

Researchers attempted to maintain "objectivity" by meeting and developing "shared and negotiated understandings of the papers"... "collaborative interpretive analysis and critique". They acknowledge that their personal histories influenced the meaning that they constructed.