Assessing Reading Metacognitive Strategy Awareness of Young Children: The Reading Metacognitive Strategy Picture Protocol

The Reading Metacognitive Strategy Picture Protocol (RMSPP) is an informal, authentic, naturalistic diagnostic tool for classroom teachers and clinicians to use with children as young as kindergarten to assess students’ knowledge and awareness of metacognitive strategies. This article describes the RMSPP and how it was implemented in one informal project with 139 kindergarten – third graders, illustrating the usefulness of this picture protocol response technique with young children.

This article describes an informal, authentic, naturalistic diagnostic tool for classroom teachers to use with children as young as kindergarten to assess students' knowledge and awareness of metacognitive strategies. The instrument -The Reading Metacognitive Strategy Picture Protocol (RMSPP) uses photographs of good readers in informal conversations about children's habits and behaviors before, during, and after reading, with an optional question which integrates children's representational drawings. The picture protocol response technique can also be implemented by special educators or clinicians as well as classroom teachers for instructional planning, having also been used successfully in individual clinical assessments. This article describes the RMSPP and how it was implemented in one informal project with 139 kindergarten -third graders, illustrating the usefulness of the RMSPP with young children.
Experienced reading teachers today know from the well-established research base that efficient, strategic readers display awareness of comprehension strategies, and struggling readers often do not. Instruction in this important component has proven to be an important aspect of classroom curricula (Alexander & Jetton, 2000;Almasi, 2003;Baumann, Jones, & Seifert-Kessell, 1993;Fielding & Pearson, 1994;Paris, Cross, & Lipson, 1984;Paris & Jacobs, 1984;Pressley, 2000;Schmitt, 1990). However, this was not always the case. Over three decades ago Delores Durkin (1978Durkin ( -1979) spent many hours observing in elementary classrooms and was concerned that she found little time being devoted to actual instruction in comprehension or to reading actual texts. She called reading researchers' attention to these unfortunate circumstances with respect to the lack of comprehension instruction, pointing to a dearth of explicit instruction on strategies for understanding text. The previous decades had primarily focused on practical instructional methods for teaching reading, but the decade of the 1980's ushered in an explosion of research investigating comprehension, with contributing studies from cognitive and developmental psychology on perception, memory, and thinking processes for making sense of text (Carr, 1981;Flavell & Wellman, 1977;Pearson & Gallagher, 1983). From these seminal interdisciplinary research studies emerged the conclusions that strategic readers know and use "fix-up" strategies as they read; poor readers often do not. Terms that are now quite familiar, but sometimes confusing, to reading teachers became commonplace in educational dialogues -metacognition, metacomprehension, and comprehension monitoring. John Flavell (1979;Flavell & Wellman, 1977) was the first researcher to use the term metacognition which refers to the ability to know about and monitor one's own learning and cognitive processes, in simple terms, awareness of one's thinking. To clarify the distinctions between these processes, Brown's research (1975) was clarifying as she selected three cognitive skill areas as the title of her classic article on memory: knowing, knowing how to know, and knowing about knowing. Her explanations helped to lay the early foundations for our understandings of the differences among these thinking processes so essential to strategic reading -metacognition, metacomprehension, and comprehension monitoring. To further highlight the differences, Flavell (1981) suggested that readers experience a heightened reaction or awareness while engaged in comprehending a passage -metacomprehension. Closely related to this reaction is comprehension monitoring ability, knowing how to comprehend and knowing some fix-up strategies to use when a reader perceives a breakdown in understanding. Baker and Brown (1984) further researched the relationship between metacognition and reading comprehension, stating that readers who are aware of their intentional activities possess metacognitive awareness and employ problem solving techniques or comprehension monitoring; the result is enhanced understanding of textcomprehension.
We often see metacognition and metacomprehension used interchangeably in the research literature. Metacognition is the general umbrella term overlaying the other two processes; metacomprehension and comprehension monitoring are both specifically used to refer to the reading process (Schraw, 2009;Stewart & Tei, 1983;Wagoner, 1983.) Readers who possess metacognitive awareness are cognizant of their understandings of text and are able to intentionally select the appropriate comprehension strategies needed to proceed with their reading.

Assessment of Metacognitive Strategy Knowledge
The literature underscores the need for comprehension instruction in these metacognitive strategies and emphasizes their central role in efficient reading (Alexander & Jetton, 2000;Almasi, 2003;Baumann & Schmitt,1986;Block & Pressley, 2002;Duffy et al., 1987;National Institute of Child Health and Human Development, 2000;Paris, Cross, & Lipson, 1984;Taylor & Duke, 2013). Responsive, engaged, and thoughtful readers who critically evaluate texts will go beyond pronouncing words correctly and quickly because they are actively involved in constructing meaning and responding appropriately. If teachers are to foster this strategic reading, comprehension instruction must include knowledge and use of metacognitive strategies for use before, during, and after reading. Measures for assessing and monitoring acquisition of these strategies are essential (Alexander & Jetton, 2000;Allington, 2000;Alvermann & Guthrie, 1993;Pressley & Afflerbach, 1995).
Consequently, there is currently a need for user-friendly instruments to measure the developing comprehension strategy awareness of very young children. Duffy et al. (1987) reported few, if any, instruments for measuring this important component of reading comprehension, and the majority of recent research conducted on measurement of this knowledge has focused on more mature students. Block (2005) presented several recently designed informal metacognitive assessments such as, "What do we need to fill in?" and "What's the problem?" She reiterates that research is limited and emphasizes the need for many other tests which can be validated, standardized, and a hierarchy established. Bauserman (2005) proposed the Metacognitive Processes Inventory (MPI) which was field tested with 20 students in kindergarten through ninth grade using an informal interview protocol with questions and recorded student responses. She agrees that more research is needed and states that the question remains as to whether or not the acquisition of metacognitive knowledge is developmental and, if so, how can research document this continuum. Mokhtari and Reichard (2002) have developed an instrument entitled Metacognitive Awareness of Reading Strategies Inventory (MARSI) for assessing 6 thgrade -12th-grade students' awareness and perceived use of reading strategies while reading academic or school-related materials. Kolic-Vehovec and Bajsanski (2006) have developed a Strategic Reading Questionnaire for measuring comprehension monitoring and strategy use in older elementary students, 5 th -8 th grades. Afflerbach and Meuwissen (2005) designed a self-evaluation instrument for assessing metacognitive strategy use by middle school students -Analytic Reading Strategy Inventory. They stress the need for a variety of informal and formal assessment tools for classroom use. (For additional reviews of instruments measuring comprehension monitoring and metacognitive strategies, see Dunlosky & Lipko, 2007, Mokhtari & Reichard, 2002Pressley, 2000).
Perhaps, the most widely used instrument for the purpose of ascertaining comprehension strategy awareness of elementary age children is the Metacomprehension Strategy Index (Schmitt, 1990). This instrument is a valid and reliable questionnaire, providing diagnostic information that can be used by teachers in planning programs of instruction in comprehension strategies. Although the instrument has been used with middle-elementary and intermediate-aged students, the author of this article, the director of a university reading clinic has found it to be time consuming to administer and difficult for very young children to understand. Emergent readers and struggling readers often cannot read the items and must have the questions read aloud to them. Young children have a difficult time understanding the question prompts and do not always respond appropriately, rather only guessing at an answer. Informal strategies such as think alouds (Baumann, Jones, & Seifert-Kessell, 1993), reciprocal teaching (Palincsar & Brown, 1984), and look-backs (Franks et al., 2013;Kinnunen & Vauras, 1995) have been used successfully with primary grade children, but do not provide specific feedback on metacognitive strategy awareness.
Since developing readers, both younger emergent and older struggling readers, are unable to complete more structured, lengthy questionnaires or reading tasks, and busy teachers with multiple mandated assessments have little time for these more timeconsuming assessments, a more user-friendly technique is needed. Block (2005), in stressing the need for better instruments to assess metacognitive strategy awareness, encouraged researchers to go beyond revising existing measures and to create new, developmentally appropriate assessment tools that give students sufficient encouragement and support so that they can communicate effectively what they know about their own comprehension monitoring processes and awareness. Vygotsky's (1978) sociocultural theory is particularly useful in choosing a design format for an assessment tool, one that will be developmentally appropriate for young children who lack independent reading skills and who often respond with answers they believe will please their teachers. Vygotsky believed children acquire much knowledge from interpersonal interactions with more experienced individuals. These interactions shape a child's perceptions and result in concept formation. As a result of these interactions, children form concepts about their world, which may then be interpreted in multiple ways. One venue for young children to communicate that conceptual understanding would be through viewing pictures of children of the same age to focus their attention on what they already know about a topic. A logical next step would be to afford opportunities for the children to talk informally about the pictures with an experienced, familiar adult. Although self-report measures have often been questioned, Kyronlampi-Kylmanen and Maatta (2011) support their use with young children because of the unique perspectives they can bring to the topic under investigation:

Image-Based Research -A Feasible Design Choice
Researchers and educators should have the courage to collect information directly from the children, which often requires untraditional methods. Studying the children should start from the children's culture. The children's and adult's ways of thinking are dissimilar, which requires effort from the adults in order to determine the children's message and opinions. (p. 93) Other early childhood researchers have pointed to the use of pictures, manipulatives, and/or puppets along with informal conversations to supplement selfreport assessments which serve to assure more valid responses from young children and to focus the young child's attention on the researcher's specific task (Causey & Dubow, 1992;Columbus, 2006;Greene & Hogan, 2005: Marsh, Ellis, & Craven, 2002McConaughy, 2013;Schwartz, 1997). These researchers point to the value of these concrete objects and pictures for establishing rapport, putting children at ease, and for encouraging more "talk" and interactions appropriate to the task in a way that more structured, standardized tests cannot. Prosser (1998), a proponent of image-based research, believes that images may provide researchers with unique ways of viewing reality and insights into the ways in which participants (i.e. young children) see themselves and others. Eisner (1985) pointed to the need to utilize different sensory systems in rethinking the way in which we view literacy learning. Silverman (1993) gave additional support to the value of image, reiterating that imagery has been a neglected data source for field studies in the past. More recently, Richards (2006) utilized image-based research in a study with preservice teachers and their self-portraits and affirmed its value for educational research. Following a thorough review of image-based research design principles, it was determined this would provide a developmentally appropriate medium for assessing metacognitive strategy awareness of young children. Space limitations prevent a detailed discussion of image-based design research, but the reader is referred to an article by Cobb (2012) for discussion and the following sources: (Cahnmann-Taylor & Siegesmund, 2013;Gardner, 1980;Hobbs, 1997;Knowles, & Cole, 2008;McKay & Kendrick, 1999, 2001a, 2001bMcNiff, 1998;Prosser, 1998;Richards, 2006;Silverman, 1993;Wetton & McWhirter, 1998).
To summarize the research informing the design of an informal assessment tool for measuring metacognitive strategy awareness, it was concluded that a self-report instrument, utilizing imagery might prove to be a viable alternative for students who have difficulty with the demands of more complex assessment tools. The assessment is proposed here since a thorough literature review from the 1970's to the present revealed few instruments to measure the metacognitive strategy knowledge of very young children or struggling readers. Early intervention and the need for gaining information about young students' developing strategy awareness necessitate the use of a strategy assessment tool if teachers are to plan the best comprehension instruction. The RMSPP offers a possible way for teachers of children, as young as kindergarten, to assess the developing knowledge of these strategies, so essential for critical reading.

Description of the Reading Metacognitive Strategy Picture Protocol (RMSPP)
This instrument was devised out of an obvious need to measure the growth in young children's metacognitive strategy awareness following a research project with early childhood preservice teachers using manipulatives to teach before, during, and after reading strategies (Cobb, 2001). The preservice teachers used the Metacomprehension Strategy Index (MSI) (Schmitt, 1990) with first and second graders, but they found it to be cumbersome and too lengthy since all items had to be read aloud to the students. In a conversation with Dr. Schmitt, author of the MSI, (M. Schmitt, personal communication, November 29, 2001), she suggested that a possible direction for a new assessment design would be a picture protocol format -more engaging and appealing for younger, struggling readers and more applicable for that age. This protocol (RMSPP) has been evolving over a period of several years and includes pictures of good readers and simple questions asked of the child while the child is viewing a picture. The child's responses are written down by the teacher and later tallied using the rubric (Appendix B), with total numbers of research-based strategies mentioned by the child calculated in three categories: before, during, and after reading. The rubric was designed after a review of literature from researchers and literacy practitioners documenting the most effective comprehension strategies used by proficient readers (Bereiter & Bird, 1985;Blachowicz & Ogle, 2008;Duke & Pearson, 2002;Farstrup & Samuels, 2002;Gambrell & Bales, 1986;Graves, Juel, & Graves, 1998;Harvey & Goudvis, 2006;Pearson & Dole, 1987;Schmitt, 1990;Stahl, 2013). Prosser's (1998; work affirmed the validity and importance of data yielded from image-based research methodology, on which the picture protocol is grounded. He states that photo-elicitation engages respondents in conversations while they view photographs so that researchers may gain important insights into a young child's world and her emerging perceptions of constructs which are often difficult to describe.

Procedures for Administering the Reading Metacognitive Strategy Picture Protocol
(RMSPP) Four choices of pictures were available: Hispanic boy, Caucasian girl, African American boy, and Asian American girl, but more photographs might be used at the teacher's discretion based on the diversity of her classroom. (The pictures used by the researcher were acquired from friends and students of the author with written permissions given for the pictures to be used for educational purposes, but not for publication or distribution.) Teachers are encouraged to solicit volunteers from their friends, families, and students who are willing to provide a picture of a child reading a book and who agree to sign a permission for classroom use (for an alternative, choices are widely available on Google Images by searching: "good readers children" and then specifically by ethnicity, gender). Ideally, the picture chosen should match the ethnicity of the child or children being assessed as well as the gender.
The child is asked specific questions about the picture they are viewing and what the pictured good reader would do before, during, and after reading a book. Their responses are informally recorded on the protocol form (Appendix A). The child is then asked if he/she is a good reader and to describe which of those strategies he/she uses in reading. An optional final question asks the child to draw a "good reader" and tell about his/her picture. The children's verbal responses and their visual representations of good reading are analyzed and compared using the rubric (Appendix B) which lists researchbased strategies for before, during, and after reading. Tallies are calculated for all relevant, "on-task" strategies mentioned in each category -before, during, and after reading metacognitive strategies, as well as a total tally for "off-task" or irrelevant comprehension strategies mentioned, one point per each strategy named. Teachers may add any strategies to the rubric they feel should be included based on their own instructional activities or curriculum and are also encouraged to develop their own set of "off task" responses based on their particular classroom context. Some differences of opinion may exist depending on the teacher's unique perceptions of strategies that would lead to a child's being an efficient, strategic comprehender and possibly to their own individual perceptions of the reading process. Not all teachers will agree on either set of "relevant" or "irrelevant" strategies. The advantage of the open-ended assessment framework proposed here is that individual classroom teachers can modify it to fit their instructional contexts and curriculum.
If teachers are concerned about the time involved in administering individual assessments, several alternatives are possible. The RMSPP can be used with small guided reading groups and can yield valuable insights without the teacher actually interviewing each child individually; rather, general anecdotal notes about the group's awareness following the informal group conversation can be taken. Also, parent volunteers or instructional aides can be quickly trained to administer the assessment, allowing the teacher to review the data at a later time. Another option is to have audiorecordings of the students' responses done by an aide or parent volunteer for the teacher to listen to the answers at a convenient time.

Field Testing: Reading Metacognitive Strategy Picture Protocol
There is support for the credibility and usefulness of the RMSPP because it has been used extensively in reading clinics and in classroom settings with positive feedback from teachers; however, more data is needed to establish reliability and validity. Data from one informal project using the RMSPP is briefly reported here to illustrate one example of how it might be used in a school district. This project was designed in response to the districts' request for assistance and to field test the effectiveness of the RMSPP, providing the schools involved with guidance for instructional decisions about the feasibility of teaching metacognitive strategies in early childhood classrooms. The project's findings would lay the foundation for future instructional discussions and inform professional development sessions within the schools.
A total of 139 children in grades kindergarten through three in 14 schools in a rural area of the southwestern United States were interviewed using this picture protocol format (RMSPP). The researcher and her graduate student research assistants conducted all the interviews. Participants included: 71 male, 68 female; 84 Caucasian, 48 Hispanic, 6 African American, 1 Asian American; 17 kindergarteners; 16 first graders; 56 second graders; 50 third graders. The children in kindergarten through grade three represented low, middle and high reading ability students. The teachers in the schools volunteered for their classrooms to be part of the study, and the students were randomly selected from each classroom. The teachers who volunteered their students were seeking information about their young children's developing awareness of comprehension monitoring and metacognitive strategy awareness in settings where the curriculum focused on phonics, word analysis, and decoding. For this project, the teachers indicated that the data would be used to advocate for a stronger focus on comprehension within the early childhood reading curriculum. The school sites were involved in partnerships with the researcher's University College of Education, and all had hosted interns and practicum students. The majority of the teachers knew the researcher personally; some were former graduate students in a Master's program in which the researcher was an instructor.

Data Analysis
A constant comparative method (Bogdan & Biklen, 1982) was used to evaluate and analyze the children's oral responses and verbal descriptions of good reader behaviors and strategies. Data sources included the recorded verbal responses of the children who were interviewed and the specific strategies mentioned in their responses to Questions 1 and 2 of the RMSPP. Although Question 3, which requires drawings of good readers, provides valuable information about children's perceptions of good readers and their metacognitive strategy awareness, the analysis presented in this article does not include this data. For a detailed analysis of data from the drawings -the optional final third question on the protocol (Appendix A), the reader is referred to Cobb, 2012. The coding process was carried out as the researcher and two graduate research assistants read through each protocol and assigned either a relevant /research-based or non-relevant/non-research-based code to each child's response in each of the three areas -before, during, after reading. Using just the responses coded as relevant, the child's strategies were then compared to the rubric and a tally mark placed beside any matches to the rubric. Each child's rubrics were tallied for totals in each of the three areas of reading.
Tallies for each grade level were calculated and tables constructed to display a summary of data. Tallies were also calculated for the non-research based strategies mentioned and summary tables constructed by grade level for developmental comparisons.

Discussion of Findings -Analysis of Children's Developing Metacognitive Strategy
Awareness After examination and close analysis of the children's responses to the questions about metacomprehension strategies of good readers, it was apparent that some, but not all, children in each grade level had developing awareness of research-based metacognitive strategies. It was also apparent that there was a developmental trend toward heightened awareness as students progressed through the primary grades, with second and third grade students displaying more knowledge of before, during and after strategies than the younger kindergarteners and first graders. Table 1 displays the before, during, and after research-based strategies mentioned by the students, disaggregated by grade levels and summarized. In reviewing the students' responses across the three categories, it appears that before reading strategies were more familiar to the children. For example, some of the young children mentioned a "picture walk" when asked about good readers' behaviors before reading. When asked to describe a picture walk, some of the children were able to be more explicit about looking at the title, the cover, the author's name, and looking at the pictures throughout the book. Less familiar to the majority of the children were the after reading and during reading strategies, with during reading research-based strategies being the most problematic. However, there was a substantial increase in the numbers of known during and after reading strategies named by the second and third grade level students. The mean total number of strategies named per child varied from 1.2 to 2.8 in grades K-3.  Table 2 shows the numbers of "off-task" or irrelevant strategies that could not be classified as research-based and those which did not specifically apply to comprehension, using the scoring rubric. Analysis of this data is also revealing in that the mean/average number of non-research-based strategies named per child was about equal to the number of research-based strategies named, leading us to infer from this data that the children in grades K-2 in this study were not certain of what would be a helpful strategy and what would not be useful to help them become good readers. However by grade three, there was a noticeable decrease in the number of "off-task" strategies mentioned, and the third graders' knowledge of research-based strategies exceeded their mention of irrelevant strategies. It is important to ponder if this finding is indicative of a developmental trend pointing to third grade as a possible milestone in the acquisition of metacognitive awareness. Although the second graders' overall mean number of named research-based strategies was slightly higher than the third graders' responses, the third graders named fewer "off task" strategies. This may indicate a more discriminating stance and a heightened awareness at the third grade level about helpful strategies which contribute to efficient comprehension.
Another interesting point from the data was that the highest category of "off-task" strategies reported by the kindergarteners through third graders pertained to phonics and decoding strategies. Obviously, opinions may differ here among teachers, but this researcher believed that these responses should not be classified as relevant comprehension strategies for the purpose of assessing metacognitive strategy awareness if we view reading as a meaning-making process (ex. "I sound out the words."). Certainly, accurate decoding is necessary for comprehension, but the purpose of this assessment is to ascertain strategies for understanding text. Project Outcomes -Teacher Feedback This informal project confirmed the usefulness of the RMSPP as a classroombased diagnostic tool which enabled the teachers who participated in the project to assess not only individual children's knowledge but also to assess instructional needs for their classes and guided reading groups. One teacher provided the following as one useful example of how the RMSPP had guided her instruction in her classroom. To illustrate the utility of the RMSPP, Table 3 summarizes her feedback and compares responses from a high achieving reader in her top guided reading group to a struggling reader in her lowest guided reading group. Both children have been in the same second grade class from the beginning of the year with this same teacher for reading instruction. It is apparent from the differences in the responses of the two children that the RMSPP does discriminate between the strategy knowledge of the two children. Child A knows appropriate strategies for before, during, and after reading (Score = 6) and considers herself to be a good reader. Child B does understand that there are different strategies or behaviors that should be happening at different times in the reading act but does not know any appropriate research-based strategies to use (Score= 0). Child B does not consider herself to be a good reader and is unable to articulate why she does not rate herself as a good reader.
From the assessment, this classroom teacher was able to view the two children from their differing responses and to plan to focus more intensely on strategies with Child B's guided reading group. Child B's group needs detailed modeling and instruction in before, during, and after reading strategies. Child B's group also needs support to enhance their sense of selves as readers, perhaps with more opportunities for successful reading experiences with independent level texts. Child A's group most likely needs continued support to develop broader understandings of a variety of appropriate strategies. The RMSPP assisted the teacher in taking a closer look at the ways in which the children view themselves as readers and in their understandings of reading as a meaning-making process. "Gets a snack." "Then go out to play." All responses were "off target" and did not relate to good comprehension.

Implications from Project/Conclusions
Although we have made tremendous strides in our understandings of the comprehension processes since the late 1970's when Durkin (1978Durkin ( -1979 visited elementary classrooms and discovered comprehension strategy instruction to be virtually non-existent, more work remains to be done. Certainly, we cannot over generalize from this research to a broader geographical area, but it does point to a need for intentional assessment of metacognitive strategy knowledge in our early childhood classrooms. The students in this study, with only a few exceptions, were strong in their knowledge of before reading strategies, but in need of more instruction of how to be more proficient comprehenders during reading and after reading. Also, many of the early childhood students across the grade levels, kindergarten -grade three, exhibited some confusion as to what constitutes an effective strategy and what is not so essential to enhance comprehension. One implication from this research points to essential questions teachers should ask about their reading programs: what message about good reading is being communicated to my students? Do my students perceive that correct and quick word pronunciation is what makes a reader a good reader? Or do my students see that comprehension is our desired outcome?
Research has pointed to best practices in enhancing comprehension for our students and has lifted up comprehension as our ultimate end goal; yet, mandated, scripted instruction sometimes leaves little time for focusing on comprehension, as was expressed by the teachers in this project. The simple assessment tool proposed in this article may provide teachers with a quick and easy method for evaluating what metacognitive strategy knowledge their students have gained and which areas are in need of additional instructional attention. It also provides teachers with a tool for assessing students' perceptions of "good reading" and how their reading curriculum is shaping their students' views. Armed with this information, teachers may be better equipped to make diagnostic decisions as they design comprehensive reading programs which include metacognitive strategy awareness and application.
Directions for Future Research with RMSPP As previously stated, the Reading Metacognitive Strategy Picture Protocol has been used informally with beneficial outcomes for teachers and clinicians and has provided useful instructional guidance. However, additional well-designed research projects to document its reliability and validity are needed. One avenue for investigation will be to compare the students' scores on the RMSPP with scores on a standardized test of reading comprehension. This will establish predictive validity. This will also give convincing evidence of a student's ability to apply their metacognitive strategy knowledge in connected text. Also needed to establish reliability would be readministration of the RMSPP to the same students, in one -two weeks and again after one month for comparison of responses. The Reading Metacognitive Strategy Picture Protocol holds promise for filling a need for a naturalistic, authentic tool for use in classrooms with children, as young as kindergarten, to assess students' knowledge and awareness of metacognitive strategies, but more documentation is needed to support its usefulness.