RELIABILITY & VALIDITY OF THE WILATA
Dr. Randall J. Ryder
The Wisconsin Learning and Thinking Assessment (WILATA) incorporate a qualitative measure for
both cognitive and metacognitive performance. It is a tool that helps teachers embed within their
classroom assignments higher intellectual demands that lead students to produce authentic intellectual
work. WILATA’s innovative taxonomy lends itself to a more organized way of thinking about how
and what students think. The WILATA was designed to address the gap in assessment tools and
provide educators with a means for determining their own expectations of students, as well as
measuring the quality of students’ application of higher order thinking strategies in various content
areas. Moreover, knowledge and use of WILATA for assessing students’ cognitive and metacognitive
strategy use offers educators an opportunity to weave into their instructional planning the types of
activities and assignments that will engage students in using higher order thinking skills. Thus, WILATA
is an authentic assessment that leads to (1) the creation of intellectually demanding assignments and (2)
students’ application of higher order cognitive and metacognitive strategies in classroom activities.
WILATA was designed with flexibility and generalizability in mind. This rubric allows for the assessment of
cognitive strategies and metacognition found in any type of assignment, project, or activity.
Developmentally, studentsbegin to explore and use critical thinking and problem solving skills
more frequently in the middle school grades. Abstract reasoning and metacognitive
awareness is an important and necessary component for WILATA to be used successfully. Since
authentic intellectual work undoubtedly varies greatly in the type, number, and quality of task
components, WILATA’s recommended use is with middle grade, high school, and college levels. Ultimately,
it is a tool to assist educators in better understanding the learning expectations
they place on students, as well as students’ understanding of those expectations, and how these
develop along the child’s academic career.
The WILATA is the outcome of a Federal grant assessing the impact of technology on 3200 middle schools
students’ learning using three models of instruction. Initially, the WILATA was a measurement tool used
by the researchers to provide an authentic measurement of students’ classroom learning. Subsequently,
validation and reliability studies were conducted over the period of one academic year with 12 fourth,
sixth, and eight grade teachers in a suburban school district in the Midwest. The 36 teachers were
provided one week of training in the use of the WILATA prior to the beginning of the school year and were
provided weekly assistance by three coaches trained by the experimenters. On a weekly basis, teachers
submitted one classroom assignment to the researchers. This assignment was scored, using the WILATA
scoring rubric, and then scored by two doctoral students trained by the researcher. The scores of each
student were then analyzed to determine the reliability and validity of the WILATA.
Inter-rater reliability was computed by comparing each student’s scores submitted by the teacher to those
of the two research staff raters. Once reliability coefficients were computed for individual student ratings
they were summed and averaged for each teacher and an overall reliability was obtained by computing
an overall reliability coefficient for the 36 teachers. The inter-class correlation coefficient was .91. Validity
estimates were computed by computing a mean level of performance for each student on the WILATA then
correlating that score with the students overall score on the Wisconsin Knowledge and Concept Examination
(WKCE). Overall correlations for grade 4 were .87, for grade 6 the correlation was .91, and for grade 8 the
correlation was .92.