Reliability

RELIABILITY & VALIDITY OF THE WILATA

Dr. Randall J. Ryder

 

The Wisconsin Learning and Thinking Assessment (WILATA) incorporate a qualitative measure for

both cognitive and metacognitive performance. It is a tool that helps teachers embed within their

classroom assignments higher intellectual demands that lead students to produce authentic intellectual

work.  WILATA’s innovative taxonomy lends itself to a more organized way of thinking about how

and what students think. The WILATA was designed to address the gap in assessment tools and

provide educators with a means for determining their own expectations of students, as well as

measuring the quality of students’ application of higher order thinking strategies in various content

areas. Moreover, knowledge and use of WILATA for assessing students’ cognitive and metacognitive

strategy use offers educators an opportunity to weave into their instructional planning the types of

activities and assignments that will engage students in using higher order thinking skills. Thus, WILATA

is an authentic assessment that leads to (1) the creation of intellectually demanding assignments and (2)

students’ application of higher order cognitive and metacognitive strategies in classroom activities.

 

WILATA was designed with flexibility and generalizability in mind. This rubric allows for the assessment of

cognitive strategies and metacognition found in any type of assignment, project, or activity.

Developmentally, studentsbegin to explore and use critical thinking and problem solving skills

more frequently in the middle school grades. Abstract reasoning and metacognitive

awareness is an important and necessary component for WILATA to be used successfully. Since

authentic intellectual work undoubtedly varies greatly in the type, number, and quality of task

components, WILATA’s recommended use is with middle grade, high school, and college levels. Ultimately,

it is a tool to assist educators in better understanding the learning expectations

they place on students, as well as students’ understanding of those expectations, and how these

develop along the child’s academic career.

 

The WILATA is the outcome of a Federal grant assessing the impact of technology on 3200 middle schools

students’ learning using three models of instruction. Initially, the WILATA was a measurement tool used

by the researchers to provide an authentic measurement of students’ classroom learning. Subsequently,

validation and reliability studies were conducted over the period of one academic year with 12 fourth,

sixth, and eight grade teachers in a suburban school district in the Midwest. The 36 teachers were

provided one week of training in the use of the WILATA prior to the beginning of the school year and were

provided weekly assistance by three coaches trained by the experimenters. On a weekly basis, teachers

submitted one classroom assignment to the researchers. This assignment was scored, using the WILATA

scoring rubric, and then scored by two doctoral students trained by the researcher. The scores of each

student were then analyzed to determine the reliability and validity of the WILATA.

 

Inter-rater reliability was computed by comparing each student’s scores submitted by the teacher to those

of the two research staff raters. Once reliability coefficients were computed for individual student ratings

they were summed and averaged for each teacher and an overall reliability was obtained by computing

an overall reliability coefficient for the 36 teachers. The inter-class correlation coefficient was .91. Validity

estimates were computed by computing a mean level of performance for each student on the WILATA then

correlating that score with the students overall score on the Wisconsin Knowledge and Concept Examination

(WKCE). Overall correlations for grade 4 were .87, for grade 6 the correlation was .91, and for grade 8 the

correlation was .92.