Why should barley inform our analysis of test data?

 At the end of the month, DESE will release results of last year’s MAP test to the public. Folks who see themselves as logical, bottom line thinkers will try to make judgements based on the data. Which school, teacher, method is most effective? Some of this will even be done by educators, under the banner of “data driven decision making.” Very little of these judgements should be taken seriously. The simplistic thinking is demonstrated in the equation most folks will use.

Proficiency rate of school/teacher/method A > = < proficiency rate of school/teacher/method/ B

Anyone with a marginal background in statistical analysis will see flaws here. There’s no accounting for sample sizes or variables which might impact learning.

Consider William Sealy Gosset who invented the T Test in 1908. Gosset was serious about statistical analysis. After all, he was involved in important work: brewing beer for Guinness. He needed to make decisions about the quality of Barley A compared to Barley B. Because he took statistical analysis seriously, he wrote an equation that factored in sample sizes and accounted for accidental variations, while assuming variables were held mostly constant. You may not be familiar with the T Test, but it is widely used in finance, science, and academic research. The formula is a bit more sophisticated than
A > = < B.

​      m- μ

 ​t = s/√n
Gosset's T test formula

Gosset realized some variance would naturally occur so he took the exercise a step further. He set thresholds to determine if the findings were significant, not incidental. Nowadays we declare victory or defeat over any difference in test scores.

The purpose of state testing is not to compare teachers, districts, or methods. The purpose of state testing is to compare a student’s performance on a test to accepted criteria which demonstrates proficiency or mastery. One problem Missouri educators face is the legislature changed the standards for evaluation multiple times, and DESE changed test formats and the criteria for mastery. The data returned to parents and educators has no value for comparative analysis. To be honest, the purpose of state testing is primarily to comply with federal legislation that drives funding.

 Pretend we sought to compare schools, teachers, or methods through rigorous statistical methodology. Pretend we took the task as seriously as William Gosset took beer. Let’s say we sat down with Gosset and discussed our plan over a cup of coffee, or something.

Us: We want to use statistics to determine which schools, teachers, and methods work best.

Gosset: Tell me more.

Us: Well, we’re going to give a test and see which group has the most kids who pass.

Gosset: How do you plan to account for sample size variances?

Us: Not going to. We’ll treat pass rates of a 4th grade class of 10 the same as we do as the pass rates for all 4,000 4th graders in a large district.

Gosset: I see. How will you hold variables constant and account for variances that might naturally occur by accident?

Us: We’re not doing anything about that either, though we admit it’s a limitation.

Gosset: I see. Are you at least using the same instrument for measurements?

Us: Not really. We’ve changed the test instrument every year, but it’s sort of the same.

Gosset: Yeah, I don’t think this will produce valuable data.

Us: Really, I think we’ll use it for scathing editorials in the paper, bullet points in faculty meetings, and for a Great School Rating which helps determine the market value of real estate on Zillow.

Gosset: All of those are terrible applications of what you are suggesting. I really must be going.

I'm not out to write a take down of standardized testing. Educators know results are important to parents and policy makers. We all want our students to excel on the tests. Test results can help educators, working collaboratively in settings that respect the voices of teachers, triangulate data to identify trends, strengths, and weakness. I wouldn't advocate to throw out testing completely, but we do need to throw out the hyperbole in which arm chair statisticians occasionally indulge. There are limits to the usefulness of test data and the example of Gosset, the brew master, helps us see them.

A few thoughts on collegiality and a story about wrecking the homecoming float

My career began in 1999 at Simonsen 9th grade center in Jefferson City, and I didn’t have a clue what I was doing. I was overwhelmed, partially because of a homecoming float. The principal offered to “sweeten the pot” on my salary if I agreed to be stuco sponsor. That allowed me to bring home an extra $36 a month, but it came with expectations- the freshman class always won the homecoming float contest.

I became obsessed- not with classroom management or assessment, but with float construction. I spent hours placing pomp in chicken wire with a student named Terrance and his cohorts. On parade day our entry was solid, with a skirt that read “Jays Beat the Eagles.” Controversy surrounded who would ride as the mascot, but I stuck to my guns (I got paid big dollars to set the starting lineup for the frosh float) and Terrance suited up as Jasper the Jay. We took off down the parade route, pulling the float behind my Mazda pickup- bound for glory.

At the first turn, disaster struck. I cut the corner too sharp. The trailer rocked. The skirt caught the curb. Soon the pomp read “ ays  eat the gles!” Terrance almost went to the emergency room.

I felt like a failure. 

In the classroom, it wasn’t much better. I didn’t know very much, but I hoped if I could keep the 14 year olds busy, they wouldn’t eat me. During the first quarter the kids did 17 worksheets a day and we read half the literature anthology. I spent every minute I wasn’t building floats grading each of the 17 worksheets my 120 students did everyday.

Fortunately, talented educators intervened. The chair of the English department, Laramie Thompson, and my mentor, Kim Sellers, are two of the most thoughtful teachers I’ve known. They took time to speak wisdom about what was worth stressing over and what wasn’t. Laramie showed me what leadership looked like. She ensured each of the 700 freshman were challenged and supported in their growth as readers and writers. There were six English teachers in our department and our time together was focused on student learning. We didn’t always agree, but we did work together to craft common assessments and evaluate data. Years later I went to my first PLC meeting- I was convinced that someone had stolen all of Laramie’s ideas.

I left Simonsen for coaching opportunities. I taught in great schools with talented staffs, but I did not find the same professional challenge and support I experienced at Simonsen. Camaraderie existed, but not the same level of professional collegiality. We met for administrator led meetings and at lunch to share grievances, but we did not have deep, teacher led conversations about teaching and learning.

Fortunately, I found that same level of collegiality again before leaving the classroom at Harrisburg Middle School. Teachers had real conversations about supporting and challenging our kids, which included a significant number of students in foster care. As a staff we worked to ensure our school was a great place for all students to learn, regardless of their life experiences. Our conversations were collegial, centered on work, not grievances. We didn’t always agree, but we worked together for the kids. I remember my friend Brian’s response to the suggestion that the staff be on the same page: “That’s fine, but maybe we should be on the page the kids need.” The point was taken. In the end, the work, guided by collegiality, produced a strong sense of camaraderie. We were serving our students well and we knew it.

I’m not as involved in building level work as I used to be. Busses, budgets, and building maintenance dominate my workdays. I miss the collegiality and camaraderie of teachers when they are focused on teaching and learning. As much as I can, I try to support practices that allow those conversations to take place.  Good collaborative work is occurring in the Norborne district right now. I’m thankful for the experiences I had as a teacher that allow me to recognize it.