Making Data Work for You

What is your data trying to tell you? In this episode, John Dues talks to Andrew Stotz about why most leaders misread data, overreact to single results, and miss the real story. Discover how Deming thinking exposes when change is truly happening and how to use a process behavior chart to listen to the real story. Plus, find out why nine years of ‘stable’ results may still demand transformation. Tune in and rethink data-driven leadership! 0:00:02.2 Andrew Stotz: My name is Andrew Stotz, and I'll be your host as we dive deeper into the teachings of Dr. W. Edwards Deming. Today, I'm continuing my discussion with John Dues, who is part of the new generation of educators striving to apply Dr. Deming's principles to unleash student joy in learning. The topic for today is understanding variation is the key to data analysis. John, take it away. 0:00:27.8 John Dues: Andrew, it's good to be back. Yeah. So, we've just started the school year in Ohio, so I thought doing a session on goal setting would be a good place to kick off the year. And I was thinking a lot of leaders, school leaders and leaders in general, are setting goals around this time period. And I was really thinking about having this Deming lens. I was thinking, how did I set goals before I sort of started understanding this approach? And it's, you know, this is one of those things where if you really stop and think about it, goal setting is a lot harder than it seems at first glance. Things like, how do you set a reasonable goal? And then once you've gotten to that place, how do you know if things are improving? How do you know if things are getting worse? And I was thinking how powerful this understanding variation method is for folks that may be struggling with those questions. 0:01:32.9 Andrew Stotz: Yeah. In fact, that's a great question for the listener and the viewer. Like, how do you set goals? How did you set goals in the past? How have you improved that? And I was thinking when you were speaking, I didn't set goals. I gave proclamations. You know, 20% of I want to see this and that. And they were just stretch targets without any means or methods. So yeah, interesting. 0:01:55.2 John Dues: Yeah. How do you set the target? Was it arbitrary? Is it based on some standard that you heard somewhere? A lot of times you have no idea sort of what's behind that target or you've sort of associated it to something that's familiar. Like in my case, we often sort of set goals that sort of mimic the grade scale. So, you know, 80% is a common goal for something like test scores, you know. 0:02:23.7 Andrew Stotz: But they don't even call them goals anymore. They call them, let me remember, I think it's called KPIs. 0:02:30.0 John Dues: KPIs, targets, you know, lots of different things for sure. And I think what I've seen is that a lot of the reason that goal setting is so hard is because you, well, one, you misinterpret your data in the first place. And a lot of that misinterpretation, at least in the education sector, is because leaders don't have the knowledge. They don't know about natural variation. They're typically making comparisons between some current performance level, some previous value. But those two things, those two data points don't show you, don't convey the behavior of that data across time. So, what we do and what I did before I sort of discovered this method is you overreact to a single data point. Probably less frequently, you underreact to the data because you don't have this understanding of, you know, how much is the data moving up and down sort of naturally almost no matter what you're doing. Now, that's not always the case, but that's the case that I've found in a lot of situations. And so until you start to take that into account, those natural ups and downs, then you just misinterpret the data over and over again, usually by overreacting is what I've seen. 0:03:54.9 Andrew Stotz: Yeah. 0:03:56.0 John Dues: So there's, you know, I think as a starting point, people in the Deming community will be familiar with, a lot of people. But others listening to this probably have never heard of this idea of dividing variation into, I've heard it described as like two flavors. There's the routine variation, what I call natural variation, things vary naturally no matter what you're doing. And then there's exceptional variation where things are so different that there is reason to pay attention to this. And what I found through studying this is, the key is knowing how to tell the difference between those two types of variation. And don't do that, lots of confusion, lots of wasted effort. And so that's really where the power of this methodology comes into play. And for anybody that's studied this, you sort of realize that you have to have a tool to make that differentiation. It's not arbitrary. And so that's where what I call the process behavior chart, some people call the control chart, where that comes into play...