Classroom Data That Helps (Not Hurts): Using Behavior and Learning Data as Support

So this week, I have got Julian Golder with me. He's a former educator and administrator turned ed tech founder.And Julian, you do a ton of work with districts, with building down to the practitioner level around data. And I love how you look at data because you don't just see data as a point or as a byproduct of things; it is so much bigger, and there's so much life in the actual dataset that gives so much opportunity. So I'm curious for you, as you're thinking about data, what are some of those first thoughts for you that are showing up to make sure that it is actually useful and that we're not accidentally kind of veering over into some of these ugly underbellies of how data can be used?

Yeah. So, I think the first thing that comes up, and you used the word useful, and I think that oftentimes we think about how we talk about how data can be useful. And I think it shines light on what I see as a misconception around the word data, that there are really 2 parts to the idea of data. There's what you're sitting there collecting- the numbers, the categorical data, IEP, whatever it may be- where the conversation often lives. And then there's a whole other side to it, which is how that data is being used. And so, I think that carrying those 2 lenses is so important. I mean, whatever we talk about today, I think oftentimes that lens of utility and utilization falls behind when we talk about data. So, that's the first thing that comes to mind is just really being able to hold both of them as part of data and not how does data just become useful.

Well, no, I mean, I think all of our conversations are a little bit of this and that also thinks of that. And there are so many different directions that this can go. There's more to data than just outcomes measurement, and it's gotta be useful. It serves a purpose. It's either proving a point or disproving a point. Like, there's that. But with a lot of what you've built, you've breathed life into meaningful data that then provides the opportunity for decision-making; it's not just this almost deterministic outcomes or lack thereof in terms of the usefulness of data, but it's really so much bigger than that. So I'm curious if you can put some more words around that and share that thinking, because I think that's maybe not always where people start when they're thinking about data collection.

Yeah, no, again, that makes sense. So, I think a good place to start around how I see the transformation... So, I mean, just as a little, I mean, background for listeners listening, I think where, where Emote started was the experience of a lot of educators that saw that there was a lot of data being collected, and oftentimes that data wasn't supporting the practice on a day-to-day basis. Maybe for another day, there's the origin story, but really that's what it comes down to is that, on a day-to-day basis, I'm dealing with a variety of both behavioral, academic, and emotional opportunities. And I don't even say needs; opportunities.

And the data is not informing that. And I think there's a journey that started with how do you build the anti-data tool data tool? What does it look like to build a data tool that focuses on action? And, and to be totally honest,we didn't want to build a data platform. We wanted to build a platform that facilitated all of the educators and adults in the student's life to be able to act far more proactively and support the student more cohesively across that system. And so it turns out we had to build a data tool to do that.

I think that a place that my thinking starts is around the separation that often comes up between the idea of data, so kind of this record-keeping, and the people that are delivering the support, whether it's academic, non-academic, tier 1, tier 3, therapeutic, whatever it may be, and that those two lanes are often treated very separately. We have discussions around what should our systems of support look like?Awesome discussion, one of my favorites. Discussions around what data should we collect? Really less fun of a discussion, but also really important to have. And those often take different tracks. And even worse, when they cross, they often, I feel like, will end up in the wrong direction.

So the question that will be asked is, how can our systems of support support the data that we're tracking, or be in service of the data we're tracking? And it should be the exact opposite- the question, which is, how can the data we track be in service of our systems of support? And I really do think everything starts with that question.I'm sure we'll touch on the idea of fidelity, on what makes good data, etc, etc. And I think it all is rooted in what the systems of support look like today. What do they need? Where are they at? And what type of data can serve them today? And just as importantly, what does that journey look like? How does data today look different than data in a month versus a year or two from now? So, I'll pause here because I see nodding, and I'm curious to what your thoughts are on that.

I cannot echo that more. The number of times that I've walked into teams or districts who have data out the wazoo. Like, they have so many data points, so much information everywhere, but to what end? What are they doing with it, and is that actually informing decisions that are being made, or is it just affirming decisions that were going to be made? Or, just those kind of like, ‘what are we actually doing here?’ So I could not echo that more; let's reverse this entire conversation. You're already thinking about services. You're already thinking about intervention. You're thinking about what you can and can't do, how you can or can't show up. You're already figuring all of that out, so can we just start there?

And then just make sure that we've got the metrics, we've got the data, we've got whatever those other pieces are embedded into it so that we can show is it working, is it not, right? If you want to go the fidelity route or the treatment integrity, or all of those types of things. We can go that direction for sure. But it's like, if all we have is just a bunch of random numbers and then we have this question, and so then we just reach into our bag of numbers and find the numbers that align with whatever decision we want to make, well, that's a completely different system than when we've just got metrics tucked in that we can very honestly and open-handedly look at to determine if the decision that we are making is founded, is not, if the services we're providing are working or not, because it's just part of it.It's all the same, where it's not, you know, "Oh, we're taking this data, and we're making these decisions." It's both/and. Like, it's completely connected.

But I think that is so protective because then you kind of miss some of those landmines where really data can be misused. And I have watched a variety of decisions made that were harmful, but it really is from the, "I know the decision I need to make. I just need to reach into the bag of numbers, get the numbers that back up what I'm going to say, and move on with life," as opposed to, "Maybe I look at the numbers because it's embedded in it and it's not what I was thinking, and then that gives me other decisions I can make." That's really protective, not just for the student in that situation or the family, the network, but also the practitioners, that they're not carrying kind of the liability of where the data could really be speaking for them. So, no, I could not agree with you more in terms of how you're, yeah, chicken or the egg on this.

I like what you were saying about... What did you just say? You said the practitioners and how the data can be speaking for them. Is that what you were just-

Yeah. Like, in my opinion, when you tuck data into practices, it's just business as usual. I'm providing these services, and I'm tracking this information along the way. That is really protective. When it's set up systematically, set up really well, that is a safeguard on behalf of that practitioner, on behalf of that system, on behalf of that district, when it's actually directly connected and embedded in a meaningful way. Like, then it's not just because I said so or my professional judgment. It's so integrated that then I just get to be the mouthpiece on behalf of the services and the data and what that is saying, as opposed to that whole interpretive capacity of ‘let me make this make sense.’ So it's just a different way of showing up in the profession.

Yeah. I mean, I also, I'd go even a step further to say that, I mean, not only is it protective of that individual, the data, and can support their choices and their perspectives, but it also actually creates a pretty profound cultural shift we've seen in schools, is that when you have data be framed as a resource for the individual, it actually creates an environment where we see... It was last year, but there was a school that was having trouble. It felt like there was one data owner for the whole school, essentially. This was actually a BCBA that was an interventionist at a school. And we hear this all the time. And so we were trying to think of, like, she asked, "What are some strategies to have data be more distributed across the team?" And because I'm sitting here, I'm the one that's pulling up all the stats on which students should we be talking about, what should our tier 3 look like, or tier 2. And I want to help with that. And I think one of the things that we explored is that, well, maybe the starting point for data collection for your school and your team isn't replicating what she was doing but instead finding what are the ways that data can create more agency for the individuals in that team.

And so we ran just kind of an impromptu training where I went around and asked everyone, what is the question that they worry about most every day? And we just found what the data today is that they're collecting that supports not just that question, but the successes. So, if one student or one teacher is like, "I'm really worried about the students that are very disengaged and how much they're falling behind," being able to not just flag who those students are, but being able to flag what the efforts are that the teacher's been successful in around the students.

And that's where there was a really big turning point, is when a teacher has realized that there's an opportunity for the data to not just be centered around their needs, but also to support their successes. And so that's when we started to see many more people in that school becoming involved in- and the word data is so singular, but having, having conversations that are informed and feeling excited about that. So, anyways, this is what just kind of came to mind as you were talking about that: how data can actually change the culture from being defensive to oftentimes pretty curious in a school.

Well, it does, though. And I will say, as a teacher, people will come in and take data on me, on my students, on my practices, on their performance. All of those things, we're constantly used to being on the receiving end of data being used for scrutiny. That's when you can flip that script and say, "Actually, now..." That may be part of it, because I don't know that we'll ever be able to fully get away from that. But also, right now it's documenting here are all of the practices. Here are all of the things that you've been doing. Here is how that's showing up. Here are those opportunities or those things that when you're just in the mix of it, you're missing it. But because we're actually tracking it, we can now start seeing things that we aren't putting mind to during the day-to-day.That does completely shift the dynamics.

And I will also share that we do a lot in program evaluation and development. And so when we go in for our cold baseline, it feels like there's scrutiny. It feels like people are being judged; they're a little guarded, or you get the dog and pony show. Because there's just sort of this norm in education that outsiders come in and they take data, and they tell you how bad you are and all the things you're not doing and all of the ways that you should have been better, and then they leave.

But one of my favorite things to do is then immediately after is to walk that through and say, "So what were we looking for and what does that mean and how can we..." And you start seeing this shift over the course of a development effort where it went from outsider, cold, being judged, being evaluated to, "Oh, check out all this stuff we're doing. Oh, look at how this is working." And it does, it shifts the, the entire atmosphere of the professional engagement when people can see the data's not being done to them, but instead it's actually documenting and bolstering all of the things that they're already doing and giving them a platform from which to continue to develop and grow and be able to see for themselves the outcomes, not just in the student outcomes, but also within their practices.

It's a really, really cool opportunity to walk that trajectory. And it's one of those things, like, I know you've walked this with teams, so you already see it, you already get it. But so many people, when all they're used to is the corrective use of data against them. It's really hard to even perceive that there could be a different experience where data is still involved, but it's not around harm, you know?

Absolutely. Man, it's, it's interesting. I'm thinking about when you enter data from the lens of the need to evaluate, that is really tough. It makes me wonder what you see around the ideal entry points. I know what we see on our end, but what do you see as the ideal entry point if there's a school that wants to be more data-driven, or a district, let's say, that wants to be more data-driven? How would you help them get a sense of where to start?

Yeah. I think especially because so much data drives a kind of the door opening or closing into whatever is next, I think there's just a natural predisposition in education that we view data as the necessity to either gain access or to restrict access from whatever is next. And that perspective, I think, is one of the hardest things to break down. And so for us, one of the things we're looking at is just their openness to engage in change. We actually run some change diagnostic pieces of how open you are to seeing things differently, or how many people do you have around you that see the world the way you have grown to see it within your local?

Because that is really the door opener for whatever effort, whatever outcomes they're trying to make happen, however they're engaging with what we're bringing to the table, the ultimate outcomes of what's actually gonna happen with that. So for us, it has less to do with data itself and more to do with what your perspectives are around change. What are some of the norms, and how firm is that gonna be?

Because at the end of the day, any data, in my opinion, can be used for growth and development and really unlocking and making really cool things. Ideas become opportunities, but it is a double-edged sword. It can also turn around and be used in pretty hurtful ways or malicious ways or be twisted to say things that are not accurate. So, for us, we're very careful about how we document things, what we put in reports, what we don't, and how we communicate those just because we know it is a double-edged sword. And it really truly could go either direction.

Yeah, no, it's interesting. I think when you're talking about a willingness for change, and I think that... or even just change as a factor to drive adoption of being data-driven, let's say. I think we will often see is probably not something totally dissimilar, but where can data collection align with the practice that already exists? And I think that often we think through the lens of curiosity is that where is there already curiosity? Good data should always start with a question, and it should always create a question. And so if the question is like, "Why are we having such high rates of ODRs?" Or, "Why are we having such low rates of attendance- high truancy rates?" I think oftentimes the data gets stuck around the event that you're trying to avoid.

There's this whole thing where this is like learning to ride a bike. It's like, if you look at the tree, you'll hit the tree. And so there's this question. I'm like, "That is really awesome that you're trying to reduce the number of ODRs, or you're accountable to do that, but what is the data that'll help you to understand the factors that are driving those ODRs?" We had one school, like, ODRs are a common thing. Either ODRs or attendance are a common thing that we get approached with initially. It's like we need to figure out how to get a handle on those.

And one of the things that this school is noticing, they started using Emote, and they... we set up alerts for them so that any time there are patterns in ODRs or increases, they get notified. And so one of those alerts went off that for October and November, there was like a 30% or something increase in the number of ODRs. And what is great is that this led them to bring this up in a meeting. And so they looked into the Emote data, and they're like, "Why is this happening?" They're really quickly able to see like, "Well, this only started in October." "Okay, where did this start?" “This seems to start really around recess for this school, around October." Like, "That's weird." And then they looked at the behaviors, and they saw a lot of physical confrontation or physical-based behaviors.

And so really quickly, in like 30 seconds in that meeting, they were able to identify the patterns that indicated early root cause and some of the antecedents that were leading in. So they asked the person on recess duty, like, "What has changed? What's been difficult? What's changed in the past couple of months?" And so I just want to note that's the question that the data created. The data didn't answer that.

You know, the person said is like, "Oh, we had an early snow these last 2 months, and the snowplows have been shooting all of the snow into the play yard, and it's absolute chaos out there because all the kids are just climbing these mounds of snow." And so the intervention then becomes actually pretty straightforward. I mean, the number of kids that were getting referrals that we know lead to increased rates of disengagement, we know reduces attainment academically, just by being able to have proactive data at a resolution that allowed you to keep asking why, led to a pretty straightforward intervention, which is just using the west play yard instead of the east one.

Well, but I think that curiosity, though, that curiosity is a professional choice. And I will say, we have worked with districts where they can be curious, and they can then tuck in that data collection so that then they can lean on the data to give them that ability to answer some of those questions that they're thinking. But I mean, I'll also say that that exact same scenario could have gone a very different direction where it's like, "Oh, well, it's always those students," or, "It's always that time of year," without anything to lean back on.

And then all of a sudden, we're just affirming what we think as opposed to being able to approach it with curiosity, wonder what's going on, and then go find that answer. I think it's a very different conversation, because I've been around a lot of people who are like, "Oh, yeah. That's just that teacher," or, "That's just that family," or, "That kid's from that part of town." Like, there are all of these justifications that just sort of sneak in, and in my opinion, sort of remove that curiosity. So then it becomes more deterministic as opposed to, "I wonder." It's, "Oh, well, that's probably what it is." Period, full stop. Move on with life.

Which is why a lot of the work that we do, we approach it from that change perspective. Do you see other potential realities that could also be true, and are you able or willing to engage in that? Because then, even if they're not currently being curious, that would let us know, but we could get there. But there is an opportunity for that, versus in some systems, that isn't an opportunity, and it's sad, and it's rough, and we'll do a very different approach in the way that we engage with districts there. But yeah, I think it's that double-edged sword. The exact same scenario could go two completely, like, very drastically different directions.

Yeah, and it's interesting. I think about what you're saying, and I'm very fortunate having been in education now for... I'm actually not even gonna count the years. Anyone who's tried to implement anything in education knows the mountain that you have to climb. And it is interesting because we... like, one of the things I get the joy of doing from the distance supporting implementation is seeing year over year how phased implementation looks. There are always the people that have had nothing to do with any sort of a change. That's like collecting new data, implementing Emote. There are the people that are really excited and leading it, people in the middle.

I don't know if I've seen that the curiosity doesn't exist necessarily with individuals. I think that what I find is that the fear is entrenched in those individuals. And so there's an, there's a different initiative that that person could be very pumped about and leading and excited, and it just happens that the initiative that we're leading, which might be bringing data into conversations around student support, let's say, is threatening. And so one of my favorite parts in the school year is right around November, where we start to see the people that were against Emote turning to supporting it, and that turn can sometimes take a couple of months.

But for them to be able to see how an effort that was threatening can become validating, honestly, is the shift that I think we started to see the most. And it really does speak to having very thoughtful implementation that puts practice first over compliance. And I know that's a really touchy thing to say, because compliance is very important, and I'm all there for people not getting sued. But I think oftentimes compliance can overshadow opportunities that exist, especially in the space that we work in around non-academic, social-emotional behavioral supports.

Yeah, but that's where I will completely back you up on that, because part of what TLC does is we're supporting due process cases. That is inherently one of the most litigious outcomes that can happen when you're looking at compliance issues. And almost always, the solution is so much bigger than just that situation, that it is within the system, within the practices that are around that student. It's rarely just that one exact case in this one exact situation. That it truly is so much bigger than that. And a lot of times we do find, not all the time, but many times, there is a fear aspect, especially once it's actually become litigious. Then there's just a ton of fear everywhere.

And that even if we've had staff who were open to innovating and trying new things and trying to be creative and look for opportunities and solutions, now all of a sudden there's a lot of fear with that. And we have walked that where we're joining teams typically because it's already become litigious, and now they need that third-party outside support. Part of that effort is walking that back in the exact same way that you're discussing: September, October, now we're kind of leaning into November. Like, we're coming in cold, resistant, fear-laden for lots of good reasons. It is litigious.It is difficult. They are under the magnifying glass.

But part of that is then walking that back to say, "Okay, yes to that and also." Right? So yes, we can do the compliance piece, and also. That is where you see practitioners step back into it and that life kind of, like, comes back into them as practitioners knowing that what they're doing is good. And now they've got some backing, and now they've learned some new strategies, or now they're approaching something a little differently, but they're founded. And that's the difference, I think, where you're seeing compliance systems being used strictly for compliance.

I can go into a data set like Emote, and I can go with very specific questions of, like, "What is this teacher doing?" Or, "What is it..." You know, you can get real kind of rough in terms of how that would be approached, but you can also then say, "Okay, now we've got all of this. Now what is this actually telling us?" And that's from my researcher side: it's that, like, deterministic- ... or is not? Like, what direction are we going? Are we going in with seeking answers to questions we have, or are we going in to find answers to questions we don't even know yet?It's a very different approach, but to walk that with staff, to see that their system around them also engage in that change process, gosh, it's like some of the most fulfilling work out there.

No, totally agree. I think, I think where we run into this similarly when I think about working with our therapeutic schools, they'll come in, and they say, "We just wanna be able to streamline our tracking around IEP goals, around time out of class, etc." And one of the things with Emote is that we require antecedents to be tracked. It's impossible to only track outcomes based on how it's built. And so they'll go through and track their early indicator behaviors and the Likert scale. And there's always this moment where they're looking at the goal attainment for these goals, and they'll look at the data and see where we have our dashboards set up that, "Hey, this goal is not being met consistently on Tuesdays and Thursdays, and it seems that the student is coming to school tired on those days." And so there's this light bulb that'll always happen where they're moving from the practice of just updating this goal over and over and over again and realizing that they actually have to consider the context.

I think that's one of the most important things with data, is data should capture context. And I mean, at the end of the day, data is essentially the effort to bridge the distance between noticing and caring, or noticing and doing.And so context is what really allows you to do that much more effectively. So it is this delightful part of the job when someone sees that the data that they came to collect isn't necessarily the data that's changing their work. I think that the data people come to collect is always, or very often, outcome data. And the data that changes your work is the context that drives those outcomes. And if you can build a relationship with the context, the work you do becomes aligned with the data you collect. You're already there because you care. The majority of educators care a lot, and the more difficult the environment you work in, there must be a reason that you're there, or at least that you started there.

And I think that when you delve into data that highlights the context of the students and of the environment that you're in, that's when the data starts to connect. It's the student coming to school tired, maybe coming to school tired on Tuesdays and Thursdays because they're staying at one parent's house on those days, and there's a new child there, there's an infant there that's keeping them up. It's the ability to connect these outcomes with the humans that actually allows data to build relationships. The intervention is the people. Not the data.

Absolutely. Oh my gosh. I'm listening to you talk, being like, I don't think I fully recognized how on the same page we are. Because one of the things that's really important to me is I'm a contextual behavior analyst, and so I do not believe that behavior can happen in a vacuum. It has to be responsive to something. And so, if we are not quantifying the environment, the context, then we cannot analyze behavior. That's not possible. So I'm sitting here like I cannot nod more vigorously because I'm coming at that from that behaviorally analytic perspective; embedded within educational systems is just like how I see the world. And that's why I think I've so enjoyed getting to know you and learning what you've built because you've baked that in. You cannot leverage your platform to answer questions outside of context. You can ignore the context. You can make that choice, but you actually cannot only have these outcomes, data pieces without context.So it becomes a professional choice to overlook the other information that is also available that isn't true in all data platforms.

So, no, I'm loving this conversation, so thank you so much for choosing to spend some time with me today and talk about kind of that heart behind the data and making sure that it really is doing what it's supposed to do and not some of the little murky sides of things that the data can be used for. So I really appreciate you bringing your perspective and your heart for making sure that we're serving education well and caring for those adults so that they can continue to care for the kids that they're serving, too.

You are absolutely welcome, and thank you so much for giving me a chance to talk about something that exists in our minds all the time. So, thank you.