Brad Weiner @ CU Boulder | Using data to influence institutional decisions | Data Science Hangout

Transcript#

This transcript was generated automatically and may contain errors.

Welcome to the Data Science Hangout and hope everybody's having a great week. I'm Rachel. I lead our pro community at Posit. And if you have not been here before, this is our open space to chat about data science leadership, questions you're facing and forgetting to hear about what's going on in the world of data across different industries.

So we're here every Thursday at the same time, same place. So if you are somewhere in the future watching this recording on YouTube, you can always add it to your calendar with the details and include it below. Together we're all dedicated to making this a welcoming environment for everyone. And we all love to hear from everybody, no matter your level of experience or area of work.

It's also totally okay to just listen in here if you don't want to join in the conversation, that's okay too. But there's three ways you can jump in and ask questions or provide your own perspective. So you can always jump in by raising your hand here on Zoom and I'll be on the lookout. You can put questions in the Zoom chat. And if you do want me to read it out instead of you, maybe you're in a cafe or your dog's barking or something, just put a little star next to your question in the Zoom chat. And then we also have a Slido link, which my colleagues will be sharing, oh, Tyler just shared it right now, which they shared in the chat so you can ask questions anonymously too.

With all that, thank you all so much for joining us here on this Thursday. I'm so excited to have Brad Weiner here with us as my co-host today. Brad is the Chief Data Officer at the University of Colorado, Boulder. And Brad, to get us going here, I'd love to have you just kind of jump in and introduce yourself, share a little bit about your role and also something you like to do outside of work too.

Getting people to actually use the data and sort of inform policies is much trickier. And I also always like to point to the fact that it really does take a village. We need all kinds of people to be invested and interested in a project.

But there is one that we've sort of been pointing to lately, which was just released publicly to kind of quietly because it happened during finals. One of the things that we've thought a lot about is ways to make it so that students have as easy of an experience as possible. Because we really, really want students to be successful. Not just because that's like our core mission, but we're measured on that. All universities are measured on that in a lot of ways. And students want to go to a place where they're going to be successful.

So we're constantly looking for what I would consider, and I know this is kind of gross terminology for people in higher ed, but building a college degree really is a supply chain problem. You know, you need to have all of these constituent parts, and they need to land at the right time, and there's tons of dependencies, and those dependencies need to be built out years in advance. So if we don't have enough chemistry labs, that student doesn't get to take chemistry. And if not enough students take chemistry, they don't have the required courses, and they can't graduate. So we're constantly looking for these things.

And one of the things that we have found is that students found that the prices of their textbooks to be something of a barrier. And one of the things that we actually found, we're not only trying to graduate students, but we're trying to graduate everybody as equitably as possible. If the experience is one way for some students and one way for another group of students, that is not considered success. And so what we were finding and what we had heard and what we learned is that students were potentially choosing which courses to take based on the cost of the textbooks. And there's also a little bit of this mode of understanding now that the students of today are used to subscription services. Nobody ever thinks like, oh, how many songs did I listen to on Spotify? Am I getting my money's worth? They're just used to Netflix or these models where you just pay one price and you get everything. And it doesn't matter how much you use it or how little you use it.

So Patrick, to answer your question, this is kind of the long way around, but to answer your question, we just recently deployed, and you can read about it publicly, but we're one of the first handful, maybe a dozen or 15 schools that have gone in on this kind of bookstore equity access model, which is basically students pay one price. And that gives them sort of financial stability, the predictability in their budgets, their ability to budget on books, but also the ability to be really flexible in what they take. And by paying that one price, they get day one, what we call day one digital access. And there's also evidence that students who, I mean, think about your college days, right? You go and you take a class. The professor gives the old like, look to your left, look to your right, one of you is going to fail. And you're like, I'm out of here. And so then what you have to do is you have to go, you have to go back to the bookstore, you have to return that book.

And so we have found, and there's evidence that having digital immediate access to your books is actually a student success lever. And so for a variety of reasons, we wanted to implement this program. And one of the things that we were asked to do is, what price should the bundle be? And if any of you have worked in pricing, it's a really complicated problem. If the price is too high, nobody participates and the whole thing kind of collapses on itself. And if it's too low, you don't cover your costs. And so we had a whole team of people who are sort of thinking through that problem. And Rachel was very clear. She's like, you do not need to shill for Posit. But we are happy to say that we used our super sweet Posit workbench setup to simulate a whole bunch of different options. The data science team did that. And we were able to help them provide, to guide them on what that price was. And it got deployed out. And so next year, from what I understand, all undergraduates at CU Boulder will pay $279 a semester and get all of their books on day one. And if they drop a class, then those books go away and the new ones come in and they just have access to their books.

Patrick, I think it'll be valuable to follow up with me a year from now and see how well we did. Because this is a known unknown. We're going to have to see where I landed.

Data visualization tools in higher ed

This is something like multiple organizations I've been part of, I've been trying to figure out like what to use for data viz. So I see you guys are doing like a lot of Tableau dashboards and Tableau public. So how did you think that through as an organization, like what data viz tools to use?

So that is a great question. Data viz is eternally complicated because the technology is always moving. And then I think it's also fair to point out that higher ed tends to be a fairly conservative industry and it tends to buy things in like big ways. So hey, we want to spin up a Posit server and just test it out on 10 users is usually a little bit different than like, what is the best way that we can like use our tremendous buying power to have everybody in the same tool? And so it's always kind of striking a balance, but a lot of universities have gone in pretty deep on, I don't know if I'm allowed to say it, but on Tableau.

Tableau is kind of an industry standard across what's called institutional research. So universities have all these compliance regulations, they need to report on a bunch of stuff. And so Tableau is just like pretty commonly used and has been for about the last 10 years. But one of the things that we're constantly thinking through are ways to leverage more modern technology. And then one of the things that's a really interesting problem space is that Tableau doesn't necessarily provide semantic HTML underneath as part of its API calls. And what that means is that it's actually a little bit difficult to access with screen readers. And of course, we're a public university, we put out a bunch of public data. And one of the things we want to make sure we do is have as accessible of visualizations and accessible access to our information as possible.

And so Tableau provides a ton of value for a lot of reasons, and people really like being able to explore the data. But we are also looking at other ways to potentially visualize information and put out data. There's also just people who want to use open source tools. And so there's always the conversation of like, how big can we scale things like Flask or Streamlit or Shiny ? And I know that those are amazing options, and I love them. But we haven't gotten to a point of like, sort of like institution wide buying. So we pick Abigail in the way that we pick a lot of things, which is through a process called satisficing, which is like, well, the university up the road is doing it seems good enough. It'll be fine. We'll go with that. And then we're mostly happy with what we get until something better comes along.

And all of the open source questions I totally agree with. And we just haven't gotten there yet. So those of you who work in like your cool startup with 50 people, and you're like, oh, well, we were in Streamlit for three weeks, and it didn't work for this. And now we're going to pivot over to Shiny or whatever. It's just a lot harder for us to move the sort of gravity of such a big organization. And I will say Tableau has a lot of advantages, it has a lot of functionality, although it does not have a spell check.

Textbook pricing outcomes and student success

I see Lisa had a follow up to your story that you shared about the book pricing. Lisa, do you want to jump in? I'm lunching right now, which is rather early for Minnesota, but so it is. Yeah, I was just curious if you shared out and maybe it's a little too early, any of the findings about the books, because I know having been in academia, that, you know, at least McAllister was sort of thinking about that problem as well.

Yeah, we're still pretty new to it. And higher ed, like seemingly has, like never has a mythical first mover. But there are a couple of schools that have gone in on this, including University of California at Davis. And so they're the ones who have sort of the most information about it. I would be happy to share and, you know, talk with the people who actually built this out. But right now we are launching it for this fall. And so it'll be a little while before we have any like real outcome data. But of course, the really exciting part will be, can we measure whether or not that particular intervention had an impact on student success? And so probably a year or two years from now, we'll have some like real data on what it all meant. Right now, we were just trying to get the thing out the door and put a price on it. But all the cool stuff is yet to happen.

Where the data office sits organizationally

Brad, I'm really curious where your role lives within the institution. Like is it, are you strongly connected to IT? Are you in administration? And I've got to think that where you kind of live organizationally has got impact on like how work comes to you, how you get to decide what to do, how you're prioritized, all those kinds of things. Just if you could kind of expand on that, that'd be great.

Yeah, that's a great question, Alan. I work in the sort of administrative side. But I have a really good, I have a colleague who's a chief data officer, and he will basically say that if you run or work with a data organization, and I would probably apply this to any of your organizations, although I only, I know my industry best, but if you are a centralized data team, and you're not serving everybody, you're failing. And maybe that's a little bit harsh, because it might not be set up that way everywhere. But for us, we want to support academic units, we want to support academic leadership, we want to support colleges and schools and programs, just as much as we want to support the auxiliary units parking in the bookstore and finance and budget and enrollment and all of those things.

So our role in the most big picture way is to support students, faculty and staff, and to engage in those three missions as much as we possibly can. And parsing out where the boundaries are is really, really tough. So people on my team, for instance, we have a whole team that does surveys. And those surveys help understand how students are engaging on campus, what their sense of belonging is, what type of activities they're participating in. But one of the things we do is we run what are called the faculty course questionnaires for all of campus. So every time you get to the end of semester, and you go, you know, this professor was great, or this professor wasn't so great, all of those data are sort of publicly available and come through this data group.

And that provides us opportunities to really talk about student success through the lens of teaching and learning and through the lens of how students are engaging in the classroom. And then, of course, there's a whole discipline of kind of what's known as learning analytics. So if you Google that, there's all kinds of papers that are out there. But you know, the degree to which students in different classes perform on different types of exercises or different types of, you know, in different types of learning environments, also matters.

So I, a bunch of years ago, I had the opportunity to work with a kind of multi-institutional team from around the Big Ten. And we were trying to understand whether or not there was any difference in academic performance based on a student's self-reported gender identity in introductory STEM classes. And it was really fascinating, because one, it was like an amazing data engineering problem. And you're like, well, we have a five-hour chemistry class, and they have a three-hour chemistry class. Are those the same? So like, harmonizing data across a bunch of universities was like, insanely complicated. But it delved really deeply into, you know, the unit of analysis of like the individual course. And then, of course, you can go even lower, like, what about the section? What about the instructor? And we are here to support efforts like that as well.

Which by the way, is why this job is so epically fun, because for three weeks, you're talking about bookstore prices, and then you're trying to figure out like, why courses that have just a midterm and a final are different from courses that have a variety of assessments along the way. So I don't know, that was kind of a long answer. But the gist is like, we live where we live organizationally, but we have touch points across the university. And if we didn't, we wouldn't have as good of an ability to answer the really hard questions where those two things land in exactly the same spot. You know, a student might not be successful because they might have basic needs problems, they might need additional financial aid, they might not have a sense of belonging, they might not be engaged in the community. They might also just really not be that good at math and need help with that. And all of that is part of the kind of interconnected sort of supply chain.

The rise of the chief data officer in higher ed

I think when we were talking a bit earlier, Brad, you mentioned that there aren't that many schools today that have a chief data officer. And I was wondering for schools who are maybe like in that spot of starting out to or like starting this role of chief data officer, if you have any tips to share them?

Yeah, I get this question pretty often. And so I'm gonna get my numbers a little bit wrong, but there's roughly 4000 or 5000 institutions in the United States that receive title for funding. Maybe it's actually probably a little bit higher, but that's everything from like, the California Barber College to Stanford. And that's basically title for funding means, you know, that they're eligible for federal student loans, or, well, that's, that's probably it. They're probably in the, you know, in the Higher Education Act, which Congress never reauthorizes. So we're always under like 20 year old rules. But there's 1000s and 1000s and 1000s of universities, and I'll just get on my soapbox for four seconds to say, remember that number, the next time you read in the New York Times or the Wall Street Journal or the Washington Post that it's so hard to get into college. It's not. We have so many choices, and most schools admit most students. So I always like to just tell people like, the national media does not paint a very good picture of the higher ed landscape at all.

Off the soapbox of those two, from what I understand, there's probably 50 or so chief data officers or fewer people with that title. There is a very traditional function in higher ed called institutional research, which sort of reports out all the numbers to the federal government, and they typically do this kind of work. So I'm not trying to pick on you, Lisa, but they're at Macalester in St. Paul, they actually have a really amazing institutional research office that's run by really cool people. But if you've worked in higher ed, you've at some point bumped into your kind of IR group. If you want to know anything about a university, these data are publicly available. And so there's someone who submits all of those data, and it's usually IR.

But what's happening more and more is institutions are realizing that their data is this core institutional asset, and that the more value that can be gotten out of those data, the better the institution can do, and the better that they can support students. There's also a growing sort of sense that with the regulatory framework and with compliance frameworks and what the need for data governance and what the need for sort of more standardized policies around data, that, oh, look, we have an iPads key holder, look at that, cool, nice to see you, George. But the conversation around how do we broadly manage our data is something that a chief data officer should be responsible for. And I wouldn't say it's like a weekly thing, but I'd say that probably like once a month, some university reaches out to me on LinkedIn and is like, should we have a chief data officer? What does that mean? What does the sort of construct look like in higher ed? And just like it was a growing trend in industry probably over the last 10 years, it's now sort of catching up in higher ed.

Prioritizing projects and getting people to use data

This has been really fascinating to learn about the setting of university and kind of problems that you have. I'm a data scientist at Microsoft, and usually in a business setting, I know the challenge that you spoke of is someone actually taking the action on the analysis you're doing is a really big challenge. And so on here, a lot of times we don't pick up analysis that could be interesting because we don't have the identified stakeholders that would actually take the action or willing to take the action, but it's also chicken and egg. Like if they don't see the analysis, they don't know what possible action they could take. So how do you overcome that challenge? How do you pick out the projects? How do you prioritize them considering you mentioned such a vast variety of projects you're able to take?

Yeah, I would say carefully and imperfectly, you know, and I think it's one of the coolest things and I have to give a hat to deposit into Rachel. One of the coolest things about getting in a room like this is realizing that we all have the same challenges. You know, I always had this idea like, oh, you work at Amazon, you work at Microsoft, like they've solved everything. And I've worked in higher ed, right? Like this place, it's been here since 1876. It's not going anywhere soon, but it's certainly decentralized and compartmentalized and has those same type of organizational challenges.

And I would say that mostly we take on projects in a what will ostensibly have the largest potential impact for students or for what we're trying to do. There's also a little bit of a conversation that I think I've thought deeply about DeepShot, which is like, what is within your span of control? And sometimes it's like, it's not, it's, you know, and this might come across as the easy road out, but it's like, what are things that we could directly impact or influence?

And then there are also times where it is the decision is made in ways that I refer to as like, you're bringing water to the desert, where occasionally there are people that have nothing, they are just totally out in the open. And so on that like dimension of like, what's easy for us versus what's valuable to them is like such in the sweet spot, because the simplest, most basic thing, it's like someone's just been asking forever, like, all I need is this one thing and to be able to get at it routinely. And you're like, hold my beer, let's focus the sprint, let's focus the cycles on that one thing. And those tend to be the situations where people come back and they're like, thank you so, so much, we just needed this one dashboard.

So I would say like, you know, if there's the thing that's easy for you and high value to someone else, that's worthwhile. If there's something where you really feel like it's totally within your galaxy to be like, hey, we had this finding, that finding suggests we shouldn't do this again. You're the person who's directly responsible for that. Don't do that again. And like, that's right within your sort of frame of reference or within your span of control.

And then, of course, as someone who just in case you can't tell, I'm kind of a wacky guy. I really value fun and joy and being a little bit weird, which I think is okay. And that one tiny corner of your whole portfolio should just be that random thing that seems like it's going to be a ton of fun to work on because those are the kinds of things that keep people engaged. And a lot of times it isn't like fun in some whimsical way, but it's just new, like this bookstore thing. That was hard work. It was a complicated problem, but it was probably like five weeks of work. It was time boxed and it was that one thing. And then we passed it on and we're like, oh, that was cool. We got to run tens of thousands of simulations on student opt-out behavior. Now back to our regularly scheduled program and we kind of can focus on the core stuff again.

And you did ask another question about like, how do you convince people to use the data? And that is, I think, you know, it just takes time and it takes trust and it takes effort. And it's also a recognition that it is not a linear path of like, we're getting better and better and better. And every single time there's more trust. It's like you go from step one to step two, and that's great. And step two to step three, and that's great. And then people start coming to you more. And then you totally box something and you're back to step one because the thing went sideways. And so I like to think that organizations are more resilient than we think they are. And that mistakes are sort of less damaging than we think they are in most cases.

And so people shouldn't be too hard on themselves. If you're like, we found this thing and then people go, oh, that's cool. Right on. We're still going to do this other thing. And you're like, but you might not want to do that because the data say this. And I will say that like higher ed is especially prone to folklore and to, you know, and to people believing that their experience is sort of outweighing data. That's changing a lot. But one of the most fun parts about working in higher ed, and this is one of the things that I was talking to Rachel about, is that if you work in some industries, finding the inefficiencies is really hard because all of the inefficiencies have already been found. And in higher ed, they're kind of like all over the place. So it's pretty cool.

Testing the campus visit weather myth

But to give you guys a quick story about folklore, has anyone ever heard the myth that students choose which college they're going to go to based on the weather on the day that they visit campus? Anyone ever been down that one? I was, I really wanted to go to school, but it was rainy and terrible. There's no way I'm going there. So it's maybe a little bit of an admissions joke, but there is some mythology around that. I happen to used to work at the University of Minnesota. Minnesota has famously great weather. And I tested that assumption and it turns out to be false.

And that included pulling hour by hour data, weather data for every single day for like going back to years and years and years and matching that up with when students visited. But to make it even more interesting, I know I'm talking to a bunch of data people, so you're all gonna be like, how did you do that? I actually calculated not the weather on campus, but I calculated the delta of the weather on campus from where they came. Because I'm imagining someone from Southern California being like, wow, this is a shocking difference. And I modeled that like in every possible way. It was full on p-hacking data torture and I couldn't find anything to support that assumption. So the next time you hear that one, college admissions people, I tested it in truly one of the worst climates or the most extreme climates in America. And it's not true.

It was full on p-hacking data torture and I couldn't find anything to support that assumption. So the next time you hear that one, college admissions people, I tested it in truly one of the worst climates or the most extreme climates in America. And it's not true.