Data Science Hangout | Joel Pepera, GEICO | Fundamentals of Data Strategy & Data Science Maturity

Welcome, everyone, to the Data Science Hangout. And welcome back to many of you have joined us here in the following past few weeks. But just for anyone who's new to this, and to Joel, thank you so much for joining us. This is an open space for current and aspiring data science leaders to just be able to connect and ask each other questions. And really, today, Joel, we're excited to be able to pick your brain a little bit. And so for the sessions, we really want to focus on any questions that are most important to you all here. So you can jump in live or put questions in the chat. And we also have a Slido link, which Rob will put in the chat window right now. So you can ask anonymous questions there too. But just a little heads up that it will be recorded for people that missed it. But I'm so grateful to be joined by my co host for today, Joel Pepera, the Director of Data Science at GEICO. And Joel, I'd love to have you introduce yourself and maybe share a bit about your team and the work that you do.

Sure, absolutely. And thank you for having me. I'm excited to talk to everyone today. As Rachel mentioned, I am the Director of Data Science at GEICO. Pretty much all of the data science activities roll up through me. We do have other analytics teams and professionals scattered throughout other parts of the organization. But all the core data science work that we do falls under my purview. I've been at GEICO for quite a long time, coming up on 14 years. My professional background is primarily in the actuarial space. So I'm a credentialed actuary, which is an insurance specific profession. It deals with a lot of things that are, let's say, similar to data science. So using data and predictive analytics and statistics to estimate things, predict values of interest. And in insurance, it's primarily assessing customer risk so that we can understand that from a pricing and financial planning perspective. So that's what I did for most of my early career. And then about three or four years ago, pivoted into a broader data science role. And we've been building up our team and our capabilities within GEICO pretty aggressively.

At GEICO, we focus on three main areas of impact for data science. One is in our product development area. The second is in our customer experience, dealing with how we interact with customers in marketing channels, customer service channels, et cetera. And then the third major area is our claims division, which deals with how we assess damages and injuries when people are involved in auto accidents. There's a lot of interesting use cases there around fraud detection or triaging, estimating payment amounts and things like that. Some use cases, I think dealing with data and statistics is not new, but certainly a lot of the technological evolution going on, not just in insurance, but more broadly is causing us to think a little bit differently about things than we have in the past. And it opens up a lot of really exciting opportunities for us to mature and move forward as a company.

Telematics and data at scale

One of the primary areas that I'm most excited about right now is telematics, which for those who aren't familiar is an initiative within insurance and other industries as well, where due to the technology on modern smartphones or increasingly technology embedded in vehicles directly, we can, with the customer's consent, of course, understand in a very detailed granular way how our customers or potential customers are operating their vehicle and get a much more granular assessment of the risk profile. So that's something that I think in general is a big focus for companies in auto insurance and certainly GEICO is among them. There's a lot of really exciting potential for data science in that domain in particular.

That's awesome. And so the team's just starting now to use telematics or how are you starting to do this today? Yeah, well, I wouldn't say we're starting. We've been working with this for a little bit now. It's not, I think, a new phenomenon in insurance. Some companies have been, at least in an R&D capacity, been working on this for several years. But I think several market factors are increasing the focus. One is just the technology is changing. Smartphones are obviously ubiquitous now, and that provides a cost-effective way of collecting data. I also think that consumer attitudes around privacy and sharing data are evolving. Of course, there are still concerns, and that will continue to be a factor. But I really think the pandemic has impacted that in some meaningful ways.

Obviously, when lockdown started occurring, people's driving behavior changed, and therefore, their risk exposure changed. And one of the things we saw was people were realizing, my insurance rates aren't as responsive as maybe they ought to be to how my use of my vehicle is changing. And you saw a lot of companies do some large-scale measures to provide people discounts or refund premiums and things like that. But I think it's also pointing to an opportunity to just have a pricing model that is, in general, more responsive to how people are operating their vehicle. And that creates a value proposition that I think more people are comfortable with in terms of exchanging some of their data for that value.

Yeah, it's a good question. So you pretty much have to have a cloud-based approach here. We're talking about sensor-level data. It's very large volumes, and we're talking petabyte scale. It really kind of depends on how many customers and what your enrollment rates look like. But for us, it's very large. So it's a significant data engineering effort to collect that data, curate it, make it ready for applying models, training, and inferencing on those data feeds. So it's a big effort. I would say it's at least as big on the data engineering side as it is on the data science and machine learning side. So those two things go hand in hand. I mean, I think that's true in general with data science, but in particular, at the scale we're talking, that's a critical consideration.

Tools and languages for data science

Definitely. I'm starting to see a few questions coming in from the audience. And Chris, I see you just asked one in the chat. Do you want to ask that live and introduce yourself?

Sure, absolutely. Hello, everyone. I'm Chris Scheibel. I have been doing data analytics for the past 24 years, active duty military, retired, and now got myself slid into a data science role. So some of the things that I'm trying to get more collective on besides all the statistical stuff in that is understanding more of the tools that my fellow data scientists out there that are using, besides the fact of, you know, things like Python, Pandas. I know RStudio is a big thing. I'm currently working on Shiny applications where I'm at as well. But I was just kind of curious as far as maybe some of the other tools out there that folks are using that I might be able to dive into and utilize.

Yeah, good question. I think you listed off a lot of the key ones, you know, from a, certainly from a machine learning model training perspective, R and Python are kind of the core languages. Our team, I would say, mostly develops in Python, but there are some users in GEICO that are heavy R users as well. So both of those, I think, are great languages to learn. You know, sort of the dirty secret, although I don't think it's a secret anymore about data sciences, you spend a lot of your time in data preparation. So I think the more adept you are with languages like SQL or Spark is a big one that we use a lot just based on the scale of our data. So learning languages, like Scala, or, you know, the SQL interface into Spark, we often use PySpark pretty heavily. So you know, there are different interfaces into Spark, but just being comfortable with optimizing data processing jobs in a distributed setting, I think is a good investment of time. For us, it's mainly a combination of Spark and Python with a good amount of SQL as well.

And then I think a lot of it sort of depends on, you know, beyond that, which I think is the core, a lot of it depends on what the role of a data scientist is in a particular company. And that can vary quite a bit. Some of them find themselves more involved in some of the low-level data engineering, where you might want to familiarize yourself with data ingestion technologies, you know, Kafka or, you know, some of the streaming technologies. Some data scientists tend to find themselves getting more involved in the front-end development, you know, where they might need to use, you know, web development technologies, JavaScript, those sorts of things. So some of that will depend on just where you're situated in the overall kind of development stack. But I would say Spark, SQL, and Python are kind of the core technologies that I would focus on.

Data analytics vs. data science

I'm just very curious, you know, the field of data science continues to evolve. I've seen so many definitions of data science, so many definitions of data analytics. I think the conclusion I'm working with in terms of defining data science is more around building predictive models. I mean, that's where I see the literature going towards. So I know that not all the problems that we try to solve in a company can be solved by data science. Some of it is actually analyzing basic data, basic descriptions. So I really wanted to find out, most times when I talk to people on data science teams, I really want to find out how much of their time is really spent building, fine-tuning, maintaining models as in data science versus trying to answer everyday basic questions.

Yeah, I think a good question. As you mentioned, this can vary quite a bit across different industries or different companies within the industry. And I think a lot of it depends on the overall maturity of their organization from a sort of data and analytics perspective. At GEICO, we have a lot of traditional analytics teams that will apply statistical methods or various analytical techniques to derive insights, make recommendations to some business consumers, build reports and those sorts of things. So at GEICO, our data science team tries to focus more on core data science work, which I would consider to be, like you mentioned, training predictive models, building machine learning systems, and most of what we do is large-scale systems that can directly impact business processes through some kind of production implementation.

I think one differentiator I would also draw is our data scientists tend to be more comfortable working with lower-level data or data in different formats. So not only are they more comfortable working with machine learning, but they're also more comfortable working with data in different formats. So not just consuming data from databases, but getting into semi-structured, unstructured data, being able to curate those views themselves, clean the data, manipulate the data, build features out of it. Most of the analysts we see tend to be more comfortable consuming data that's already been structured into a database, and they're doing more of the high-level analysis on top of it. And then I also think that the focus on putting code into production is something that data scientists tend to do more so than analysts. So building a model, deriving insights from it, building reports is important, but there's sort of another level of consideration when you have to think about how a model is going to interact in a production process. What data is going to be available? What's the context of a prediction within a business process? How is the data going to be fed through that model? Those are some really important and can be very tricky considerations that if you're not building models for production, you don't really have to think about.

So that's sort of how I think about the distinction within GEICO. That said, these terms are fluid across the industry. What I described in terms of a data scientist at GEICO, another company might think of as an ML engineer. So I think it's really important if you're interviewing for a company or considering different jobs to really understand what is the role for this person in this company, regardless of what the title is.

And I feel like this question of data analysts or data science seems to come up in a lot of conversations as well. And I know it might be different for some larger organizations versus smaller startups, but would data analysts sit on a much different team than yours? Within GEICO, they're in a different team, but we would work collaboratively with them. So anytime we take on a use case, we would typically identify partners on a business analyst team, a business process team, data engineering, and if there's an application team involved, we sort of build these cross-functional teams to address a particular problem.

I don't think that's the only model. I think a lot of companies combine them on the same team, and we've considered the pros and cons of that as well. So I think a lot of times there's fluidity between the roles. And especially at smaller companies, I think data scientists are often asked to wear many hats, just depending on what that company needs at the time. Maybe the best way to help them is by creating a dashboard or putting out some basic summary statistics and analysis on that. I think a lot of data scientists I've run into and interviewing in just different settings kind of get frustrated by that because they come into the job, the industry thinking, you know, I'm going to build machine learning models, or I'm going to do deep learning. And I certainly think it's important to find a role that fits in with what you're looking for. But I would also stress that, especially early in your career, being willing to kind of roll your sleeves up and get involved in whatever's needed is really critical. And you'll develop a lot of good experience in that as well. So even if you find yourself doing a lot of low level database work, and it's not really what you're looking for long term, I think there's a lot of valuable learning to that. So I would certainly encourage you to take advantage of those opportunities and be flexible and willing to contribute value however you can for the company that you're working with.

Pandemic impact and data governance

And kind of going back to what you were initially talking about, about things that changed during the pandemic and looking more into, or when people were driving less, did the pandemic force more data science resources to be put into front facing customer experience initiatives? It's hard for me to say broadly. That was certainly not the case at GEICO. We're a bigger organization, so we have more specialization and teams that kind of focus on those aspects. It definitely impacted us to a degree. You know, initiatives that we were working on were maybe steered in a direction that could directly contribute to something more pandemic related. So I think there were some small examples of that, but I don't think it was a wholesale redirection of resources within our company, at least. I would strongly suspect that that answer would be different at other companies. I think in insurance, we were certainly fortunate to not be as directly impacted as some other companies like service industry, for example, hotel industries. There were huge impacts for some of those areas, and I expect the answer would be different for people working there.

Joel, you got me really curious talking about data and models and then figuring out how to give safer drivers better rates. It got my mind thinking again about this is sort of a data governance question, and you could say, you know what, we're only using the data for people that opt in, but sort of automatically by doing that, you're learning about the people that don't opt in. I'm kind of curious, what is your experience? Do you have any comments or thoughts on what I see as sort of a moral hazard there?

Yeah, that's a great question. That's something I think we had talked about for years as a sort of complicating factor for moving forward with telematics. If it's an opt-in program, you've got some self-selection bias, and you sort of have to think through whether the economic incentives of that align with your business goals. I guess one thing I would say is there are different ways of getting telematics information, and some of them have higher adoption rates, higher opt-in rates than others. So, you know, for example, one source is connected vehicles, and people may or may not know this, but when you're at the dealership buying a car, and you're signing through the million sheets of paper or whatever, one of those is in all likelihood opting into some sort of monitoring for your driving behavior. It might be bundled in with some kind of entertainment package or OnStar or SiriusXM or whatever. So, you know, there's some ethical considerations there, of course, but that also does kind of shape a little bit of the objectivity or the comprehensiveness of the data that you have access to.

To the larger question, though, I think this honestly just has to, it's an evolving picture, right? So, initially, you'll have some limited adoption, and you know your data is biased by that, but it's also the population that you're trying to estimate, right? I mean, the way you're applying it is the people opting in. So, in some sense, that is the most relevant population to analyze, but that will evolve over time as there are other incentives for people to kind of join the party and contribute their data. So, I just think you have to keep considering the evolving profile of people that opt in. And, you know, I think the larger point in this scenario and others is making sure that there is a value to the customer so that it feels like a worthwhile trade-off for them. You know, people, millions of people, most everyone has a smartphone these days. You're giving up a whole lot of data just by owning that smartphone. And, you know, I think some of the smartphone companies have made a lot of progress in giving people more control of what information they're giving up, which is important. But, overall, I think a lot of people feel like they're getting value out of services like Facebook or YouTube, TikTok, whatever. And, you know, as long as they feel like there's an equal exchange for what they're giving up, I think people are increasingly getting comfortable with that.

Considering how models are used

I don't know if that was a direct answer to your question, but it's not an easy thing to consider, but it's an important thing when you're doing practical data science, practical machine learning to kind of think about these second order effects of, you know, not only who is opting into this, but what impact does my model have, for example, right? You come in with some historical data about who buys what products and whatever, you build a recommender system, but that system itself is going to be biasing who purchases what products. And, you know, how do you plan for that bias when you're trying to, you know, retrain models or continue to develop them and prevent, you know, building echo chambers, for example. So, I think it requires a lot of thought, a lot of it's use case specific, but it's a really important consideration for people doing practical data science work.

Yeah, I think you mentioned second and third order effects, that's probably the more interesting point because I think all of the single applications that we create and all the models that we're building, we could probably justify each of them individually, right? There's probably always a good reason. And to be honest, the person that's driving 60 miles an hour down my residential street, I hope they're paying way more in car. You should actually charge that person more money, but yeah, it's probably a matter of like the micros are okay, but then you put those all together and you're like, oh man, that's a, now we have a moral conundrum that we need to deal with more as an industry or society.

Absolutely. I think as a data scientist, you always have to be considering the broader picture. How is my model going to be used? Is that a valid use of the model given what I know about the data that went into it and how it was trained? I think there are a lot of pitfalls that several people have pointed out in recent years for misapplying models. So, it's definitely something that data scientists need to be aware of. Kind of to your point though about incentives and making sure people are being charged the right amount for how they're driving. I think one of the real true selling points about telematics for insurance pricing is that the user is in control. And that wasn't always the case for traditional characteristics that are used to segment pricing. If you don't like the fact that you're paying more because you're driving 90 miles an hour down the highway, well, you can change your behavior and your price will adapt. And I think that is a satisfying realization for people as well. If you feel like you have control and you understand why you're paying what you are, there's sort of a feeling of justice with that, I think.

If you feel like you have control and you understand why you're paying what you are, there's sort of a feeling of justice with that, I think.

Building out a data science team

Well, data science really relies on a solid foundation of data engineering. So I would say, you know, if you think data science or machine learning is eventually going to be a big contributor to your bottom line, getting really good at collecting and organizing data is a great place to start. Because I think some companies who haven't really dealt with that fundamental infrastructure hear a lot about all the cool things that data scientists are doing and big technology companies or internet companies, and they think, oh, I'm just going to hire a bunch of data scientists and I'm going to be able to do all these really exciting things. And what that often leads to is a frustrated data science team because they find out that the data they need is scattered across dozens of different databases and different environments, different formats, and that's where they end up spending all of their time pre-processing and building databases just before they can even get to delivering insights or building models. So I think thinking through your data architecture and your data strategy early on is pretty critical in my view.

Data science maturity

Could you maybe define like levels of data science maturity for companies?

Ah, that's a good question. I would say probably the level one is you've got one, maybe a couple of data scientists, and you're really just looking for them to derive insights. This is probably a company that doesn't have a lot of existing analytics support. They're making decisions based on experience or domain knowledge or things like that, and they really don't have a data-driven approach at all. And so they hire data scientists and they say, well, what does the data say about the decisions I'm trying to make? How could I improve upon a very heuristic based decision making process? And there, I think, the bar is pretty low. You could use a basic data set, some pretty basic modeling techniques, and probably improve upon what they're doing.

And from there, I mean, there's sort of almost infinite stages in between all the way up to it's very machine learning driven where they've got hundreds of models that are running that determine pretty much every aspect of how the customer is interacting with their systems, from recommender systems to fraud detection to all kinds of personalization opportunities. I would say GEICO is somewhere in between those two extremes, probably further toward the mature data science organization, but probably not as mature as a company like Netflix or Amazon or Uber or something like that.

Model monitoring and AI fairness

This question is a really, really, really important question to me, because the models that we build, we don't just build them in a lab, right? They impact lives. You know, they impact lives. So essentially, in this case, whatever the model predicts is impacting, for example, I'm assuming the rate that someone pays for insurance, right? So how do you know that your model is actually making the right decision? And so that's key for me, because I guess I'm from Jamaica. I'm a person of color, and I've read many cases where there are models that don't behave as well as we'd want them or they should, and the impact on people's lives is negative.

Yeah, it's a great question. So I'll answer kind of the general question of, you know, how do you know your models are working? And then, you know, I think there was certainly an ethical AI component to your question, too, which we can talk about. In general, when you're evaluating a machine learning system, I think of three levels of monitoring. One is the sort of infrastructure monitoring, the technical system monitoring, you know, this is a piece of software, it's a, you know, maybe a web service or whatever, and you need to understand how that system is performing and making sure that, you know, it's able to handle the volume of requests that are coming into it, you're not seeing, you know, large error rates, things like that. So there's a sort of low-level technical monitoring that's important. Then there is what I would call statistical monitoring, which is, is the model reliably predicting what it was trained to predict? You know, are the distribution of the feature inputs changing over time? Is the distribution of score changing over time? You know, that's, I think, a very important diagnostic to understand, you know, whether it's time to retrain your model because you see drift or, you know, whether there's some change in the population that's being scored through your model. And then there's the third, which I think gets closer to what you're asking about, which is what are the downstream business impacts of the prediction? So, you know, you're predicting whatever it is that you're trying to predict, but then there's often actions that are taken based on that prediction. And typically there are business KPIs or maybe other considerations that are associated with those actions. And so we would typically set up monitoring for some of those things as well. And I think it's important to have all three levels.

To your point about, you know, how models impact people's lives and, you know, how they might disparately impact certain groups over others. I mean, it's very important and I think a field that's, or a topic that's getting a lot of consideration over the recent months and years. Personally, I don't think there's an easy answer to it. I think it requires an investigation of specific use cases, you know, how the model interacts with different groups of people. What is the intended impact versus what the actual impact is? You know, so for example, so there's a lot of talk about AI bias and, you know, how predictions skew toward one group or another. And, you know, one answer might be saying, well, we don't want there to be any skew across certain demographic groups. But I think, and I understand what's behind that, and I think in some cases that might be the answer. But I also think in some cases that's not really the intent. You know, if you think about, for example, a product recommendation engine and something like gender. I think a lot of people would expect and would want there to be some bias in product recommendations across gender. But in some cases, they might have problems with certain recommendations being targeted at one gender or another. So there's some nuance there, and it's hard to set a global policy that I think covers all cases. So again, I think it's one of those things that a data scientist needs to be aware of. They need to know how their model is going to be used, and they need to ensure that that is consistent with the data that's being fed into it and the methodology that's used to make those predictions.

Team structure and standards

The way we approach it, I mentioned we had three domains, and those more or less operate as autonomous teams, but we do coalesce around languages and some of the primary tools. I don't really see that as an impairment for speed to market or development speed. I don't see setting up standards around how we're going to structure our code repositories, or that we're all going to use Python, we're all going to use Databricks as our spark development environment. I don't really see those as impediments to speed. I'd be curious to know what the questioner has in mind or what their experience is in terms of standards getting in the way of speed to market.

On that first part of the question, the need for structure, how do you handle that with teams using R or Python or other languages? Does that matter? Within my team, we're pretty consistent with what tools we use. It's pretty much a Python and Spark shop. We don't collaborate as directly with some of the other teams in GEICO that use R. I suppose that would be down the line if we wanted to go there, just because the code is different. I don't know that those things aren't overcomable, but it would certainly be a little bit longer to get up to speed on either side, because of that language difference.

Hiring and what GEICO looks for

So we are interested in hiring across the spectrum and certainly bringing in more experienced candidates that could come in at a senior level is a big focus, but we're also interested in people who are just entering the data science profession. We look for people who have a pretty solid understanding of machine learning fundamentals. So they're familiar with various algorithms and understand how they work, what some of the major hyper parameters are and how to tune them, what some of that stuff means. That's, I think, a pretty fundamental requirement. We like people to have some experience in working with different data sets, whether it's just SQL experience or to have experience working in big data ecosystems, that would be great as well. Beyond that, I honestly, I think it's just really important to have candidates who are enthusiastic, able to think through and solve problems in efficient ways. So even if you don't have five years of experience developing in Spark or Python, but you have some baseline knowledge of that and you're really eager and enthusiastic to learn and you've got a good sense about data and you're able to frame analytical questions well and create data products that answer those questions. Those are sort of the core skills that we look for.

I'll say most of the people, even at the entry level that we bring in, do have some sort of advanced degree in some STEM related field and that could vary quite a bit, honestly. I think there are an increasing number of data analytics or data science programs at the master's level in particular, but we've also brought in some pretty strong candidates who have PhDs in physics or theoretical chemistry or even as diverse as music history. So I don't necessarily think there's one path into data science and we consider candidates with a diverse range of backgrounds.

Measuring model impact and knowledge sharing

I think the stronger the connection between the model prediction and the action, the easier it is to measure, but that's not always the case. I mean sometimes you're providing guidance or recommendations and it's still ultimately up to the decision maker to factor that in along with many other things and so attributing a specific action to the model can be difficult. I don't know that there's an easy answer to that all the time, some of that can be handled though through good experimental design. So you know maybe, you know in your case in retail, maybe there there's a way of only rolling out a product in a subset of stores and then you compare that to the stores where it's not. So even if you can't grow stores, had access to that prediction and this other group didn't and you hope that through your experimental design you've randomized enough for some of the other confounding factors and you're able to draw valid inferences about the impact of that. So I think you just have to think through the experimental design, how you're going to measure that, keeping in mind just what the business case is and how the prediction is being used.

I don't think we have explicit mechanisms for that right now. You know we don't have like trainings that we lead for other analyst groups. It might be something we consider as we kind of expand our organization, but most of that happens I think within the context of teams. You know we're just by the nature of you know needing to communicate results or communicate how things work to a team that's going to consume the output. So you have to explain how these things work, how you should interpret them, you know what does precision mean, what does recall mean, how do these features relate to the target or you know stuff like that. So it's I would say more informal typically, at least the way that we operate.

I mean aside from training do you have some sort of like internal community or group where people share different things that they work on or like a slack channel for people to help each other? Yeah, so we use WebEx Teams which is sort of a slack clone I guess or like Microsoft Teams, very similar. That's something that we really adopted during the pandemic and I think it's actually been a great addition to our workflow. So we definitely communicate a lot through that and share information through different team channels. So that's a good avenue. Within our data science organization we have you know regular team meetings where different teams are sharing what they're working on. We often you know get access to different vendor products that you know will organize different calls around. So there's some information sharing there, you know attending conferences. So I guess there's a lot of different mechanisms where people sort of share information or get information from outside groups that we have available.

Advice for aspiring data science leaders

Really pay attention to the business, right? Because ultimately you're there to drive business results and you know the technology, the methodologies, the statistics and data science are really interesting. You can dive as deep as you want in that area and have a great career and be fulfilled in that area. But if you want to move into leadership, you need to learn to speak the business language, understand the needs of the business that you're trying to serve, and you know through that I think you can be an effective leader and understand how to bridge the gap between the technology and the business process that can leverage that technology. Because a lot of times, I mean that's honestly a very valuable skill, because a lot of people are good at the technical side. They can you know tell you how machine learning algorithms work and how to optimize you know the different parameters to predict the value but they can't really relate to the business problem. And then on the other side, there's a lot of business leaders who have a very high level understanding of machine learning or AI. They've read you know Harvard Business Review or something like that, but they don't really understand it in meaningful terms. So being someone who can bridge that gap I think is a great way to advance into leadership within data science.

But if you want to move into leadership, you need to learn to speak the business language, understand the needs of the business that you're trying to serve, and you know through that I think you can be an effective leader and understand how to bridge the gap between the technology and the business process that can leverage that technology.

So at my level I don't do any of that professionally. I really just don't have the time. Sometimes in my free time you know, actually I'm currently taking a Coursera course right now, so I, you know, I can work that in sometimes in the margins, but professionally honestly just don't have the time for it. So that, I mean that's, I think one of the choices you make when you decide to go into data science leadership versus, you know, being a senior technical contributor or something like that, is you recognize that your hands-on time is going to diminish the higher that you advance. But I think it's important to invest enough in at least understanding the technologies. I mean data science is a rapidly developing space, technologies, platforms, tools, methodologies are always moving. I think as a leader you need to invest enough of your time to understand how to make good decisions in your job and how to utilize those technologies, where to invest, and those sorts of things to do that well. So you can't completely, you know, ignore what's going on. I think you have to maintain some connection to some of the lower level details. So it's a balancing act.

Thank you, Joel. And I know we are a little bit past the hour, so there's a few other questions, but I want to be conscious of your time too, so see if you have a few more minutes to run. I do have to run unfortunately, but like I said, if you want to follow up with me on LinkedIn, I'd be happy to, you know, extend the conversation there. Awesome. Well, thank you so much, Joel, for your time. Really appreciate all the insights, and hopefully we'll see you back on one of these in the future too. But thank you so much for joining.