Sharpening your axe and the BAU trap | Steph Locke | Data Science Hangout

Transcript#

This transcript was generated automatically and may contain errors.

Welcome back to the Data Science Hangout, everybody. Happy Halloween. If we haven't met before, I'm Rachel. I lead Customer Marketing at Posit. Posit builds enterprise and open source tools for people who do data science with R and Python. We are also the company formally called RStudio .

I'm joined by my lovely co-host here today, Libby. Hello, everybody. I am a Community Manager working with Posit on the Data Science Hangout. So I work with Rachel. And I also work for Posit Academy. I help teach R and Python for professionals so that they can do more with data.

We're so happy to have you joining us here today. The Hangout is our open space to hear what's going on in the world of data across all different industries, chat about data science leadership, and connect with others who are facing similar things as you. And we get together here every Thursday at the same time, same place, except quick note, not next Thursday, because we do have a webinar with GSK happening at the same time.

But if you're watching this as a recording and want to join in the future, there's details to add it to your own calendar below. But thank you so much to those who have made this the friendly and welcoming space that it is today. And we're all dedicated to keeping it that way.

At the Hangout, we love hearing from you. So no matter your years of experience, your title, the industry you work in, the languages that you use, we want you to connect with each other in the chat. Say who you are, where you're from, what your role is. If you are hiring for any roles, please, please, please share that. If you're looking for a role, please also share that. Share your LinkedIn profile. Share a link to your website, your repos, whatever it is. This is a space for everybody to connect in.

So there are three ways to jump in and either ask a question or share your experience. You can raise your hand on Zoom. You can put a question in the Zoom chat. Feel free to put an asterisk next to it if you want us to read it. And then we can also ask a question for you if you would like to post it anonymously on Slido.

So it can feel slower at first, but you're going slower to go fast, longer term. Competency and quality baked into everything we do enables more innovation in the long term.

The BAU trap

Yeah, so I was really glad to finally write that article because it's been something that stuck with me for years is you see people who, when they don't necessarily have time to invest in doing things in a way that keeps quality, that's gonna have high quality and low maintainability requirements and is easy to extend and create new things. When people aren't doing that, anything they ship is going to then cost them more time to do something, to look after whatever they've shipped. That then gives them less time to ship the next thing.

So the next thing takes longer to deliver. And then they also have to do the maintenance on that thing. And as they keep going, they spend more and more of their time kind of like hugging, loving, maintaining.

Yeah, so the amount of continued efforts something takes after you initially develop it for it to continue being useful to other people takes up your time away from building the next new useful thing. And it's slowly but surely stacks up and stacks up until the only thing you're able to do with your day is maintain things.

And most people don't get into their jobs to be maintainers for systems. They get into their jobs to build great things and add value. And it's really hard to feel like you're adding value when you're just keeping all your previous work going.

So what we can do is, and this comes down to your question of, convincing people to have that time for learning. It also comes down to how do we design things and work and how do we work so that the thing that we output at the end is gonna take less time to keep it adding value. Because the less time that we can make that take, the more new things we can build and the more things that we can support for the same amount of time.

And so that's something that we all need to be thinking about is, are you currently spending too much time just keeping the lights on and putting out fires? With a bit of time, could you get that time burden down? Could you improve your situation? And could you change the way that you do things so that you spend less time going forward on that maintenance, on those activities? Because those activities aren't adding value, adding new value, they're just hopefully keeping old value continuing.

Work at Microsoft: app modernization and developer enablement

Yeah, so I work primarily in software and the use of AI capabilities, but a lot of this still ends up being really relevant for data scientists. So we focus on a number of areas, and one of them is what we'd call application modernization. Somebody built something a while ago, or in an enterprise, lots of people have built things and bought things, and they're all sitting there and running, but they all have technology that was current at the time they've employed and haven't necessarily had a lot of updates since.

And they're going into those kind of maintenance problems of either people just spending all their time maintaining things and not shipping value for the company. Or everybody's stopped maintaining things, and there's a whole bunch of fires that people are desperately going, not me, don't make me fix it.

So we start working with customers to think, okay, how can you change your perceptions around people and processes to solve these? Because it's always a people problem. Processes support people, and then tools support processes. So it's how do you keep the culture of people, processes, peace first, but then also how do you bring technology to help change things?

So we work with a lot of customers on modernization. And this is in kind of the data world. This might be all databases. You might have Hadoop clusters running with MapReduce jobs. You might have old versions of Python, like version two running and not version three. And software, even when it is, every software ages and rusts and technology moves on.

The other, we also help people build new things. Well, and better, and kind of think about, okay, how do you proactively optimize and how do you make sure you've got great observability and monitoring to know it's working and things.

And then the final area that we do a lot of work with is how does your team use, have the right processes and tools to continue shipping value? So some of you might have heard Microsoft has a lot of co-pilots. I'm proud of myself, it's only 24, it's 24 whole minutes in, and that's the first time I said the word co-pilot.

But we have GitHub co-pilots to help developers, data scientists, data engineers, kind of get some of the grunt work of writing code that makes a difference, quicker and more effective. So how technology teams provision environments for data scientists, developers to work in, what does that look like for third parties and contractors? What about when you have like a Java estate and a .NET estate and a Spark estate and like all of these disparate development stacks?

How do you work and enable somebody to switch from all of those? How do you build a consistent compliance and security backbone automated inside your development and collaboration environment like GitHub or GitLab and things? And so we help customers tackle these types of problems because there's no silver bullet on any of this. It's all kind of hard work, doing culture and process work to enable people as well as tools.

Killing projects and defining done

Yeah, so I very proudly, I'm very happy to say that in Microsoft, which can have a culture of you need to own things, I've killed two things, two of my babies this year, this fiscal year. And it really helps to be able to kill something when you have had a solid definition of death.

So like with everything, it is amazing if you can get clarity on what good looks like and especially when you're thinking about software or processes or new initiatives. So one of the things that I killed is we have discovery processes to help different parts of customers' estate whether it's their infrastructure, their data, their data science or their applications, be able to understand where their technical data is and what the value is that they're going to be able to get in the cloud.

We inherited this activity when we got created as a new team last year. And so I formed an initiative to help us be able to onboard this new capability, help us learn quickly, share good practices and be able to like really embed this as part of our kind of commitment to excellence.

By the time a year had rolled on, we had learned everything we needed to learn as a broader group, the enablement had happened and people were kind of rolling well or rolling all right. But then the bits that were still a challenge wasn't really something that we could do centrally. It needed distributed efforts.

So I killed the central piece because I was able to line up accountable people elsewhere who were thinking about things from the infrastructure side, the data side, the app side, for them to be able to go and actually go, okay, so we're good at this. What does great look like for us when we do the input piece, the data piece? And so now that I killed the central piece because I was able to put it into different activities, other initiatives that are running because those are gonna be able to take things to the next level. So my definition of done, get everybody equipped and enabled to execute was sorted. And now other people are responsible for driving their individual areas of excellence.

Transitioning from running your own business to a large organization

Yeah, moving to Microsoft was both awful and amazing at the same time from coming from a business that I ran and that had like 10 people. I'd always worked in organizations that were quite small, like up to 250 people in HQ kind of thing. So it was really, I've always been used to being able to execute quickly, to get stuff done. And so going to Microsoft, which is such a large organization, that there are almost inevitably five different departments who are doing roughly the same thing that you need to collaborate with to help go further.

But the idea, but coming into a culture of teamwork and being able to bring a group together to get economies of scale and to be able to build something bigger was fantastic. And of course, whilst I work for myself, I love being able to set like the direction and strategy and everything. I'll be honest, pay pensions, being able to afford my own house, sweet, sweet benefits.

In the UK, private medical insurance isn't usually a thing, but we have fantastic private medical insurance with Microsoft that's helped cover my partner's transition entirely instead of it being like a 10 year agonizingly slow waitlist hell. Things like that have been so much better that I wouldn't have been able to achieve kind of running my own business than paying other people first. So big culture, pros and cons, and it's been a really useful enabler for kind of my circumstances to be more positive.

Career progression: from IC to directing

So I think the problem in many organizations is actually people go in management's frame as the way to have more influence. So, you know, you can have leadership without authority and as a technical person, what that often looks like is to a certain extent, moving away from being excellent at the coding side to becoming a deciding factor in what gets coded.

So this is, you know, things like architectural design, what do we research? What do we invest our time in? All of these things, kind of some organizations conflate with management, but they're technical leadership pieces. And so you can continue being an amazing coder for sure, but the challenge is, and some companies will push it as you can only progress if you have, if you go into management, but really the push is for you to go from being quite directed to directing.

That's the setting strategy to having impact at a larger scale. If you want to continue solving these technical problems or building new algorithms or kind of being on the technology cutting edge, there is nothing wrong with that. You can get amazingly good, but there are only certain types of organizations that will respect those boundaries.

But you might, for various reasons, prefer to think about how can I have a greater scale of impact? And that's where management or product management architecture, various bits and pieces will evolve your role.

Yeah, so part of it is, unfortunately, fortunately, people skills, but a large proportion of it is actually, how do you approach your job in terms of how are you scaling your impact? So, unfortunately, a whole lot of applications or machine learning solutions and everything are kind of like fundamentally the same. You're gonna go through the same process, you're gonna write roughly the same code and it's gonna produce kind of roughly the same outputs.

The way you can scale is, okay, how do we build the framework to stop people doing roughly the same thing bespoke every time? How do I change the way that we work to deliver more business value more quickly? And then how do I tell people that we're having more business value more quickly? Because you can't just do it for the funsies, you need a measurement and you need to be able to communicate it in terms of like positive things.

So like being able to automate the production, but there's also scaling up your impact. So you might build that one solution but then how do you make that one solution have more impact? Could it generalize to a slightly different part of the business? Could you publish the results? Do internal knowledge transfer? Could you help get that through regulatory sign-off processes when that's not currently your job?

So you can think about how can you add more business value that is more strategically useful than the initial act of writing the code is useful. Because writing the code is useful, but it's kind of similar with research in that the act of research is useful, but almost until something has gone through its commercialization and commoditization phases and is turned into something practical. It's only that practical piece that proves the value of the research in many cases.

It's how you turn that one piece, that one-to-one time-to-value effort into a one-to-many value that you really start moving up the stack in terms of influence and perceived impact.

AI's impact on data science roles

So this will partly depend on what people's definitions of AI are. But I think generative AI has some amazing advancements on natural language processing and how we can interact in a multimodal capability is really groundbreaking.

And I think we should all invest some time in knowing how we can leverage whether it's large language models or small language models and how you can consume these things. Because they can change our interfaces. They can start supporting some reasoning and co-generation for us. But I don't see it replacing data scientists.

It might replace some of the coding we do as data scientists. And I think of this kind of the same way. Like low code interfaces to data science. So like we had Azure Machine Learning Studio, you drag and drop, and you can put a bunch of boxes next to each other. And you go, hey, I've got a model. I don't understand it, but I've got something. You can now call an API.

But the data scientist value for me isn't how they solve the problem. It's how they help people understand and define the problem and bring the insight to light. So I think we might change. Like we might all write less code. We might be able to get the code part solved sooner. But our merits in understanding domains, turning that into scoped challenges and problem statements, and working out the appropriate steps to solve that with solid quality outcomes, that isn't gonna change. The what won't change, but the how might.

But the data scientist value for me isn't how they solve the problem. It's how they help people understand and define the problem and bring the insight to light. The what won't change, but the how might.